Application of a methodological framework for the development and multicenter validation of reliable artificial intelligence in embryo evaluation
Abstract Background Artificial intelligence (AI) models analyzing embryo time-lapse images have been developed to predict the likelihood of pregnancy following in vitro fertilization (IVF). However, limited research exists on methods ensuring AI consistency and reliability in clinical settings durin...
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
BMC
2025-01-01
|
| Series: | Reproductive Biology and Endocrinology |
| Subjects: | |
| Online Access: | https://doi.org/10.1186/s12958-025-01351-w |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1832571179614863360 |
|---|---|
| author | D. Gilboa Akhil Garg M. Shapiro M. Meseguer Y. Amar N. Lustgarten N. Desai T. Shavit V. Silva A. Papatheodorou A. Chatziparasidou S. Angras J. H. Lee L. Thiel C. L. Curchoe Y. Tauber D. S. Seidman |
| author_facet | D. Gilboa Akhil Garg M. Shapiro M. Meseguer Y. Amar N. Lustgarten N. Desai T. Shavit V. Silva A. Papatheodorou A. Chatziparasidou S. Angras J. H. Lee L. Thiel C. L. Curchoe Y. Tauber D. S. Seidman |
| author_sort | D. Gilboa |
| collection | DOAJ |
| description | Abstract Background Artificial intelligence (AI) models analyzing embryo time-lapse images have been developed to predict the likelihood of pregnancy following in vitro fertilization (IVF). However, limited research exists on methods ensuring AI consistency and reliability in clinical settings during its development and validation process. We present a methodology for developing and validating an AI model across multiple datasets to demonstrate reliable performance in evaluating blastocyst-stage embryos. Methods This multicenter analysis utilizes time-lapse images, pregnancy outcomes, and morphologic annotations from embryos collected at 10 IVF clinics across 9 countries between 2018 and 2022. The four-step methodology for developing and evaluating the AI model include: (I) curating annotated datasets that represent the intended clinical use case; (II) developing and optimizing the AI model; (III) evaluating the AI’s performance by assessing its discriminative power and associations with pregnancy probability across variable data; and (IV) ensuring interpretability and explainability by correlating AI scores with relevant morphologic features of embryo quality. Three datasets were used: the training and validation dataset (n = 16,935 embryos), the blind test dataset (n = 1,708 embryos; 3 clinics), and the independent dataset (n = 7,445 embryos; 7 clinics) derived from previously unseen clinic cohorts. Results The AI was designed as a deep learning classifier ranking embryos by score according to their likelihood of clinical pregnancy. Higher AI score brackets were associated with increased fetal heartbeat (FH) likelihood across all evaluated datasets, showing a trend of increasing odds ratios (OR). The highest OR was observed in the top G4 bracket (test dataset G4 score ≥ 7.5: OR 3.84; independent dataset G4 score ≥ 7.5: OR 4.01), while the lowest was in the G1 bracket (test dataset G1 score < 4.0: OR 0.40; independent dataset G1 score < 4.0: OR 0.45). AI score brackets G2, G3, and G4 displayed OR values above 1.0 (P < 0.05), indicating linear associations with FH likelihood. Average AI scores were consistently higher for FH-positive than for FH-negative embryos within each age subgroup. Positive correlations were also observed between AI scores and key morphologic parameters used to predict embryo quality. Conclusions Strong AI performance across multiple datasets demonstrates the value of our four-step methodology in developing and validating the AI as a reliable adjunct to embryo evaluation. |
| format | Article |
| id | doaj-art-860e1142de49498e9c6cc65ab8f03546 |
| institution | Kabale University |
| issn | 1477-7827 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | BMC |
| record_format | Article |
| series | Reproductive Biology and Endocrinology |
| spelling | doaj-art-860e1142de49498e9c6cc65ab8f035462025-02-02T12:47:59ZengBMCReproductive Biology and Endocrinology1477-78272025-01-0123111210.1186/s12958-025-01351-wApplication of a methodological framework for the development and multicenter validation of reliable artificial intelligence in embryo evaluationD. Gilboa0Akhil Garg1M. Shapiro2M. Meseguer3Y. Amar4N. Lustgarten5N. Desai6T. Shavit7V. Silva8A. Papatheodorou9A. Chatziparasidou10S. Angras11J. H. Lee12L. Thiel13C. L. Curchoe14Y. Tauber15D. S. Seidman16AIVF LtdIVIRMA ValenciaAIVF LtdIVIRMA ValenciaAIVF LtdAIVF LtdDepartment of Obstetrics and Gynecology, Division of Reproductive Endocrinology and Infertility, Women’s Health Institute, Cleveland ClinicIn Vitro Fertilization (IVF) Unit, Assuta Ramat HaHayalFerticentro - Centro de Estudos de FertilidadeEmbryolab Fertility ClinicEmbryolab Fertility ClinicFIRST IVF ClinicMaria Fertility HospitalPraxis Dres.med. GöhringArt Compass, an AIVF TechnologyAIVF LtdAIVF LtdAbstract Background Artificial intelligence (AI) models analyzing embryo time-lapse images have been developed to predict the likelihood of pregnancy following in vitro fertilization (IVF). However, limited research exists on methods ensuring AI consistency and reliability in clinical settings during its development and validation process. We present a methodology for developing and validating an AI model across multiple datasets to demonstrate reliable performance in evaluating blastocyst-stage embryos. Methods This multicenter analysis utilizes time-lapse images, pregnancy outcomes, and morphologic annotations from embryos collected at 10 IVF clinics across 9 countries between 2018 and 2022. The four-step methodology for developing and evaluating the AI model include: (I) curating annotated datasets that represent the intended clinical use case; (II) developing and optimizing the AI model; (III) evaluating the AI’s performance by assessing its discriminative power and associations with pregnancy probability across variable data; and (IV) ensuring interpretability and explainability by correlating AI scores with relevant morphologic features of embryo quality. Three datasets were used: the training and validation dataset (n = 16,935 embryos), the blind test dataset (n = 1,708 embryos; 3 clinics), and the independent dataset (n = 7,445 embryos; 7 clinics) derived from previously unseen clinic cohorts. Results The AI was designed as a deep learning classifier ranking embryos by score according to their likelihood of clinical pregnancy. Higher AI score brackets were associated with increased fetal heartbeat (FH) likelihood across all evaluated datasets, showing a trend of increasing odds ratios (OR). The highest OR was observed in the top G4 bracket (test dataset G4 score ≥ 7.5: OR 3.84; independent dataset G4 score ≥ 7.5: OR 4.01), while the lowest was in the G1 bracket (test dataset G1 score < 4.0: OR 0.40; independent dataset G1 score < 4.0: OR 0.45). AI score brackets G2, G3, and G4 displayed OR values above 1.0 (P < 0.05), indicating linear associations with FH likelihood. Average AI scores were consistently higher for FH-positive than for FH-negative embryos within each age subgroup. Positive correlations were also observed between AI scores and key morphologic parameters used to predict embryo quality. Conclusions Strong AI performance across multiple datasets demonstrates the value of our four-step methodology in developing and validating the AI as a reliable adjunct to embryo evaluation.https://doi.org/10.1186/s12958-025-01351-wArtificial intelligenceTime lapse imagingEmbryo rankingMachine learning |
| spellingShingle | D. Gilboa Akhil Garg M. Shapiro M. Meseguer Y. Amar N. Lustgarten N. Desai T. Shavit V. Silva A. Papatheodorou A. Chatziparasidou S. Angras J. H. Lee L. Thiel C. L. Curchoe Y. Tauber D. S. Seidman Application of a methodological framework for the development and multicenter validation of reliable artificial intelligence in embryo evaluation Reproductive Biology and Endocrinology Artificial intelligence Time lapse imaging Embryo ranking Machine learning |
| title | Application of a methodological framework for the development and multicenter validation of reliable artificial intelligence in embryo evaluation |
| title_full | Application of a methodological framework for the development and multicenter validation of reliable artificial intelligence in embryo evaluation |
| title_fullStr | Application of a methodological framework for the development and multicenter validation of reliable artificial intelligence in embryo evaluation |
| title_full_unstemmed | Application of a methodological framework for the development and multicenter validation of reliable artificial intelligence in embryo evaluation |
| title_short | Application of a methodological framework for the development and multicenter validation of reliable artificial intelligence in embryo evaluation |
| title_sort | application of a methodological framework for the development and multicenter validation of reliable artificial intelligence in embryo evaluation |
| topic | Artificial intelligence Time lapse imaging Embryo ranking Machine learning |
| url | https://doi.org/10.1186/s12958-025-01351-w |
| work_keys_str_mv | AT dgilboa applicationofamethodologicalframeworkforthedevelopmentandmulticentervalidationofreliableartificialintelligenceinembryoevaluation AT akhilgarg applicationofamethodologicalframeworkforthedevelopmentandmulticentervalidationofreliableartificialintelligenceinembryoevaluation AT mshapiro applicationofamethodologicalframeworkforthedevelopmentandmulticentervalidationofreliableartificialintelligenceinembryoevaluation AT mmeseguer applicationofamethodologicalframeworkforthedevelopmentandmulticentervalidationofreliableartificialintelligenceinembryoevaluation AT yamar applicationofamethodologicalframeworkforthedevelopmentandmulticentervalidationofreliableartificialintelligenceinembryoevaluation AT nlustgarten applicationofamethodologicalframeworkforthedevelopmentandmulticentervalidationofreliableartificialintelligenceinembryoevaluation AT ndesai applicationofamethodologicalframeworkforthedevelopmentandmulticentervalidationofreliableartificialintelligenceinembryoevaluation AT tshavit applicationofamethodologicalframeworkforthedevelopmentandmulticentervalidationofreliableartificialintelligenceinembryoevaluation AT vsilva applicationofamethodologicalframeworkforthedevelopmentandmulticentervalidationofreliableartificialintelligenceinembryoevaluation AT apapatheodorou applicationofamethodologicalframeworkforthedevelopmentandmulticentervalidationofreliableartificialintelligenceinembryoevaluation AT achatziparasidou applicationofamethodologicalframeworkforthedevelopmentandmulticentervalidationofreliableartificialintelligenceinembryoevaluation AT sangras applicationofamethodologicalframeworkforthedevelopmentandmulticentervalidationofreliableartificialintelligenceinembryoevaluation AT jhlee applicationofamethodologicalframeworkforthedevelopmentandmulticentervalidationofreliableartificialintelligenceinembryoevaluation AT lthiel applicationofamethodologicalframeworkforthedevelopmentandmulticentervalidationofreliableartificialintelligenceinembryoevaluation AT clcurchoe applicationofamethodologicalframeworkforthedevelopmentandmulticentervalidationofreliableartificialintelligenceinembryoevaluation AT ytauber applicationofamethodologicalframeworkforthedevelopmentandmulticentervalidationofreliableartificialintelligenceinembryoevaluation AT dsseidman applicationofamethodologicalframeworkforthedevelopmentandmulticentervalidationofreliableartificialintelligenceinembryoevaluation |