Performance of a deep learning based neural network in the selection of human blastocysts for implantation.

Data collection and preparation

Data were collected at the Massachusetts General Hospital (MGH) fertility center in Boston, Massachusetts. We used 3469 recorded videos of embryos collected from 543 patients with informed consent for research and publication, under an institutional review board approval for secondary research use. Videos were collected for research after institutional review board approval by the Massachusetts General Hospital Institutional Review Board (IRB#2017P001339 and IRB#2019P002392). All the experiments were performed in compliance with the relevant laws and institutional guidelines of the Massachusetts General Hospital, Brigham and Women's Hospital, and Partners Healthcare. The videos were collected using a commercial time-lapse imaging system (Vitrolife Embryoscope). The imaging system used a Leica 20x objective that collected images at 10-min intervals under illumination from a single 635 nm LED. Each patient’s set of embryos were exported as videos (.avi) using the imaging system software. The videos of individual embryos were broken down into their respective frames to extract images from all timepoints post insemination. The images were identified by their timestamps and only images collected at 113 ± 0.05 hr post insemination were processed and used in this study. The extracted images were 250 × 250 pixels and they were cropped to 210 × 210 pixels. The cropping removed both the timestamps and identifiers present in the frame. All embryos used in the study were annotated using images from the fixed time-points (113 hpi) by senior-level embryologists with a minimum of 5 years of human IVF training. Annotations for embryo implantation were assigned based on clinical outcomes. Out-of-focus images were included in the datasets and used for both testing and training. Only images of embryos that were completely non-discernable were removed from the study as part of the data cleaning procedure. Hierarchical categorization The two networks in this study used two categorization systems. The network focused on the rank ordering of embryos used a hierarchical categorization system. The embryo images at 113 hpi time point were categorized between training classes 1 through five as described in detail elsewhere ( Thirumalaraju et al., 2020 ). Briefly, degenerated embryos, which did not begin compaction formed Class 1 while Class 2 embryos were those that reached the morula stage by 113 hpi. Classes 1 and 2 together formed ‘non-blastocysts’ inference class. Class three embryos exhibit features of an early blastocyst which is highlighted by the presence of blastocoel cavity and thick zona pellucida but lack expansion. Class four embryos were blastocysts with blastocoel cavities occupying over half of the embryo volume but either their inner cell mass (ICM) or trophectoderm (TE) was of poor quality. They are non-freezable quality embryos (3 CC) and included full blastocysts to hatched blastocysts. Classes 3, 4, and 5 together formed ‘blastocysts’ inference class. The two inference classes are used since the differentiation of blastocysts and non-blastocysts is a universally accepted categorization that is relevant to embryologists, while the five class categorization is specific to the neural network training, performance and evaluation ( Thirumalaraju et al., 2020 ). Networks that were focused on estimating an embryo’s implantation potential used a two-class training and inference system- positive for implantation and negative for implantation. Neural network training for 113 hpi The 113 hpi evaluation dataset included images of 2440 embryos categorized across five classes post-cleaning based on their clinical annotations made at 113 hpi. Our training set for this classification task used 1188 images with a validation dataset of 510 images obtained at 113 hpi. With the availability of unskewed validation sets prior to augmentation, we used a data generator during training, which performed random rotations and flips across all classes on the fly. The system performing with an accuracy of 90.97% was used in this study in combination with our genetic algorithm. The genetic algorithm was trained and tested with the training data prior to testing it with our independent test data. No human interaction was required/performed once the images were provided to the system during testing, as the entire process was fully automated. The independent non-overlapping test set consisted of 742 images of embryos originating from 97 patients. The selections were compared with embryologist selections. The network was also trained to classify embryos with successful and unsuccessful implantation. 281 embryo images with known implantation outcomes were used for training. Implantation signifies the attachment of a blastocyst into the endometrium. The status of implantation was clinically verified by ultrasound ~6 weeks after embryo transfer. Ninety-seven euploid embryos were evaluated by 15 embryologists, including director level embryologists from five different fertility centers. Embryo selection algorithm development A genetic algorithm was designed to perform selections in combination with the neural network. The genetic algorithm component utilizes the probability scores of every embryo belonging to each of the five different classes to generate a transfer score that can be used to effectively identify the best embryo available in a cohort. For system evaluations, we used an independent set of embryos (100 patients; 2–12 embryos per patient), with no overlap with the training data set used for any prior exercise. The patient cohorts were chosen under the following criteria: (i) each patient embryo cohort had to possess at least two 2PN embryos, and (ii) at least one embryo of the patient embryo cohort developed to blastocyst stage by 113 hpi.

Genetic algorithm

We trained a genetic algorithm to select the morphologically highest quality embryo from a given cohort. There are four phases namely initialization, selection, crossover, and mutation. The classified embryos for each patient were sorted according to their identifier numbers allotted by the deep neural network. A population of weights was generated at random during initialization. A population size of 100 was generated with a 5 × 1 matrix representing each weight. Each weight defined a possible solution for the rank-ordering of embryos based on their quality using the five training classes. The dot product of the weights with the output logits provided by the CNN was used in the calculation of the fitness. The algorithm runs multiple cycles to select the optimal set of weight towards achieving the appropriately suitable rank order of embryos based on their qualities. At each cycle, all the weight sets obtained using the given population were used rank-ordering embryos within the training set. The best 20 wt sets were selected in each cycle. These selected weights (specimens) were then bred with each other with a probability set to 20%. It randomly selected two specimens from the selected top pool and created a random binary 5 × 1 matrix, where one represents that the given element should be switched in cohort and 0 represents that given element should not be switched within the cohort. The fitness function checks if the selected embryo belongs to the highest class available within the tested cohort. It checks if the selected solution (specimen) picked the embryo belonging to the top class in a given cohort of patient embryos. If the selected embryo belonged to the top class, the score was increased and if it did not, the score was not modified. After iterating for all patients’ cohorts, the total scores were used to select the best 20 weights of the given population and were taken for crossover and mutation to repeat the process. The new specimens replaced their parents in the top selected group of embryos. Otherwise, the matrix remained the same. After breeding, each specimen from the top selected group was mutated to give five mutations by adding a random float 5 × 1 matrix with a probability of 20%. These mutations were then added to the new population and the selection step was repeated with the new population of 100. The genetic algorithm ran until the entire population converged to the same score after which a random weight was selected from the population as the final weight. Thus, final generated weights were used to further test the embryo cohorts within our test set.

Additional files Supplementary file 1. Patient and Cycle characteristics. ( A ) Patient population characteristics. All embryo images (except the PGT screened embryos) utilized for experiments reported in the study were obtained from cycles that belong to the presented distribution of parameters. All values in table are presented as median along with the range unless noted otherwise. ( B ) Total number of transfer outcomes for embryos selected by the network. A total of 102 fresh-transfer embryos had known implantation outcomes (45 embryos implanted). Twenty-eight frozen transfers were performed by the clinic where 18 implanted. The table lists only embryos which were selected by the network with known outcomes for both fresh cycles and in frozen subsequent transfers. ( C ) Cycle characteristics of the euploid test set. Embryos used in the euploid embryo differentiation experiment based on the implantation outcomes, originated from cycles that belong to presented distribution of characteristics. These cycles are independent of the original 97 patient cohort test set and also the training data sets. All values in table are presented as median along with the range unless noted otherwise. Transparent reporting form