Attention-Based Deep Neural Networks for Detection of Cancerous and Precancerous Esophagus Tissue on Histopathological Slides.

Data Set For this diagnostic study, whole-slide images were collected from patients who underwent endoscopic esophagus and gastroesophageal junction mucosal biopsy between January 1, 2016, and December 31, 2018, at Dartmouth-Hitchcock Medical Center, a tertiary academic medical center in Lebanon, New Hampshire. The use of data collected for this study was approved by the Dartmouth Institutional Review Board, which waived the requirement of informed consent as the collected data were deidentified. The study is in compliance with the Declaration of Helsinki on Ethical Principles for Medical Research Involving Human Subjects. 35 In addition, the study followed the Standards for Reporting of Diagnostic Accuracy ( STARD ) reporting guidelines. 36 A scanner (Aperio AT2; Leica Biosystems Inc) was used to digitize hematoxylin-eosin–stained whole-slide images at 20× magnification. Scanning with 20× magnification is routinely performed in the clinical workflow for faster scanning throughput and efficient file size. We had a total of 180 whole-slide images, of which 116 (64.4%) were used as the training set and 64 (35.6%) were used as the testing set. Of the training set, 23 whole-slide images (19.8%) were reserved for validation. These whole-slide images can cover multiple pieces of tissue. Therefore, the whole-slide images were separated into 379 high-resolution images later in the preprocessing step, with each image covering a single piece of tissue. To determine labels for whole-slide images and to train the existing state-of-the-art sliding window approach as the baseline, 2 of our expert pathologists from the Department of Pathology and Laboratory Medicine at Dartmouth-Hitchcock Medical Center (A.S., B.R.) annotated bounding boxes around lesions in these images (eMethods 1 in the Supplement ). We considered these labels as the reference standard, as any disagreements in annotation were resolved through further discussion among our senior domain-expert pathologist annotators. These bounding boxes were not needed in training the proposed attention-based model. This study used categories of esophageal dysplasia and carcinoma based on the Vienna classification system. 37 The normal class included normal squamous epithelium, normal squamous and columnar junctional epithelium, and normal columnar epithelium. Barrett esophagus negative for dysplasia was included in the BE-no-dysplasia class. Barrett esophagus is defined by columnar epithelium with goblet cells (intestinal metaplasia) and preservation of orderly glandular architecture of the columnar epithelium with surface maturation. The BE-with-dysplasia class included low-grade dysplasia (noninvasive low-grade neoplasia) and high-grade dysplasia (noninvasive high-grade neoplasia). Columnar epithelium with low-grade dysplasia is characterized by nuclear pseudostratification, mild to moderate nuclear hyperchromasia and irregularity, and the cytologic atypia extending to the surface epithelium. High-grade dysplasia demonstrated marked cytologic atypia, including loss of polarity, severe nuclear enlargement and hyperchromasia, numerous mitotic figures, and architectural abnormalities such as lateral budding, branching, and villous formation as well as variation in the size and shape of crypts. In contrast to the Vienna classification system, we merged BE with low-grade dysplasia and high-grade classes into 1 class owing to the low number of collected samples for each class. The adenocarcinoma class included invasive carcinoma (intramucosal carcinoma and submucosal carcinoma and beyond) and high-grade dysplasia suggestive of invasive carcinoma. Cases in the adenocarcinoma class may present the following features: single-cell infiltration, sharply angulated glands, small glands in a back-to-back pattern, confluent glands, cribriform or solid growth, ulceration occurring within high-grade dysplasia, dilated dysplastic glands with necrotic debris, or dysplastic glands undermining squamous epithelium.

Two-Step Method and Testing

The proposed attention-based model has 2 steps, which are shown in Figure 1 . The first step is the extraction of grid-based features from the high-resolution image, at which point each grid cell in the whole slide is analyzed to generate a feature map ( Figure 1 A and B). The second step is the application of the attention mechanism on the extracted features for slide classification ( Figure 1 C). The feature extractor is jointly optimized across all the grid cells with the attention module in an end-to-end fashion. In the end-to-end training pipeline, the cross-entropy loss over all classes is computed on class predictions. The loss is backpropagated to optimize all parameters in the network without any manual adjustment for attention modules. The model does not need bounding box annotations around ROIs, and all optimization is done to only the labels at the tissue level. Further details of the model architecture of the grid-based feature extraction and attention-based classification are provided in eMethods 2 in the Supplement . Figure 1. Overview of Proposed Attention-Based Model A, An input image is divided into r × c grid cells (dividing lines are shown only for visualization). B, Features extracted from each grid cell build a grid-based feature map tensor U. C, Learnable 3-dimensional convolutional filters of size k × d × d (where d denotes the height and width of the convolutional filters) are applied on U feature map to generate an attention map α, which operates as the weights for an affine combination of feature vectors in U. The α represents a 2-dimensional attention map whose size is r in height and c in width; CNN, convolutional neural network; r and c, the number of rows and columns of input tissue grid; U, a tensor of features extracted from each grid cell, and its size is r in height, c in width, and k in depth; and z, a vector of features representing a whole-input image. To evaluate the attention-based classification model for high-resolution microscopy images, we applied the steps to high-resolution scanned slides of tissues endoscopically removed from patients who were at risk for esophageal cancer. We compared the performance results of the proposed model with those of the state-of-the-art sliding window approach. 22 For preprocessing, we removed the white background from the slides and extracted only regions of the images that contained tissue. eFigure 1A in the Supplement shows a typical whole-slide image from the data set. These whole-slide images can cover multiple pieces of tissue, so we separated them into subimages with each covering only a single piece of tissue. The median (interquartile range) width of the tissues was 4500 (3000-6500) pixels and the median (interquartile range) height was 5500 (4000-7500) pixels. Every tissue image was given an overall label based on the labels of its lesions. If multiple lesions with different classes were present, we used the class with the highest risk as the corresponding label, as that lesion would have the highest implication clinically. If no abnormal lesions were found in an image, it was assigned to the normal class. After this preprocessing step, each image was assigned to 1 of 4 classes: normal, BE-no-dysplasia, BE-with-dysplasia, and adenocarcinoma (eFigure 1B in the Supplement ). The data set included 379 images after preprocessing. One-third of the data set was reserved for testing. To avoid possible data leakage, we placed all tissues extracted from 1 whole-slide image into the same set of images when the training and testing sets were split. The Table summarizes the results of the testing set. Table.

Classification Results for the Testing

Set a Metric Sliding Window Approach Performance (95% CI) b Attention-Based Model Performance (95% CI) Normal class Accuracy 0.63 (0.56-0.69) 0.70 (0.64-0.76) Recall 0.62 (0.53-0.71) 0.69 (0.61-0.77) Precision 0.60 (0.51-0.69) 0.68 (0.59-0.76) Specificity 0.63 (0.57-0.72) 0.71 (0.62-0.79) F1 score 0.61 (0.53-0.68) 0.68 (0.61-0.75) BE-no-dysplasia class Accuracy 0.85 (0.80-0.89) 0.85 (0.81-0.90) Recall 0.43 (0.31-0.56) 0.77 (0.66-0.87) Precision 0.87 (0.73-0.97) 0.68 (0.57-0.78) Specificity 0.98 (0.95-1.00) 0.88 (0.83-0.93) F1 score 0.58 (0.45-0.69) 0.72 (0.63-0.80) BE-with-dysplasia class Accuracy 0.72 (0.66-0.77) 0.89 (0.84-0.92) Recall 0.36 (0.18-0.54) 0.21 (0.07-0.38) Precision 0.16 (0.08-0.26) 0.50 (0.20-0.80) Specificity 0.76 (0.70-0.82) 0.97 (0.94-0.99) F1 score 0.22 (0.11-0.33) 0.30 (0.11-0.48) Adenocarcinoma class Accuracy 0.87 (0.83-0.91) 0.88 (0.84-0.92) Recall 0.52 (0.37-0.68) 0.71 (0.57-0.85) Precision 0.65 (0.48-0.80) 0.63 (0.49-0.76) Specificity 0.94 (0.90-0.98) 0.91 (0.87-0.95) F1 score 0.58 (0.44-0.70) 0.67 (0.54-0.77) Mean Accuracy 0.76 (0.73-0.80) 0.83 (0.80-0.86) Recall 0.48 (0.41-0.56) 0.60 (0.53-0.66) Precision 0.57 (0.51-0.63) 0.62 (0.53-0.71) F1 score 0.50 (0.43-0.56) 0.59 (0.52-0.66) Abbreviation: BE, Barrett esophagus. a The proposed attention-based model's performance was assessed on the basis of accuracy, recall, precision, specificity, and F1 score. Results were rounded to 2 decimal places. The model outperformed the sliding window baseline in both accuracy and F1 score for all classes. b The sliding window approach is explained in Wei et al. 22 Sliding Window Approach as Baseline To compare the proposed model with previous methods for high-resolution image analysis, we implemented the current state-of-the-art sliding window approach 22 as a baseline. For this method, we used the annotated bounding box labels to generate small, cropped images of 224 × 224 pixel size for training a cropped-image classifier. For preprocessing, we normalized the color channels and performed standard data augmentation, including color jittering, random flips, and rotations. For training, we initialized ResNet-18 with MSRA (Microsoft Research Asia) initialization. 38 We optimized the model with a cross-entropy loss function for 100 epochs, using standard weight regularization techniques and learning rate decay. We trained the cropped-image classifier to predict the class of any given window on a high-resolution image. For whole-slide inference, we performed a grid search of the validation set for optimal thresholds to filter noise. Then, our 2 pathologists (A.S., B.R.) were consulted to develop heuristics for aggregating cropped-image predictions. We chose the thresholds and heuristics that performed the best on the validation set and applied those to the whole-slide images in the testing set.

Attention-Based Model

We implemented the attention-based model as described. Given the size of features extracted from the ResNet-18 model, we used 512 × 3 × 3, 3-D convolutional filters in the attention module, with implicit zero padding of 0 for depth, 1 for height, and 1 for width dimensions. We used 64 of these filters to increase the robustness of the attention module, as patterns in the feature space are likely too complex to be recognized and attended by a single filter. To avoid overfitting and encourage each filter to capture different patterns, we regularized the attention module by applying dropout 39 with P = .50 after concatenating all of the feature vectors. We initialized the entire network with MSRA initialization for convolutional filters, 38 unit weight and zero bias for batch normalizations, 40 and Glorot initialization for fully connected layers. 41 Only the cross-entropy loss against class labels was used in training. Other information, such as the location of bounding boxes, was not given to the network as guidance to optimal attention maps. The model identified such ROIs automatically. We initialized the feature extraction network with weights pretrained on the ImageNet data set. 42 Input for the network was extracted grid cells of 492 × 492 pixels that were resized to 224 × 224 pixels. We normalized the input values by the mean (SD) of pixel values computed over all tissues in the training set. In training, the last fully connected layer of the network was removed, and all residual blocks except for the last one were frozen, serving as a regularization mechanism. We trained the entire network on large, high-resolution images. For data augmentation, we applied random rotation and random scaling, with a scaling factor between 0.8 and 1.2 during training. We used the Adam optimizer with an initial learning rate of 1 × 10 −3 , decaying by 0.95 after each epoch, and reset the learning rate to 1 × 10 −4 every 50 epochs in a total of 200 epochs, similar to the cyclical learning rate. 43 , 44 We set the mini-batch size to 2 to maximize the use of memory on the graphic processing unit (Nvidia Titan Xp; NVIDIA Corporation). The model was implemented in PyTorch. 45 At testing, the network took a mean 0.34 seconds to analyze a high-resolution image.

Statistical Analysis

Data were analyzed in October 2018. For quantitative evaluation, 4 standard metrics were used for classification under a 1-vs-rest strategy: accuracy, recall, precision, and F1 score. To estimate 95% CIs, bootstrapping was used for all metrics. The 2-tailed McNemar-Bowker test was used, and α = .05 was considered statistically significant. Statistical analysis was carried out with SciPy, version 1.0.0 (SciPy developers).

Two-Step Method and Testing

The proposed attention-based model has 2 steps, which are shown in Figure 1 . The first step is the extraction of grid-based features from the high-resolution image, at which point each grid cell in the whole slide is analyzed to generate a feature map ( Figure 1 A and B). The second step is the application of the attention mechanism on the extracted features for slide classification ( Figure 1 C). The feature extractor is jointly optimized across all the grid cells with the attention module in an end-to-end fashion. In the end-to-end training pipeline, the cross-entropy loss over all classes is computed on class predictions. The loss is backpropagated to optimize all parameters in the network without any manual adjustment for attention modules. The model does not need bounding box annotations around ROIs, and all optimization is done to only the labels at the tissue level. Further details of the model architecture of the grid-based feature extraction and attention-based classification are provided in eMethods 2 in the Supplement . Figure 1. Overview of Proposed Attention-Based Model A, An input image is divided into r × c grid cells (dividing lines are shown only for visualization). B, Features extracted from each grid cell build a grid-based feature map tensor U. C, Learnable 3-dimensional convolutional filters of size k × d × d (where d denotes the height and width of the convolutional filters) are applied on U feature map to generate an attention map α, which operates as the weights for an affine combination of feature vectors in U. The α represents a 2-dimensional attention map whose size is r in height and c in width; CNN, convolutional neural network; r and c, the number of rows and columns of input tissue grid; U, a tensor of features extracted from each grid cell, and its size is r in height, c in width, and k in depth; and z, a vector of features representing a whole-input image. To evaluate the attention-based classification model for high-resolution microscopy images, we applied the steps to high-resolution scanned slides of tissues endoscopically removed from patients who were at risk for esophageal cancer. We compared the performance results of the proposed model with those of the state-of-the-art sliding window approach. 22 For preprocessing, we removed the white background from the slides and extracted only regions of the images that contained tissue. eFigure 1A in the Supplement shows a typical whole-slide image from the data set. These whole-slide images can cover multiple pieces of tissue, so we separated them into subimages with each covering only a single piece of tissue. The median (interquartile range) width of the tissues was 4500 (3000-6500) pixels and the median (interquartile range) height was 5500 (4000-7500) pixels. Every tissue image was given an overall label based on the labels of its lesions. If multiple lesions with different classes were present, we used the class with the highest risk as the corresponding label, as that lesion would have the highest implication clinically. If no abnormal lesions were found in an image, it was assigned to the normal class. After this preprocessing step, each image was assigned to 1 of 4 classes: normal, BE-no-dysplasia, BE-with-dysplasia, and adenocarcinoma (eFigure 1B in the Supplement ). The data set included 379 images after preprocessing. One-third of the data set was reserved for testing. To avoid possible data leakage, we placed all tissues extracted from 1 whole-slide image into the same set of images when the training and testing sets were split. The Table summarizes the results of the testing set. Table.

Classification Results for the Testing

Set a Metric Sliding Window Approach Performance (95% CI) b Attention-Based Model Performance (95% CI) Normal class Accuracy 0.63 (0.56-0.69) 0.70 (0.64-0.76) Recall 0.62 (0.53-0.71) 0.69 (0.61-0.77) Precision 0.60 (0.51-0.69) 0.68 (0.59-0.76) Specificity 0.63 (0.57-0.72) 0.71 (0.62-0.79) F1 score 0.61 (0.53-0.68) 0.68 (0.61-0.75) BE-no-dysplasia class Accuracy 0.85 (0.80-0.89) 0.85 (0.81-0.90) Recall 0.43 (0.31-0.56) 0.77 (0.66-0.87) Precision 0.87 (0.73-0.97) 0.68 (0.57-0.78) Specificity 0.98 (0.95-1.00) 0.88 (0.83-0.93) F1 score 0.58 (0.45-0.69) 0.72 (0.63-0.80) BE-with-dysplasia class Accuracy 0.72 (0.66-0.77) 0.89 (0.84-0.92) Recall 0.36 (0.18-0.54) 0.21 (0.07-0.38) Precision 0.16 (0.08-0.26) 0.50 (0.20-0.80) Specificity 0.76 (0.70-0.82) 0.97 (0.94-0.99) F1 score 0.22 (0.11-0.33) 0.30 (0.11-0.48) Adenocarcinoma class Accuracy 0.87 (0.83-0.91) 0.88 (0.84-0.92) Recall 0.52 (0.37-0.68) 0.71 (0.57-0.85) Precision 0.65 (0.48-0.80) 0.63 (0.49-0.76) Specificity 0.94 (0.90-0.98) 0.91 (0.87-0.95) F1 score 0.58 (0.44-0.70) 0.67 (0.54-0.77) Mean Accuracy 0.76 (0.73-0.80) 0.83 (0.80-0.86) Recall 0.48 (0.41-0.56) 0.60 (0.53-0.66) Precision 0.57 (0.51-0.63) 0.62 (0.53-0.71) F1 score 0.50 (0.43-0.56) 0.59 (0.52-0.66) Abbreviation: BE, Barrett esophagus. a The proposed attention-based model's performance was assessed on the basis of accuracy, recall, precision, specificity, and F1 score. Results were rounded to 2 decimal places. The model outperformed the sliding window baseline in both accuracy and F1 score for all classes. b The sliding window approach is explained in Wei et al. 22