Label-free intraoperative histology of bone tissue via deep-learning-assisted ultraviolet photoacoustic microscopy.

Label-free reflection-mode UV-PAM. The reflection-mode UV-PAM system used an Nd: YLF (neodymium-doped yttrium lithium fluoride) Q-switched 266 nm nanosecond pulsed laser (QL266-010-O, CrystaLaser). A bandpass glass filter (FGUV5, Thorlabs) was placed at the laser output to reject the leaked pump light. After passing the colored glass filter, a small portion of the beam was reflected by a UV fused silica beam sampler (BSF10-UV, Thorlabs) and directed to a Si photodiode (PDA36A, Thorlabs) for pulse-to-pulse fluctuation compensation. The UV laser beam was expanded by a pair of plano-convex lenses and spatially filtered by a 15 μm high-energy pinhole (900PH-15, Newport). The expanded and collimated beam was then focused onto the sample by a custom-made water-immersion UV objective lens (consisting of an aspheric lens, a concave lens, and a convex lens (NT49-696, NT48-674, NT46-313, Edmund Optics)) with a numerical aperture (NA) of 0.16. A customized ring-shaped ultrasonic transducer (42 MHz center frequency, 76% −6 dB two-way bandwidth) with a central aperture was used to detect the PA signal, which allows the optical and acoustical confocal alignment. The detected signal was amplified by two low noise amplifiers (ZFL-500LN+, Mini-Circuits) and digitized by the data acquisition card (ATS 9350, Alazar Technologies) at a 500 MHz sampling rate. The PAM image was acquired by scanning the water-immersed sample mounted onto a customized 3D scanner (consisting of 3 step motors, PLS-85, PI Micos, GmbH). The reconfigurable I/O device (myRIO-1900, National Instruments) with a field-programmable gate array (FPGA) was used to control and synchronize laser pulses, motor movements, and data acquisition. Real-time 3D contour-scanning UV-PAM. To allow imaging of the rough surface of unprocessed thick samples like bone, we developed the contour scanning mechanism without prior knowledge of the sample surface using a 3-axis motorized stage, which ensures consistent lateral resolution within a large field of view. For consistent and optimized optical resolution, the distance between the sample and optical focus should be maintained within the DOF during scanning. In contour scanning, the time-of-flight information of PA signals is used to calculate the distance between the sample and the focal spot, which can be adjusted by the z -motor during scanning. With the numerical aperture of 0.16, the DOF of our UV-PAM microscope is only about 9 μm, which corresponds to 6 ns ultrasound propagation for the speed of sound at 1500 m/s in room temperature water. The acquired PA signal was digitized at the sampling rate of 500 MHz (ATS9350, AlazarTech). The z -profile of the sample surface can be accurately calculated using the time-of-flight information of PA signals, enabling contour scanning for z -position compensation. Prior to PAM imaging, the optical and acoustic foci are confocally aligned, while the propagation time of the acoustic signal from the optical focus is recorded to determine the focal spot position. To extract the ultrasound propagation time, we calculate the centers of positive and negative peak positions in PA A-line signals, which are converted to the sample position. Without prior knowledge of the sample surface profile, one seed B-scan with the z -motor disabled is implemented to calculate the starting contour trajectory. To avoid potential noise interference and remove outliers, a 100-point moving average is used to generate a smooth z scanning trajectory. During the contour scanning, both the x -axis motor and the z -axis motor move simultaneously. After the first contoured B-scan, the z -motor trajectory and the distance between the sample surface and the ultrasonic transducer are calculated and used to compute the accurate surface profile. Due to the small y step (0.625 μm), we set the second z -motor trajectory to follow the surface profile from the previous contoured B-scan 43 . Then, the surface profile is updated according to the second contoured B-scan. This process is repeated until the whole scanning is completed. Real-time data processing and system control are implemented using MATLAB and LabVIEW hybrid programming. Bone specimen preparation and H&E imaging. The bone specimens for UV-PAM imaging were procured from larger specimens in the pathology laboratory with informed consents of patients, surgically removed from patients at the City of Hope and UCLA medical center. All bone specimens were fixed in 10% buffered formalin prior to any other procedures. For thick undecalcified specimens in this study, the specimens were mounted to the sample holder for imaging without further processing. To decalcify specimens, we treated the bone specimens with a decalcifying solution containing chelating agents in dilute HCl, while the treatment time varied depending on the size and hardness of the specimens. After fixation and decalcification, the specimen was embedded in paraffin wax, sectioned into 5-micron thick slices, and placed on glass slides. Specimens with less calcification were sectioned without decalcification. These slices were then processed with standard H&E staining and cover-slipped. The H&E-stained slides were imaged using either the standard optical microscope or the digital whole slide scanning (Leica Aperio AT2) with a 40X objective. UV-PAM virtual histology via CycleGAN. To reconstruct the UV-PAM images, we first calculated the PA amplitude of each A-line signal after the Hilbert transform. The pulse energy measured by the photodiode was used to normalize the PA amplitude and compensate for the laser pulse fluctuation. The axial position of the specimen surface was calculated by detecting the peak of the A-line signal after the Hilbert transform. The 2D MAP (maximal amplitude projection) image was self-normalized. Since the PA amplitude of the contrast is proportional to its absorption coefficient, it can be used to effectively differentiate cell nuclei, cytoplasm, and the background. The cell nuclei have the largest absorption coefficient at 266 nm and the highest PA signals. After calculating the grayscale UV-PAM, we use a trained neural network to perform virtual H&E staining, which is more familiar to pathologists and thus easier for them to interpret. We use the CycleGAN architecture 36 , shown in Fig. 6 , which can learn how to map images from the UV-PAM domain, PA, to the H&E domain, HE, without the need for well-aligned image pairs. We use an adversarial loss to learn the transformations G: PA HE and F: HE PA, such that the images G(PA) and F(HE) are indistinguishable from HE and PA, respectively. The discriminators are trained to distinguish between real images and those produced by the generators. The loss function for D H E is given by 44 (1) l D H E = D H E ( G ( P A ) ) 2 + ( 1 − D H E ( H E ) ) 2 , where P A is a UV-PAM image patch, and H E is an H&E-stained image patch. Similarly, the loss function for D P A is given by (2) l D P A = D P A ( F ( H E ) ) 2 + ( 1 − D P A ( P A ) ) 2 . The generators are trained to try and fool the discriminators by producing images that match the statistical properties of the target domain. To ensure G doesn’t simply produce convincing but irrelevant H&E images, an additional loss term is necessary. Conventionally, this would be the l 2 or l 1 norm loss between the network output and some known ground truth image. However, this requires well-aligned paired datasets, which are challenging to acquire after sample preparation. Instead, the CycleGAN architecture learns the inverse transformation so that cycle consistency can be used to ensure the images are of the same structures. The total loss for the generators is (3) l G = ( 1 − D H E ( G ( P A ) ) ) 2 + ( 1 − D P A ( F ( H E ) ) ) 2 + λ ∣ F ( G ( P A ) ) − P A ∣ + λ ∣ G ( F ( H E ) ) − H E ∣ , where the regularization parameter λ is set to 10. The generators are residual networks consisting of an input convolutional layer, two convolutional layer and downsampling blocks, nine residual network blocks, two convolutional and upsampling blocks, and finally, an output convolutional layer 45 . Instance normalization and rectified linear unit (ReLU) layers are used after each convolutional layer. For the discriminator, we use PatchGAN consisting of convolutional layer and downsampling blocks, which classify whether the image is real on overlapping 70x70 pixel image patches 46 . This patch size is a compromise between promoting high spatial frequency fidelity and avoiding tiling artifacts. In the discriminator networks, instance normalization and leaky ReLU (lReLU) layers, l R e L U ( x ) = max ( 0.2 x , x ) are used after each convolutional layer. Anti-alias downsampling and upsampling layers are used in both the generators and discriminators to improve shift invariance 47 . The training dataset consisted of UV-PAM images of undecalcified bone specimens. These images were converted into 17940 and 26565 286x286 pixel image patches for UV-PAM and H&E histology, respectively. During training, these were further randomly cropped to 256x256 for data augmentation. The training was performed with the Adam solver with a batch size of 4 and an initial learning rate of 0.0002, decaying to zero over 100 epochs 48 . Once trained, we used the generator G to transform UV-PAM data in overlapping 256x256 pixel image patches, which were recombined with linear blending. To validate the virtual histology performance, we have quantitatively assessed the accuracy of our virtual staining method. We have segmented the cell nuclei in comparative regions of interest to compare their numbers, sizes, and densities. The nuclear segmentation was performed via Qupath 49 , using the default cell detection settings with the threshold set to 0.3 to reduce false positives. The cell counts, average nuclear areas, and average nearest neighbor internuclear distances were calculated for comparison of the UV-PAM virtual histology and H&E results.

Supplementary Material Supplementary Information Supplementary Movie 1 Supplementary Movie 2 Supplementary Movie 3