Machine Learning Based Real-Time Image-Guided Cell Sorting and Classification.

System design of machine learning based real-time image-guided cell sorter As shown in Figure 1 (a) , the image-guided cell sorting and classification system consists of: (1) an imaging optical system with a spatially coded optical filter to perform spatial-temporal transformation, (2) a real-time image processing and feature extraction module, (3) an off-line post processing module for construction of cell images for human vision and generation of cell classification criteria, and (4) a microfluidic chip integrated with an on-chip piezoelectric (PZT) cell sorting actuator. The optical filter encodes the fluorescent or light scattering signal of a cell into a temporal photocurrent waveform in the output of a photomultiplier tube (PMT) detector. Through a mathematical transformation described in [ 22 , 25 ], the 1D time domain signal is transformed into a 2D cell image. Due to the simplicity of the transformation algorithm, real-time signal processing can be implemented to extract image features of each cell to allow cell sorting based on these image features. Machine learning is needed to generate and adjust image features to guide cell sorting [ 26 – 28 ]. To start, training samples are flowed through the system to produce a set of training data. Off-line processing is employed to construct high resolution, cell images to interface with users whose inputs will aid the selection and adaptation of cell classification criteria for real-time sorting, a method of supervised machine learning. During the real-time cell sorting experiments, real-time processing module reconstructs cell images, extracts image features and makes sorting decisions based on the off-line trained sorting criteria. When a decision is made to sort a cell, a voltage pulse is applied to the on-chip PZT actuator, which instantaneously bends the bimorph PZT disk to deflect the cell away from the central flow into the sorting channel. For a proof-of-concept prototype, we have used a field-programmable-gate-array (FPGA) platform to implement real-time image processing which produces a latency of a few milliseconds. Higher performance GPU processors can reduce the processing time by 100 folds or more to microseconds.

Optical imaging setup

In the optical imaging system, suspended single cells flow in a microfluidic channel made of soft-molded polydimethylsiloxane (PDMS) bonded to a glass substrate. Sheath flow is used to hydro-dynamically focus the travelling cells to the center of the microfluidic channel. At the interrogation zone, each flowing cell is illuminated simultaneously by a 500mW 455nm LED (Thorlabs) and a 100mW 488nm laser (iBeam-SMART, Toptica) to generate bright field and fluorescent images. The output beam of 488nm laser is collimated, focused and expanded to illuminate a 100μm (x-direction) by 250μm (y-direction) area. The LED light is collimated and focused at the laser illumination area. Both the fluorescent emissions and the transmitted bright field signal are detected by PMTs (H9307–02, Hamamatsu). To accommodate the geometry of the microfluidic device, the laser beam is introduced to the optical interrogation area by a 52-degree miniature dichroic mirror positioned in front of a 50X objective lens (NA=0.55, working distance=13 mm, Mituyoyo). The LED is placed at the opposite side of the channel and the light is focused to the same position as the laser beam. The spatially coded optical filter is inserted at the image plane in the detection path. The pattern of the filter is shown in Figure 1 (b) . With the spatial filter, fluorescence/scattering from different parts of the cell will pass different slits at different times. As a result, the waveform of the fluorescent/scattering signal from the PMT consists of a sequence of patterns separated in time domain, and each section of the signal in the time domain corresponds to the fluorescent/scattering signal generated by each particular segment of the cell. After the light intensity profile over each slit is received, the cell image of the entire cell can be constructed by splicing all the profile together. The image resolution in x- (transverse to the flow) direction is primarily determined by the number of slits on the spatial filter, and the resolution in y- (flow) direction is mainly determined by the sampling rate and cell flow speed. In our system, the raw image resolution is 2 μm in x-direction and 0.4 μm in y-direction. Dichroic mirrors are used to route the desired emission bands to their respective PMTs.

Real-Time image processing

The real-time image processing module is implemented in a field-programmable-gate-array (FPGA) platform (National Instrument cRIO-9039). The processing module performs the functions of cell detection and image processing, to be discussed next. Both the bright field and fluorescent images of a cell are reconstructed from respective PMT readouts. Each image covers a field-of-view of 20X20µm 2 with 50X50 pixels, matched to the finest spatial resolution achievable by the system. All relevant image features are calculated and compared against the cell selection (gating) criteria to make sorting decision.

Cell detection algorithm

The cell detection function determines whether a cell exists within a certain time interval, and subsequently instructs the system whether to store and transmit the signal over this time interval. The system records the outputs from PMTs in the First-in First-out (FIFO) data structure over a chosen length of time. Each time a new set of PMT readout enters the FIFO, cell detection algorithm is activated to determine whether there is a cell within the optical system’s field of view. As soon as the system detects the presence of cell within the data set, the system processes the data immediately to construct images and extract image features. Otherwise, the system continues to examine the next set of PMT readout. Since fluorescent and bright-field images are generated simultaneously, only one fluorescent signal is used in cell detection algorithm. To shorten the processing time, we calculate the Brightness of signal by integrating the fluorescent intensity stored in the FIFO. The cell detection algorithm is shown in Supplementary Figure 1 . First, the Brightness is compared to a preset threshold defined as Threshold1 . If the Brightness is greater than Threshold1 , the system assumes a cell is entering the field of view. The calculated Brightness enters a FIFO data structure named FIFO Brightness . The length of FIFO Brightness is 100. Second, the time derivative of Brightness is evaluated to check if the cell is within the field of view. The time derivative of Brightness can be represented by the quantity of “difference”, defined as (max Brightness -min Brightness )/max Brightness , where max Brightness and min Brightness are the maximum and minimum elements in FIFO Brightness . If the “difference” is smaller than a preset Threshold2 (e.g. Threshold2 = 5%), the system considers the cell is within the field of view, and the function of real-time image processing is activated. The values of max Brightness and min Brightness are updated each time a new element enters FIFO Brightness . The algorithm to calculate max Brightness and min Brightness is shown in Supplementary Figure 2 . Third, Brightness is compared to Threshold1 again as time goes on. If the Brightness falls below Threshold1 at some point of time, the algorithm determines the cell has left the field of view. Brightness is calculated from the PMT readout stored in the FIFO data structure. The FIFO for fluorescent signal is referred as FIFO PMT . Each time when a new PMT readout enters FIFO PMT , Brightness value is updated by including the latest PMT readout and excluding the dequeued element in FIFO PMT . Brightness is calculated by equation 1 . (1) B r i g h t n e s s [ t n ] = B r i g h t n e s s [ t n − 1 ] + V o l t a g e P M T − F I F O P M T [ d e q u e u e ] where B r i g h t n e s s [ t n ] is the updated Brightness , B r i g h t n e s s [ t n − 1 ] is Brightness of last time step, V o l t a g e P M T is the incoming PMT readout, which is the enqueued element of FIFO PMT , and F I F O P M T [ d e q u e u e ] is the dequeued element in FIFO PMT .

Image processing algorithm

For all applications, we essentially follow the same flow for real time image processing, involving denoising, image resizing, contour definition, area calculation, feature enumeration, etc. Some of the processes can run in parallel to simultaneously extract multiple image derived features pertinent to image-guided sorting. In the following, we depict the specific processes applicable to each specific experiment. Real time image processing algorithm for protein translocation experiment. The image processing algorithm is illustrated in Supplementary Figure 3 . The image processing algorithm includes the following steps: (1) Denoise PMT signals with a 10 th -order Hamming low-pass filter. (2) Reconstruct both bright-field and fluorescent images from PMT signals. Since both bright-field and fluorescent signals are generated by the same slit although from different light sources, they are well synchronized. Thus the image reconstruction algorithm only needs to be launched once for both bright-field and fluorescent images. (3) Resize the images from 10×50 pixels (due to asymmetric resolution in raw images) to 50×50 pixels. (4) Detect contours of cell images by first converting the grayscale images to binary images, then eliminating spurious noise with open filter, and finally applying the contour detection algorithm to the binary images. (5) Extract all image-derived parameters. Examples of image-derived parameters that can be extracted from each cell in real time are shown in Supplementary Table 1 . As discussed later in this paper, these image-derived parameters are evaluated with Receiver Operating Characteristics (ROC). The 3 parameters receiving the highest ROC score are used as default parameters for real-time image-guided sorting. The latency of real-time processing algorithm is 5.8ms/cell with the current FPGA system, and the latency can be reduced by over 100 times with high performance GPU(e.g. NVIDIA QUADRO P6000). Real time image reconstruction and speed detection. An optical spatial filter consisting of 10 slits is placed at the image plane of the signal. With a 50X (or 20X if desired for extended focal depth) objective lens, the image projected onto the optical spatial filter is 50X (20X) times greater than the object in the microfluidic channel. For each cell travelling in the microfluidic channel, its PMT readout produces10 peaks, each corresponding to the cell’s fluorescent or bright-field transmitted signal passing one of 10 slits on the spatial filter. At a cell travel speed of 8 cm/s and at 200 kSamples/s, each peak consists of 50 sampling points. To reconstruct cell images, two factors need to be considered. First, since cells do not travel at a perfectly uniform speed, the actual number of sampling points for each of 10 peaks may vary slightly. Second, since both cell travelling speed and cell position within the 20 μm by 20 μm image area can vary, the starting time point of the PMT readout also varies. In the image reconstruction algorithm, we refer “m” to be the starting point of PMT readout, “n” to be the number of sampled points in each peak. Based on cell speed variations, n ranges from 46 to 51. This leads to a range of m from 0 to 519–10n. The algorithm sweeps m and n to assure the best combination of (m,n) to reconstruct the cell image. Summation of intensities at the starting point of each peak is calculated for every (m,n) combination. The combination yielding the smallest sum is the right (m,n) which we use to reconstruct the image. Travelling speed of the cell can be detected based on the calculated value “n”. The algorithm is shown in equation 2 and Supplementary Figure 4 . In Supplementary Figure 4 , The black “*” are starting points for each peak found by the reconstruction algorithm. (2) ( m , n ) = { ( m , n ) | min ∑ i = 0 10 ∑ m = 0 519 − 10 n F I F O P M T 1 [ m + i × n ] } After image reconstruction, grayscale cell images are resized to 50×50 pixel images by linear interpolation. Then resized grayscale images are converted to binary images based on the preset intensity threshold. The conversion is described as follows: (3) b i n a r y _ i m a g e ( i , j ) = { 1 , i f g r a y s c a l e _ i m a g e ( i , j ) > t h r e s h o l d 0 , i f g r a y s c a l e _ i m a g e ( i , j ) < t h r e s h o l d where ( i , j ) refers to the pixel located at row i and column j . In the step of open filter, we use a 3×3 pixel neighborhood in our image processing algorithm. Real time image contour detection. As shown in Supplementary Figure 5 , in the contour finding algorithm all the pixels in a binary cell image are scanned. For those pixels of non-zero value, the algorithm checks all eight pixels surrounding the center pixel in a 3×3 matrix. If the number of non-zero neighboring pixels is between 1 and 7, then this pixel is determined to be on the cell contour. Otherwise, the pixel is either inside or outside the contour. The criteria can be described in (4) : (4) i m a g e _ c o n t o u r ( i , j ) = { 1 , i f 0 < n < 8 & b i n a r y _ i m a g e ( i , j ) = 1 0 , O t h e r w i s e where n is the number of non-zeros pixels surrounding the center pixel of the 3×3 matrix.

Real time image processing algorithm for sorting

MDCK cells according to the number of particles on cell surfaces. The image processing algorithm for sorting MDCK Madin-Darby Canine Kidney Epithelial Cells based on the number of beads bonded to the cells is essentially the same as the previous cases except a top-hat filter with 7X7 pixels is used to extract features of small particles. The entire algorithm takes about 6 ms with current hardware. The flow chart of real-time image processing algorithm is shown in Supplementary Figure 6 . Real time image processing algorithm for sorting human glioblastoma cells by the extent of radiation induced DNA damage. The image processing algorithm is illustrated in Supplementary Figure 7 . The same top-hat filter with 7X7 pixels used before is also applied to remove background in gamma-h2ax image. Also the same algorithm is used to convert both the GFP image and the background removed gamma-h2ax image into binary images, and extract all image-derived parameters. The latency of the algorithm is about 6.7ms/cell.

Cell sample preparation

Translocating pEGFP-GR plasmids to HEK-297T human embryonic kidney cells. First GR-GFP plasmid DNA is obtained from bacterial culture. Then two plates of HEK 293T cells are cultured and transfected with GR-GFP. After transfection, cells are cultured for 2~3 days. Then one plate of cells is treated with dexamethasone. The dexamethasone treatment is supposed to cause migration of pEGFP-GR protein from cytoplasm to nucleus. For untreated cells, pEGFP-GR protein stays in cytoplasm. Bond fluorescent beads to MDCK cells. MDCK cells are cultured in a 10cm diameter dish. Then add 100µL solution of 1μm diameter fluorescent beads to the culture dish and keep the cell culture overnight. In the final step, cells are fixed and stained with Carboxyfluorescein succinimidyl ester (CFSE). Irradiation and antibody staining of Human Glioblastoma Cells. To induce DNA double-strand breaks (DSB), GFP labeled Human Glioblastoma Cells (GBM-CCC-001) are treated with 6Gy irradiation. The treated cells are washed once with phosphate buffered saline (PBS) and fixed with 1% paraformaldehyde 30 minutes post irradiation. The fixed cells are washed with PBS twice. Then 70% ethanol is added to the cells and the cells are incubated on ice for 1 hour. After ethanol treatment, cells are washed with PBS twice and incubated in 1% TritonX-100 at room temperature for 10 minutes. Then cells are washed with PBS once and incubated in 5% Bovine Serum Albumin (BSA) in PBS for 30 minutes at room temperature on shaker. Then cells are washed with PBS once and incubated in Anti-phospho-Histone H2A.X (Ser139) Antibody, clone JBW301 at 1:300 dilution on ice on shaker for 1 hour. After the primary antibody treatment, cells are washed twice with 5% BSA and incubate in PerCP/Cy5.5 anti-mouse IgG1 Antibody at 1:100 dilution on ice on shaker for 1 hour. At last, the stained cells are washed twice with 5% BSA and resuspended in 1:3 diluted stabilize fixative buffer in MilliQ water.

Supplementary Material Supp info