Superresolution structured illumination microscopy reconstruction algorithms: a review.

Development of SIM reconstruction methods

Development of OS-SIM reconstruction methods

In addition to commonly used RMS algorithms, there are other OS-SIM reconstruction methods, including projection (i.e., sum, maximum, and super-confocal), scaled subtraction of the out-of-focus estimation, and a modified version of Fourier-space treatment (known as patterned excitation microscopy processing). Heintzmann et al. 30 discussed and analysed these three strategies based on a regular array of point illumination patterns. Simulation and experimental results demonstrated that while projection methods, especially the super-confocal method, have exceptional optical sectioning characteristics, they also exhibit more residual patterning than the other two methods. PEM processing can provide high resolution but is computationally expensive. In 2014, Schropp & Uhl 31 pointed out an alternative structured illumination microscopy employing square or hexagonal illumination patterns (Fig. 8a–d ). Rather than shifting a regular array of point illumination patterns row by row along the x- and y-axes, they shifted 2D illumination patterns unidirectionally along pattern-dependent angles. This approach results in an isotropic power spectral density and opens new possibilities for high-resolution imaging in biological and materials science applications. Fig. 8 Optical section images reconstructed under different OS-SIM algorithms. a – d Comparison between single phase images illuminated by square a or hexagonal c illumination patterns and quasiconfocal images b or d 31 . e HiLo technique applied to a pair of fluorescent pollen grains located at slightly different depths 36 . Top row: speckle-illumination image (left), uniform illumination image (right); middle row: intermediate low-pass image (left), intermediate high-pass image (right); bottom row: composite full resolution optically sectioned image (left), extended focus image obtained from a maximum intensity projection of 80 slices separated by 0.5 μm steps. f Whole-brain imaging of the anterograde projections of AAV-YFP-labelled neurons in the motor cortex 43 . a – d © The Journal of Microscopy, e © The Optical Society, f © Nat Methods To streamline the measurement setup and enhance imaging speed, a two-frame OS-SIM algorithm was proposed, such as the HiLo algorithm 36 or the amplitude demodulation algorithm based on the Hilbert-Huang transform 76 , 77 . HiLo microscopy employs a nonuniform (fixed-frequency or speckle) image to provide low-resolution information with spatial frequencies below a user-specified cut-off frequency, while a uniform illumination image is acquired to provide high-resolution information with a spatial frequency above the cut-off frequency. By appropriately setting the cut-off frequency and fusing the low- and high-resolution information, a full-resolution optically sectioned image can be recovered (as shown in Fig. 8e ). However, in the HiLo algorithm, additional attention must be paid to the process of fusing the low- and high-resolution information. Moreover, the amplitude demodulation method based on the Hilbert-Huang transform necessitates two mutually π phase-shifted raw structured images. However, the imaging speed of this two-frame OS-SIM method is still constrained by the camera’s capabilities (~34 fps). Subsequently, a single-frame OS-SIM algorithm was proposed. In 2019, Wang et al. 78 proposed a Fourier bandpass filtering algorithm to reconstruct optical section images by shifting the in-focus signals to the +1st order in the Fourier domain. However, this method requires perfect separation of Fourier spectrum components using a bandpass filter, which can reduce the lateral resolution. In 2021, Zhong et al. 43 further developed a high-definition fluorescent micro-optical sectioning tomography (HD-fMOST) method for whole-brain optical imaging with submicrometer-voxel resolution, based on the LiMo method (Fig. 8f ). In addition, deep learning has demonstrated its effectiveness in OS-SIM techniques. It can be used to solve issues such as high computational costs and oversimplification of optical systems for some deconvolution techniques 79 , 80 (such as Wiener filtering 81 and Richardson–Lucy (RL) deconvolution 82 , 83 ). For example, in 2018, Zhang et al. 84 developed a deep learning-based computational algorithm, which only requires a single wide-field image and a corresponding optical sectioning reference image to train a convolutional neural network (CNN). This algorithm can reconstruct optical section images with lower noise, fewer artifacts, and higher imaging depth at an optimized frame rate of 14 Hz. In 2021, Chai et al. proposed a one-shot optically sectioned method called Deep-OS-SIM 85 , which is based on deep-learning techniques. Unlike other methods that use low entropy wide-field images, this approach takes full advantage of the high entropy properties of structured illumination images to train a CNN model. Optical-sectioning imaging using this method only requires a single image for decoding, thereby improving the raw image acquisition efficiency by 50% compared to the two-frame HiLo method. However, similar to other deep learning-based methods, this method requires expertise in deep learning, and it is currently restricted to specific projected illumination patterns and samples for OS-SIM. If the statistical characteristics of samples or the illumination pattern modes change, retraining for variations in imaging parameters will be necessary.

Development of 2D-SIM reconstruction methods

Parameter estimation SR-SIM reconstruction is essentially an ill-posed inverse problem. As mentioned in the subsection “Basic SR-SIM algorithm”, solving and separating the spectra of the sample and then moving them back to their correct positions is crucial during SIM reconstruction. This process requires precise knowledge of structured illumination patterns, especially in those techniques that rely on high-order harmonics to improve resolution 86 , 87 . Even slight deviations in the reconstruction parameters from the correct ones can lead to noticeable artifacts in the reconstructed images, such as ghosting and fringing. The periodic illumination pattern parameters include the illumination frequency vector, angle, phase, and modulation depth. Although the illumination frequency vector can be determined with high precision and reproducibility using structured illumination generators such as SLM 52 – 54 and DMD 55 , 88 , the initial phase is difficult to determine accurately without prior knowledge. Moreover, factors such as sample movement, system-dependent optical aberrations, and photobleaching can cause the pattern position to shift in the raw images, making it challenging to estimate the parameters based on prior knowledge. As a result, numerous algorithms have been proposed for postprocessing parameter estimation of periodic illumination patterns. Based on the work of Gustafsson et al. 12 , the frequency vector can be retrieved by iteratively maximizing the cross-correlation (COR) of the overlap areas between the first separated and zero orders, assuming an equidistant phase distribution (Fig. 9e ). The modulation depth and phase offset can then be obtained by calculating the absolute value and angle of the complex factor of the overlap areas, respectively. One advantage of this method is that the separation of orders only relies on the relative phase between individual images. To speed up the parameter estimation process, a notch filter (Fig. 9d ) was later introduced to roughly extract the peak position of the +1 order-separated spectrum before optimizing the maximum cross-correlation 89 – 91 . Fig. 9 Parameter estimation in 2D-SIM. a Raw image of actin from a homemade sample U2OS cell acquired using a 2D-SIM system. b The corresponding Fourier spectrum distribution of ( a ). c One of the separated spectra, +1 order spectrum. When estimating the frequency vector k , there are two steps: d Rough estimation 91 . Combined with a notch filter, the peak position in the +1 order-separated spectrum is extracted; e Precise estimation 12 . The cross-correlation between the separated 0- and +1 order spectra is maximized. There are four methods when estimating the initial phase φ: e COR 94 . Sum the pixel values of the overlap areas between the separated 0 and +1 order spectrum, and extract the angle of the complex factor obtained by summation; f POP 92 , 93 . Directly extract the phase at the peak in the raw Fourier image; g ACR 95 . Sum the overlapping areas of the autocorrelation of the respective raw Fourier images; h IRT 96 . Sum the raw images with phase shift, transform the summation into Fourier space, and then extract the phase of the estimated peak In many cases, residual orders may exist in the spectra separated by unmixing due to imprecise individual illumination phases, which can be minimized by estimating the phase offset of each illumination pattern. Shroff et al. 92 , 93 proposed a Fourier domain phase of the peak (POP) estimation method without prior knowledge of the phase shifts, which is suitable for live-cell imaging, as shown in Fig. 9f . However, this method is inappropriate for high-frequency illumination patterns such as those in TIRF mode. As the phase information is directly estimated from the raw image, three conditions should be simultaneously met: the raw image has a high SNR, the high-frequency component decays rapidly enough, and the illumination pattern frequency is lower than the cut-off frequency determined by the support area for OTF detection. In 2013, Wicker et al. 94 developed an iterative optimization method to determine the pattern phases using the COR between separated components in cases where the illumination pattern was too fine to detect. Although this method can robustly determine the relative pattern phase in SIM raw images with a precision below λ/100, its iterative nature inevitably results in longer computation times. In a later study, Wicker 95 presented a faster and more robust noniterative autocorrelation reconstruction (ACR) method for determining a pattern’s phase, as shown in Fig. 9g . This method calculates each illumination pattern’s phase from the autocorrelation of its corresponding raw Fourier image and typically achieves precision less than λ/500 at realistic SNR levels. In 2016, Lal et al. 27 provided a comprehensive theoretical overview of 2D-SIM algorithms, including determining the illumination frequency vector using the ACR method and estimating the phase offset through iterative optimization of the correlation function between the illumination pattern and the sample’s Fourier image. In 2016, Zhou et al. 96 proposed an image recombination transform (IRT) algorithm (Fig. 9h ), which utilizes the phase difference among three raw images to obtain a high-precision initial phase. By combining this algorithm with their own DMD-projection-based, multicolor LED-illumination SIM system, they achieved low excitation intensity fluorescence imaging even less than 1 W∕cm 2 . However, the IRT algorithm only considered a phase shift of π/2 and two orientations separated by 90°, limiting its application in general scenarios. To overcome this problem, Zhao et al. 97 reported an enhanced IRT algorithm that can handle arbitrary phase shifts. It should be noted that the POP, ACR, or IRT algorithms cannot guarantee accurate phase estimation when the raw images have low SNR or weak modulation depth. In addition, for certain periodic samples, the ACR algorithm requires the modulation vector to be distinct from the spatial frequency vector. To address these issues, Cao et al. 98 proposed a noniterative phase estimation method based on an inverse matrix by incorporating extra matrices into the phase estimation algorithm. However, since the parameters of the inverse matrix can affect the phase estimation error, it is essential to select an appropriate parameter set to decrease the average phase error, which lacks objectivity. In 2022, Qian et al. 99 , 100 introduced principal component analysis (PCA) into SIM for the first time to identify the frequency vectors and pattern phases of the illumination pattern (Fig. 10 ). They demonstrated that PCA-SIM can achieve fast and accurate noniterative parameter estimation (with frequency vector precision below 0.01 pixels and relative phase precision of 0.1% of 2π under typical noise levels) that is also robust at low SNRs. This allows for real-time superresolution imaging of live cells in complicated experimental scenarios. Fig. 10 Schematic diagram illustrating the PCA-SIM estimation algorithm. a Flow chart of the parameter estimation in PCA-SIM 100 . b – d Output results of red boxes in a . b The first-order spectrum after shifting the integer-pixel frequency vector. c The phases of the phasor matrix obtained from b after different operations: the original phases c1, phases after applying the masking operator c2, phases after applying PCA c3, and phases after least-squares fitting c4. d Obtained frequency vector with subpixel accuracy Fourier domain reconstruction algorithms FDR algorithms, also known as direct methods, are the most used methods for SIM reconstruction. As mentioned earlier, the first proposed FDR algorithm was the generalized Wiener filtering algorithm. However, this algorithm is not only affected by imprecise parameter estimation but also vulnerable to systematic aberrations and the SNR of raw images 101 . In addition, its reconstruction speed does not meet the requirements of real-time imaging. To address these issues, various improved FDR algorithms have been proposed, which can be classified into the following three parts. Part 1: SIM reconstruction for suppression of optical aberrations Optical aberrations, such as systematic spherical aberration and sample-induced aberrations, not only cause artifacts, loss of resolution, and reduced image contrast in SIM-reconstructed images but also limit the technique’s application to samples thinner than a single cell 102 , 103 . Even small optical aberrations, which have minimal influence on a diffraction-limited image, can cause severe artifacts in SIM-reconstructed images. Moreover, the degree of image distortion caused by spherical aberration is influenced by a range of physical parameters, including cover glass thickness, a refractive index of the sample embedding medium/immersion oil, and sample temperature, all of which are empirical and add to the complexity of the problem. Typically, there are two physical ways to improve spherical aberration: choosing an appropriate immersion oil or adding adaptive optics in the imaging path 87 , 104 – 107 . In recent years, algorithms have been proposed to improve SIM imaging quality, including PSF engineering 91 , 108 and tiled reconstruction methods 109 , 110 . In 2016, Perez et al. 108 proposed an RL-based deconvolution 111 filtering step for both raw and reconstructed images. This method depends only on unbiased filtering steps during reconstruction, without requiring any parameter tuning. However, it does not suppress the effects of out-of-focus background and spectral inhomogeneity on the reconstructed image. In 2020, Wen et al. 91 presented a high-fidelity SIM (HiFi-SIM) reconstruction algorithm, which engineers the effective PSF into an ideal form. By combining a normalized cross-correlation method with a spectrum notch, HiFi-SIM can automatically estimate the illumination pattern parameters. Furthermore, it can effectively reduce common artifacts without sacrificing delicate structures and improve axial sectioning for samples with a strong background. Here, we present schematic diagrams of two FDR reconstruction procedures, the generalized Wiener filtering method (OpenSIM) 27 and HiFi-SIM (Fig. 11 ). Apparently, the spectrum reconstructed by HiFi-SIM is flatter and smoother than that reconstructed by OpenSIM. Additionally, the reconstructed image produced by HiFi-SIM has higher contrast while preserving details. Fig. 11 Fourier domain-based SIM reconstruction methods. The 2D superresolution SIM reconstruction process using two different Fourier domain reconstruction methods: the generalized Wiener filter (OpenSIM) 27 and high-fidelity SIM (HiFi-SIM) 91 In 2020, Hoffman et al. 109 developed a tiled reconstruction method to achieve artifact-free whole-slide imaging with a large field-of-view in SIM, which can alleviate many common SIM reconstruction artifacts caused by global parameter estimation errors. In this method, each raw image was divided into overlapping tiled subsets, and each subset was reconstructed using independently measured or user-optimized parameters. These subsets were then reassembled into a composite superresolution image covering the original field of view. Furthermore, Johnson et al. 110 proposed a Bayesian estimation-based SIM reconstruction method that combined SIM with image-stitching and devignetting methods to provide artifact-free stitched images with optical sectioning and superresolution properties. The results of five different samples demonstrated that the stitched SIM images were useful for intraoperative histology. Part 2: SIM reconstruction under a low SNR situation When acquiring raw SIM images, using a higher signal level can result in better-quality reconstructed images. However, this can accelerate sample photobleaching and limit the number of time points for live-cell images. On the other hand, acquiring raw images at low signal levels can result in considerable noise, leading to artifacts in the reconstructed image. To minimize these artifacts, the parameters in the Wiener filter 27 are typically set manually, which is user-dependent and lacks objectivity. Subsequently, a series of regularization-based iterative optimization methods were proposed based on the prior knowledge of structured illumination patterns. For example, in 2014, Chu et al. 112 introduced a total variation (TV) constraint for SIM reconstruction. The algorithm can image at least 15 times more time points than a traditional Wiener filtering reconstruction method. However, the reconstructed image contains stepped artifacts due to the overcorrection of edge information. Lukeš et al. 113 proposed a SIM method based on the maximum a posteriori (MAP) probability. Combined with homodyne detection, this method can suppress out-of-focus information, improve spatial resolution, and enable the reconstruction of 2D and 3D images of cells, even with weak signals. They later developed an open-source, modular function set, SIM-Toolbox for MATLAB, which supports OS-SIM and SR-SIM image reconstruction 114 . In 2018, Huang et al. 115 reported a deconvolution method for SIM, called Hessian-SIM, which utilized prior knowledge of the continuity of multidimensional biological structures based on Hessian matrices. This method enabled ultrafast live-cell superresolution imaging (such as structural dynamics of mitochondrial cristae) with a spatiotemporal resolution of 88 nm and 188 Hz. Moreover, compared with TV-SIM, Hessian-SIM can retain more image details while reducing noise. In the same year, Boulanger et al. 116 proposed a nonsmooth convex optimization method for SIM reconstruction. However, this method requires heavy computation and takes a long time to converge. In 2020, Yu et al. 117 implemented a second-order optimally regularized SIM (sorSIM) method, which utilizes second-order partial derivatives to suppress the stepped artifacts that appear in TV-SIM. This method achieves a balance between resolution enhancement and noise immunity. In 2021, Zhao et al. 118 added the sparsity of biological structures to Hessian-SIM and proposed a Sparse-SIM deconvolution algorithm, which can achieve a resolution of ~60 nm at a frame rate of up to 564 fps. This method also enables four-color, 3D live-cell superresolution imaging at ~90 nm resolution. However, the resolution enhancement of Sparse-SIM will depend on factors such as the image SNR and optimal parameter selection, which can be cumbersome for different biological samples. In 2022, Zhou et al. 119 established a nonuniform sCMOS noise model and proposed a corresponding noise-corrected SIM reconstruction algorithm based on the stable biconjugate gradient descent algorithm (Bi-CGSTAB) 120 and split Bergman algorithm 121 . Simulation results indicated that this noise-corrected SIM reconstruction algorithm can effectively suppress sCMOS noise-related reconstruction artifacts. Recently, Hou et al. 122 developed an MRA deconvolution algorithm for fluorescence images, which uses framelet and curvelet domain sparsity to regularize the solution. This algorithm allows fine detail recovery even with a negative SNR and provides more than twofold physical resolution enhancement with fewer artifacts than maximum likelihood estimation (MLE) methods. Furthermore, they developed a DeepMRA deconvolution algorithm, which can address severer backgrounds and better preserves high-frequency and low-intensity details that are commonly disrupted by other algorithms. Part 3: SIM reconstruction speed improvement To enable real-time observation for live-cell imaging, various attempts have been made to enhance the SIM imaging speed. In addition to improvements in optical system hardware, several algorithm optimizations have been proposed, such as frame reduction of raw images, rolling reconstruction, and GPU acceleration. Preliminary results have demonstrated that a superresolution image can be reconstructed from four raw images 123 , 124 . In 2017, Ströhl and Kaminski proposed acquiring three raw images under different illumination orientations, and indicated that the frame rate can be doubled by using the joint RL deconvolution algorithm 125 . However, in 2018, Lal et al. 126 found only approximately a 1.5× resolution enhancement in the final image reconstructed using three raw images and concluded that at least four raw images are required to double the resolution. In 2022, Zeng et al. 127 introduced polarization modulation to the frame reduction imaging model, proposing a complete and versatile imaging model called PRSIM. They indicated that for polarized samples, polarization-related artifacts can be reduced by combining a Fourier domain iterative reconstruction algorithm. However, these frame reduction methods rely on assumptions about the image-formation process, and the final reconstructed results are limited by the type of noise. In 2017, Ma et al. 128 proposed the combination of SIM with an interleaved reconstruction strategy (SIMILR) to maximize the use of each subframe of the acquisition series. This method enabled the observation of highly dynamic structures, such as the endoplasmic reticulum, which undergoes continuous rapid growth or shape changes. Later, in 2018, Guo et al. 129 employed SIMILR in grazing incidence SIM, which utilizes highly inclined laminar illumination 130 . This method achieved dynamic imaging of events near the basal cell cortex at 97 nm resolution and 266 fps over thousands of time points. In addition, combining Hessian-SIM with a ‘rolling’ reconstruction procedure allowed for a maximum number of frames in time-lapse imaging of up to 291 fps 115 . Similarly, Sparse-SIM achieved a frame rate of 564 fps 118 . In terms of GPU acceleration, various SIM reconstruction methods have been developed using programming languages such as CUDA C++, Java, and Python 89 , 131 – 133 . For example, in 2019, Markwirth et al. 132 proposed a video-rate immediate GPU-accelerated open-source reconstruction (VIGOR) method by recreating, modifying, and extending the fastSIM 134 approach and image reconstruction software. The results demonstrated that multicolor SR-SIM can be reconstructed at video frame rates (25 reconstructed fps or more), with a delay of less than 250 ms between measurement and reconstructed image display. In 2021, Gong et al. 131 presented a GPU-accelerated SIM method using a hexagonal illumination pattern based on the Python language. This method can process over 239 input raw images (512 × 512 pixels) per second and generate over 34 superresolution frames per second at 1024 × 1024 pixels. However, it should be noted that in these GPU acceleration methods, the illumination parameters are estimated and calibrated in advance (e.g., using the COR algorithm) and then reused in the subsequent reconstruction, making it challenging to address complex dynamic operating environments, such as artificial interference and environmental perturbations, which can lead to drift in illumination patterns. Spatial domain reconstruction The SDR approach was initially proposed by Cragg & So during the development of SIM 135 , 136 . They created an enhanced 2D image by taking images at different phases and directions of a structured illumination pattern and forming a weighted sum. SDR requires the same number of raw images as the generalized Wiener filtering algorithm but is faster because it does not require Fourier transform operations. In 2021, Manton et al. 137 reported an equivalent AM signal demodulation method that used the structured illumination pattern as the carrier signal, the sample as the message signal, and the recorded data as the product of these two, i.e., the AM signal. They then heterodyned the AM signal with another sinusoidal pattern with the same phase and period as the carrier signal, realizing spectrum separation and recombination. Inspired by the series expansion of a function in mathematics 136 , Dan et al. 138 proposed another SDR method by computing the coefficient matrix (Fig. 12 , second row). Their results showed that this method reconstructed a superresolution image sevenfold faster than the FDR algorithm. However, this method did not address out-of-focus backgrounds. In 2022, Wang et al. 139 developed a joint space and frequency reconstruction (JSFR)-SIM by combining spatial domain processing with optical sectioning superresolution SIM implemented in the frequency domain (Fig. 12 , third row). By utilizing multithreading, they were able to reduce the execution time of reconstruction to 10.2 ms for raw images that were 512 × 512 pixels in size. Recently, the JSFR algorithm has been integrated with HiFi-SIM to form the joint space-frequency reconstruction-based artifact reduction algorithm for SR-SIM (JSFR-AR-SIM) 140 . However, all three SDR methods require parameter estimation, which is the most time-consuming step in the reconstruction process. Fig. 12 Spatial domain-based SIM reconstruction methods. The process of reconstructing SIM in the spatial domain involves several methods, including: spatial domain reconstruction SIM (SDR-SIM) 137 , 138 , joint space and frequency reconstruction SIM (JSFR-SIM) 139 , and shifting phase SIM (SP-SIM) 141 In contrast to these methods, Tu et al. 141 developed a parameter-free algorithm called shifting phase SIM (SP-SIM), which directly reconstructs superresolution images in the spatial domain (Fig. 12 , last raw). However, similar to the OS-SIM algorithm, the SP-SIM algorithm only preserves first-order spectral band information and discards the zero-order spectral band information during the derivation process. Consequently, low-frequency components of the reconstructed superresolution image using SP-SIM may be lost, resulting in lower image contrast compared to the FDR algorithm.

Open-source software

Open-source and open-access software packages for SR-SIM reconstruction have become more prevalent, and the once-opaque algorithms are now accessible to ordinary users. In this section, we provide a summary of existing 2D-SIM open-source reconstruction algorithms in Table 1 , which includes information on the number of raw images needed, achievable resolution, implementation methods, and more. Table 1 Open-source reconstruction algorithms for the 2D-SIM Method N/frames Language Resolution in X, Y/nm, or t/Hz Property Categories OpenSIM 27 9 MATLAB ~2-fold of the diffraction limit Classical 2D-SIM FDR OpenSIM-4 126 4 MATLAB ~2-fold of the diffraction limit Frame reduction FDR Hessian-SIM 115 9 MATLAB A spatiotemporal resolution of 88 nm and 188 fps Ultrafast and hour-long dynamic superresolution imaging Iterative algorithm Sparse-SIM 118 9 MATLAB ~60 nm resolution at a frame rate of up to 564 fps Sparse deconvolution Iterative algorithm fairSIM 89 , 90 9/15-3D slice Java ~2-fold of the diffraction limit Classical 2D-SIM with notch filter FDR VIGOR 132 9/15-3D slice Java Multicolor SR-SIM imaging at video frame- rates (25 reconstructed fps or more) GPU-accelerated FDR HexSIM 131 7 Python Over 239 input raw images per second at 512 × 512 pixels, generating over 34 SR fps at 1024 × 1024 pixels GPU-accelerated Frame reduction Hexagon illumination SDR + FDR SP-SIM 141 9 MATLAB ~2-fold of the diffraction limit No parameter estimation Suitable for speckle illumination pattern SDR JSFR-SIM 139 9 MATLAB The reconstruction time is 10.2 ms for raw images with 512 × 512 pixels OS-SR-SIM SDR + FDR SIM-Toolbox 114 9 MATLAB ~2-fold of the diffraction limit classical OS-SIM 2D-SIM with homodyne detection Iterative algorithm HiFi-SIM 91 9/15-3D slice MATLAB ~2-fold of the diffraction limit High fidelity PSF engineering FDR MRA 122 &DeepMRA 9 MATLAB ~70 nm fidelity-ensured resolution High fidelity Suppression of defocus background FDR Although some of the 2D-SIM reconstruction algorithms listed in Table 1 can reconstruct a single 3D image slice using a 2D-OTF, this does not constitute a true 3D-SIM image stack. This is because 3D-SIM reconstruction requires a system-specific 3D-OTF, preferably experimentally measured, to achieve high spatial resolution along the vertical axis. Without this information, the spatial resolution along the vertical direction may be limited. In addition, several open-source ImageJ plugins offer tools for assessing the quality of SIM images. For example, SIMcheck can analyse both SIM raw and reconstructed data, providing system calibration to help users acquire optimum raw data for successful image reconstruction 142 . Another plugin, NanoJ-SQUIRREL (superresolution quantitative image rating and reporting of error locations), can quantify artifacts in SIM-reconstructed images 143 . By comparing a reference image (generally diffraction-limited) with a superresolution image, a quantitative map of localized image artifacts can be generated and used to guide researchers in optimizing imaging parameters. Regarding resolution assessment methods, Koho et al. 144 proposed a method based on Fourier ring correlation (FRC) analysis, where a single image is divided into four subsets (i.e., two-image pairs) and used to estimate the effective PSF in Wiener and iterative RL deconvolution. In 2019, Descloux et al. 145 proposed a rapid image resolution estimation method called decorrelation analysis, which also uses a single image without prior knowledge. This method explores the highest frequency from the local maxima of the decorrelation functions, avoiding user-defined parameters. However, these methods may not be suitable for images with low SNR or artifacts, which could be interpreted as detailed information of the samples, leading to inaccurate estimated resolution. To analyse and compare the advantages and disadvantages of several open-source algorithms listed in Table 1 , we processed two sets of raw data of actin filaments labelled with AF-568 phalloidin dye collected under high SNR (i.e., SNR = 7.7 dB, PSNR = 20.53 dB for OpenSIM) and low SNR (i.e., SNR = −2.2 dB, PSNR = 15.75 dB for OpenSIM). It is clear in Fig. 13a–c that in the case of a high SNR of the raw images, HiFi-SIM can better eliminate defocus information while maintaining high image resolution. Although the fairSIM algorithm has a faster reconstruction speed than HiFi-SIM and OpenSIM, the reconstructed image contains more artifacts. Regarding the SDR algorithm, JSFR-SIM provides a resolution improvement effect comparable to OpenSIM, but it cannot suppress defocus information as effectively as HiFi-SIM. SP-SIM has the fastest reconstruction speed but lacks zero-order information, causing a discontinuity in the reconstructed images, making it difficult to assess the resolution improvement effect. In the case of low SNR, as shown in Fig. 14a–c , noise introduced during the reconstruction process can reduce the resolution of the final reconstructed image. Overall, the FDR algorithm is more robust to noise than the SDR algorithm. Fig. 13 Reconstruction results of different 2D-SIM algorithms under high SNR conditions. a The results reconstructed under averaging, OpenSIM, HiFi-SIM, Fair-SIM, JSFR-SIM, and SP-SIM. b Intensity profiles along the red line in ( a ). c SNR and PSNR statistical diagrams of each result in ( a ). d The results obtained by postprocessing OpenSIM results using regularization-based iterative algorithms, such as TV, Hessian, sparse, MRA, and DeepMRA. e Intensity profiles along the red line in ( d ). f SNR and PSNR statistical diagrams of each result in ( d ) Fig. 14 Reconstruction results of different 2D-SIM algorithms under low SNR conditions. a The results reconstructed under averaging, OpenSIM, HiFi-SIM, Fair-SIM, JSFR-SIM and SP-SIM. b Intensity profiles along the red line in ( a ). c SNR and PSNR statistical diagrams of each result in ( a ). d The results obtained by postprocessing OpenSIM results using regularization-based iterative algorithms, such as TV, Hessian, sparse, MRA, and DeepMRA. e Intensity profiles along the red line in ( d ). f SNR and PSNR statistical diagrams of each result in ( d ) A regularization-based iterative algorithm is applied based on the reconstruction output of OpenSIM. In the case of a high raw image SNR (Fig. 13d–f ), TV-SIM results in reduced resolution due to stepped artifacts, while Hessian-SIM maintains the resolution of the original OpenSIM output. Sparse-SIM, MRA, and DeepMRA can further enhance image resolution. In addition, Sparse-SIM and DeepMRA can effectively suppress defocus information. When the raw image SNR is low (Fig. 14d–f ), denoising reduces the resolution of the final image, regardless of the algorithm used. There is a trade-off between noise suppression and contrast enhancement. We organize the evaluation results of the above various methods in Appendix 3 .

Development of 3D-SIM reconstruction methods

Recently, the problem of multilayer 3D-SIM image reconstruction has been addressed and implemented. For example, in 2015, based on the generalized Wiener reconstruction theory mentioned in ref. 12 , Shao et al. developed and shared the 3D-SIM reconstruction software with CUDA acceleration 133 . This method has been widely applied in biological study 146 , 147 . In 2021, Smith et al. 148 presented a physically realistic noise model and provided three complementary reconstruction methods: true-Wiener-filtered SIM, flat-noise SIM, and notch-filtered SIM. Experimental results demonstrated that introducing notch filtering can partly overcome the trade-off between increasing contrast and suppressing noise. In the same year, Zhu et al. proposed an iterative algorithm called NGD-SIM based on gradient descent and a nonlinear optimizer RMSprop 149 . However, this algorithm is time-consuming. In 2022, Cai et al. proposed a TV-FISTA-SIM algorithm that combines TV with the fast iterative shrinkage threshold algorithm (FISTA) 150 to further improve imaging speed. Compared to the NGD-SIM algorithm, this algorithm achieves faster convergence speed and higher reconstruction fidelity when the SNR is as low as 5 dB. Additionally, in 2022, Cao et al. 151 proposed an Open-3DSIM algorithm by introducing spectrum filtering to further optimize the reconstruction results of traditional 3D-SIM and improve its friendliness to general users. They provided a MATLAB code, ImageJ version, and Exe application simultaneously. Experimental results demonstrated that Open-3DSIM has superior performance in suppressing artifacts and removing defocus information. It is important to consider the effect of motion artifacts on the quality of a reconstructed image if the sample is moving during imaging 54 . In wide-field microscopy, small sample movements may go unnoticed if they are smaller than the resolution limit. However, in SIM reconstruction, even low velocities may introduce artifacts, leading to a reduction in resolution and potentially misleading interpretations. Thus, Förster et al. proposed a frame difference method (FDM) 152 and its improved versions 153 to detect and locate motion artifacts in SIM images. However, these methods require 3D-stack data and are not executable for two-beam methods, such as nonlinear SIM. Furthermore, to reduce artifacts resulting from optical aberrations and enable 3D-SIM imaging in thick tissues, Lin et al. 105 proposed the AO-3DSIM system, which combines adaptive optics with 3D-SIM and processes 3D-stack data using the generalized Wiener filtering method. The AO-3DSIM system achieved a resolution of 150 nm laterally and 570 nm axially, along with optical sectioning, at a depth of 80 μm through Caenorhabditis elegans , compared to a resolution of 280 nm laterally and 930 nm axially in wide-field imaging. Table 2 summarizes the corresponding 3D-SIM open-source reconstruction methods for 3D-SIM. Table 2 Open-source reconstruction algorithms for the 3D-SIM Method Categories Language Property 3D-SIM 133 FDR CUDA C++ acceleration Classical 3D-SIM AO-3DSIM 105 Python Classical 3D-SIM Open-3DSIM 151 Fiji/MATLAB Suppression of noise artifacts Spectrum optimization True-Wiener-filtered SIM 148 MATLAB High contrast imaging Flat-noise SIM 148 Suppression of structural noise artifacts Notch filtered SIM 148 Higher image contrast than flat-noise SIM TV-FISTA-SIM 150 Iterative algorithm Fast convergence speed We compared and analysed three open-source 3D-SIM reconstruction algorithms, AO-3DSIM, SIMnoise (i.e., true-Wiener-filtered SIM), and Open-3DSIM, by testing another actin filament sample obtained from the open-source data of ref. 105 . As shown in Fig. 15 , the image reconstructed under the AO-3DSIM algorithm contained some artifacts caused by out-of-focus information (Fig. 15a ). Although these artifacts can be effectively removed by using SIMnoise, some details of the sample are also lost (Fig. 15b ). Open-3DSIM can better retain the detailed information of the sample while suppressing the defocus information (Fig. 15c ). However, it should be noted that the comparison results may vary depending on the specific sample and imaging conditions. Therefore, users should choose the appropriate algorithm based on their own requirements and considerations, such as speed, accuracy, noise suppression, and artifact reduction. Moreover, further development and optimization of 3D-SIM reconstruction algorithms are still needed to achieve higher resolution, faster computation, and higher robustness. Fig. 15 Reconstruction results under three typical 3D-SIM algorithms. a AO-3DSIM, b True-Wiener-filtered SIM, and c Open-3DSIM. a 1– c 1 Magnified images from the orange box in ( a – c ); a 2– c 2 Magnified images from the green box shown in ( a – c ) Development of blind-SIM reconstruction methods If the raw image SNR is too low, or the illumination patterns are distorted due to the inhomogeneity of the sample refraction index, parameter estimation-based reconstruction algorithms may fail to work. To overcome this problem, Mudry et al. 154 developed a blind-SIM reconstruction method in 2012 for illuminating samples with random light speckles. This method dramatically simplifies the experimental setup by not requiring knowledge of the illumination pattern. However, the temporal average of speckle illumination must be roughly homogeneous over the sample for it to work, limiting its wide application. In 2013, Min et al. 155 presented another speckle illumination microscopy by implementing a multiple sparse Bayesian learning (M-SBL) algorithm 156 . A threefold resolution gain was reported under the joint support constraints. Moreover, Ayuk et al. 157 extended the application of blind-SIM to periodic illumination patterns by introducing an additional Gaussian filter during the inversion procedure. It was shown that this filtered blind-SIM is as efficient as traditional SIM when the illumination pattern is periodic. Additionally, it is robust to distortion and misalignment. In 2014, by improving the Fourier ptychography (FP) algorithm proposed by Zheng et al. 158 , Dong et al. 159 proposed a pattern-illumination Fourier ptychography (piFP) method. This method is applicable to any unknown illumination pattern and has been used in computational photography and image-based rendering 160 . In 2015, Ströhl et al. 161 presented a jRL-MSIM plan to suppress out-of-focus signals. Later, Chakrova et al. 162 compared the piFP and jRL algorithms by formulating a generalized maximum likelihood estimation (MLE). They found that the piFP method can resolve periodic and isolated structures equally well, while the jRL method is more suitable for processing isolated objects. In a similar fashion to the piFP method, subsequent methods such as PE-SIMS 163 (a self-calibration strategy for SIM) and TIRF-piFPM 164 were proposed. However, these methods require prior knowledge of the illumination pattern. In 2021, Samanta et al. 165 envisaged the utility of optical lattice illumination patterns generated by phase-engineered interference of coplanar beams 166 and presented a blind reconstruction approach combined with a multiple signal classification algorithm (MUSICAL) 167 . The results demonstrated that using sinusoidal and multiperiodic illumination patterns, a maximum of three- and six-fold resolution enhancement beyond the diffraction limit could be obtained, respectively. While out-of-focus signals can be addressed by introducing OTF attenuation 89 or RL deconvolution 161 , 162 , they are only suitable for imaging relatively thin samples. To address out-of-focus in thicker samples, several blind-SIM algorithms have been proposed 168 – 171 . For example, Jost et al. 168 proposed the thick slice blind-SIM algorithm, which considers several additional planes to collect out-of-focus light and processes monofocal layer data to bridge 2D- and 3D-SIM reconstructions. In 2019, Soubies et al. 171 improved upon this method by proposing an inner-loop-free alternating-direction method of multipliers (ADMM) 172 , which relies on a specific formulation of the optimization problem and closed-form expressions of proximal operators, resulting in faster computation. By considering additional planes in the model, they demonstrated improved image quality for slice-by-slice computational sectioning.

Development of OS-SIM reconstruction methods

In addition to commonly used RMS algorithms, there are other OS-SIM reconstruction methods, including projection (i.e., sum, maximum, and super-confocal), scaled subtraction of the out-of-focus estimation, and a modified version of Fourier-space treatment (known as patterned excitation microscopy processing). Heintzmann et al. 30 discussed and analysed these three strategies based on a regular array of point illumination patterns. Simulation and experimental results demonstrated that while projection methods, especially the super-confocal method, have exceptional optical sectioning characteristics, they also exhibit more residual patterning than the other two methods. PEM processing can provide high resolution but is computationally expensive. In 2014, Schropp & Uhl 31 pointed out an alternative structured illumination microscopy employing square or hexagonal illumination patterns (Fig. 8a–d ). Rather than shifting a regular array of point illumination patterns row by row along the x- and y-axes, they shifted 2D illumination patterns unidirectionally along pattern-dependent angles. This approach results in an isotropic power spectral density and opens new possibilities for high-resolution imaging in biological and materials science applications. Fig. 8 Optical section images reconstructed under different OS-SIM algorithms. a – d Comparison between single phase images illuminated by square a or hexagonal c illumination patterns and quasiconfocal images b or d 31 . e HiLo technique applied to a pair of fluorescent pollen grains located at slightly different depths 36 . Top row: speckle-illumination image (left), uniform illumination image (right); middle row: intermediate low-pass image (left), intermediate high-pass image (right); bottom row: composite full resolution optically sectioned image (left), extended focus image obtained from a maximum intensity projection of 80 slices separated by 0.5 μm steps. f Whole-brain imaging of the anterograde projections of AAV-YFP-labelled neurons in the motor cortex 43 . a – d © The Journal of Microscopy, e © The Optical Society, f © Nat Methods To streamline the measurement setup and enhance imaging speed, a two-frame OS-SIM algorithm was proposed, such as the HiLo algorithm 36 or the amplitude demodulation algorithm based on the Hilbert-Huang transform 76 , 77 . HiLo microscopy employs a nonuniform (fixed-frequency or speckle) image to provide low-resolution information with spatial frequencies below a user-specified cut-off frequency, while a uniform illumination image is acquired to provide high-resolution information with a spatial frequency above the cut-off frequency. By appropriately setting the cut-off frequency and fusing the low- and high-resolution information, a full-resolution optically sectioned image can be recovered (as shown in Fig. 8e ). However, in the HiLo algorithm, additional attention must be paid to the process of fusing the low- and high-resolution information. Moreover, the amplitude demodulation method based on the Hilbert-Huang transform necessitates two mutually π phase-shifted raw structured images. However, the imaging speed of this two-frame OS-SIM method is still constrained by the camera’s capabilities (~34 fps). Subsequently, a single-frame OS-SIM algorithm was proposed. In 2019, Wang et al. 78 proposed a Fourier bandpass filtering algorithm to reconstruct optical section images by shifting the in-focus signals to the +1st order in the Fourier domain. However, this method requires perfect separation of Fourier spectrum components using a bandpass filter, which can reduce the lateral resolution. In 2021, Zhong et al. 43 further developed a high-definition fluorescent micro-optical sectioning tomography (HD-fMOST) method for whole-brain optical imaging with submicrometer-voxel resolution, based on the LiMo method (Fig. 8f ). In addition, deep learning has demonstrated its effectiveness in OS-SIM techniques. It can be used to solve issues such as high computational costs and oversimplification of optical systems for some deconvolution techniques 79 , 80 (such as Wiener filtering 81 and Richardson–Lucy (RL) deconvolution 82 , 83 ). For example, in 2018, Zhang et al. 84 developed a deep learning-based computational algorithm, which only requires a single wide-field image and a corresponding optical sectioning reference image to train a convolutional neural network (CNN). This algorithm can reconstruct optical section images with lower noise, fewer artifacts, and higher imaging depth at an optimized frame rate of 14 Hz. In 2021, Chai et al. proposed a one-shot optically sectioned method called Deep-OS-SIM 85 , which is based on deep-learning techniques. Unlike other methods that use low entropy wide-field images, this approach takes full advantage of the high entropy properties of structured illumination images to train a CNN model. Optical-sectioning imaging using this method only requires a single image for decoding, thereby improving the raw image acquisition efficiency by 50% compared to the two-frame HiLo method. However, similar to other deep learning-based methods, this method requires expertise in deep learning, and it is currently restricted to specific projected illumination patterns and samples for OS-SIM. If the statistical characteristics of samples or the illumination pattern modes change, retraining for variations in imaging parameters will be necessary.

Development of 2D-SIM reconstruction methods

Parameter estimation SR-SIM reconstruction is essentially an ill-posed inverse problem. As mentioned in the subsection “Basic SR-SIM algorithm”, solving and separating the spectra of the sample and then moving them back to their correct positions is crucial during SIM reconstruction. This process requires precise knowledge of structured illumination patterns, especially in those techniques that rely on high-order harmonics to improve resolution 86 , 87 . Even slight deviations in the reconstruction parameters from the correct ones can lead to noticeable artifacts in the reconstructed images, such as ghosting and fringing. The periodic illumination pattern parameters include the illumination frequency vector, angle, phase, and modulation depth. Although the illumination frequency vector can be determined with high precision and reproducibility using structured illumination generators such as SLM 52 – 54 and DMD 55 , 88 , the initial phase is difficult to determine accurately without prior knowledge. Moreover, factors such as sample movement, system-dependent optical aberrations, and photobleaching can cause the pattern position to shift in the raw images, making it challenging to estimate the parameters based on prior knowledge. As a result, numerous algorithms have been proposed for postprocessing parameter estimation of periodic illumination patterns. Based on the work of Gustafsson et al. 12 , the frequency vector can be retrieved by iteratively maximizing the cross-correlation (COR) of the overlap areas between the first separated and zero orders, assuming an equidistant phase distribution (Fig. 9e ). The modulation depth and phase offset can then be obtained by calculating the absolute value and angle of the complex factor of the overlap areas, respectively. One advantage of this method is that the separation of orders only relies on the relative phase between individual images. To speed up the parameter estimation process, a notch filter (Fig. 9d ) was later introduced to roughly extract the peak position of the +1 order-separated spectrum before optimizing the maximum cross-correlation 89 – 91 . Fig. 9 Parameter estimation in 2D-SIM. a Raw image of actin from a homemade sample U2OS cell acquired using a 2D-SIM system. b The corresponding Fourier spectrum distribution of ( a ). c One of the separated spectra, +1 order spectrum. When estimating the frequency vector k , there are two steps: d Rough estimation 91 . Combined with a notch filter, the peak position in the +1 order-separated spectrum is extracted; e Precise estimation 12 . The cross-correlation between the separated 0- and +1 order spectra is maximized. There are four methods when estimating the initial phase φ: e COR 94 . Sum the pixel values of the overlap areas between the separated 0 and +1 order spectrum, and extract the angle of the complex factor obtained by summation; f POP 92 , 93 . Directly extract the phase at the peak in the raw Fourier image; g ACR 95 . Sum the overlapping areas of the autocorrelation of the respective raw Fourier images; h IRT 96 . Sum the raw images with phase shift, transform the summation into Fourier space, and then extract the phase of the estimated peak In many cases, residual orders may exist in the spectra separated by unmixing due to imprecise individual illumination phases, which can be minimized by estimating the phase offset of each illumination pattern. Shroff et al. 92 , 93 proposed a Fourier domain phase of the peak (POP) estimation method without prior knowledge of the phase shifts, which is suitable for live-cell imaging, as shown in Fig. 9f . However, this method is inappropriate for high-frequency illumination patterns such as those in TIRF mode. As the phase information is directly estimated from the raw image, three conditions should be simultaneously met: the raw image has a high SNR, the high-frequency component decays rapidly enough, and the illumination pattern frequency is lower than the cut-off frequency determined by the support area for OTF detection. In 2013, Wicker et al. 94 developed an iterative optimization method to determine the pattern phases using the COR between separated components in cases where the illumination pattern was too fine to detect. Although this method can robustly determine the relative pattern phase in SIM raw images with a precision below λ/100, its iterative nature inevitably results in longer computation times. In a later study, Wicker 95 presented a faster and more robust noniterative autocorrelation reconstruction (ACR) method for determining a pattern’s phase, as shown in Fig. 9g . This method calculates each illumination pattern’s phase from the autocorrelation of its corresponding raw Fourier image and typically achieves precision less than λ/500 at realistic SNR levels. In 2016, Lal et al. 27 provided a comprehensive theoretical overview of 2D-SIM algorithms, including determining the illumination frequency vector using the ACR method and estimating the phase offset through iterative optimization of the correlation function between the illumination pattern and the sample’s Fourier image. In 2016, Zhou et al. 96 proposed an image recombination transform (IRT) algorithm (Fig. 9h ), which utilizes the phase difference among three raw images to obtain a high-precision initial phase. By combining this algorithm with their own DMD-projection-based, multicolor LED-illumination SIM system, they achieved low excitation intensity fluorescence imaging even less than 1 W∕cm 2 . However, the IRT algorithm only considered a phase shift of π/2 and two orientations separated by 90°, limiting its application in general scenarios. To overcome this problem, Zhao et al. 97 reported an enhanced IRT algorithm that can handle arbitrary phase shifts. It should be noted that the POP, ACR, or IRT algorithms cannot guarantee accurate phase estimation when the raw images have low SNR or weak modulation depth. In addition, for certain periodic samples, the ACR algorithm requires the modulation vector to be distinct from the spatial frequency vector. To address these issues, Cao et al. 98 proposed a noniterative phase estimation method based on an inverse matrix by incorporating extra matrices into the phase estimation algorithm. However, since the parameters of the inverse matrix can affect the phase estimation error, it is essential to select an appropriate parameter set to decrease the average phase error, which lacks objectivity. In 2022, Qian et al. 99 , 100 introduced principal component analysis (PCA) into SIM for the first time to identify the frequency vectors and pattern phases of the illumination pattern (Fig. 10 ). They demonstrated that PCA-SIM can achieve fast and accurate noniterative parameter estimation (with frequency vector precision below 0.01 pixels and relative phase precision of 0.1% of 2π under typical noise levels) that is also robust at low SNRs. This allows for real-time superresolution imaging of live cells in complicated experimental scenarios. Fig. 10 Schematic diagram illustrating the PCA-SIM estimation algorithm. a Flow chart of the parameter estimation in PCA-SIM 100 . b – d Output results of red boxes in a . b The first-order spectrum after shifting the integer-pixel frequency vector. c The phases of the phasor matrix obtained from b after different operations: the original phases c1, phases after applying the masking operator c2, phases after applying PCA c3, and phases after least-squares fitting c4. d Obtained frequency vector with subpixel accuracy Fourier domain reconstruction algorithms FDR algorithms, also known as direct methods, are the most used methods for SIM reconstruction. As mentioned earlier, the first proposed FDR algorithm was the generalized Wiener filtering algorithm. However, this algorithm is not only affected by imprecise parameter estimation but also vulnerable to systematic aberrations and the SNR of raw images 101 . In addition, its reconstruction speed does not meet the requirements of real-time imaging. To address these issues, various improved FDR algorithms have been proposed, which can be classified into the following three parts. Part 1: SIM reconstruction for suppression of optical aberrations Optical aberrations, such as systematic spherical aberration and sample-induced aberrations, not only cause artifacts, loss of resolution, and reduced image contrast in SIM-reconstructed images but also limit the technique’s application to samples thinner than a single cell 102 , 103 . Even small optical aberrations, which have minimal influence on a diffraction-limited image, can cause severe artifacts in SIM-reconstructed images. Moreover, the degree of image distortion caused by spherical aberration is influenced by a range of physical parameters, including cover glass thickness, a refractive index of the sample embedding medium/immersion oil, and sample temperature, all of which are empirical and add to the complexity of the problem. Typically, there are two physical ways to improve spherical aberration: choosing an appropriate immersion oil or adding adaptive optics in the imaging path 87 , 104 – 107 . In recent years, algorithms have been proposed to improve SIM imaging quality, including PSF engineering 91 , 108 and tiled reconstruction methods 109 , 110 . In 2016, Perez et al. 108 proposed an RL-based deconvolution 111 filtering step for both raw and reconstructed images. This method depends only on unbiased filtering steps during reconstruction, without requiring any parameter tuning. However, it does not suppress the effects of out-of-focus background and spectral inhomogeneity on the reconstructed image. In 2020, Wen et al. 91 presented a high-fidelity SIM (HiFi-SIM) reconstruction algorithm, which engineers the effective PSF into an ideal form. By combining a normalized cross-correlation method with a spectrum notch, HiFi-SIM can automatically estimate the illumination pattern parameters. Furthermore, it can effectively reduce common artifacts without sacrificing delicate structures and improve axial sectioning for samples with a strong background. Here, we present schematic diagrams of two FDR reconstruction procedures, the generalized Wiener filtering method (OpenSIM) 27 and HiFi-SIM (Fig. 11 ). Apparently, the spectrum reconstructed by HiFi-SIM is flatter and smoother than that reconstructed by OpenSIM. Additionally, the reconstructed image produced by HiFi-SIM has higher contrast while preserving details. Fig. 11 Fourier domain-based SIM reconstruction methods. The 2D superresolution SIM reconstruction process using two different Fourier domain reconstruction methods: the generalized Wiener filter (OpenSIM) 27 and high-fidelity SIM (HiFi-SIM) 91 In 2020, Hoffman et al. 109 developed a tiled reconstruction method to achieve artifact-free whole-slide imaging with a large field-of-view in SIM, which can alleviate many common SIM reconstruction artifacts caused by global parameter estimation errors. In this method, each raw image was divided into overlapping tiled subsets, and each subset was reconstructed using independently measured or user-optimized parameters. These subsets were then reassembled into a composite superresolution image covering the original field of view. Furthermore, Johnson et al. 110 proposed a Bayesian estimation-based SIM reconstruction method that combined SIM with image-stitching and devignetting methods to provide artifact-free stitched images with optical sectioning and superresolution properties. The results of five different samples demonstrated that the stitched SIM images were useful for intraoperative histology. Part 2: SIM reconstruction under a low SNR situation When acquiring raw SIM images, using a higher signal level can result in better-quality reconstructed images. However, this can accelerate sample photobleaching and limit the number of time points for live-cell images. On the other hand, acquiring raw images at low signal levels can result in considerable noise, leading to artifacts in the reconstructed image. To minimize these artifacts, the parameters in the Wiener filter 27 are typically set manually, which is user-dependent and lacks objectivity. Subsequently, a series of regularization-based iterative optimization methods were proposed based on the prior knowledge of structured illumination patterns. For example, in 2014, Chu et al. 112 introduced a total variation (TV) constraint for SIM reconstruction. The algorithm can image at least 15 times more time points than a traditional Wiener filtering reconstruction method. However, the reconstructed image contains stepped artifacts due to the overcorrection of edge information. Lukeš et al. 113 proposed a SIM method based on the maximum a posteriori (MAP) probability. Combined with homodyne detection, this method can suppress out-of-focus information, improve spatial resolution, and enable the reconstruction of 2D and 3D images of cells, even with weak signals. They later developed an open-source, modular function set, SIM-Toolbox for MATLAB, which supports OS-SIM and SR-SIM image reconstruction 114 . In 2018, Huang et al. 115 reported a deconvolution method for SIM, called Hessian-SIM, which utilized prior knowledge of the continuity of multidimensional biological structures based on Hessian matrices. This method enabled ultrafast live-cell superresolution imaging (such as structural dynamics of mitochondrial cristae) with a spatiotemporal resolution of 88 nm and 188 Hz. Moreover, compared with TV-SIM, Hessian-SIM can retain more image details while reducing noise. In the same year, Boulanger et al. 116 proposed a nonsmooth convex optimization method for SIM reconstruction. However, this method requires heavy computation and takes a long time to converge. In 2020, Yu et al. 117 implemented a second-order optimally regularized SIM (sorSIM) method, which utilizes second-order partial derivatives to suppress the stepped artifacts that appear in TV-SIM. This method achieves a balance between resolution enhancement and noise immunity. In 2021, Zhao et al. 118 added the sparsity of biological structures to Hessian-SIM and proposed a Sparse-SIM deconvolution algorithm, which can achieve a resolution of ~60 nm at a frame rate of up to 564 fps. This method also enables four-color, 3D live-cell superresolution imaging at ~90 nm resolution. However, the resolution enhancement of Sparse-SIM will depend on factors such as the image SNR and optimal parameter selection, which can be cumbersome for different biological samples. In 2022, Zhou et al. 119 established a nonuniform sCMOS noise model and proposed a corresponding noise-corrected SIM reconstruction algorithm based on the stable biconjugate gradient descent algorithm (Bi-CGSTAB) 120 and split Bergman algorithm 121 . Simulation results indicated that this noise-corrected SIM reconstruction algorithm can effectively suppress sCMOS noise-related reconstruction artifacts. Recently, Hou et al. 122 developed an MRA deconvolution algorithm for fluorescence images, which uses framelet and curvelet domain sparsity to regularize the solution. This algorithm allows fine detail recovery even with a negative SNR and provides more than twofold physical resolution enhancement with fewer artifacts than maximum likelihood estimation (MLE) methods. Furthermore, they developed a DeepMRA deconvolution algorithm, which can address severer backgrounds and better preserves high-frequency and low-intensity details that are commonly disrupted by other algorithms. Part 3: SIM reconstruction speed improvement To enable real-time observation for live-cell imaging, various attempts have been made to enhance the SIM imaging speed. In addition to improvements in optical system hardware, several algorithm optimizations have been proposed, such as frame reduction of raw images, rolling reconstruction, and GPU acceleration. Preliminary results have demonstrated that a superresolution image can be reconstructed from four raw images 123 , 124 . In 2017, Ströhl and Kaminski proposed acquiring three raw images under different illumination orientations, and indicated that the frame rate can be doubled by using the joint RL deconvolution algorithm 125 . However, in 2018, Lal et al. 126 found only approximately a 1.5× resolution enhancement in the final image reconstructed using three raw images and concluded that at least four raw images are required to double the resolution. In 2022, Zeng et al. 127 introduced polarization modulation to the frame reduction imaging model, proposing a complete and versatile imaging model called PRSIM. They indicated that for polarized samples, polarization-related artifacts can be reduced by combining a Fourier domain iterative reconstruction algorithm. However, these frame reduction methods rely on assumptions about the image-formation process, and the final reconstructed results are limited by the type of noise. In 2017, Ma et al. 128 proposed the combination of SIM with an interleaved reconstruction strategy (SIMILR) to maximize the use of each subframe of the acquisition series. This method enabled the observation of highly dynamic structures, such as the endoplasmic reticulum, which undergoes continuous rapid growth or shape changes. Later, in 2018, Guo et al. 129 employed SIMILR in grazing incidence SIM, which utilizes highly inclined laminar illumination 130 . This method achieved dynamic imaging of events near the basal cell cortex at 97 nm resolution and 266 fps over thousands of time points. In addition, combining Hessian-SIM with a ‘rolling’ reconstruction procedure allowed for a maximum number of frames in time-lapse imaging of up to 291 fps 115 . Similarly, Sparse-SIM achieved a frame rate of 564 fps 118 . In terms of GPU acceleration, various SIM reconstruction methods have been developed using programming languages such as CUDA C++, Java, and Python 89 , 131 – 133 . For example, in 2019, Markwirth et al. 132 proposed a video-rate immediate GPU-accelerated open-source reconstruction (VIGOR) method by recreating, modifying, and extending the fastSIM 134 approach and image reconstruction software. The results demonstrated that multicolor SR-SIM can be reconstructed at video frame rates (25 reconstructed fps or more), with a delay of less than 250 ms between measurement and reconstructed image display. In 2021, Gong et al. 131 presented a GPU-accelerated SIM method using a hexagonal illumination pattern based on the Python language. This method can process over 239 input raw images (512 × 512 pixels) per second and generate over 34 superresolution frames per second at 1024 × 1024 pixels. However, it should be noted that in these GPU acceleration methods, the illumination parameters are estimated and calibrated in advance (e.g., using the COR algorithm) and then reused in the subsequent reconstruction, making it challenging to address complex dynamic operating environments, such as artificial interference and environmental perturbations, which can lead to drift in illumination patterns. Spatial domain reconstruction The SDR approach was initially proposed by Cragg & So during the development of SIM 135 , 136 . They created an enhanced 2D image by taking images at different phases and directions of a structured illumination pattern and forming a weighted sum. SDR requires the same number of raw images as the generalized Wiener filtering algorithm but is faster because it does not require Fourier transform operations. In 2021, Manton et al. 137 reported an equivalent AM signal demodulation method that used the structured illumination pattern as the carrier signal, the sample as the message signal, and the recorded data as the product of these two, i.e., the AM signal. They then heterodyned the AM signal with another sinusoidal pattern with the same phase and period as the carrier signal, realizing spectrum separation and recombination. Inspired by the series expansion of a function in mathematics 136 , Dan et al. 138 proposed another SDR method by computing the coefficient matrix (Fig. 12 , second row). Their results showed that this method reconstructed a superresolution image sevenfold faster than the FDR algorithm. However, this method did not address out-of-focus backgrounds. In 2022, Wang et al. 139 developed a joint space and frequency reconstruction (JSFR)-SIM by combining spatial domain processing with optical sectioning superresolution SIM implemented in the frequency domain (Fig. 12 , third row). By utilizing multithreading, they were able to reduce the execution time of reconstruction to 10.2 ms for raw images that were 512 × 512 pixels in size. Recently, the JSFR algorithm has been integrated with HiFi-SIM to form the joint space-frequency reconstruction-based artifact reduction algorithm for SR-SIM (JSFR-AR-SIM) 140 . However, all three SDR methods require parameter estimation, which is the most time-consuming step in the reconstruction process. Fig. 12 Spatial domain-based SIM reconstruction methods. The process of reconstructing SIM in the spatial domain involves several methods, including: spatial domain reconstruction SIM (SDR-SIM) 137 , 138 , joint space and frequency reconstruction SIM (JSFR-SIM) 139 , and shifting phase SIM (SP-SIM) 141 In contrast to these methods, Tu et al. 141 developed a parameter-free algorithm called shifting phase SIM (SP-SIM), which directly reconstructs superresolution images in the spatial domain (Fig. 12 , last raw). However, similar to the OS-SIM algorithm, the SP-SIM algorithm only preserves first-order spectral band information and discards the zero-order spectral band information during the derivation process. Consequently, low-frequency components of the reconstructed superresolution image using SP-SIM may be lost, resulting in lower image contrast compared to the FDR algorithm.

Open-source software

Open-source and open-access software packages for SR-SIM reconstruction have become more prevalent, and the once-opaque algorithms are now accessible to ordinary users. In this section, we provide a summary of existing 2D-SIM open-source reconstruction algorithms in Table 1 , which includes information on the number of raw images needed, achievable resolution, implementation methods, and more. Table 1 Open-source reconstruction algorithms for the 2D-SIM Method N/frames Language Resolution in X, Y/nm, or t/Hz Property Categories OpenSIM 27 9 MATLAB ~2-fold of the diffraction limit Classical 2D-SIM FDR OpenSIM-4 126 4 MATLAB ~2-fold of the diffraction limit Frame reduction FDR Hessian-SIM 115 9 MATLAB A spatiotemporal resolution of 88 nm and 188 fps Ultrafast and hour-long dynamic superresolution imaging Iterative algorithm Sparse-SIM 118 9 MATLAB ~60 nm resolution at a frame rate of up to 564 fps Sparse deconvolution Iterative algorithm fairSIM 89 , 90 9/15-3D slice Java ~2-fold of the diffraction limit Classical 2D-SIM with notch filter FDR VIGOR 132 9/15-3D slice Java Multicolor SR-SIM imaging at video frame- rates (25 reconstructed fps or more) GPU-accelerated FDR HexSIM 131 7 Python Over 239 input raw images per second at 512 × 512 pixels, generating over 34 SR fps at 1024 × 1024 pixels GPU-accelerated Frame reduction Hexagon illumination SDR + FDR SP-SIM 141 9 MATLAB ~2-fold of the diffraction limit No parameter estimation Suitable for speckle illumination pattern SDR JSFR-SIM 139 9 MATLAB The reconstruction time is 10.2 ms for raw images with 512 × 512 pixels OS-SR-SIM SDR + FDR SIM-Toolbox 114 9 MATLAB ~2-fold of the diffraction limit classical OS-SIM 2D-SIM with homodyne detection Iterative algorithm HiFi-SIM 91 9/15-3D slice MATLAB ~2-fold of the diffraction limit High fidelity PSF engineering FDR MRA 122 &DeepMRA 9 MATLAB ~70 nm fidelity-ensured resolution High fidelity Suppression of defocus background FDR Although some of the 2D-SIM reconstruction algorithms listed in Table 1 can reconstruct a single 3D image slice using a 2D-OTF, this does not constitute a true 3D-SIM image stack. This is because 3D-SIM reconstruction requires a system-specific 3D-OTF, preferably experimentally measured, to achieve high spatial resolution along the vertical axis. Without this information, the spatial resolution along the vertical direction may be limited. In addition, several open-source ImageJ plugins offer tools for assessing the quality of SIM images. For example, SIMcheck can analyse both SIM raw and reconstructed data, providing system calibration to help users acquire optimum raw data for successful image reconstruction 142 . Another plugin, NanoJ-SQUIRREL (superresolution quantitative image rating and reporting of error locations), can quantify artifacts in SIM-reconstructed images 143 . By comparing a reference image (generally diffraction-limited) with a superresolution image, a quantitative map of localized image artifacts can be generated and used to guide researchers in optimizing imaging parameters. Regarding resolution assessment methods, Koho et al. 144 proposed a method based on Fourier ring correlation (FRC) analysis, where a single image is divided into four subsets (i.e., two-image pairs) and used to estimate the effective PSF in Wiener and iterative RL deconvolution. In 2019, Descloux et al. 145 proposed a rapid image resolution estimation method called decorrelation analysis, which also uses a single image without prior knowledge. This method explores the highest frequency from the local maxima of the decorrelation functions, avoiding user-defined parameters. However, these methods may not be suitable for images with low SNR or artifacts, which could be interpreted as detailed information of the samples, leading to inaccurate estimated resolution. To analyse and compare the advantages and disadvantages of several open-source algorithms listed in Table 1 , we processed two sets of raw data of actin filaments labelled with AF-568 phalloidin dye collected under high SNR (i.e., SNR = 7.7 dB, PSNR = 20.53 dB for OpenSIM) and low SNR (i.e., SNR = −2.2 dB, PSNR = 15.75 dB for OpenSIM). It is clear in Fig. 13a–c that in the case of a high SNR of the raw images, HiFi-SIM can better eliminate defocus information while maintaining high image resolution. Although the fairSIM algorithm has a faster reconstruction speed than HiFi-SIM and OpenSIM, the reconstructed image contains more artifacts. Regarding the SDR algorithm, JSFR-SIM provides a resolution improvement effect comparable to OpenSIM, but it cannot suppress defocus information as effectively as HiFi-SIM. SP-SIM has the fastest reconstruction speed but lacks zero-order information, causing a discontinuity in the reconstructed images, making it difficult to assess the resolution improvement effect. In the case of low SNR, as shown in Fig. 14a–c , noise introduced during the reconstruction process can reduce the resolution of the final reconstructed image. Overall, the FDR algorithm is more robust to noise than the SDR algorithm. Fig. 13 Reconstruction results of different 2D-SIM algorithms under high SNR conditions. a The results reconstructed under averaging, OpenSIM, HiFi-SIM, Fair-SIM, JSFR-SIM, and SP-SIM. b Intensity profiles along the red line in ( a ). c SNR and PSNR statistical diagrams of each result in ( a ). d The results obtained by postprocessing OpenSIM results using regularization-based iterative algorithms, such as TV, Hessian, sparse, MRA, and DeepMRA. e Intensity profiles along the red line in ( d ). f SNR and PSNR statistical diagrams of each result in ( d ) Fig. 14 Reconstruction results of different 2D-SIM algorithms under low SNR conditions. a The results reconstructed under averaging, OpenSIM, HiFi-SIM, Fair-SIM, JSFR-SIM and SP-SIM. b Intensity profiles along the red line in ( a ). c SNR and PSNR statistical diagrams of each result in ( a ). d The results obtained by postprocessing OpenSIM results using regularization-based iterative algorithms, such as TV, Hessian, sparse, MRA, and DeepMRA. e Intensity profiles along the red line in ( d ). f SNR and PSNR statistical diagrams of each result in ( d ) A regularization-based iterative algorithm is applied based on the reconstruction output of OpenSIM. In the case of a high raw image SNR (Fig. 13d–f ), TV-SIM results in reduced resolution due to stepped artifacts, while Hessian-SIM maintains the resolution of the original OpenSIM output. Sparse-SIM, MRA, and DeepMRA can further enhance image resolution. In addition, Sparse-SIM and DeepMRA can effectively suppress defocus information. When the raw image SNR is low (Fig. 14d–f ), denoising reduces the resolution of the final image, regardless of the algorithm used. There is a trade-off between noise suppression and contrast enhancement. We organize the evaluation results of the above various methods in Appendix 3 .

Development of 3D-SIM reconstruction methods

Recently, the problem of multilayer 3D-SIM image reconstruction has been addressed and implemented. For example, in 2015, based on the generalized Wiener reconstruction theory mentioned in ref. 12 , Shao et al. developed and shared the 3D-SIM reconstruction software with CUDA acceleration 133 . This method has been widely applied in biological study 146 , 147 . In 2021, Smith et al. 148 presented a physically realistic noise model and provided three complementary reconstruction methods: true-Wiener-filtered SIM, flat-noise SIM, and notch-filtered SIM. Experimental results demonstrated that introducing notch filtering can partly overcome the trade-off between increasing contrast and suppressing noise. In the same year, Zhu et al. proposed an iterative algorithm called NGD-SIM based on gradient descent and a nonlinear optimizer RMSprop 149 . However, this algorithm is time-consuming. In 2022, Cai et al. proposed a TV-FISTA-SIM algorithm that combines TV with the fast iterative shrinkage threshold algorithm (FISTA) 150 to further improve imaging speed. Compared to the NGD-SIM algorithm, this algorithm achieves faster convergence speed and higher reconstruction fidelity when the SNR is as low as 5 dB. Additionally, in 2022, Cao et al. 151 proposed an Open-3DSIM algorithm by introducing spectrum filtering to further optimize the reconstruction results of traditional 3D-SIM and improve its friendliness to general users. They provided a MATLAB code, ImageJ version, and Exe application simultaneously. Experimental results demonstrated that Open-3DSIM has superior performance in suppressing artifacts and removing defocus information. It is important to consider the effect of motion artifacts on the quality of a reconstructed image if the sample is moving during imaging 54 . In wide-field microscopy, small sample movements may go unnoticed if they are smaller than the resolution limit. However, in SIM reconstruction, even low velocities may introduce artifacts, leading to a reduction in resolution and potentially misleading interpretations. Thus, Förster et al. proposed a frame difference method (FDM) 152 and its improved versions 153 to detect and locate motion artifacts in SIM images. However, these methods require 3D-stack data and are not executable for two-beam methods, such as nonlinear SIM. Furthermore, to reduce artifacts resulting from optical aberrations and enable 3D-SIM imaging in thick tissues, Lin et al. 105 proposed the AO-3DSIM system, which combines adaptive optics with 3D-SIM and processes 3D-stack data using the generalized Wiener filtering method. The AO-3DSIM system achieved a resolution of 150 nm laterally and 570 nm axially, along with optical sectioning, at a depth of 80 μm through Caenorhabditis elegans , compared to a resolution of 280 nm laterally and 930 nm axially in wide-field imaging. Table 2 summarizes the corresponding 3D-SIM open-source reconstruction methods for 3D-SIM. Table 2 Open-source reconstruction algorithms for the 3D-SIM Method Categories Language Property 3D-SIM 133 FDR CUDA C++ acceleration Classical 3D-SIM AO-3DSIM 105 Python Classical 3D-SIM Open-3DSIM 151 Fiji/MATLAB Suppression of noise artifacts Spectrum optimization True-Wiener-filtered SIM 148 MATLAB High contrast imaging Flat-noise SIM 148 Suppression of structural noise artifacts Notch filtered SIM 148 Higher image contrast than flat-noise SIM TV-FISTA-SIM 150 Iterative algorithm Fast convergence speed We compared and analysed three open-source 3D-SIM reconstruction algorithms, AO-3DSIM, SIMnoise (i.e., true-Wiener-filtered SIM), and Open-3DSIM, by testing another actin filament sample obtained from the open-source data of ref. 105 . As shown in Fig. 15 , the image reconstructed under the AO-3DSIM algorithm contained some artifacts caused by out-of-focus information (Fig. 15a ). Although these artifacts can be effectively removed by using SIMnoise, some details of the sample are also lost (Fig. 15b ). Open-3DSIM can better retain the detailed information of the sample while suppressing the defocus information (Fig. 15c ). However, it should be noted that the comparison results may vary depending on the specific sample and imaging conditions. Therefore, users should choose the appropriate algorithm based on their own requirements and considerations, such as speed, accuracy, noise suppression, and artifact reduction. Moreover, further development and optimization of 3D-SIM reconstruction algorithms are still needed to achieve higher resolution, faster computation, and higher robustness. Fig. 15 Reconstruction results under three typical 3D-SIM algorithms. a AO-3DSIM, b True-Wiener-filtered SIM, and c Open-3DSIM. a 1– c 1 Magnified images from the orange box in ( a – c ); a 2– c 2 Magnified images from the green box shown in ( a – c )

Development of blind-SIM reconstruction methods

If the raw image SNR is too low, or the illumination patterns are distorted due to the inhomogeneity of the sample refraction index, parameter estimation-based reconstruction algorithms may fail to work. To overcome this problem, Mudry et al. 154 developed a blind-SIM reconstruction method in 2012 for illuminating samples with random light speckles. This method dramatically simplifies the experimental setup by not requiring knowledge of the illumination pattern. However, the temporal average of speckle illumination must be roughly homogeneous over the sample for it to work, limiting its wide application. In 2013, Min et al. 155 presented another speckle illumination microscopy by implementing a multiple sparse Bayesian learning (M-SBL) algorithm 156 . A threefold resolution gain was reported under the joint support constraints. Moreover, Ayuk et al. 157 extended the application of blind-SIM to periodic illumination patterns by introducing an additional Gaussian filter during the inversion procedure. It was shown that this filtered blind-SIM is as efficient as traditional SIM when the illumination pattern is periodic. Additionally, it is robust to distortion and misalignment. In 2014, by improving the Fourier ptychography (FP) algorithm proposed by Zheng et al. 158 , Dong et al. 159 proposed a pattern-illumination Fourier ptychography (piFP) method. This method is applicable to any unknown illumination pattern and has been used in computational photography and image-based rendering 160 . In 2015, Ströhl et al. 161 presented a jRL-MSIM plan to suppress out-of-focus signals. Later, Chakrova et al. 162 compared the piFP and jRL algorithms by formulating a generalized maximum likelihood estimation (MLE). They found that the piFP method can resolve periodic and isolated structures equally well, while the jRL method is more suitable for processing isolated objects. In a similar fashion to the piFP method, subsequent methods such as PE-SIMS 163 (a self-calibration strategy for SIM) and TIRF-piFPM 164 were proposed. However, these methods require prior knowledge of the illumination pattern. In 2021, Samanta et al. 165 envisaged the utility of optical lattice illumination patterns generated by phase-engineered interference of coplanar beams 166 and presented a blind reconstruction approach combined with a multiple signal classification algorithm (MUSICAL) 167 . The results demonstrated that using sinusoidal and multiperiodic illumination patterns, a maximum of three- and six-fold resolution enhancement beyond the diffraction limit could be obtained, respectively. While out-of-focus signals can be addressed by introducing OTF attenuation 89 or RL deconvolution 161 , 162 , they are only suitable for imaging relatively thin samples. To address out-of-focus in thicker samples, several blind-SIM algorithms have been proposed 168 – 171 . For example, Jost et al. 168 proposed the thick slice blind-SIM algorithm, which considers several additional planes to collect out-of-focus light and processes monofocal layer data to bridge 2D- and 3D-SIM reconstructions. In 2019, Soubies et al. 171 improved upon this method by proposing an inner-loop-free alternating-direction method of multipliers (ADMM) 172 , which relies on a specific formulation of the optimization problem and closed-form expressions of proximal operators, resulting in faster computation. By considering additional planes in the model, they demonstrated improved image quality for slice-by-slice computational sectioning.