A fast blind zero-shot denoiser.

Methods that require a training set The first attempt to apply convolutional neural networks (CNNs) to the task of denoising was in ref. 15 . This was heavily refined in refs. 3 , 16 (DnCNN) to achieve performance that is still competitive today. Zhang et al. later released FFDNet 17 , a denoising CNN designed with speed in mind which, similar to our method, also uses downsampling, although in a different manner and to an entirely different end (see ref. 18 ). The main benefit of using trained methods, outside of their outstanding performance, is that they do not require assumptions about the type and structure of the noise, they can simply be trained on noisy/clean pairs of images. However, their reliance on noisy/clean image pairs can be considered a limitation in situations where we do not have access to ground truth images to train on. To overcome this limitation, Noise2Noise was developed 4 . Noise2Noise can be trained exclusively on pairs of noisy images without any access to ground truth data. It is especially useful in biological imaging where, often, imaging trade-offs dictate that ground truth data cannot ever be obtained. However, paired noisy images are not always easy to obtain, so there was interest in developing methods that could denoise on unpaired training sets of noisy images from some desired domain. The first method capable of this without having sensitive hyperparameters was Noise2Void 6 . Noise2Void works by training the network to learn a mapping from the noisy image back to itself, masking the centre of each receptive field so as to avoid learning the identity. This basic model of masking the input is known as a blind-spot network, and was heavily refined and expanded upon in ref. 19 and much more recently applied in BP-AIDE 20 in a manner that is specifically tailored to Gaussian–Poisson noise. In ref. 21 they demonstrate a retooled version of BP-AIDE with much faster inference time. A recently developed alternative to blind-spot networks is Neighbor2Neighbor 9 which underlies the method we present in this paper. Neighbor2Neighbor learns to map adjacent pixels in the image to one-another, with the idea being that, except in the most highly dynamic regions of the image, adjacent pixels tend to have a similar underlying signal. Recorrupted-to-Recorrupted 22 is another recent denoiser. Recorrupted-to-Recorrupted attempts to corrupt single noisy images into noisy image pairs, and then apply a Noise2Noise-like network. Recorrupted-to-Recorrupted is not blind, and requires an estimate of the underlying noise variance and also contains a sensitive ’coefficient of recorruption’ parameter. Ultimately, all methods listed in this section require a representative training set of noisy images to train on before being applied. In the next section we describe methods that were specifically developed for denoising single noisy images without a training set.

Zero-shot methods

The first method that directly applied itself to the task of blind zero-shot denoising is Noise2Self 7 . Noise2Self is a very similar method to Noise2Void that achieves slightly better performance, and includes a very thorough mathematical justification for the principles underlying the success of masking-based denoising techniques. Self2Self 8 was the first blind zero-shot method whose performance approaches fully trained methods. Self2Self is a blind-spot method, however instead of replacing masked pixels, it ignores them altogether by using partial convolutions 23 , 24 . Self2Self also introduces the innovative step of adding dropout and averaging across multiple runs of the same image. However, this comes at a high computational cost.

Non-blind zero-shot methods BM3D 25 is one of the gold standards for pure Gaussian noise. It works by unfolding the image into interleaved square patches, clustering those patches based on similarity, and then filtering them before reconstructing the image. BM3D, however, is not blind and takes, as a parameter, an estimate of the standard deviation of the underlying noise. Moreover, BM3D does not work on Poisson noise. A much more recent learning-based method is Deep Image Prior (DIP) 26 . DIP works by taking a neural network with randomly initialized weights, and training to reconstruct the noisy image. Similar to Noise2Noise, it will fail to learn the underlying noise (at least at first) and instead learn to output the signal. DIP is highly sensitive to the number of iterations, and will quickly overfit if trained too long, for this reason it is not completely practical as a blind denoiser. For our experiments, we force it to be blind by using a fixed iteration number, however, the results it attains are far below what a non-blind version of this algorithm can reach. Since it is easy to confuse the various different types of unsupervised denoiser, we have included a chart in Extended Data Fig. 2 to clarify the distinctions.

Methods

Noise2Fast implementation details Here we outline the specifics of our neural network and training scheme, giving the implementation details of the process outlined earlier. We note that for all zero-shot denoisers we test, noise generation is carried out separately from denoising. That is to say, each method is only exposed to one single noisy instantiation of one single image during training. For our neural network, we use a simple CNN architecture which we explain briefly here, and illustrate in Fig. 1b . We start by performing two 32 channel 3×3 convolutions with ReLU activation. We repeat this step three more times, each time doubling the number of channels. In the final step, we do 1×1 convolution followed by sigmoid activation. In our initial testing we found that this much simpler architecture outperformed the classical U-net architecture used in the original Noise2Noise paper 4 . Although the results are not that sensitive to the number of hidden layers, we do find a noticeable, albeit small, drop in performance as we add more to our current model. A possible reason for this is that it causes our network to overfit the data much too quickly. This architecture is similar in its simplicity to DnCNN, one major difference being our lack of batch normalization. The main novelty of our method is how we train it. Consider a 2D image documentclass[12pt]{minimal} usepackage{amsmath} usepackage{wasysym} usepackage{amsfonts} usepackage{amssymb} usepackage{amsbsy} usepackage{mathrsfs} usepackage{upgreek} setlength{oddsidemargin}{-69pt} begin{document}$${{{bf{x}}}}in {{mathbb{R}}}^{mtimes n}$$end{document} x ∈ R m × n . Recall from the theoretical background that we can divide our image in two by using chequerboard downsampling. By taking the even or odd pixels and squeezing them up to fill in the spaces, as depicted in Fig. 1 , we can generate two downsampled documentclass[12pt]{minimal} usepackage{amsmath} usepackage{wasysym} usepackage{amsfonts} usepackage{amssymb} usepackage{amsbsy} usepackage{mathrsfs} usepackage{upgreek} setlength{oddsidemargin}{-69pt} begin{document}$$mtimes frac{1}{2}n$$end{document} m × 1 2 n images 16 documentclass[12pt]{minimal} usepackage{amsmath} usepackage{wasysym} usepackage{amsfonts} usepackage{amssymb} usepackage{amsbsy} usepackage{mathrsfs} usepackage{upgreek} setlength{oddsidemargin}{-69pt} begin{document}$${{{{bf{x}}}}}_{{{{rm{even}}}}}(i,j)={{{bf{x}}}}(i,2j+(i,{{{rm{mod}}}},2)),$$end{document} x even ( i , j ) = x ( i , 2 j + ( i mod 2 ) ) , 17 documentclass[12pt]{minimal} usepackage{amsmath} usepackage{wasysym} usepackage{amsfonts} usepackage{amssymb} usepackage{amsbsy} usepackage{mathrsfs} usepackage{upgreek} setlength{oddsidemargin}{-69pt} begin{document}$${{{{bf{x}}}}}_{{{{rm{odd}}}}}(i,j)={{{bf{x}}}}(i,2j+(i,{{{rm{mod}}}},2)+1).$$end{document} x odd ( i , j ) = x ( i , 2 j + ( i mod 2 ) + 1 ) . We can call these the ‘up’ chequerboard downsamples. Notice that we can also squeeze the pixels left to generate two documentclass[12pt]{minimal} usepackage{amsmath} usepackage{wasysym} usepackage{amsfonts} usepackage{amssymb} usepackage{amsbsy} usepackage{mathrsfs} usepackage{upgreek} setlength{oddsidemargin}{-69pt} begin{document}$$frac{1}{2}mtimes n$$end{document} 1 2 m × n images 18 documentclass[12pt]{minimal} usepackage{amsmath} usepackage{wasysym} usepackage{amsfonts} usepackage{amssymb} usepackage{amsbsy} usepackage{mathrsfs} usepackage{upgreek} setlength{oddsidemargin}{-69pt} begin{document}$${{{bf{x}}}}^{prime} (i,j)={{{bf{x}}}}(2i+(j,{{{rm{mod}}}},2),j),$$end{document} x ′ ( i , j ) = x ( 2 i + ( j mod 2 ) , j ) , 19 documentclass[12pt]{minimal} usepackage{amsmath} usepackage{wasysym} usepackage{amsfonts} usepackage{amssymb} usepackage{amsbsy} usepackage{mathrsfs} usepackage{upgreek} setlength{oddsidemargin}{-69pt} begin{document}$${{{bf{x}}}}^{prime} (i,j)={{{bf{x}}}}(2i+(j,{{{rm{mod}}}},2)+1,j).$$end{document} x ′ ( i , j ) = x ( 2 i + ( j mod 2 ) + 1 , j ) . Giving us the ‘left’ chequerboard downsamples. Using these we construct a four-image-pair training set (see Fig. 1c for an overview of our training scheme). We then feed this training data one-by-one into our neural network (batch size = 1). At each iteration we compute the binary cross-entropy (BCE) loss between the target and the output of our neural network, and adjust our weights using the Adam optimizer 33 , 34 with learning rate set to 0.001. When we downsample our image we fundamentally distort the pixel lattice and the relationship between adjacent pixels. For example, in a normal image, suppose each pixel is 1×1 μm. Then the pixel immediately above will be 1 μm away, and the pixel immediately to the right will also be 1 μm away. In the case of a chequerboard downsampled image, for example x even in Fig. 1 , depending on where you are the relationship might be something like: Up: 2 μm, Left: documentclass[12pt]{minimal} usepackage{amsmath} usepackage{wasysym} usepackage{amsfonts} usepackage{amssymb} usepackage{amsbsy} usepackage{mathrsfs} usepackage{upgreek} setlength{oddsidemargin}{-69pt} begin{document}$$sqrt{2}$$end{document} 2 μm. The relationship becomes even more complex as you move further away and the net effect is a serious disruption of the underlying relationship between a pixel and neighbours. We note that this is different from adopting a chequerboard masking scheme and training Noise2Void, where the relationship and adjacency between pixels is preserved. And even Neighbor2Neighbor at least preserves ’ratios’ in the sense that the vertical and horizontal scale are divided by a common number, and these proportions do not vary from pixel to pixel. This has one important effect: while blind-spot methods based on masking get more and more accurate over time without overfitting because they are not given sufficient information to overfit the data, our method performs more like DIP, where accuracy reaches a maximum very quickly, before it starts to plummet as it ultimately overfits the data, this effect happens particularly quickly on our small four-image training set. However, since our distorted downsampled data looks nothing like the original noisy image at a local level, we observe that if we train our four-image chequerboard downsampled set, while we do inevitably overfit our small training set, the original noisy image is basically unseen data for all intents and purposes. Therefore, we can actually use this image as a validation set. More specifically, after each iteration we monitor how our neural network maps our original noisy image x to itself, that is how close f ( x ) is to the identity mapping on x . Our observation is that the optimal time to ‘stop’ training is quite close to the point where the output of the neural network, when applied to original noisy image, most resembles itself. We can validate in this way only because our training set images look so vastly different at a local level than the image we are trying to denoise. In Extended Data Fig. 5 we illustrate how this validation strategy works by comparing over time known ground truth PSNR to the validation PSNR determined by comparing our output to the original noisy image. As can be seen in this figure, the two lines move together and achieve their peaks at roughly the same time. Ultimately, our ability to validate in this way results in a method that converges both quickly and accurately.

Compared datasets

For blind Gaussian denoising we use the greyscale BSD68 35 dataset, as was used in ref. 6 and a multitude of other denoising papers. BSD68 consists of 68 clear 481×321 photographs to which we add synthetic Gaussian noise. However, to show the effect of spatial resolution on speed and performance, we additionally tested the methods on Set12 which contains a mixture of 256×256 and 512×512 images. For performance on real-world confocal microscopy, we used a subset of the confocal microscopy images in Fluorescent Microscopy Dataset (FMD) 2 that we refer to as ‘Confocal’. This dataset contains, among other things, images of biological materials such as cells, zebrafish and mouse brain tissues acquired using commercial confocal microscopes. As described in their paper, ground truth values are estimated by averaging together all 60,000 noisy images in a given set.

Compared methods

We compare denoising and speed performance against five other blind zero-shot denoisers: Noise2Self 7 , Noise2Void 6 , Self2Self 8 , Neighbor2Neighbor 9 and DIP 26 . Not all of these methods were originally designed for zero-shot denoising. We will describe how we configured each of these methods in turn, we adhere to published code as much as possible. Self2Self For Self2Self we use the default published settings of 150,000 iterations and a learning rate of 1 × 10 –4 . We standardize our images differently than Self2Self and some of these other methods. For example, we do not clip our input noisy data [0, 255] at any point. To account for this difference, we have rewritten the dataloaders for Self2Self and other methods to ensure consistency of comparison. Noise2Self For Noise2Self the only change we make from their published single-shot denoising notebook is to increase the number of iterations from 500 to 20,000, as we found that 500 iterations were not nearly enough to achieve good results on these datasets. Noise2Void For Noise2Void we found that their ImageJ plugin worked much better than their GitHub code for zero-shot denoising. We therefore used the ImageJ version for benchmarking purposes, which is why our results on this method deviate so much from previous publications. We used a patch size of 64 with 100 epochs and 100 steps per epoch, a batch size of 16 per step, and a neighbourhood radius of 5. DIP If we fix the maximum number of iterations, DIP becomes a blind denoiser. However, as noted in ref. 8 , it performs better as a non-blind denoiser. For comparison purposes however, we will set the maximum number of iterations at 3,000, as the authors of DIP have done in their example code on GitHub. This turns it into a blind single-shot denoiser, fully comparable in scope to our method. Neighbor2Neighbor For Neighbor2Neighbor we used the adaptation of the code found here: https://github.com/neeraj3029/Ne2Ne-Image-Denoising . We adapted the script to zero-shot denoising and attempted in good faith to optimize for the task as best we could, however, we found that the results were inconsistent. We believe that this method is probably best suited to datasets as the authors intended and not single images. We include these results only to illustrate the need to change Neighbor2Neighbor in order to achieve fast and accurate zero-shot denoising results, as we have done in this paper. We do not believe our results are a fair illustration of the power of Neighbor2Neighbor when applied to the tasks it was designed for and we have therefore excluded it from our visual illustrations. We used a learning rate of 0.0003 and trained for 100 epochs, as suggested in their paper for synthetic datasets.

Fluorescence microscopy images

For fixed immunofluorescence microscopy, RPE-1 cells were fixed with 4% paraformaldehyde at room temperature for 10 min. The cells were then blocked with a blocking buffer (5% BSA and 0.5% Triton X-100 in PBS) for 30 min. Cells were washed with PBS and subsequently incubated with phalloidin-Alexa488 (Molecular Probes) and DAPI in blocking solution for 1 hour. After a final wash with PBS, the coverslips were mounted on glass slides by inverting them onto mounting solution (ProLong Gold antifade; Molecular Probes). For the fixed imaging in Fig. 3a , single Z slices of cells were imaged using Nikon Ti2E/CREST X-Light V2 LFOV25 spinning disk confocal microscope in widefield mode using a 60×/1.4 NA oil-immersion Plan-Apochromat lambda objective. The microscope was outfitted with a Photometrics Prime95B 25 mm FOV ultra-high sensitivity sCMOS camera and images were captured with no binning using the full 25 mm diagonal FOV area at 1,608 px by 1,608 px with a bit depth of 16 bit. After capture, 500 px by 500 px areas were cropped and used as our input dataset. For live imaging in Fig. 3b , endogenously tagged MDA-MB 231 cells were seeded in Nunc Lab-Tek Chamber Slides and imaged on the Nikon Ti2E/AIR-HD25 scanning confocal microscope with temperature and CO 2 control, using a 40×/1.15 NA water-immersion objective Apochromat lambda S objective. High-speed image acquisition was carried out with the resonance scan head with 2× averaging at 1,024 px by 1,024 px. Full volumes of cells were captured ( Z total = 20 μm, Z interval = 0.5 μm) every 5 minutes for 24 hours. For Fig. 4 , single Z slices of cells were imaged using Nikon Ti2E/CREST X-Light V2 LFOV25 spinning disk confocal microscope in widefield mode using a 60×/1.4 NA oil-immersion Plan-Apochromat lambda objective. The microscope was outfitted with a Photometrics Prime95B 25 mm FOV ultra-high sensitivity sCMOS camera and images were captured at two different exposures (0.1 and 100 ms) with no binning using the full 25 mm diagonal FOV area at 1,608 px by 1,608 px with a bit depth of 16 bit. After capture, 500 px by 500 px areas were cropped (this you specify to your cropping in this figure) and used as our input dataset. Images were denoised as individual Z -slices and max projected. All are displayed with auto scaled LUTs. Ablation study For our ablation study, we compare three different refinements of the model. First, we replace our simple neural network with a U-net architecture, which is the standard network used in Self2Self and Noise2Void. Again, our performance drops (Extended Data Fig. 6 , U-net). Also, using known ground truth values, we manually subtract out the s odd − s even term in equation ( 12 ) and show that this has virtually no impact on our denoising results, hence this term is not having a significant impact on our algorithm (Extended Data Fig. 6 , Exact). Finally, we test how well Noise2Fast works if instead of applying it to the original noisy image, we apply it to the chequerboard downsampled images and reassemble them into the full-sized image (Extended Data Fig. 6, Split). We also investigate what happens if we replace our unusual chequerboard downsampling with a more conventional downsample where we divide our image into 2×2 blocks, as used in Neighbor2Neighbor and also 3×3 blocks for fourfold and ninefold downsampling, respectively. This has the advantage making our training set consist of even smaller images to further reduce computation time. We test this on the confocal dataset, and as can be seen in Extended Data Fig. 7 , fourfold downsampling doubles speed at only a small drop in accuracy, making this perhaps an attractive solution to those looking for even more speed. Ninefold sampling on the other hand only increases speed marginally and with a much steeper drop in accuracy. The diminishing returns in speed gain are likely a result of the neural network requiring longer to converge when there is less training data per iteration. We also compare the effect our small architecture has on our results by adding the DIP hourglass architecture to Noise2Fast (Extended Data Fig. 8 ), and also by inserting our architecture into Self2Self (Extended Data Fig. 9 ) and running it on the parrot image from Set12. Source of cell lines MDA-MB 231 cell line was a gift from R. S. Kerbel (Sunnybrook Health Sciences Centre, Toronto, Canada) and cultured at 37 °C in a humidified environment containing 5% CO 2 . MDA-MB 231 cells were grown in Roswell Park Memorial Institute (RPMI) 1640 medium (Life Technologies) supplemented with 10% fetal bovine serum (FBS). RPE-1 (CRL-4000) cell line was acquired from the American Type Culture Collection (ATCC) and grown in Dulbecco’s Modified Eagle Medium/Nutrient Mixture F12 (DMEM/F12 1:1; Life Technologies) supplemented with 10% FBS. To generate our endogenous fluorescent cell lines, CRISPR-Cas9 was paired with a repair construct to insert sequences encoding fluorescent proteins via homology-directed recombination into the N- and C-terminus of ACTB and H3-3B, respectively. Briefly, sgRNAs targeting the N-terminus of ACTB (GCCGTTGTCGACGACGAGCGCGG) and the C-terminus of H3-3B (CAGTTGGCTCGCCGGATACGGGG) were cloned into a pX330 plasmid (Addgene plasmid #42230) following the Zhang protocol 36 . To generate the repair constructs, 10,00 bp of homologous genomic sequence surrounding the sgRNA cut site of ACTB and H3-3B was amplified from the genome of RPE-1 cells and subsequently Gibson (M5510AA; NEB) assembled to flank a cassette containing mNeon or mScarlet followed by a 2A peptide into a puromycin or blasticidin resistance gene. Cells were co-transfected with a sgRNA-cloned pX330 plasmid and its matching repair construct using Lipofectamin3000 (Invitrogen) at a 1:1 ratio. An editing period of 72 hours was allotted prior to selection with puromycin (2 mg ml –1 ) or blasticidin (10 mg ml –1 ) to cull non-integrated cells. Knock-in positive cells were subsequently FACS sorted and inspected via fluorescence microscopy. Reporting summary Further information on research design is available in the Nature Research Reporting Summary linked to this article.

Compared methods

We compare denoising and speed performance against five other blind zero-shot denoisers: Noise2Self 7 , Noise2Void 6 , Self2Self 8 , Neighbor2Neighbor 9 and DIP 26 . Not all of these methods were originally designed for zero-shot denoising. We will describe how we configured each of these methods in turn, we adhere to published code as much as possible. Self2Self For Self2Self we use the default published settings of 150,000 iterations and a learning rate of 1 × 10 –4 . We standardize our images differently than Self2Self and some of these other methods. For example, we do not clip our input noisy data [0, 255] at any point. To account for this difference, we have rewritten the dataloaders for Self2Self and other methods to ensure consistency of comparison. Noise2Self For Noise2Self the only change we make from their published single-shot denoising notebook is to increase the number of iterations from 500 to 20,000, as we found that 500 iterations were not nearly enough to achieve good results on these datasets. Noise2Void For Noise2Void we found that their ImageJ plugin worked much better than their GitHub code for zero-shot denoising. We therefore used the ImageJ version for benchmarking purposes, which is why our results on this method deviate so much from previous publications. We used a patch size of 64 with 100 epochs and 100 steps per epoch, a batch size of 16 per step, and a neighbourhood radius of 5. DIP If we fix the maximum number of iterations, DIP becomes a blind denoiser. However, as noted in ref. 8 , it performs better as a non-blind denoiser. For comparison purposes however, we will set the maximum number of iterations at 3,000, as the authors of DIP have done in their example code on GitHub. This turns it into a blind single-shot denoiser, fully comparable in scope to our method. Neighbor2Neighbor For Neighbor2Neighbor we used the adaptation of the code found here: https://github.com/neeraj3029/Ne2Ne-Image-Denoising . We adapted the script to zero-shot denoising and attempted in good faith to optimize for the task as best we could, however, we found that the results were inconsistent. We believe that this method is probably best suited to datasets as the authors intended and not single images. We include these results only to illustrate the need to change Neighbor2Neighbor in order to achieve fast and accurate zero-shot denoising results, as we have done in this paper. We do not believe our results are a fair illustration of the power of Neighbor2Neighbor when applied to the tasks it was designed for and we have therefore excluded it from our visual illustrations. We used a learning rate of 0.0003 and trained for 100 epochs, as suggested in their paper for synthetic datasets.

Supplementary information Reporting Summary

Supplementary information The online version contains supplementary material available at 10.1038/s42256-022-00547-8.