If it ain’t broke you’re not trying hard enough

Science Highlights
June 1, 2021
Thomas M Johnson

Microscopists push neural networks to the limit to sharpen fuzzy images

Fluorescence imaging uses laser light to obtain bright, detailed images of cells and even sub-cellular structures. However, if you want to watch what a living cell is doing, such as dividing into two cells, the laser may fry it and kill it. One answer is to use less light so the cell will not be damaged and can proceed with its various cellular processes. But, with such low levels of light there is not much signal for a microscope to detect. It’s a faint, blurry mess. 

mitochondrial membrane image with AI resolution
DENOISING. Images of the outer mitochondrial membrane of a cell are difficult to see because of the very low signal to noise ratio (SNR) in the left (Raw) image. Using high power, a much clearer image (known as the ground truth or GT image) can be obtained. At the far right is the image created by the RCAN network, which was shown the raw image and predicted the sharper image. Scale bar = 5 micrometers. Credit: Jiji Chen.

In new work published in the June issue of Nature Methods a team of microscopists and computer scientists used a type of artificial intelligence called a neural network to obtain clearer pictures of cells at work even with extremely low, cell-friendly light levels.1

The team, led by Hari Shroff, PhD, Senior Investigator in the National Institute of Biomedical Imaging and Bioengineering, and Jiji Chen, PhD, of the trans-NIH Advanced Imaging and Microscopy Facility call the process “image restoration.” The method addresses the two phenomena that cause low-light fuzzy images—low signal to noise ratio (SNR) and low resolution (blurriness). To tackle the problem they “trained” a neural network to “denoise” noisy images and “deblur” blurry images.

So what exactly is training a neural network? It involves showing a computer program many matched pairs of images. The pairs consist of a clear, sharp image of, say, the mitochondria of a cell, and the blurry, unrecognizable version of the same mitochondria. The neural network is shown many of these matched sets and therefore “learns” to predict what a blurry image would look like if it were sharpened up. Thus, the neural network becomes capable of taking blurry images created using low-light levels and converting them into the sharper, more detailed images scientists need in order to study what is going on in a cell. 

image of cell nuclear pores
DEBLURRING. Images of nuclear pores created with diffraction-limited confocal microscope (left) are blurry. Using a super-resolution microscope the nuclear pores are much better resolved (GT, ground truth image).  At the far right the RCAN network was shown the blurry confocal image and predicted the sharp image, which much better resembles the high resolution GT image. Scale bar = 5 micrometers. Credit: Jiji Chen.

To work on denoising and deblurring 3D fluorescence microscopy images, Shroff, Chen and their colleagues collaborated with a company, SVision (now part of Leica), to refine a particular kind of neural network called a residual channel attention network or RCAN.

In particular, the researchers focused on restoring “super-resolution” image volumes, so-called because they reveal extremely detailed images of tiny parts that make up a cell. The images are displayed as a 3D block that can be viewed from all angles as it rotates.

The team obtained thousands of image volumes using microscopes in their lab and other laboratories at NIH. When they obtained images taken with very low illumination light, the cells were not damaged, but the images were very noisy and unusable—low SNR. By using the RCAN method, the images were denoised to create a sharp, accurate, usable 3D image.

“We were able to ‘beat’ the limitations of the microscope by using artificial intelligence to ‘predict’ the high SNR image from the low SNR image,” explained Shroff. “Photodamage in super-resolution imaging is a major problem, so the fact that we were able to circumvent it is significant.” In some cases, the researchers were able to enhance spatial resolution several-fold over the noisy data presented to the 3D RCAN.

Another aim of the study was determining just how messy of an image the researchers could present to the RCAN network—challenging it to turn a very low resolution image into a usable picture. In an “extreme blurring” exercise, the research team found that at large levels of experimental blurring, the RCAN was no longer able to decipher what it was looking at and turn it into a usable picture.

“One thing I'm particularly proud of is that we pushed this technique until it 'broke',” explained Shroff. “We characterized the SNR regimen on a continuum, showing the point at which the RCAN failed, and we also determined how blurry an image can be before the RCAN cannot reverse the blur. We hope this helps others in setting boundaries for the performance of their own image restoration efforts, as well as pushing further development in this exciting field.”

This research was supported by the intramural research programs of the National Institute of Biomedical Imaging and Bioengineering and the National Heart, Lung, and Blood Institute within the National Institutes of Health, and grants from the National Institute of General Medical Sciences, the NIH Office of Data Science Strategy, and the National Science Foundation.

1. Three-dimensional residual channel attention networks denoise and sharpen fluorescence microscopy image volumes. Jiji Chen, Hideki Sasaki, Hoyin Lai, Yijun Su, Jiamin Liu, Yicong Wu, Alexander Zhovmer, Christian A. Combs, Ivan Rey-Suarez, Hung-Yu Chang, Chi Chou Huang, Xuesong Li, Min Guo, Srineil Nizambad, Arpita Upadhyaya, Shih-Jong J. Lee, Luciano A. G. Lucas & Hari Shroff. Nature Methods, 2021-May 31. DOI:10.1038/s41592-021-01155-x

This Science Highlight describes a basic research finding. Basic research increases our understanding of human behavior and biology, which is foundational to advancing new and better ways to prevent, diagnose, and treat disease. Science is an unpredictable and incremental process — each research advance builds on past discoveries, often in unexpected ways. Most clinical advances would not be possible without the knowledge of fundamental basic research.