Adaptive Optics Retinal Imaging with Eye Tracking

The goal of this project with the Clinical and Translational Imaging Unit, NEI, is to design and integrate both closed-loop optical stabilization, using real-time retinal tracking, and image registration into an existing adaptive optics scanning light ophthalmoscope to improve image quality and reduce data loss due to eye motion.

A custom built, research grade adaptive optics scanning light ophthalmoscope is in use at the Eye Clinic in the NIH Clinical Center. This instrument enables the noninvasive visualization of retinal cells. However, since this is a temporal imaging modality along with an image acquisition period (e.g., sequential scanning), eye motion causes unique distortions within the image captures and, in some cases, loss of data. Correcting or compensating for this eye motion is essential for maintaining imaging performance and analyzing the image data. NEI and SPIS staff have started to implement a method of real-time retinal tracking and image registration to correct for eye motion, extending on developments at the University of Rochester (Yang, Q., et al. "Closed-loop optical stabilization and digital image registration in adaptive optics scanning light ophthalmoscopy." Biomedical optics express 5.9 (2014): 3174-3191). This work will require substantial hardware and software development for real-time eye movement measurement and subsequent correction calculations using a graphical processing unit (GPU). In addition, new circuitry will be designed and built as required to interface with a field programmable gate array (FPGA) board, which will modify the image scanning process in real-time via custom algorithms running on the FPGA hardware.

Correction for eye motion will be accomplished by combining two approaches, within-frame motion correction at the strip-level and final correction at the frame-level. Incoming image frames are acquired in strips, such that each strip consists of a defined number of rows within a single image frame. Immediately after each strip is acquired, the strip will be quickly copied to the processing thread (ensuring frame acquisition is not interrupted), which uses a GPU to cross-correlate the current strip to the corresponding strip in a previously selected reference frame. The cross-correlation will yield the spatial offset between the strips, which is used to adjust the scanning mirror pattern control (and thus the image scanning area) to compensate for the detected motion while acquiring subsequent strips. Once all strips are acquired for an image frame, the algorithm will calculate the motion for the entire frame relative to the reference frame. This frame motion is used as an offset in the real-time strip motion calculations for the next frame. The calculated real-time strip motions for the current frame are then used for digital image registration to align the entire frame to the reference image. Finally, the algorithm will perform a second round of cross correlation between multiple strips in the current frame and reference frame to filter out any spurious motion and image artifacts from the digital image registration step if they are found to be significant.

A key aspect of real-time eye motion correction for adaptive optics instruments is proper control of the scanning mirrors which are responsible for sweeping the light source across the retina during imaging. Implementation of eye tracking in hardware is achieved by modulating the behavior of these scanning mirrors based on the eye motion offsets calculated in software. These modulations effectively cancel out eye movement by shifting the image scanning pattern in a direction that is opposite of the actual eye motion. This ‘additional’ motion is seamlessly combined with the original scanning motion of the mirror such that the summation of the two motions results in stabilized imaging of the retina, despite the actual eye motion. The horizontal scanning is accomplished by a high frequency resonance scanner, which operates at approximately 15 kHz (i.e., lines/sec). This resonance scanner also establishes the master clock for the entire adaptive optics instrument. Custom electronics are required to interface with this fast horizontal scanner, as well as with the slower vertical scanner, to allow for both control of the scanners and broadcasting of the master clock to other critical components of the adaptive optics instrument.

Once the above system modifications are completed, future development will expand the capabilities of the existing hardware with further innovations.