A Hybrid Deep Learning–Based Approach for Visual Field Test Forecasting

Conventional Trend-Based Regression Methods

Whether forecasting individual test points of a VF test result (pointwise forecasting)¹⁹ or a global index summarizing the test²⁰ at a specific future time, the trend-based approach, which is the current clinical standard, uses linear regression and requires a minimum of 3 prior tests to provide a rough estimate of the future test result. Considering a typical patient visit interval of 6 months, the linear regression methods require at least 1 and a half years to make the first prediction, which is neither time nor cost-efficient, regardless of the accuracy. A comprehensive study on the number of prior tests needed to forecast the future test results found that at least 10 prior VFs are required to create an ordinary linear regression model with a mean absolute error (MAE) of <2.5 decibels (dB),²¹ which needs a 4.5-year patient follow-up.

Although Lasso regression²² can slightly reduce the number of required prior VFs, recent studies have consistently shown that deep learning–based methods outperform regression-based methods in both pointwise and global forecasting while requiring fewer prior VF tests,12, 13, 14^,17,23,24 which makes these methods more time and cost-efficient.12, 13, 14, 15, 16, 17 This performance gap can be partially due to the nonlinear nature of VF changes in glaucoma^25,26 and the limited scalability of linear models in harnessing large training datasets.

Deep Learning for VF Forecasting

Wen et al¹² proposed CascadeNet-5, which includes 5 convolutional blocks and is essentially derived from the DenseNet architecture²⁷ with no downsampling to maintain spatial information. Its input is a 2 × 8 × 9 tensor, where the first 8 × 9 matrix contains raw sensitivity values, which are 0-padded and arranged like a typical 2-D grid of a VF test result, and the second matrix is similarly filled with the future age. The output is a similar matrix that carries the forecasted future VF test result for the requested age. An average MAE around 2.5 dB over a relatively large dataset has been reported, which is, in comparison to conventional trend-based methods (Conventional Trend-Based Regression Methods), very promising, especially considering the fact that CascadeNet-5 only requires 1 prior VF, and it was developed and tested over a much bigger dataset. However, CascadeNet-5 is unable to perform a temporal and patient-specific analysis because it depends on only 1 prior VF test result. In contrast, Park et al¹³ proposed an RNN architecture using long short-term memory cells that can forecast the sixth total deviation outcomes using 5 prior observations. Even though this method can capture temporal dependencies in a sequence of VF tests, its predictive performance is reported to be considerably poorer than CascadeNet-5,^14,15 even with the use of a bidirectional-gated network.¹⁴ To explore latent representations in VF tests for progression prediction, a variational autoencoder was trained to reconstruct future VFs using a single prior VF test.¹⁷ However, this method is less effective than the RNN approach and cannot predict VFs at specific times because its input does not explicitly include temporal information.¹⁶ In the following, we propose an improved hybrid method that combines spatial and temporal modeling to deliver personalized forecasting with greater flexibility and performance using a few prior VF tests while reducing the waiting time needed for VF forecasting.

Dataset

We gathered data from our longitudinal glaucoma cohort, which included 1750 subjects—both healthy subjects and patients with glaucoma. The data were collected at the University of Pittsburgh and New York University in accordance with the tenets of the Declaration of Helsinki for research involving human participants. The study complied with the regulations of the Health Insurance Portability and Accountability Act and received approval from the institutional review board at each institution. Patients with best-corrected visual acuity of 20/60 or better and refractive error between −6.0 and +3.0 diopters were included, and those with a history of intraocular surgery or any ocular pathological conditions, other than glaucoma, that could affect OCT scanning, retinal layer thickness measurements, or both were excluded.

For each subject, a series of Humphrey VF tests with the 24-2 Swedish Interactive Threshold Algorithm (Zeiss) had been taken. The test intervals were at least 3 months, and the average number of days between visits was 342 days (range 90–2400). We excluded VF tests with more than 33% fixation losses (FLs) or with false-positive or false-negative errors exceeding 15%. Our final dataset was comprised of 19 437 VF tests from 1750 subjects. The average age at the first visit was 63.2 ± 12.7 years, the number of VF visits per subject was 6.26 ± 3.42, and the average mean deviation (MD) was −5.5 ± 7.4.

Proposed Method

The overall structure of the proposed hybrid VF forecasting neural network (Hybrid-VF-Net) method is shown in Figure 2. Assume that the input includes a sequence of prior VF vectors, their corresponding test dates (in ascending order), and a specified future date for which the forecasting will be requested. A VF test vector has 52 sensitivity values (excluding 2 blind spots). Then, in the preprocessing and encoding module, each VF sensitivity value is normalized by dividing it by 48, which was the maximum sensitivity value that we observed in the training dataset. This step ensures that all sensitivity values are scaled within a consistent range, facilitating more effective model training and prediction. All test dates are transformed into time displacements, where each time displacement is defined by computing the number of days between the date of the latest VF test (as a baseline date) and others in such a way that negative, 0, and positive numbers are assigned to the past, latest, and future dates, respectively.^13,14 The time displacement values are also scaled similarly. Then, each VF test vector is appended by the corresponding time displacement to form a vector with 52 + 1 entries. Because, at this step, the future sensitivity values are unknown, they are filled with zeros while the normalized time displacement for the requested future date is included.

After the preprocessing and encoding module, a temporal processing module was employed to encode the temporal relationships among the input data and summarize them into an intermediate output with 52 entries. We implemented this module using an RNN, followed by a fully connected layer and a Gaussian error linear unit activation function. For the RNN implementation, a popular RNN architecture can be used, like long short-term memory²⁸ or gated recurrent unit.²⁹ Here, we opted for gated recurrent unit due to its simplicity and computational efficiency.

The intermediate output of the temporal processing module (Fig 2) is in the form of a VF test vector. A VF test vector is typically represented on a 2-D grid to interpret sensitivity variations across the vision field (Fig 1, left). To align with this representation and make VF test vectors suitable for further processing, a module called make (2-D) representation reshapes the intermediate output into an 8 × 9 matrix with extra locations padded with zeros (shaded matrix entries in Fig 2). This module also converts the latest VF test vector and the requested future time displacement into the same matrix representations and concatenates all of them. Note that the future time displacement is a single number that is transformed into a matrix by repeating this number to fill all nonzero entries of the matrix. The concatenated matrices form a tensor that is now suitable for local modeling using a CNN module. This arrangement preserves the structural integrity of the VF grid while enriching the input data for more robust forecasting.

In Figure 3, the CNN architecture that we propose to implement the local modeling module in our framework (Fig 2) is shown. In this architecture, both convolution and depthwise transformer layers are used to implement a CNN with a self-attention mechanism³⁰ equipped with a global residual connection.³¹ This design allows for both local and global processing of the input features. Although it was not shown in the figure, we found that input padding was beneficial for the task of VF forecasting, probably due to mitigating the edge effects. Therefore, we symmetrically pad a 3 × 8 × 9 input tensor to 3 × 14 × 14 before feeding it to the local modeling module. Finally, we remove the padding from the output tensor (1 × 14 × 14) and restore it to its original size (8 × 9) to obtain the forecasted VF sensitivity values.

Given a full training set of input and target (corresponding VF at specified future time) pairs, we trained the Hybrid-VF-Net architecture by minimizing MAE using ADAM³² for 1000 epochs. The batch size was set to 64, and the learning rate to 0.0001. To realize the CNN (Fig 3), assuming an input with a spatial size of 14 × 14, the first convolutional stage—which includes 2 blocks of convolutional layers, each one composed of a convolution followed by batch normalization and an activation—has 196 features. The sizes of subsequent feature maps are set to 392, 392, 784, and 196 for the remaining convolutional and depthwise transformer blocks, respectively. All hyperparameters for training and architecture design were manually tuned through iterative adjustments to optimize cross-validation performance.

Error Comparisons

Figure 4 displays the average and pointwise MAEs for the compared methods. The pointwise error plot of the identity method (Fig 4A) shows that noncentered locations typically have the largest differences (between input and target values), which is expected due to the greater test–retest variability in VF test results at peripheral locations. Unsurprisingly, the error plots of other methods (Fig 4B–D) also indicate higher errors at noncentered locations, suggesting that predictions in areas with higher variabilities are more challenging for all methods.

The pointwise plots in Figure 4B and C show CascadeNet-5 (MAE = 2.48) performed slightly better on average than RNN (MAE = 2.54), though the overall difference is not substantial. However, the proposed Hybrid-VF-Net method (Fig 4D) surpassed RNN and CascadeNet-5 by combining temporal and local modeling approaches.

VF Test Reliability and Forecasting Performance

The reliability of VF test results is generally assessed using 3 indices: FLs, false-positive rate, and false-negative rate (FNR). Among these, FL has recently been reported to have minimal impact on reliability, whereas false-positive rate and FNR are more influential in contributing to the test–retest variability of VF test results.³³ More specifically, FNR is reported to significantly affect VF assessment outcomes.³⁴ Therefore, in Figure 5, we demonstrated how variations in FNR influence the overall performance of the compared methods across different FNR ranges, with overlaid histograms showing the distribution of samples.

Figure 5 demonstrates that when FNR is less than 6%, all the compared methods have relatively low MAEs (<2.5 dB), with RNN and CascadeNet-5 closely aligned, while Hybrid-VF-Net achieved a lower MAE. However, as FNR increases to [6, 9] percent range and beyond, the gap between the proposed Hybrid-VF-Net method and the other methods widens slightly, suggesting that Hybrid-VF-Net is more robust to increasing FNR. The overlaid bar plot illustrates the number of samples within each FNR range, clearly indicating that not only is VF forecasting more challenging when FNR increases, but also the number of samples decreases.

Forecasting Performance and MD

The MD measures the overall deviation of a patient's VF from the same-age normative dataset of healthy subjects. The greater the VF loss, the lower the MD value, which generally indicates greater glaucoma severity. In Figure 6, we show how different VF forecasting methods perform in predicting VF changes across varying levels of disease severity.

Figure 6 shows that overall, the proposed Hybrid-VF-Net demonstrated superior performance in maintaining lower errors for negative MD values (i.e., when VF is worse than the age-corrected average). However, for positive MD values (i.e., when VF is better than the age-corrected average), the proposed Hybrid-VF-Net method is not advantageous. For negative MD values, the RNN method shows higher error, while CascadeNet-5 and the proposed Hybrid-VF-Net method are more stable.³⁵ Additionally, this figure shows the distribution of samples (yellow bars), which is biased toward samples with mild vision loss (with MD values higher than −6 dB).

Effects of Number of Input VFs on Performance

In Conventional Trend-Based Regression Methods, we briefly discussed the number of prior tests needed to forecast VF test results using conventional trend-based regression and recent deep learning-based methods. While reducing the number of prior tests is desirable to decrease the long wait times required for VF forecasting, it is not always optimal. For example, using only 1 prior VF test as input, CascadeNet-5 has been reported to have better results than the trend-based regression and RNN methods (Deep Learning for VF Forecasting). However, CascadeNet-5 has very limited patient-specific analysis capability due to its inability to incorporate past information. In Figure 7, we investigate how altering the amount of prior information in the input affects the overall performance of VF forecasting methods.

Figure 7 demonstrates that, although RNN, unlike CascadeNet-5, can incorporate past information, its performance changes slightly across different numbers of prior VF tests, and in all cases, the RNN's performance is below that of both CascadeNet-5 and the proposed Hybrid-VF-Net method. The best result for the RNN is achieved when using 3 prior VF tests, whereas the proposed Hybrid-VF-Net method consistently outperformed the RNN and CascadeNet-5 methods, regardless of the number of prior VF tests used as input. Notably, Hybrid-VF-Net excels particularly well in settings where only 1 or 2 prior VF tests are used.

VF Forecasting Examples Using Hybrid-VF-Net

This section aims to qualitatively demonstrate usage examples of the proposed Hybrid-VF-Net method to forecast future VF tests. Three scenarios are considered to highlight the strengths and limitations of the proposed Hybrid-VF-Net method: success examples, failure examples, and VF test forecasting over time.

Figure 8 presents some success examples, where the grayscale sensitivity heatmaps of the outputs closely resemble the target heatmaps. The examples have various levels of vision loss and VF deficit patterns. The first row shows VF tests of a relatively stable case with minimal vision loss. The second row depicts mild loss, while the third and fourth rows display signs of moderate vision loss. Additionally, the second and third rows are examples of rapid progressors (as considerable changes are evident within a relatively short period of time) with 2 different deficit patterns (superior and inferior quadrantopia deficits, respectively). The last row shows an example of profound vision loss.

Figure 9 presents some failure examples. In the first row, clearly the time intervals are so big. The method is asked to forecast the VF test of >2.5 years later using 2 inputs that are more than 2 years apart. In the second row, there is an inconsistency between the 2 inputs. For example, generally the second VF shows a considerably better VF test result than the first one. In the nasal part of the latest VF (just below the horizontal meridian), there are 2 bright locations that were previously dark in the first VF. Additionally, we noticed that the FNRs of these 2 VF tests are relatively high (14 and 12, respectively). The third row shows an example in which the 2 inputs are almost 2 years apart. Moreover, the FNRs for the inputs and target are 9, 13, and 7, respectively.

Figure 10 illustrates how the proposed Hybrid-VF-Net method can be applied to forecast future VF tests gradually over time. With each new test result, the method can use the 2 most recent VF tests to forecast the next one. In the first row of the figure, the proposed Hybrid-VF-Net method took 2 prior VF tests and forecasted the next VF test. Then, in the subsequent rows, the second input VF and the target VF from the previous row are used as the 2 most recent inputs, and the method is tasked with predicting the future VF test.

Conclusion

In this study, we presented Hybrid-VF-Net, a hybrid deep learning model for VF forecasting. Hybrid-VF-Net combines spatial and temporal modeling to enhance flexibility and accuracy compared with the existing deep learning-based VF forecasting methods. We conducted extensive experiments to demonstrate, compare, and analyze the performance of Hybrid-VF-Net, highlighting the merits and challenges of VF forecasting methods as well as the opportunities in applying deep learning to VF forecasting. The experimental results suggest that the proposed Hybrid-VF-Net method has greater resilience to the variability in data reliability and disease severity, enabling more reliable predictions even in challenging cases. This underscores its potential for practical application in glaucoma management. Future work should focus on further refining temporal modeling and leveraging larger datasets.