A global review of publicly available datasets for ophthalmological imaging: barriers to access, usability, and generalisability

Search strategy and selection criteria

This Review forms part of The Lancet Global Health Commission on Global Eye Health, which is examining some of the central issues in global eye health.¹⁹ Any form of ophthalmological imaging was eligible for inclusion. Datasets containing non-ophthalmological images, text, or numerical-only data and images from non-human patients were excluded. No datasets were excluded on the basis of age, sex, or ethnicity of the patients from whom data were collected. Datasets of all languages and geographical origin were included.

The search consisted of two parts. First, we did a literature search of MEDLINE to identify studies de scribing ophthalmological imaging datasets that were publicly available and then attempted to access the datasets at source. The search combined terms describing various types of ophthalmological imaging, such as “eye”, “fundus”, and “retina”, with “optical coherence tomography”, “retinal images”, “biometry”, and “topography”, as well as terms such as “dataset” and “databases”. The MEDLINE search strategy is provided in the appendix (p 1). Second, we did a targeted search with similar search terms using Google’s Dataset Search and the Google search engine, to identify ophthalmological imaging datasets directly. Google’s Dataset Search is designed specifically for the discovery of online repositories and supports locating tabular, imaging, and text datasets. Additionally, indexing is provided to those interested in publicising their dataset with a metadata reference schema. For all the results returned from the search, it provides a description of the dataset content, direct links, and the file format. The Google searches also included terms relating to ocular diseases and ophthalmological imaging, and terms relating to datasets. For both the Google search and the Google Dataset searches, results returned from the first ten pages for each search were systematically collated and screened. The choice of ten pages was tested on the basis of several pilot searches at the beginning of this study to estimate the number of pages needed to capture the relevant results.

Our search was additionally supplemented by manually screening the references of relevant articles, the proceedings of relevant meetings, and consulting clinical experts in the field. The original search for all three sources, including all results from MEDLINE database inception, was done on Dec 3, 2019, and the MEDLINE search was updated on May 11, 2020. No restrictions were placed on language. Because of the absence of time-stamping for Google searches, we were unable to update the Google searches.

Identification of ophthalmological imaging datasets

Search results from MEDLINE were screened by the primary reviewer (SMK) to identify the name and source of any relevant datasets. Where status of availability was unclear, we included these datasets and attempted to access their source, reporting any further barriers to access in our results. The Google and Google Dataset Search results were also screened by the primary reviewer to directly identify relevant datasets. Where there was ambiguity regarding whether a dataset fit the inclusion criteria, a second reviewer (XL) independently reviewed the dataset and if this could not be resolved, a third reviewer (SN) was consulted.

Dataset access and description

Various classifications have been proposed for levels of data accessibility, including a tiered grading system where data accessibility is described as open, safe-guarded, or controlled.^20,21 In this Review, we defined the accessibility of datasets as either: (1) open access, for which there were no requirements or minimum requirements for access (eg, submission of personal information, an email request, creation of an account); (2) open access with barriers, which were datasets fulfilling the theoretical criteria for open access, but being inaccessible because of unpredictable reasons (eg, no response to requests or broken hyperlinks); and (3) regulated access, which required the fulfilment of formal agreements, approvals, or payment. For the open access category where the dataset access required an email request, we allowed a 2-week period for email response; where the website was unresponsive or the download link was not functional, this was checked by a second and third reviewer, before the attempt to access was abandoned.

Extraction of dataset characteristics

A prespecified data extraction form was developed and piloted on the first 20 datasets (by SMK and XL). Information characterising each dataset was recorded, including the direct links to the data source, accessibility, content (in terms of population, pathology, and imaging), and associated metadata (including clinical data, image labels, and segmentation). Where this information was reported at source (eg, on the dataset website) or a link to the paper describing the dataset was provided, we recorded the information as provided. Additionally, we presented the completeness of the reporting for key clinical metadata items across all open access datasets. Each item was marked as reported if the information was described in the dataset documentation, description, or referencing publication and was accepted as reported even if given at the aggregate level.

Datasets identified from the literature search

The MEDLINE search identified 3542 articles, of which 2361 were deemed not relevant on the basis of screening their title and abstract. 1181 studies were obtained as full text articles, to be screened for the mention of ophthalmological imaging datasets that are publicly available. Of these, 534 records did not describe the datasets as publicly available and 151 reported non-ophthalmological datasets. Of the 496 articles that were still included, 161 potential datasets were identified. The same datasets were often referenced by multiple studies. The targeted search for datasets with the use of Google’s Dataset Search and Google search engine identified 106 datasets and after combining the results, 81 duplicate datasets were removed and 46 datasets were further excluded (35 not fitting the inclusion criteria and 11 were derived from other included datasets). 140 unique datasets were identified and included for further assessment at the data source. The dataset selection process is outlined in figure 1.

Dataset access

Of the 140 unique datasets, only 94 were open access from which the raw data could be downloaded. 27 datasets were categorised as open access with barriers, from which data could not be downloaded. 19 datasets had regulated access (12 requiring licensing agreements, six requiring an ethical committee or institutional approval, and one requiring a payment of £2250 plus value-added tax).²² Only the 94 open access datasets could be thoroughly characterised from inspection of the raw data themselves. Details of the datasets in the open access with barriers and regulated access categories are included in the appendix (pp 2–3) and are derived from information found at the source or in the description of the dataset in its associated publication, or both, rather than inspection of the raw data itself.

Characteristics of accessible datasets

Of the 94 open access datasets, we found 25 to be from within Asia (four from south Asia and 21 from southeast Asia or east Asia), nine from North Africa and the Middle East, 34 from Europe, 16 from North America, two from South America, and one from sub-Saharan Africa (figure 2). Of these, 9 datasets contained images originating from multiple countries. No data was reported to be originating from Oceania. The country of origin was unknown in 13 datasets. Dataset inception was reported by 47 datasets and ranged from 2003 to 2019. Open access dataset characteristics are summarised in the table.

From the 94 datasets, we were able to access 507 724 images and 125 videos from at least 122 364 patients (39 datasets did not record the number of patients). The number of patients across the datasets ranged from two to 85 550 (median=50; IQR=371), and the number of images ranged from eight to 109 312 (median=220; IQR=1017). The exact number of images could not be established for one dataset (CASIA Iris Ageing), so a conservative estimate was calculated from the dataset description (n=26 038). Of the total number of images, over half were contributed by three of the largest datasets: Kermany and colleagues (109 312 images),⁵ EyePACS (88 702 images),²³ and MRL Eye (84 898 images).²⁴ In contrast with these large datasets, 68 datasets had less than 1000 images, each ranging from eight to 850 images (median=111; IQR=245).

Where reported, the most common reason for image acquisition was for a research study or a clinical trial (54 of 94; 57%), and for routine clinical care or screening (23 of 94; 24%). Five of 94 (5%) datasets included images acquired from primary care (including screening programmes), 45 of 94 (48%) were from secondary care (hospital or eye clinics), 18 of 94 (19%) were collected in other settings (such as from a university, research settings, or eye banks), and one of 94 (1%) from a non-health-care setting. The setting was unreported in 25 of 94 (27%) datasets. Only 20 of 94 (21%) datasets gave information on whether patient consent was sought and 26 of 94 (28%) datasets stated details about obtaining ethical approval for obtaining or sharing the images.

Ophthalmological diseases represented in the datasets include diabetic eye disease (35 of 94; 37%), glaucoma (19 of 94; 20%), age-related macular degeneration (15 of 94; 16%), hypertensive retinopathy (six of 94; 6%), cataracts (four of 94; 4%), other eye diseases (the full list can be found in the table), and healthy eyes (58 of 94; 62%; figure 2). Moreover, 17 of 94 (18%) datasets did not specify the diseases represented in the dataset. It is possible that these datasets contained healthy eyes; however, no specific indication was given at the data source. 53 of 94 (56%) datasets contained more than one disease, including healthy eyes. Healthy eye images were intended for use in a range of biomedical applications, such as the analysis of normal anatomical structures (including endothelial cell density, the detection of photoreceptors, the assessment of nerves, and vessel morphology) and for technical uses (including denoising images, iris recognition, and eye tracking).

Imaging modalities included in the datasets included retinal fundus photographs (54 of 94; 57%), OCT or OCT angiography (18 of 94; 19%), external eye photographs (seven of 94; 7%), in vivo confocal microscopy (five of 94; 5%), scanning laser ophthalmoscopy and adaptive optics-scanning laser ophthalmoscopy (five of 94; 5%), fluorescein angiography (four of 94; 4%), slit lamp photographs (one of 94; 1%), phase contrast microscopy (one of 94; 1%), specular microscopy (one of 94; 1%), preocular tear film photograph (one of 94; 1%), and videos (two of 94; 2%; figure 2). Of these, five datasets contained images taken from more than one modality. Of the 18 imaging datasets based on OCT, half contained 2 dimensional imaging data and the other half contained 3 dimensional imaging data. Most datasets stored images in a portable network graphics, a tagged image file format, a bitmap image file, or a joint photographic experts group file format (82 of 94; 87%), ten of 94 (11%) datasets offered images as a Matlab file, one dataset offered a NumPy array file (Python), one dataset offered a portable pixmap format file (Netpbm) and 1 dataset offered a hierarchical data format file. Of these, nine of 94 (10%) datasets included images stored in multiple formats. 55 of 94 (59%) datasets included images annotated with labels (including diagnostic labels, eg, grade of diabetic retinopathy severity; or feature labels, eg, vessel labelled as an artery or vein) and 33 of 94 (35%) datasets included images annotated with manual segmentation, with 14 of 94 (15%) datasets providing both labels and segmentation. Annotations were provided by an array of experts including ophthalmology clinicians, general medical physicians, and researchers (including medical students, optometrists, operators, other undefined experts, or ground truth labellers). In addition, 15 datasets presented the images readily divided into either a training subset or a testing subset; however, they do not specify whether splits were made at the patient level.

Completeness of metadata reporting

The percentage of completion of the reporting for metadata items are shown in figure 3. Although technical details relating to the image files and their acquisition were well reported, any associated clinical information was not. The following information was consistently reported across all datasets: imaging modality (100%), number of images (100%), image format (100%), country of origin (86%), device name and manufacturer (85%), and ophthalmological disease (82%). Patient characteristics (including age, sex, and ethnicity) were particularly under-reported (these factors were reported in <20% of datasets), with 74% of the datasets not reporting any patient demographic data, even at the aggregate level. The inclusion and exclusion criteria were described for only 15% of the datasets and the data collection period was reported for only 19% of the datasets. The completeness of the metadata reported by each dataset is summarised in the appendix (pp 4–7).

Summary of findings

To the best of our knowledge, this is the first study to curate a comprehensive list of ophthalmological imaging data that are publicly available. From a search of the medical literature and dataset search engines, our Review found 94 unique ophthalmological datasets that fit in the open access category, containing over 500 000 images. Besides healthy eyes, the most common diseases represented were diabetic retinopathy, glaucoma, and age-related macular degeneration. These diseases are most likely representative of the most commonly imaged diseases in routine clinical practice and research. In particular, screening programmes for diabetic retinopathy exist in several countries, leading to the accumulation of large national-level imaging data of the diabetic population.^25–27

Across all datasets, fundus retinal photography was the most common imaging type (54 of 94 datasets), prob ably because of its widespread availability and common use across a wide range of ophthalmological diseases. The second most common imaging modality was OCT and OCT angiography (18 of 94 datasets, where 9 contained 3 dimensional OCT data). Preservation of the 3 dimensional volume data is advantageous as they give contextual information from neighbouring B-scans, allowing ML algorithms to learn key structural information that might enhance its performance.

Several variables that are clinically essential were under-reported across all datasets. Demographic data including age, sex, and ethnicity were not reported in most of the datasets (74%), even at the aggregate level. Furthermore, inclusion and exclusion criteria were only defined for 15% of the datasets. This missing information is a concern as it is unclear whether there is appropriate representation of population groups within the data, and the ability of researchers to assess the applicability to research findings from these data will be severely restricted.

For datasets with image labels (such as diagnostic or feature labels), the labelling processes were also poorly defined. Many assumptions are made during the labelling of ground truths, and therefore assurance regarding the label accuracy are paramount since they carry implications for any ML model trained with the use of these labels. Details about the labellers’ amount of expertise, the consensus process used for multiple labellers, and how discrepancies were resolved are therefore all relevant.²⁸ In the few datasets that reported this information, labellers ranged from medical students to specialist ophthalmologists, but in most cases the skills of the labellers were unknown. Although the detailed labelling of public datasets might be ambitious, a checklist of minimum reporting metadata items could drastically improve the usefulness of the data and could also potentially enable merging across multiple datasets.

Strengths and weaknesses

This is the first study to systematically identify ophthalmological imaging datasets that are publicly available. An important aspect of the work is the unrestrictive nature of the search strategy applied to a medical bibliographic dataset and online search engines, including those specifically targeting datasets. This method recognises the possibility that not all relevant datasets would be identified with the use of academic publications alone. Furthermore, we sought to verify the claims of all datasets, so that we could adjudge the extent to which such datasets were truly open access and what the user experience might be. We took reasonable measures to obtain actual copies of the data, so their contents could be examined and verified. This process gave us the ability to identify accessibility barriers where data described as being open access were difficult to access regardless. This process also enabled us to identify the extent to which key metadata were available.

We recognise several limitations to our study. Firstly, only the initial ten pages of results returned by a Google Dataset search and Google’s search engine results were screened. It is not clear how the ranking of datasets in Google Datasets Search is established, but according to the documentation, metadata quality, the number of citations, and a combination of other factors are taken into account.²⁹ The nature of the Google search engines is such that it is not possible to update the searches, whereas the MEDLINE search was originally run in December, 2019 and could be updated in May, 2020. Given the probable delays between a dataset being publicly available, and a study being completed and becoming visible on MEDLINE, it would be reasonable to assume that our Review describes the situation as of December, 2019. Secondly, we recognise that this field is moving fast and that this Review is only a snapshot in time. Unlike publications, datasets can be edited, updated, and removed without documentation. As with the timestamping issue, these alterations prohibit the ability to monitor changes in data availability over time. Thirdly, there are other sources of datasets available, such as Kaggle, but these were not explored in this Review (although the Google Dataset Search tool explicitly indexes Kaggle). Future iterations of this study will explore other search engines beyond Google Datasets Search. Lastly, our aim was specifically targeted towards identifying open access datasets. As such, some datasets with regulated access might not have been identified if they were not described as being open access. If there were requirements for accessing the data, we took reasonable measures to obtain actual copies. We emailed the authors or owners of the data (when requested) but did not go as far as completing licensing agreements or pursuing any ethical committee approvals. Although verification of access for those datasets would be welcome, doing so was outside the scope of this Review and would have imposed an additional burden on those institutions with little value other than simple verification. However, we do recognise that the datasets with regulated access might be of higher quality, and that this more restricted access might reflect governance processes associated with a stronger attention to quality and metadata reporting. Regardless, as noted earlier, ease and speed of access is an important driver for researchers, and there is a risk that the benefits of a higher quality dataset might be overlooked in favour of a lower quality dataset that has immediate, unregulated access.

Implications

Publicly available imaging datasets are increasingly being used by a range of researchers from epidemiologists to computer scientists. These datasets can be a powerful enabler to research but, as with any data source, the provenance and limitations of that dataset must be considered. In this section, we highlight three broad implications for the providers and users of such datasets: accessibility; transparency and reporting; and ensuring an adequate representation of the population.

The first implication is accessibility. It is encouraging that our Review identified 94 datasets that were potentially open access, but discoverability appears to be an issue. Although a few datasets are well known, many are not, which might lead to lost research opportunities and might result in bias because of an overuse of a few potentially non-representative datasets. There is value in having an online catalogue of such datasets, which would improve their visibility and provide some key metadata that would enable researchers to identify the most suitable dataset for their research question. Our study provides an initial point of access that will improve their discoverability. Further considerations in this regard, arising from our Review, include a greater clarity around the terms of access from some dataset providers. Although it was beyond the scope of this study, such datasets should also be accompanied by sufficient information regarding their provenance so that researchers can be assured that there is an appropriate ethical and governance framework underpinning the provision of these data.

The second important implication is the transparency and reporting of a dataset. The value of a dataset is associated with far more than just its size, and our Review has highlighted many factors that would be key considerations for a user. There are, of course, advantages to scale, for example in the development of deep learning models or when seeking to detect a modest signal in a heterogeneous population, but the usability of the dataset will also be associated with the quality, depth, and representativeness of the data. Small datasets such as DRIVE (consisting of only 40 images) are examples of a situation in which high-quality labelling and annotation outweighs quantity. DRIVE has become a popular resource for researchers for the purposes of retinal vessel segmentation, probably because of the richness and quality of the segmentation annotations.^30,31 Given the need for researchers to show the generalisability of research findings and their clinical applicability, it is essential that these digital repositories are adequately representative of the diverse population of humans and their diseases. Important characteristics should be re ported to assist the user in decisions around applicability. There is sparse reporting of data characteristics and little guidance to inform the curators of such datasets. Although these might be unimportant considerations from a technical perspective, they are crucial to consider for any clinical applications. Without key information about the population and disease, it is impossible to make assumptions on how generalisable the data are for a real world setting. Previous work outside of the field of health data, such as Datasheets for Datasets (a concept derived from the electronics industry), have previously highlighted many of the issues raised in this Review, which are prevalent across disciplines.³² Gebru and colleagues³² have proposed the reporting of considerations that can improve the transparency and accountability of datasets.

However, there are recognised challenges associated with providing richly labelled data. The curation of metadata items is demanding, costly, and requires careful consideration to ensure accuracy and completeness. The excessive inclusion of detailed metadata could also increase the chance of the reidentification of data items and pose additional privacy concerns. Therefore curation, storage, and access all require thoughtful ethical oversight. However, these risks should be balanced with the potential harm implicated by widespread use of biased and clinically unusable data. Additionally, the risk of reidentification can be mitigated with adherence to widely adopted guidelines for the sharing of raw clinical trial data.³³ The investment of time, skill, and money would generate substantial value in the data and its associated labels, therefore such a dataset is unlikely to be freely available.

The last key implication is around ensuring adequate representation of the population by such datasets. A major concern is the possibility of the underrepresentation of specific groups within public and other datasets, posing unknown biases towards some populations or disease groups. An ML algorithm developed exclusively on one population group might translate poorly beyond that population.³⁴ If an ML algorithm runs poorly on unseen data that are inadequately described, it is difficult to establish whether the poor performance is attributable to spectrum bias.^35–37 Knowledge of the populations represented is therefore important for the development of ML algorithms and even more so for their evaluation. This is a key consideration from a global perspective, as countries wishing to develop applications where there is no infrastructure to curate imaging datasets might also be most likely to access publicly available resources as a first option.

Underrepresentation of diseases is also a concern. Datasets are likely to reflect diseases of particular relevance to their country of origin; data from routine clinical care will reflect the prevalence of the disease of the attending population; and cohort data will reflect the inclusion criteria of the study, but also the health priorities of that particular country. The type of diseases within each dataset and their prevalence within the dataset will affect the generalisability of that dataset to other settings globally. It is important to note that, of the priority eye diseases highlighted by WHO, only diabetic retinopathy, glaucoma, and age-related macular degeneration were strongly represented in the public imaging datasets. For 2015, these three conditions together were estimated to account for 15% of global blindness and 5% of moderate and severe vision impairment, in contrast with the other priority diseases such as cataracts (four datasets), trachoma (one dataset), and refractive errors (three datasets), which contribute to 53% of blindness and 79% of moderate and severe vision impairment.³⁸ This mismatch might be attributed to many factors, including the relative importance of imaging in the management of the disease, the presence of well developed screening programmes for the most represented diseases (such as diabetic retinopathy) and funding available for specific research areas. Diabetic retinopathy, glaucoma, and age-related macular degeneration are more frequently imaged as part of standard care, as opposed to cataracts, trachoma, and refractive errors. If potential imaging-based solutions could improve the care of patients with cataracts, trachoma, and refractive errors by a non-specialist workforce with the use of task sharing, then perhaps a targeted global effort is required to prioritise the curation and development of imaging in these disease areas.

The publicly available datasets identified in this Review are unevenly distributed globally. There are no known publicly available datasets for ophthalmological images in 172 countries (equating to nearly 3·5 billion people, or 45% of the global population). The availability of needed data is even lower if specific use cases are considered. For understanding the healthy eye, there are 58 datasets from 20 countries, representing 54% of the population; for age-related macular degeneration, there are 15 datasets from 6 countries, representing an estimated 44% of the population; for diabetic eye disease, there are 35 datasets from 14 countries, representing 50% of the population. Inferences from data cannot be assumed to generalise across populations and might be unusable on unseen populations. We would argue that this is a form of data poverty that should be taken seriously as it might cause widening of health inequalities, as major parts of the world are unable to benefit from innovations arising in a small pool of data-rich countries.