Validity of Myocardial Infarction Diagnoses in Administrative Databases: A Systematic Review

Introduction

Cardiovascular diseases (CVD), including myocardial infarction (MI), are associated with physical disability, reduced quality-of-life, economic hardship, and death. In 2008 CVD accounted for 30% of all deaths globally [1], and annual cost estimates for CVD have recently exceeded €169 billion for the European Union [2] and $400 billion in the United States [3]. Although age is one of the primary risk factors for CVD, growing evidence suggests that chronic conditions including inflammatory rheumatic diseases [4]–[9], osteoarthritis [10], diabetes [11], and clinical depression [12] are also associated with an increased risk of CVD, independent of age.

Alongside, there is increasing recognition of the value of administrative data for use in disease surveillance [13]–[19], and this data source has been key in identifying the associations between chronic diseases and CVD as mentioned above. Administrative databases provide easy access to data for a large number of patients attending multiple centres, with longer follow-up periods at relatively low cost. For example, the universal provision of publically-funded health care in Canada allows the patient-level linkage of health resource utilization data (including hospital separations, outpatient visits, procedures and tests, and, in some provinces, dispensed prescriptions) for nearly every resident of each province to demographic and vital statistics data. Consequently, both selection and recall bias are minimized.

Despite these advantages, much uncertainty exists around the validity of diagnoses recorded in administrative data since most databases are not established for research purposes. Instead, records of each healthcare encounter are submitted by physicians and hospital staff primarily to obtain reimbursement. Thus, not all conditions may be recorded in the databases, and those recorded may not correspond to the date of disease onset or reflect the true diagnosis and assessment made by the treating physician. These errors and inconsistencies in diagnostic codes may lead to misclassification bias, impacting the quality of research using these sources and, in turn, any changes in health policy and care practices stemming from it. For example, failure to adequately capture the number of people afflicted by CVD may underestimate the burden of these diseases, thus limiting the health resources allocated to address them. Alternatively, when studying long-term health outcomes, capturing an excess number of false-positive cardiovascular events could overestimate the risks associated with an otherwise beneficial therapy or intervention.

While several assessments of the validity of cardiovascular codes have been published [20]–[23], most concerned a single CVD and were conducted within a limited geographic area, restricting their generalizability. Much inconsistency exists with regards to the methods (including the source of the population and gold standards) adopted by these studies and the way in which results are reported. To our knowledge data on the validity of these codes have not yet been synthesized on a larger scale.

As part of a Canadian Rheumatology Network for establishing best practices in the use of administrative data for health research and surveillance (CANRAD) [13], [19], [24], our objective was to conduct a systematic review of studies reporting on the validity of diagnostic codes for identifying CVD in administrative data. Data from these studies were used to compare the validity of these codes, and to evaluate whether administrative health data can accurately identify CVD for the purpose of identifying these events as covariates, outcomes, or complications in future research. We focus on MI in this paper, and will discuss two other CVD, congestive heart failure and cerebrovascular accident, in subsequent reports.

Methods

Results

Discussion

To our knowledge this is the first systematic review on the validity of MI diagnoses in administrative data. Overall, MI diagnostic codes from hospitalization data appear to be valid: in more than half of the studies, sensitivity and specificity exceeded 83%, and PPV exceeded 92%. Therefore, we believe hospitalization data can be used to identify MI either as a covariate or as an outcome. The accuracy of MI as a cause of death on death certificates was lower, with the highest PPV for definite fatal MI being 59% amongst the studies included. In comparison, the PPV was greater than 59% in three-quarters of the studies reporting on hospitalization data. Accordingly, caution should be taken when using vital statistics data to identify deaths from MI, and authors are encouraged to acknowledge this limitation.

It is possible that our findings on the accuracy of MI diagnoses were unduly influenced by publication bias or selective outcome reporting, wherein some authors who did assess the validity of MI codes in their study may have chosen not to report the statistics if they were low. But while our findings for MI in hospitalization data were generally positive, there were exceptions. For example, we observed that the accuracy of MI diagnoses was heavily influenced by the gold standard employed, with lower statistics when the previously-used, more conservative MONICA criteria [52] were applied. These criteria, developed in the 1970's and 80's from international standards, differ from more recent criteria with regards to the biomarkers of cardiac damage. The creatine kinase, lactate dehydrogenase, and aspartate transaminase enzymes are part of MONICA [33], used by 12 studies in this review [20], [21], [34], [35], [37]–[39], [41], [46], [49]–[51]. Three studies [43], [49], [53] used the 2003 American Heart Association (AHA) criteria, which consider levels of cardiac troponin [32] - a component of cardiac muscle and a more sensitive and specific indicator of myocardial damage [54] – in addition to creatine kinase. Similarly, in the Joint European Society of Cardiology/American College of Cardiology (ESC/ACC) criteria [55] - used in two studies [42], [53] - troponin levels take precedence over creatine kinase, and neither aspartate transaminase nor lactate dehydrogenase (the two other enzymes from MONICA) are considered markers of cardiac damage [56].

Support for the increased sensitivity of cardiac troponin is provided by many clinical and population-based studies [57]–[59] where more cases of MI were detected when applying the new criteria than when the MONICA. Consistent with this, some authors have shown that, when defined by the older criteria, the incidence of MI appears to have declined over the decades, but when the newer criteria are applied, the incidence appears to have remained steady [60] or even increased [61]. In other words, more cases will be classified as MI under the newer criteria than the old. Thus, given the increased sensitivity of the newer criteria, we expected to see greater sensitivity values amongst the more recently-published studies in this review, but we did not observe a trend in either direction. Amongst the ten studies reporting on the sensitivity of MI diagnoses in hospital data, sensitivity in the five earlier studies ranged from 80–94%, while in the five later studies it ranged similarly from 69–93%. This may simply be due to the comparatively small number of studies where sensitivity was reported, though heterogeneity in the study settings may also play a role. One study included in our review, by Rosamond et al [36], evaluated the sensitivity and PPV of ICD-9 410 over the period 1987–2000. They reported that while overall, these statistics remained relatively stable, amongst teaching hospitals they declined significantly (with sensitivity declining from 74% to 59%, and PPV from 80% to 71%). In contrast, in a study conducted at a university hospital in the Netherlands, both sensitivity and PPV were higher in the later period (years 1996–2003) than the earlier period 1987–1995 (with sensitivity increasing from 82% to 85%, and PPV from 94% to 99%) [62].

In addition to being more sensitive, cardiac troponin is also a more specific indicator of MI. Although few studies in this review reported specificity values directly, this statistic can be analysed by way of PPV. Specificity is equal to 1 - the number of false positives, so will increase as the number of false-positive cases decreases. PPV is the proportion of true-positives amongst all true-positive and false-positive cases, so will also increase as the number of false-positive cases decreases. The fact that the PPV's for hospitalization data generally increased over time provides support for an increase in the specificity of MI diagnoses as well.

When comparing the performance of the newer diagnostic criteria to the MONICA, the contribution of other secular changes must be considered. One factor is the use of different revisions of the ICD coding system in different time periods. Mahonen et al [35] found that the sensitivity of ICD 410 was generally lower during the period 1987–1990 (ICD-9) than 1983–1986 (ICD-8), even though the same diagnostic critera (FINMONICA, a Finnish adaptation of the MONICA criteria) were used throught the study period. In contrast, those authors found that the PPV's in the ICD-9 period were generally higher than in the ICD-8 period. However, the impact that cardiac troponin testing has on the validity of MI diagnoses is difficult to ignore. For example, Pajunen et al [43] reported higher sensitivity during the ICD-10 period (1998–2002) than the ICD-9 period (1988–1997), but the authors attribute this difference to the use of cardiac troponin testing during the ICD-10 period. We believe the introduction of cardiac troponin testing and its increasing use over time may be mainly responsible for the improvements we observed in the PPV of MI codes over time.

When examining only studies that used the MONICA criteria, we observed that the PPV's were usually higher in studies stemming from the original MONICA project compared to those just applying the MONICA criteria in other samples. This was especially apparent amongst the European studies from the MONICA project. One explanation for this may be some cross-referencing between the hospital databases and MONICA registries. It is acknowledged in these studies [21], [35] how the MONICA project itself may have influenced local coding practices. For example, some of the same physicians that were involved with the MONICA study were also treating patients hospitalized for coronary events in local centres. However the potential influence these factors may have had in Europe, they did not appear to carry over in Australia and New Zealand, where the PPV's in studies using the MONICA registries were much lower.

We observed that the accuracy of MI as a cause of death on death certificates was lower in comparison to hospitalization data. Death certificate diagnoses of MI may be less accurate because less information is available on these cases from which to determine a precise cause of death. Specifically, many deaths are not attended to by medical personnel, resulting in a lack of comprehensive documentation [39]. In support of this, Lowel et al [41] found that the PPV's were lower for cases who spent less time in hospital, and had less clinical data and test results (including electrocardiograms and enzyme levels) available, which could otherwise aide in establishing a more accurate cause of death [41].

Our review showed that the accuracy of hospitalization data for identifying MI cases is much higher than data from death certificates; consequently, we recommend that, when available, researchers attempt to confirm the cause of death by matching vital statistics death records for MI with administrative hospitalization data. At the very least, the limitations of vital statistics data should be acknowledged by these authors.

Many of the findings presented in this paper are based on PPV, which was the most frequently-reported statistic amongst the studies included in this review. PPV is relatively easy for researchers to assess since they only need to evaluate cases who initially test positive for the condition (here being MI). However, a caveat of both PPV and NPV are their dependence on the prevalence of the condition in the study population [63]. The PPV will be lower for a rare condition than for a common condition. For example, amongst all testing positive in a rare condition (those in the denominator), few are likely to be true-positives (and appear in the numerator). In this review, we expected the PPV's to be lower amongst the community-based studies than the clinic-based studies or those with otherwise more selected populations, and this was apparent in several studies. For instance, the PPV in a study of patients admitted to coronary care units was 89% [48] and in two studies that were restricted to individuals aged 65 years and older (amongst whom MI is more common) the PPV's were 95% [47] and 98% [45]. In contrast, in another study which had a younger source population (aged between 25 and 64 years), the PPV was much lower (only 67%) [37]. Consequently, differences in the expected prevalence of MI in the different source populations may have contributed to variation in the PPV's reported by the different studies in this review.

A significant research gap was identified in the course of this review, being a lack of studies reporting on the validity of codes from the ICD-10. This system has been in widespread use in Europe and Australia for at least a decade, but ICD-10 codes were evaluated in just three studies included in this review, and only two of these [42], [43] reported on the validity of ICD-10 codes separately from ICD-9 codes. One of these studies reported that the PPV for ICD-10 I21-22 was good, especially in tertiary care hospitals (PPV = 93%) [42], and findings from the other suggest that ICD-10 I21-22 is more sensitive for MI than the equivalent ICD-9 code, 410 [43]. With ICD-10 codes now a key component of health research, assessments of the validity of ICD-9 codes are quickly losing their relevance, and clearly, more investigations into the accuracy of ICD-10 codes are needed to support ongoing research endeavours.

Our systematic review has some limitations. We could not consider articles whose full-text was not available in English, and this may have introduced a language bias. We were unable to include articles that did not report or reference the diagnostic algorithms being validated, or those that were published after the conclusion of our search period (February 2011). As well, although our MEDLINE and EMBASE searches were conducted by an experienced librarian, some relevant studies may have been missed since administrative databases are not well catalogued in these indexes (e.g. no MeSH term pertaining to “administrative database”). Most of the articles included in this review were located through database searches. In these, we searched for articles that were indexed under terms relating to Administrative Data, Validation, and Cardiovascular Disease. However, in our subsequent handsearch we located several relevant articles that were not indexed under these Administrative Data or Validation categories. Thus, while our handsearches were extensive, it is possible that we still missed some relevant articles if they were not indexed in the databases with a term relating to validation or administrative data, or were published in a journal not indexed in the MEDLINE or EMBASE databases.

In summary we conclude that, based on the evidence, hospitalization data can be used to identify MI as a covariate or outcome, but the accuracy of MI as a cause-of-death in vital statistics data is limited. Authors using vital statistics data to identify MI deaths are encouraged to compare such data with hospitalization data to confirm the cause of death or use sensitivity analyses excluding cases from this source. While most administrative databases are not established for research purposes, they are increasingly being used to study long-term patient outcomes and disease burden. Therefore, in order to maximize the sensitivity of these databases, physicians and hospital coders should be encouraged to record all significant complications and comorbidities. In the meantime, authors using administrative data to identify MI deaths should acknowledge the limitations of this data source. Finally, with ICD-10 coding now commonplace, more assessments of the validity of ICD-10 codes for MI are needed to ensure the quality of future research. We believe our findings will help to increase the rigour of population-based epidemiological and outcomes research and thus potentially improve health surveillance, resource allocation and patient care.

Supporting Information

Funding Statement

This study was funded in part by the Canadian Arthritis Network (http://www.arthritisnetwork.ca). JAAZ held a salary award from the Canadian Arthritis Network and The Arthritis Society of Canada. He is currently the British Columbia Lupus Society Scholar. DL holds the Mary Pack Chair in Arthritis Research from UBC and The Arthritis Society of Canada. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.