Detecting Mild Cognitive Deficits in Parkinson’s Disease: Comparison of Neuropsychological Tests

Study Design and Participants

The IPMDS PD-MCI Validation Study Group is an international consortium that was formed to validate the IPMDS PD-MCI criteria¹⁴ in a pooled group of large extant databases from studies of PD cognition^9,17 (see Supplementary Co-Investigator File detailing the IPMDS Study Group “Validation of Mild Cognitive Impairment in Parkinson Disease”).

Data from the consortium include clinical and neuropsychological measures from a total of 24 cohort studies in PD with a focus on cognition, 13 of these studies also including data on healthy controls (HCs). Supplementary Table 1 and Table 1 show the characteristics of the included cohorts. Neuropsychological tests at each site were administered and scored by trained research personnel and in that country’s official language.

To achieve our goals, a 2-step data-inclusion process was used. First, cross-sectional baseline data on neuropsychological tests were selected from the IPMDS PD-MCI consortium database when (1) normative scores based on published norms were available, to enable comparison of neuropsychological tests to each other and across slightly different versions sometimes used in different countries, and (2) the neuropsychological tests were administered in at least 2 countries and across at least 3 studies to increase the generalizability of our results. Details on the neuropsychological tests included are shown in Table 2.

Second, eligible patients (PD) and HCs were selected from the identified studies to establish a nondemented study cohort. Patients diagnosed with PDD at baseline visit were excluded; this classification was performed according to the diagnostic criteria specified in Supplementary Table 1, and when formal diagnostic criteria were not applied, patients were excluded as having possible dementia based on scoring below recommended cutoffs on available global cognitive screening instruments¹⁸ as follows: Montreal Cognitive Assessment (MoCA; cutoff 20/21),^19,20 Mattis Dementia Rating Scale (MDRS; cutoff 129/130, a blend between the original recommended cutoff²¹ and more recent recommendation),^21,22 and Mini-Mental State Examination (MMSE; cutoff 23/24).²³ The same cutoffs on the global cognitive measures were applied to HCs. To focus on typical PD patients, patients with a disease duration > 25 years were excluded. Applying these criteria led to the exclusion of an additional 72 participants (49 PD patients, 23 HCs), and another 46 participants (36 PD patients, 10 HCs) who had neither dementia status coded nor available cognitive screening tests (Fig. 1).

Preprocessing of Neuropsychological Data

Normative neuropsychological scores were available as either z scores, t scores, or various types of scaled scores (eg, Wechsler scaled scores). The scores were transformed to a z-score scale based on their theoretical distributions. Furthermore, to avoid influential outliers, scores were winsorized to the −3.0 to 3.0 standard deviation (SD) range, which affected 2% of the data.

Incomplete Data

For quality control and to avoid incomplete data in large undefined subsets that are possibly not missing at random, data for any given test-cohort combination were not used if the test had >25% missing values within a PD cohort. Detailed study of reasons for missing data within the original studies was not feasible. In all other cases of missing data, multiple imputation was applied using the mice package in R.²⁴ Multiple imputation accounts for the relationship between variables, which is especially useful for correlated data (eg, neuropsychological test results) and takes into account uncertainty regarding incomplete data.²⁵ Incomplete data were assumed to be missing at random.²⁶ To allow for between-study heterogeneity in the imputation model, multiple imputation was applied within each cohort’s data set.²⁷ Twenty imputations were run using predictive mean matching based on a model that included age, sex, years of education, Hoehn and Yahr stage, Unified Parkinson’s Disease Rating Scale (UPDRS) part III score, disease duration, and neuropsychological measures, with PD-related measures contributing to imputations in PD patients only. Therefore, different imputation models were run for analyses A and B (described below) because of their different requirements. Summary measures were derived within each original study. Analyses were pooled over imputations using Rubin’s rules,²⁸ which provides a means to combine within- and between-imputation variability into a single estimate.

Analysis A: Neuropsychological Performance in PD Using Published Norms

For analysis A, each test (ie, subtest score) was analyzed separately (see Table 2 for sample size for each test and published norms used). To evaluate whether PD patients deviated from normal performance on each of the available neuropsychological measures, their performance in terms of normative z scores was compared relative to the expected mean under normal performance (ie, zero). Exclusion of zero from the 95% confidence interval indicated rejection of the null hypothesis of no impairment. A 2-step meta-analysis approach was used to derive these estimates, using the pooled within-study estimates for both fixed and random effects inverse variance-weighted models.²⁹ The Hartung Knapp method, as implemented in R’s meta package,³⁰ was used; this is recommended when the number of contributing studies is low.³¹ A lower bound of the 95% confidence interval of I² over 50% indicated that greater than half the observed variability in the estimate was a result of between-study differences and was interpreted as a sign of large heterogeneity in performance.

In addition, the residual influence of age, sex, and education on the normative scores of PD patients was evaluated using individual patient-level data. Models with random intercepts per cohort and fixed effects for age, sex, and years of education were fitted for each outcome of interest in analysis A. Based on the findings from these models, post hoc evaluation of the 2-step meta-analysis in sex subgroups was performed.

Analysis B: Neuropsychological Performance in PD Versus HCs Using Published Norms

This analysis compared performance on normative scores in PD patients relative to the performance of the HCs, with the latter enrolled at the same sites as the respective PD patient samples. The tests available in both PD patients and HCs were a subset of those used for analysis A (Table 2). As in analysis A, a 2-step approach was used. Pooled estimates of neuropsychological performance in PD and HCs were obtained within each of the original studies and entered into both fixed- and random-effects meta-analysis models using the Hartung Knapp variance estimator to construct the confidence intervals accounting for between-study variability.

In addition, individual patient data were used to correct the results of analysis B for possible differences in age, sex, and education between the PD and HC cohorts. Models with random intercepts and patient-versus-control effects per cohort and fixed effects for age, sex, and years of education were fitted for each outcome of interest in analysis B.

Analysis C: Neuropsychological Performance in HCs Using Published Norms

These methods were exactly similar to the 2-step meta-analyses used for analyses A, but applied to HCs instead of PD participants.

Software

All analyses were performed using the R statistical software.³² SPSS 22.0³³ was used for data management.

Analysis A (Neuropsychological Performance in PD Using Published Norms)

The total available number of PD patients was 3101 (Fig. 1). Figure 2 shows overall estimates from the random effects models for each of the tests. In total, 77% of the comparisons showed substantial signs of between-study variability, and the median I² over all comparisons in analysis A was 93%. This high degree of between-study variability precluded strong conclusions based on the overall estimates. Nonetheless, despite the observed heterogeneity, the 95% CIs still excluded zero for 6 test scores: 3 verbal memory performance indices of the Hopkins Verbal Learning Test (HVLT), both indices of the Trail-Making Test (TMT), and Stroop Color naming.

Supplementary Figure 1 shows forest plots, including overall random effects model estimates (fixed-effect estimates were omitted because of the observed heterogeneity), measures of heterogeneity, and study-specific means, standard errors, numbers of observations, and analyses weights for each individual cognitive test. To illustrate the observed heterogeneity, the Animal Fluency forest plot shows that many study-specific results deviate from zero on both sides. Prediction of results for a future study would be unreliable because most of the variability depends on unaccounted study characteristics.

Despite using standardized scores, residual effects of age, sex, and education were found on many of the test scores (Supplementary Table 2). Increasing age (14 test scores), male sex (19 test scores), and fewer years of education (25 test scores) were associated with worse performance. Exceptions were relatively better performance for men on Judgement of Line Orientation (JOLO) and Boston Naming Task and for increasing age on WAIS vocabulary. The residual effects of age were relatively small (ie, up to 0.31 SD change for a 10-year age difference), whereas some of the sex and education effects were quite large (eg, women scoring 0.7-0.8 SD lower on the HVLT- and RAVLT immediate recall and each year of education adding 0.15 SD to WAIS similarities performance). In general, these differences between age, sex, and education effects can be explained by the current state of normative data for which correction for age is always applied, but only a minority correct for all 3 characteristics.

Results for the post hoc meta-analyses in sex subgroups are shown in Supplementary Figure 2(women) and Supplementary Figure 3 (men). Heterogeneity was reduced but still problematic. Female-specific results only showed significant deviation from the reference population mean for TMT A and B. Male-specific results showed deviations for 10 test scores: 5 verbal memory performance indices (3 from the HVLT and 2 from the Rey Auditory Verbal Learning Test [RAVLT]), TMT A and B, and Stroop Word, Color, and Interference

Analysis B (Neuropsychological Performance in PD Versus HCs Using Published Norms)

A total of 1226 HCs were available (Fig. 1). Figure 3 shows the estimates for the PD-versus-HC normative score comparisons. On average, PD patients differed from HCs on all but 3 of the evaluated measures (Digit Span Forward, Modified Wisconsin Card Sorting Test perseverative errors, and Boston Naming Test). The degree of between-study variability was smaller than in analysis A, with 18% of I² CIs excluding 50% on the lower side. This is illustrated in Supplementary Figure 4 by the forest plots for analysis B.

The test scores with 95% CIs excluding zero included test scores in all 5 cognitive domains specified in the PD-MCI criteria: language (WAIS Similarities); attention and working memory (WAIS Digit-Symbol Substitution [or Coding], TMT A, 2 Stroop scores [Word and Color], and Digit Span total); executive (letter fluency [COWAT], animal fluency, Stroop Interference, TMT B, and Modified Wisconsin Card Sorting Test nonperseverative errors); memory (both WMS logical memory indices); and visuospatial (JOLO).

Correcting the analyses for overall effects of age, sex, and education yielded the same results, although quantitatively slightly different (most notably, the estimate for Digit Span Forward was more precise), as shown in Supplementary Figure 5. The effects of age, sex, and education are summarized in Supplementary Table 3 and show worse performance for increasing age on animal fluency, TMT B, Stroop Interference, and JOLO; male sex on animal fluency, COWAT, Stroop Word, Color, and Interference, WAIS-III Digit-Symbol Substitution (or Coding), and WMS logical memory part I/II; female sex on JOLO; and lower education on all test scores. As for analysis A, the age effects were relatively small, whereas some sex and education effects were substantial.

Analysis C (Neuropsychological Performance in HCs Using Published Norms)

To put the performance of PD patients relative to HCs in perspective, we examined HC normative performance separately. Examining the 14 test scores that PD patients demonstrated impairment relative to HCs, the HCs performed above average (ie, the 95% CI excluded zero) on 50% of the tests (7 of 14) and borderline (95% CI bordered on zero) on 14% (2 of 14); see Supplementary Figure 6.