Hypothesis Testing Methods for Multivariate Multi-Occasion Intra-Individual Change

Use of LGC Models for Evaluating Intra-Individual Change

Although the LGC models are usually used to evaluate population level change, they can also be used to produce estimates of intra-individual change trajectories. Specifically, a second-order LGC model with IRT θ as a dependent variable could be written as

Assuming a unidimensional ${\theta }_i$ is measured at each occasion and the total number of occasions is T, then both ${\mathit{\theta }}_i$ and ${\mathit{e}}_i$ are T-by-1 vectors. ${\mathit{X}}_i$ and $\mathit{Z}$ are the T-by-p and T-by-q design matrices, and β and ${\mathit{u}}_i$ are p-by-1 and q-by-1 vectors denoting fixed and random effects respectively. In a more general case, $\mathit{Z}$ can be different across individuals (i.e., ${\mathit{Z}}_i)$ so that it takes into account unbalanced time design. The individual random effect ${\mathit{u}}_i$ will be obtained through the following equation

In Equation 2, ${\widehat{\mathit{G}}}_i$ is the estimated covariance of random effects, ${\widehat{\mathit{V}}}_i={\mathit{Z}}_i{\widehat{\mathit{G}}}_i{\mathit{Z}}_i^{\mathrm{T}}+{\widehat{\mathit{R}}}_i,$ where ${\widehat{\mathit{R}}}_i$ is the estimated residual covariance matrix. Using the multivariate delta method, the sampling variance of ${\widehat{\mathit{u}}}_i$ can be computed from Equation 3 [see Equation (5.20) in Hoffman (2015)],

such that it could be used to evaluate whether the individual change is significant over time.

Despite of the solid statistical foundation of the LGC model and the easy-to-use closed-form equations introduced above for making statistical inference at individual level, the LGC model is not without limitations. Because the individual random effect ${\mathit{u}}_i$ is “predicted” from the population model (i.e., Equation 2), the individual growth trajectories (such as intercept and slopes in linear growth models) are actually a weighted function of that person’s ordinary least squares (OLS) estimates of his/her growth parameters (${\mathit{\gamma }}_{OLS,i}$) and the model fixed effects (i.e., β) (Hoffman, 2015, p. 200). That is, the composite individual effect is

where ${\mathit{W}}_i={\mathit{G}}_i{\left[{\mathit{G}}_i+{\mathit{R}}_i{\left({\mathit{Z}}_i^{\mathrm{T}}{\mathit{Z}}_i\right)}^{-1}\right]}^{-1}$. Indeed, ${\mathit{\gamma }}_{OLS,i}$ is a truly individual level parameter that is not affected by the groups to which the individual belongs, but it might suffer from low precision when the number of occasions is few. Instead, ${\mathit{\beta }}_i$ is affected by population parameter $\mathbf{\beta }$, which means that a person’s change trajectory is considered relative to the group in which the person happens to be embedded—the same person with a given pattern of change might be fitted to a different pattern of change in the context of a different group. Instead, this paper considers intra-individual change on a latent variable using data from a single person, and directly address the above-mentioned question of the significance of change.

Reliable Change Index

At the intra-individual level, a routinely used change statistic is the reliable change index proposed in the clinical context, (RCI; Jacobson, Follette, & Revenstorf, 1984; Jacobson & Truax, 1991), which defines whether an individual’s change after therapeutic intervention is “statistically significant.” RCI is defined as the ratio between a person’s observed change score and the standard error of measurement (SEoM) of their difference score,

In Equation 5, X_i1 and X_i2 denote the pre- and post-test summed scores for person i, SD₁ is the standard deviation of scores in the pretest sample, and r_XX′ is the reliability. Applications of RCI in the literature are unclear as to which type of reliability to use. Some users of RCI use internal consistency reliability (e,g, coefficient alpha; Cronbach, 1951) but sometimes use a reliability coefficent from other groups rather than computing it from a group to which the person belongs (e.g., Bauer, Lambert, & Nielsen, 2004)—one study (Hiller et al., 2012) used an “average” reliability from a meta analysis. Others recommend the use of test-retest reliabilities (e.g., Christensen & Mendoza, 1986), which obviously would need to be computed from an independent group in which no change was assumed to have occurred.

Over the years, various revisions of the RCI index have been proposed and a number of alternative change indices have been proposed to replace or refine it¹ (e.g., Christensen & Mendoza, 1986; Hageman & Arrindell, 1993; Maassen, 2001; Speer, 1992). In addition, there have been a number of studies that compared the RCI with other methods of determining significant individual change (e.g., Atkins, Bedics, McGlinchey, & Beauchaine, 2005; Bauer, Lambert, & Nielsen, 2004; Hiller, Schindler & Lambert 2012; McGlinchey, Atkins, & Jacobson, 2002; Speer, 1999), primarily by comparing the number of instances of “significant” change identified in some sample. Although these studies have demonstrated that different variations of RCI and other similar indices identify different numbers and combinations of “significant” change, none has been able to demonstrate that one method is better than another when compared with some external criterion of change. The general conclusion emerging from this research is that the RCI index in Equation 5 is useful and simple to compute, and should be sufficient for most applications (e.g., Speer, 1999; Speer & Greenbaum, 1995).

However, there are psychometric problems with RCI that limit its usefulness and sometimes lead to illogical results. As shown in Equation 5, a primary assumption of RCI is that the SEoM, ${\mathit{SD}}_1\sqrt{1-r_{X{X}^{\prime }}}$, is the same for all persons, a basic assumption of classical test theory (e.g., Gulliksen, 1950; Lord & Novick, 1958; Brouwer, Meijer, & Zevalkink, 2013). Modern test theory, discussed below, shows that this assumption is untenable—the SEoM can vary substantially as a function of the level of the trait being measured and even between two individuals who have the same observed test score.

A second problem with the RCI, or any of its derivative indices, is the use of reliability to obtain the SEoM. If different values of reliability are used in Equation 5, different conclusions might be drawn about the significance of change for a person with the same scores at pre- and post-test, which is clearly an illogical situation. Similarly, because Equation 5 uses the standard deviation of the pretest scores in the computation of the SEoM in its denominator, a single person with the same scores on pre- and post-tests measured in the context of two different groups with different variabilities (SDs), might have “significant” change when considered as part of one group and non-significant change when measured as part of the other group—another illogical situation.

In constrast, Weiss and Kingsbury (1984) first proposed an IRT-based confidence interval method for identifying significant intra-individual change, which was later evaluated by Kim-Kang and Weiss (2008). Suppose for a randomly selected person, the null hypothesis is and the alternative hypothesis is, where and denote the unidimensional latent trait scores for a person on two testing occasions, respectively. The change methods for testing $H_0$ against $H_a$ permit a determination of whether a single person’s change in $\theta$ is meaningful or “psychometrically significant” where “significance” is based on psychometric theory using the observed data from a single person. This is different from statistical significance that is usually tested at the population level in growth models. Confidence intervals are constructed around the two estimates ${\widehat{\theta }}_1$ and ${\widehat{\theta }}_2$ based on information in a single person’s response patterns, and the significance of change is determined when the two confidence intervals do not overlap.

Finkelman, Weiss, and Kim-Kang (2010) introduced two additional hypothesis testing methods, namely, a Z-test and a likelihood ratio (LR) test. The performance of these two change indices and the confidence interval method was evaluated in Monte Carlo simulation studies. The results showed that the two new approaches had Type I error rates close to the nominal rate and very good power to detect simulated true change in comparison to the confidence interval approach, which had very low Type 1 error rates and lower power. Lee (2015) later studied two additional hypothesis testing methods, namely, the score test and the Kullback-Leibler divergence test, and found that the LR index displayed the best balance of Type I error rates and power than the other change indices under various conditions.

The earlier research using IRT for evaluating psychometrically significant intra-individual change focused on unidimensional θ at two measurement occasions, and recent studies have extended the methods to unidimensional θ at multiple occasions (Phadke, Weiss, & Christ, 2016; Phadke, 2017; Phadke & Weiss, 2017), or multidimensional θ at two occasions (Wang & Weiss, 2018). The current study further extends the hypothesis testing methods to evaluate if a person’s multidimensional θ changes significantly across multiple occasions. The methods studied herein are the most general and subsume the methods discussed in earlier research (Finkelman, et al., 2010; Lee & Weiss, 2014; Phadke 2017; Wang & Weiss, 2018; Weiss & Kingsbury, 1984) as special cases. Studying intraindividual multivaraite change has profound significance in that the majority of the constructs measured in psychology and education are inheriently multidimesional, such as intelligence, personality, or achievement. Hence, it is importnt to evaluate whether the profile of a multifaceted construct (rather than a single unidimensional construct) of individual change over time will provide a synthesized view of change and potientally identify unique patterns of change that would otherwise be neglected when each facet is considered in isolation.

Purpose

This paper describes the development of several change indices designed to quantify and test the psychometric significance of intra-individual multivariate change for variables measured on multiple occasions. The performance of the indices is evaluated in Monte Carlo simulation studies in the context of both conventional fixed-form tests and computerized adaptive tests, and their application is illustrated with measures of patient-reported outcomes in a hospital setting.

Multidimensional Graded Response Model

This study used the multidimensional graded response model (MGRM). If $\mathit{\text{θ}}={\left({\theta }^{\text{1}}\text{, …,}{\theta }^{\text{D}}\right)}^{\text{T}}$ denotes a D-by-1 latent trait vector (i.e., a D-dimensional vector) for an examinee, and responses to item j are from ${\text{C}}_{\text{j}}$ categories, {0, …, ${\text{C}}_{\text{j}}-\text{1}$}, then the boundary response function, which gives the probability of responding with category k or above in item j with trait vector $\theta$, is computed as

where ${\mathit{\text{a}}}_{\text{j}}\text{∈}{\mathbb{R}}_{\text{+}}^{\mathit{\text{D}}}$ is the item discrimination parameter of item j, and ${\mathit{\text{b}}}_{\mathit{\text{jk}}}\text{∈}{\mathbb{R}}^{\text{1}}$ is the boundary location parameter of the kth category of item j. In Equation 6, $\mathit{\theta }-{\mathit{\text{b}}}_{\mathit{\text{jk}}}$ means subtracting a common constant ${\mathit{\text{b}}}_{\mathit{\text{jk}}}$ from the vector of $\mathit{\theta }$, resulting in a D-by-1 vector, as well. The probability of responding with the lowest category or above, $\mathit{\text{P}}\left({\mathit{\text{u}}}_{\mathit{\text{j}}}\text{≥0|}\mathit{\theta }\right)$, is set to 1.0, and the probability of responding above the highest category, $\mathit{\text{P}}\left({\mathit{\text{u}}}_{\mathit{\text{j}}}\text{≥}{\mathit{\text{C}}}_{\mathit{\text{j}}}\text{|}\mathit{\theta }\right)$, is set to 0.0. Then, the category response function, which gives the probability of responding with category k, is

It is assumed throughout the study that the item parameters are known in advance, and item parameter calibration error is not considered in θ estimation, as is usually assumed in operational analyses (e.g., NAEP, von Davier & Sinharay, 2007).

When assessing change over time, it is important to consider whether the same construct, measured by multiple indicators, has the same meaning and metric over time (McArdle, 2001; McArdle & Hamagami, 2001). This falls under the rubric of measurement invariance (Widaman, Ferrer, & Conger, 2010), and is often referred to in a longitudinal setting as longitudinal invariance. Using the same set of items or a set of anchor items allows longitudinal invariance to be tested (Grimm, Kuhl, & Zhang, 2013). A thorough examination of longitudinal invariance is beyond the scope of this study, but longitudinal invariance is assumed to hold when evaluating intra-individual change.

Multivariate Hypothesis Testing for Multi-Occasion Intra-Individual Change

In the context of testing whether change occurs across occasions and in any of the latent trait dimensions for a given examinee, the null hypothesis, ${\text{H}}_0:{\mathit{\theta }}_1={\mathit{\theta }}_2\mathrm{=\dots \; =}{\mathit{\theta }}_T$, is tested against the alternative hypothesis, ${\text{H}}_a:$ ${\mathit{\theta }}_m\ne {\mathit{\theta }}_n$ for at least one pair of (m, n) where m, n ∈ {1, …, T}, m ≠ n, and T is the number of occasions. Four methods for performing such hypothesis tests are introduced here; they are extensions to multiple occasions of the work by Wang and Weiss (2018), who proposed and evaluated tests for psychometrically significant multivariate change for two occasions. The four methods include the likelihood ratio (LR) test, the multivariate Wald test and its modification, as well as the score test (also known as the Lagrange multiplier test). In essence, the LR test computes the ratio of the likelihoods at the null and alternative hypotheses, whereas the Wald test evaluates how far the model parameters under alternative hypothesis are from the hypothesized values under the null. The score test statistic is calculated based on the slope of the likelihood function at the parameter values satisfying the null hypothesis, and it is used to estimate the improvement in model fit when the constraints on the parameter values from the alternative hypothesis are removed. Although the values of the Wald and score test statistics will become increasingly close to the test statistic from the LR test when the test length increases, it is still interesting and important to compare the performance of the three test statistics under finite test length.

Study 1: Fixed-form Test

The performance of the four hypothesis testing methods was compared with conventional fixed-form tests using Monte Carlo simulation. Both the false positive rates (FPR, or type I error) and the true positive rates (TPR, or power) were evaluated in different datasets. FPR is the proportion of simulees (i.e., simulated individuals) who had no change but were incorrectly identified as exhibiting significant change. TPR is the proportion of simulees who had a specified level of change and were correctly identified as having significant change.

Study 2: Computerized Adaptive Test (CAT)

In Study 2, the performance of the four hypothesis testing methods were further evaluated using CAT. Because CAT produces more accurate θ estimates than fixed-form tests with the same test length (e.g., Weiss, 1982), it was expected that the FPR would decrease and TPR would increase across four methods. CAT was specifically considered because the real data (see below) came from a CAT administration.

Real Data Example

Conclusions

Taken together, the results of the simulation studies and the real data application suggest that the change indices proposed and evaluated in these studies are viable methods for evaluating intra-individual multivariate change. Among the three methods, the LRT test is recommended as it provides the best balance between FPR and TPR. The application of these methods in real data studies can serve to refocus the study of individual change to a single individual at a time, rather than considering individual change through the lens of group change, such as in the LGC approach. Using these methods, an individual can be classified into those with significant change and those with non-significant change, independent of the groups to which they are assigned.

The proposed change identification methods were studied here in the context of a multidimensional graded IRT model, but they are applicable to any IRT model since all that they require is a set of IRT parameters for a given set of items and the responses of an individual to items from that subset in order to estimate trait levels (θ) and their standard errors. On the assumption that the dimensionality of the traits is constant over time, either the same or different subsets of items can be used for the longitudinal measurements, depending on the measurement objectives. Finally, as demonstrated, the methods can be applied in either conventional tests or CATs, but CATs are recommended because of their superior performance.

Limitations and Future Directions

One limitation of these studies is that the item parameters were assumed to be error-free. However, as previous research has shown (e.g., Cheng & Yuan, 2010; Patton, Cheng, Yuan, & Diao, 2013), ignoring calibration error would underestimate the standard error of $\widehat{\mathit{\theta }}$. This implies that had estimated item parameters been used instead of true item parameters, the FPR of the omnibus hypothesis testing would be inflated. This conjecture should be further verified via future simulation studies. On the other hand, if a second-order LGC model (e.g., Wang et al., 2016; Wang & Nydick, 2019) was used to estimate both item parameters and growth of $\mathit{\theta }$ simultaneously, the item parameter estimation error would automatically be accounted for in the model, albeit from a group lens. Another unaddressed issue is longitudinal invariance. Because invariance is a population level concept, invariance has to be evaluated using a group longitudinal model (e.g., Widaman et al., 2010) first before using the proposed intra-individual omnibus tests.

This research could be extended in a few directions. First, as alluded to in the introduction, a thorough comparison study between the LGC approach and the proposed approach could be conducted because the LGC approach also allows for drawing statistical inferences at the individual level (i.e., Equations 2~4). It will be particularly interesting to see if, and to what extent, the individual level inference would be affected by the group change pattern using the LGC approach. Second, within the framework of evaluating the change between two or multiple profiles for each individual, differences between two profiles can be quantified using their level (mean difference), scatter (variability) and shape (left over after accounting for level and scatter) (Cronbach & Gleser, 1953). This would offer a nuanced way to group individuals based on their unique change patterns. Third, in the current simulation setting with T = 4, the effect size of change was relatively high such that the power from different methods with CAT was very high and fairly close, although the power still differed across methods with fixed-form tests. Therefore, the simulation studies could be further expanded to identify the lower limits of individual change that is detectable using these methods with CAT.