Computing Confidence Bounds for Power and Sample Size of the General Linear Univariate Model

1.1 Motivation

Statisticians estimate power and sample size in many settings. These quantities are used in conducting a sensitivity analysis while planning a future study (prospective power analysis), and also help in the quantitative evaluation of an existing study (retrospective power analysis). Muller and Benignus (1992) recommended that any claim of no effect be substantiated by an appropriate retrospective power analysis. For example, the wish to demonstrate no appreciable effect arises in assessing toxicity of any compound proposed for commercial use. Following a negative result, a study's power to have detected an appreciable difference may be important. Regulatory decisions by agencies such as the U.S. Environmental Protection Agency often hinge on such negative evidence.

Researchers may propose fixed values as the true population means in order to conduct a power analysis. They often complete the power computation for a general linear univariate model (GLUM) test by using a variance estimate σ̂² obtained from an existing study. For example, physicians describe a “clinically significant” effect as one that changes a person's health in an important way. Combining a fixed sample size (N), a fixed, clinically significant mean difference, and a variance estimate determines a power value. With estimated error variance, a power value for a fixed N must be recognized as a random variable. Similarly the sample size needed to achieve a fixed power varies randomly. Standard statistical practice requires reporting uncertainty associated with such point estimates. A lower bound for power allows stating, with specified confidence, that a study has power of at least “P” to detect the effect. Similarly, an upper bound on N provides a sample size which ensures, with specified confidence, a desired target power.

Kupper and Hafner (1989) demonstrated the bias resulting from choosing sample size for a test statistic assuming known variance (a z test), when one will conduct the data analysis with an estimated variance (t test). Power calculations for both a z test and a t test start from an assumption of known variance, and diverge due to differences in the conduct of data analysis (test statistic computation). This highlights the importance of aligning power computations with actual analysis methods. Kupper and Hafner, as well as Beal (1989), provided closely related discussions in the context of choosing the N necessary to control the length of a confidence interval for a scalar parameter.

Other authors have addressed the question of confidence intervals for power. Dudewicz (1972) suggested substituting exact confidence bounds for σ̂² into power calculations for a t test. This yields approximate confidence limits for power. He did not present any asymptotic or simulation evidence as to accuracy, nor did he note the approximate nature of the calculations. Clark (1987) presented analytic and simulation results for generalization of the method to the fixed-effects GLUM with Gaussian errors. She proved that exact confidence bounds for estimated variance lead to asymptotically unbiased confidence intervals for both noncentrality and power. She also demonstrated, via simulation, the optimistic bias in the method. We present an alternate and straightforward formulation which allows computation of exact confidence bounds for noncentrality, power, and sample size in the GLUM. These methods are applicable when testing scalar and vector hypotheses.

1.2 An Example

Falk, Hogan, Muller, and Jennette (1992) compared treatment of deteriorating renal function with both cyclophosphamide and corticosteroid therapies to corticosteroid therapy alone. The researchers defined a clinically significant improvement in renal function as a doubling of reciprocal serum creatinine level, corresponding to an increase of .50 dL/mg. Twelve patients were randomly assigned to each treatment in a clinical trial. A power calculation was performed with a test size of α_t = .01 assuming true improvement due to dual therapies was .50 dL/mg. Based on a variance estimate of .068, power was approximated as .960. The authors used these calculations both to stop the trial early and to argue that the treatment likely had little value. Given these results, what interval of uncertainty surrounds the power estimate of .960?

1.3 Statement of the Model, Hypothesis, and Properties

The GLUM with N sampling units and q predictors can be stated (Searle 1971, chap. 5):

Here X, with rank r ≤ q < N, contains fixed, known constants and β contains fixed, unknown parameters. Also assume $\text{e}\stackrel{d}{=}{N}_N(\mathbf{0},{\sigma }^2\mathbf{\text{I}})$ with 0 < σ² < ∞. The usual hypothesis involves the secondary parameters θ = Cβ. Interest lies in testing the general linear hypothesis

versus

Both θ₀ and C contain fixed constants chosen by the data analyst. Assume full rank for C, with each row of C defining an element of θ. The usual GLUM test statistic is

in which

and (·)⁻ indicates a generalized inverse. The test statistic follows a noncentral F distribution with ν₁ (size and rank of C) numerator degrees of freedom (df), ν₂ = (N − r) denominator df, and noncentrality parameter

We use the notation SSH(·,N) to emphasize that SSH is a function of sample size by way of the matrix X. The cumulative distribution function (CDF) of a noncentral F with ν₁ and ν₂ df and noncentrality parameter ω will be represented by F_F[· | ν₁, ν₂, ω], with corresponding density function f_F[· | ν₁, ν₂, ω]. Using this notation, the power of the test can be written

in which f_crit(l − α_t) is the 100 · (1 − α_t) percentile from a central F with ν₁ and ν₂ df. The CDF and density of a non-central F may be expressed as Poisson-weighted infinite sums (Johnson and Kotz 1970, p. 192). In practice, CDF and density values may be precisely calculated with computer algorithms (Abramowitz and Stegun 1964, p. 947; Odeh and Fox 1991, chap. 5).

Some authors, notably Cohen (1987, p. 8), have discussed “effect size” as a generic concept encompassing disparate analyses (including tests of means, proportions, and categorical data). For the special cases of the GLUM considered by Cohen, effect size and noncentrality can be expressed as one-to-one functions of each other. Although distinct cases of the GLUM yield distinct formulas for effect size, they yield only a single formula for noncentrality. Hence we restrict our attention to noncentrality in order to maximize generality and avoid ambiguity.

2.1 Confidence Bounds for Noncentrality

For all calculations, a distinction must be carefully maintained between the sample size of the estimation study (which provides the variance estimate) and the sample size of the target study (for which power is desired). For example, the value of f_crit(·) is always calculated using the target study error df. In a retrospective analysis, the two sample sizes are generally the same. This may not be true, however, in a prospective analysis. To help facilitate the necessary distinction, we use the notation ν_2e and ν_2t to represent the error df associated with the estimation and target studies, respectively. As a further convenience, N_t represents the sample size of the target study.

For fixed means, confidence bounds for noncentrality are easily formulated using known distributional properties of the GLUM. Bounds for power and sample size follow directly from these results. Clark (1987) demonstrated that the approximate method suggested by Dudewicz (1972) yields optimistically small intervals in small samples, with bias decreasing as ν_2e increases. Exact bounds for ω can be calculated as follows. Assign lower (α_cL) and upper (α_cU) tail probabilities to define the confidence coefficient (1 − α_cL − α_cU), and let c_crit(p | ν) be the 100 · (p) percentile from a central χ² with ν df. From standard Gaussian theory, recognize that ${\nu }_{2e}\cdot {\widehat{\sigma }}^2/{\sigma }^2\stackrel{d}{=}{\chi }^2({\nu }_{2e}).$ Then observe that

Multiplying all terms in the inequality by SSH(θ, N_t)/SSE yields

with SSH(θ, N_t) fixed under the assumption of known (nonzero) θ and sample size N_t. Consequently,

and

provide exact confidence bounds for ω. Choosing α_cL = 0 yields a lower one-sided interval, [0, ω̂_U], while choosing α_cU = 0 yields an upper one-sided interval, [ω̂_L, ∞).

By similar argument, one can derive a minimum length interval for ω based on the shortest unbiased confidence interval for variance (Tate and Klett 1959; Juola 1993). However, since ω ∊ ℜ⁺ while power lies in the bounded interval [α_t, 1), minimum length intervals for noncentrality may not lead to minimum length intervals for power.

2.2 Confidence Bounds for Power

The strict monotone dependence of the noncentral F distribution function on noncentrality (Johnson and Kotz 1970, p. 193) ensures that an exact confidence interval for power follows from an exact interval for ω. The lower (P̂_L) and upper (P̂_U) bounds for power must satisfy the two equations

and

The fact that power is a smooth, strictly increasing function of ω ensures that

Alternately, consider using a distinct, and therefore approximate method to produce, say, ω̃_L and ω̃_U. Then an approximate confidence interval for P results by observing

2.3 Confidence Bounds for Sample Size

An upper confidence bound for sample size (N_U) provides the smallest N that ensures, with confidence (1 − α_cL), a desired target power. To achieve the result, one must solve the equation

for N_U such that

Here ν_NU is the error df for the target study, and is a function of N_U. The value of N_U is most easily obtained by iterative solution of Equations (2.9) and (2.10). The strict monotonicity of the power function makes this task straightforward. In practice, the use of discrete N introduces some inexactness in coverage.

2.4 Simultaneous Bounds for Power and SampleSize

A confidence region for an entire power curve may be desired. For example, consider power as a function of the difference between two means, δ. One may seek a confidence region for power over an interval of δ > 0. For a particular value of δ, say δ₀,

and

provide exact confidence bounds. Considering a continuous set of δ₀ generates two curves, which in turn define a confidence region. The boundaries are defined by the locus of all points generated by computing (2.11) and (2.12) for all δ₀ > 0 in the set. The strict monotonicity of the function ensures that

Hence the pointwise bounds provide simultaneous confidence intervals. See Figure 1, discussed in Section 4, for an example.

Similar logic applies to sample size and other power related functions. Consider the triplet (P, δ, N). Fixing any one of the three, and computing a second as a function of the third, leads to a potentially informative plot. Given the results of Sections 2.1–2.3, these applications lead to exact coverage (ignoring the use of discrete N).

3.1 Simulation Methods

In order to assess both the Dudewicz/Clark and exact methods, simulations were conducted using SAS IML^©. Bounds for the power of a one-way ANOVA were calculated, and results were tabulated for each method under the following conditions: significance level of the test (α_t ∊ {.01, .05}), target power (P ∊ {.2, .5, .9}), number of groups (G ∊ {2, 4}), cell size (N_G ∊ {4, 8,16}), and target coverage (C ∊ {.80, .95}, with α_cL = α_cU). Without loss of generality, σ² = 1 for all simulations. A vector of cell means was chosen as β = γ · [0 1… G − 1]′, in which γ was calculated by iterative use of a power program (Muller, LaVange, Ramey, and Ramey 1992, app. A), to ensure the desired target power for each combination of α_t, P, G, N_G, and C. A total of 1,500 random samples of Gaussian data were generated for each combination and method. The statistic σ̂² was calculated, as were lower and upper bounds for power. The proportion of events for which confidence bounds captured true power was determined for consecutive sets of 150 replications. These were used to test the significance of α_t, P, G, and N_G and their interactions as predictors of coverage using factorial ANOVA's with 35 model and 324 error degrees of freedom. The 150 cases for each binomial observation, the balanced design, and the small predicted differences in population proportions led us to expect accurate inference with the use of ANOVA's.

3.2 Simulation Results

Simulation results are summarized in Table 1. For the Dudewicz/Clark method, the main effects of G and N_G were significant at the .01 level (p ≤ .001). Results agreed with Clark's proof that the method provides asymptotically unbiased confidence intervals for power. Small error degrees of freedom, however, yielded rather liberal results. For example, with C = .95 and 6 error df, the observed coverage was only .872 (std err = .0012).

With the exact method, the α_t× P × G × N_G interaction term for C = .80 was significant at the .01 level (p = .007). We discounted this lone small p value in light of the small amount of variance associated with the effect and the multiplicity of tests conducted. Empirical results did not deviate substantially from predicted values, and accurate coverage was obtained for all conditions.