VALIDATION-STUDY CONCLUSIONS FROM DIETARY REPORTS BY FOURTH-GRADE CHILDREN OBSERVED EATING SCHOOL MEALS ARE GENERALIZABLE TO DIETARY REPORTS BY COMPARABLE CHILDREN NOT OBSERVED

Participants

The participants were 120 fourth-grade children recruited from 6 of 33 elementary schools in one school district. These schools were selected to obtain a sample of children with high participation in school meals; at these schools, 70% (from 58% to 82%) of children were eligible to receive free or reduced-price school meals during the school year of data collection. During fall, 2002, all fourth-grade children from these schools were invited to participate. The invitation explained that participating children might be watched while eating breakfast and lunch at school several times, and might be interviewed about what they eat; that not all children who agreed to participate would be interviewed; and that a $15 check would be mailed for each interview. Of the 443 children (29% black boys, 30% black girls, 20% white boys, 16% white girls, 3% other boys, 2% other girls) invited to participate, 312 (70%) agreed by returning signed child assent and parental consent forms. (Agreement rates did not depend on race or sex.²) From these 312, the 120 participants (39 black boys, 45 black girls, 18 white boys, 13 white girls, 3 other boys, 2 other girls) were randomly selected, subject to the constraints that half be girls and that all be usual eaters of school breakfast and school lunch. A child satisfied the latter criterion by eating both school meals on at least four of five practice observation days.

Design and Procedure

Within sex, children were assigned randomly to be observed or not observed, and then to one of the six interview conditions yielded by crossing two target periods (previous day; prior 24 hours) with three interview times (morning; afternoon; evening). Thus, 10 children (5 girls) participated in each of the 12 observation status × target period × interview time conditions. The distribution of children by race was essentially identical in the two observation-status conditions (observed: 42 black, 16 white, 2 other; not observed: 42 black, 15 white, 3 other).

For the previous-day target period, children were to report about midnight to midnight of the day before the interview, beginning with when they got up (e.g., for an interview on Tuesday at 6:30 p.m., from midnight Sunday to midnight Monday). For the prior-24-hours target period, children were to report about the period from 24 hours before the interview until the time the interview began (e.g., for an interview on Tuesday at 6:30 p.m., from 6:30 p.m. on Monday to 6:30 p.m. on Tuesday) (see Ref. 2). Available evidence indicates that fourth-grade children’s dietary reporting accuracy does not depend significantly on whether interviews are conducted in person or by telephone,¹⁵ so morning and afternoon interviews were conducted in person at school after breakfast and lunch, respectively, and evening interviews were conducted by telephone after 6:30 p.m.

Interviewers were blind to the observation status of the children they interviewed: Observations and interviews were conducted by different individuals; interviewers did not know at which schools observations were conducted for specific meals.

We have published analyses of the effects of target period and interview time on the reporting accuracy of the 60 observed children.²

Observations

For each observation, one of two dietitians stood, for an entire meal period, by the table(s) at which classes or groups of children regularly sat, and appeared to observe the entire class or group although she recorded information about only one or two children; during such observation periods, all fourth-grade children at that school who had agreed to participate in the study wore nametags provided by the research staff. Children knew when an observer was present, but did not know who, specifically, was being observed or who, if anyone, would be interviewed. To facilitate identifying what children ate, only children who obtained meals from the school foodservice were observed.³⁵ To familiarize children with the presence of an observer and to identify usual school-meal eaters, practice observations were conducted with each class prior to data collection. Interobserver reliability, assessed weekly using established procedures,^2,13–15,36 was acceptable; observers agreed on the identities of all items and on amounts within one-fourth serving for 90% of the items.

As children were randomly assigned to be observed or not observed, observations were conducted in all six schools. However, no observations were conducted in the schools of children who were interviewed without having been observed during meal periods that occurred during the 24-hour periods about which these children were interviewed.

Interviews

Each interview was conducted by one of two dietitians using a multiple-pass protocol modeled on that of the Nutrition Data System for Research (NDS-R; version 4.05_33; Nutrition Coordinating Center, University of Minnesota, Minneapolis). The NDS-R target period is the previous day. For interviews about the prior 24 hours, the NDS-R protocol was modified so that a child was to report intake for the interview day, from when he or she awoke until the time of the interview, followed by intake for the preceding day, from 24 hours before the interview until the time he or she went to bed. If a child reported intake prior to the beginning of the target period, the interviewer asked about the next time he or she ate or drank until report was for the target period. For a fuller description of the interview protocols, see Ref. 2.

Interviewers both recorded children’s reports on paper forms and made audio-recordings. Interview quality, evaluated daily during pre-data-collection practice and during data collection on randomly-selected interviews, was acceptable.^2,13–15,37

Response Variables

We evaluated five variables—interview length, number of meals and snacks reported for the target period, and, for the two school meals, number of meal components reported, importance-weighted number of items reported, and kilocalories reported.

For a reported meal to be counted as a school meal, a child had to indicate that the meal was eaten at school, report the time to within an hour of the observed time, and name the meal appropriately.^2,12–15 Every reported school-meal item was classified as one of 10 meal components—lunch entrée, combination entrée, bread/cereal, vegetable, fruit, dessert, condiment, beverage, breakfast meat, and miscellaneous. The importance weight assigned to combination entrées [e.g., hamburger] was 2; to condiments was 1/3; and to other meal components was 1. For example, a child who reported eating non-zero amounts of Cheerios, white milk, and orange juice at school breakfast and a hamburger, French fries, mayonnaise, corn, chocolate milk, and vanilla pudding at school lunch reported 6 unique meal components [bread/cereal, beverage, combination entrée, condiment, vegetable, dessert] and 9.33 importance-weighted items. We have used these methods of classifying and importance-weighting in other studies.^2,12–15

For each child, for each reported school-meal item, reported amount and kilocalories/serving information were used to compute reported kilocalories; these were summed across items reported for the two school meals. Children reported amounts in servings as none (0), taste (0.1), little bit (0.25), half (0.5), most (0.75), all (1.0), or the number of servings for more than one serving. We have quantified servings this way in other studies.^2,12–15 Kilocalorie information for standard school servings of school-meal items was obtained from the NDS-R database or from the school district’s nutrition program.

Analyses

Anticipating appreciable correlations between response variables, we planned to conduct principal component analysis of these variables; to retain a subset of the principal components, rotated orthogonally; and to conduct analysis of variance of scores on retained principal components. Principal component analysis transforms intercorrelated response variables into mutually uncorrelated principal components, each of which is a linear combination of the original variables. The first principal component accounts for maximum variability in the response variables; each successive principal component captures a maximum amount of remaining variability. Although five principal components are required to account for all variability in five response variables, a smaller number might capture a substantial fraction of the variability and be retained. To enhance interpretability, retained principal components are rotated orthogonally to maximize the correlations of some original variables with each component while minimizing others. For each participant, a score on each retained principal component may be calculated; this is a linear combination of that participant’s values on the original variables.^38,39

Statistical analyses were conducted with SAS (version 9.1; SAS Institute, Cary, NC).

Limitations

One limitation of this study is that these results only suggest, but do not guarantee, that observation of school meals does not affect children’s dietary reporting performance. With 60 children per observation-status condition and a two-tailed type I error rate of .10, the power of this experiment was .85 to detect a moderate effect (0.5 SD) of observation status in a conventional test of the null hypothesis of no effect. However, to have power of .80 to detect a small effect (0.2 SD) with a two-tailed type I error rate of .10 would require 310 children per observation-status condition.^41,42

On Component 1, the composite variable on which we observed the larger difference between observation-status conditions, the condition means differed by 0.0932 SD, which, as half of a conventionally-defined small effect,⁴² is likely negligible. However, to assert that the means of the two observation-status conditions were statistically equivalent, using as the equivalence criterion an interval of 0.19 SD (which is smaller than a small effect), would require finding this difference in an experiment with 831children per observation-status condition.³¹ This sample-size requirement is sobering.

The combination of no significant effect of observation status and the significant effect of target period encourages, but does not guarantee, the conclusion that school meal observation does not affect fourth-grade children’s dietary reporting performance. An open issue is how large an effect of observation status would have to be to undermine claims about the generalizability of validation-study conclusions to individuals not observed. This deserves serious attention from the community of investigators interested in validation-study methodology.^30,31

A second limitation is that we have at best established that there is not a specific effect of observation of particular school meals on children’s dietary reports. Our data do not exclude the possibility of some general effect. We compared responses of children observed at school meals during the period about which they would report to responses of children not observed at the school meals (and during which no observers were present) during the period about which they would report. However, during the study, the latter children ate other school meals during which observers were present and so during which the children followed the study protocol (e.g., wore nametags). To exclude a general effect of school meal observation would require collecting data from children who had no exposure to observers. This would likely be impossible, given that elementary-school children eating school meals are generally monitored by adults.^15,43

A third limitation is that the participants were fourth-grade children who tended to be from economically disadvantaged families in one school district. It would be valuable to broaden the investigation of the effect of school meal observation on dietary reporting performance to children of other ages and other socio-economic levels, and to assess relationships with children’s cognitive ability and scholastic achievement.

Other Relevant Research

There appears to be little published research on effects of school meal observation on children’s dietary reporting performance. However, Baranowski et al.,⁷ evaluating a software system for eliciting dietary reports from fourth-grade children, recognized that collecting reference information by observing school lunch might influence children’s reports. From each participating child, they obtained two 24hDRs—one using the self-operated software and one in a dietitian-administered interview—and evaluated the correspondence of these 24hDRs. Correspondence measures were based on matches (items reported in the software 24hDR and to the dietitian), intrusions (items reported in the software 24hDR but not to the dietitian), and omissions (items reported to the dietitian but not in the software 24hDR). To assess the effect of observation, Baranowski et al. compared correspondence between 24hDRs of children observed at school lunch on the previous day to the correspondence between 24hDRs of children who had not been observed. These investigators found that observed and unobserved children did not differ on the match and omission measures, but that observed children averaged significantly lower than unobserved children on the intrusion measure. The conclusion of Baranowski et al., that “observation of school lunch somehow intensifies the experience of lunch, which impacts accuracy of report by minimizing report of foods not eaten.…” (p. 384), differs from ours. However, in the study of Baranowski et al., observation status was partially confounded with a “bogus-pipeline” manipulation intended to enhance reporting accuracy. Data in their article permitted assessment of the unconfounded effect of observation status; analysis of these showed that the difference in the intrusion measure for observed and unobserved children, although in the direction discussed by Baranowski et al., was not statistically significant.

Final Remark

A significant research challenge for nutritional epidemiology is identifying methods that optimize the validity of dietary reports by respondents who do not anticipate reporting and who report without memory aids. This requires a research phase in which validity is assessed, and a purpose of this phase should be to estimate the response validity of individuals for whom reference information will not be available. In other words, the response validity of validation-study participants should be an estimate of the response validity of respondents in general. We have described an approach to assessing whether collecting reference information by observing school meals in a validation study with fourth-grade children influences their dietary reports. Our results suggest that it does not.

Absent a complete theory of the impact of measurement procedures on behavior, assessing the impact of measurement in validation studies is essential to generalizing the results of validation studies to research participants for whom reference information has not been collected. In addition, a consensus on the magnitude of an effect of collecting reference information that can be treated as trivial or inconsequential must be developed. These should be continuing concerns of researchers interested in the usefulness of methods used to collect dietary information.