Validity and Reliability of Single-item Questions about Physical Activity

INTRODUCTION

The health benefits of habitual physical activity have been reported by many epidemiological studies using physical activity questionnaires¹⁾. Currently, the questionnaire is the most widely used measurement method for assessing physical activity in study populations, and various questionnaires have been developed²⁾. Many of them are complex, with several questions from which to estimate a subject’s energy expenditure from activity; the validity and reliability of these have been investigated^3-10). However, in practical use, single-item questions are often included in epidemiological and public health questionnaires designed to measure health status and exposure to risk factors¹¹⁾. Measuring physical activity with a single-item question has the advantage of requiring less time and money to obtain the data. The simplicity is especially needed in a large-scale epidemiological study that uses a lengthy questionnaire to collect data on many factors, including physical activity. There are only a few reports on the validity and reliability of single-item questions about physical activity^{12, 13)}. The Japan Collaborative Cohort Study for Evaluation of Cancer Risk Sponsored by Monbusho (JACC Study), a multicenter study in Japan begun in 1986¹⁴⁾, included some single-item questions concerning physical activity in its self-administered questionnaires given at baseline and interim surveys. However, these questions seemed to have limitations, so we investigated their validity and reliability among many Japanese adults.

METHODS

RESULTS

Table 3 shows the distributions of responses from the first self-administered questionnaire by sex and age group. The distributions were biased toward inactivity in the sports and physical exercise questions and biased toward activity in the walking question.

Table 4 shows the Spearman’s rank correlation coefficients from correlating the rank scores of the single-item questions with the indices derived from the interview. The validity coefficients ranged from 0.43 to 0.60, showing moderate correlations. The coefficients of Question 1 were not calculated among women 22-39 years old because the sample size was so small that no sample was present in the most active category of Question 1. Men 40-59 years old had a slightly higher correlation coefficient than men 60-79 years old; a similar relationship was not observed in the women’s data.

Table 5 shows quartiles of time spent on sports and physical exercise, derived from the interview, for each of the response categories of Questions 1 and 3. The level of activity time derived from the interview increased in the order of the response categories of questions for both men and women. Of the subjects who reported the least activity on the response to Question 1, 5% of the men and 3% of the women had average activity times of at least 5 hours per week according to the interview. These subjects accounted for 3.3% of all men and 2.2% of all women in the study. Conversely, of the subjects who reported the most activity on the response to Question 1, 20% of the men and 21% of the women had average activity times of less than 1 hour per week. These subjects accounted for 1.9% of all men and 1.6% of all women in the study. Thus, these extreme misclassifications accounted for almost 20% of the most active category, but only accounted for a very small percentage of the total group of respondents.

The weighted kappa coefficients calculated from data obtained from the first and second surveys are shown in Table 6. The coefficients of Questions 1 and 3, classified according to sex and age, ranged from 0.39 to 0.56, showing moderate reliability; those of Question 2 ranged from 0.25 to 0.39, showing fair reliability. Thus, the reliability of the walking question was weaker than that of the sports and physical exercise questions.

The reliability coefficients did not vary substantially when they were calculated for subjects who were surveyed at an interval very close to 12 months with both surveys conducted in the same season. However, for those who were surveyed at an interval of 12.5 to 15.9 months, there was some variability, with the coefficient of Question 2 lower for men and the coefficient of Question 3 lower for women.

DISCUSSION

The validity of a survey instrument is commonly defined as the extent to which it measures what it is intended to measure²¹⁾. The phenomenon that a survey is intended to measure must be clearly defined for the subjects in a self-administered questionnaire survey. When the phenomenon being measured is sports and physical exercise (“undou”), the definition must include the purpose, type, and intensity of the activity as well as the reference period of interest^{23, 24)}. The questions on the JACC questionnaire were not accompanied by such information, which might have led to misunderstandings. Furthermore, although we intended to focus on activity over a long reference period, such as 1 year, the subject’s response might have been biased by the subjects recalling their most recent activities and attributing that frequency to the entire year.

Once the definition of “undou” (physical exercise) was determined, we compared responses to questions on the JACC questionnaire with the criterion measure derived from the interviewing method used in the Japan Lifestyle Monitoring Study. In this interview, after asking a question very similar to Question 1, the interviewer added several questions to discern which activities to exclude because they were outside of the stated definition and which activities to include because they were otherwise being overlooked. The validity correlation coefficients of the responses to Questions 1 and 3 with activity time or LTPA score derived from the interview ranged from 0.43 to 0.60. These coefficients were better than those (r = 0.25-0.31) of responses to a single-item question about usual physical activity correlated with an energy expenditure index obtained from another detailed question set, reported by Weiss et al¹²⁾. Our correlation results suggest moderate validity of Questions 1 and 3 relative to indices which reflect the act of engaging in sports and physical exercise over a year, easing concerns that those questions might have caused misunderstanding.

The correlations of Question 1 or 3 with the LTPA score were almost the same as the correlations with activity time derived from the interview. The reason for this is that the LTPA score, an index of energy expenditure, showed an extremely high correlation with activity time (Spearman’s r = 0.96 when activity time exceeded 0).

The criterion validity is generally estimated using the criterion measure currently assumed to be the gold standard. Whether the criterion measure derived from the interviewing method in the present study should be regarded as the gold standard is arguable. However, the criterion measure need not be the gold standard, but it does need to be an established and presumably better measure than the ones selected for comparison¹²⁾. We assumed that the interviewing method is more accurate than the self-administered, single-item questions, and also that it has the advantages of not altering individual behavior, not being intrusive, and being practical in a large-scale validation study, unlike other methods²⁵⁾. However, there is the disadvantage of introducing the same source of error, memory, in both the self-administered questionnaire and interview²⁶⁾.Therefore, the errors of both methods might remain correlated, and the validity we assessed in the present study might be higher than the validity assessed by criterion measures in which data are objectively recorded or sensor-monitored. In spite of this possible study limitation, we determined the validity of measuring physical activity using simple, single-item questions, relative to measuring physical activity using a detailed interview requiring more time and effort from both the participants and the interviewers.

We did not conduct a validation study of Question 2, because there was no established interviewing method that could be regarded as a criterion measure (gold standard); i.e., a method believed to be more accurate than Question 2 and practical for measuring walking time averaged over a relatively long reference period. However, it would be worthwhile to assess the validity of a single-item question about walking by comparing responses to a self-administered questionnaire with results assessed by criterion measures in which data are objectively measured, such as data obtained with a pedometer.

The reliability coefficients of Questions 1, 2, and 3 at nearly a 1-year interval, classified according to sex and age, ranged from 0.25 to 0.56. Reliability is the extent to which repeated measurements of a stable phenomenon get similar results²⁷⁾, but physical activity is not necessarily a stable phenomenon, particularly over a long time. The reliability assessed in the present study reflects both variation in response to the questions and true changes in physical activity over 1 year. The results of our study suggested that the long-term stability of physical activity assessed by these questions, including the performance of measurements, is fair to moderate.

Test-retest reliability is generally believed to become lower as the interval between tests becomes longer. However, Tsubono et al. reported that reliability of a self-administered food frequency questionnaire tended to be lower when the questionnaires were administered in different seasons at a 5-month interval than when they were administered in the same season at a 1-year interval, in their multiple-interval reliability study²⁸⁾. Given these results, it seems reasonable to speculate that seasonal variation also affects reliability of physical activity surveys. In the present study, some reliability coefficients for subjects who were surveyed at intervals of 12.5 to 15.9 months were apparently lower than those for subjects who were surveyed at intervals very close to 12 months. These results may indicate the effects of seasonal variations. However, these pairs of coefficients, obtained from repeated surveys conducted in different seasons and the same season, were measured for different subgroups, and the difference between the two coefficients did not necessarily reflect within-person variations. Further investigations will be required to clarify the effects of seasonal variations on the reliability of these questions.

In the present study population, the responses to Questions 1, 2, and 3 showed biased distributions. We thought the number of subjects in the least frequent response category was large enough to overcome effects of chance and provide appropriate evidence of validity and reliability, such as a significant level of criterion measure in each response category and meaningful coefficients. The sample size of the present study was sufficiently large for 40- to 79-year-old people.

Most of the subjects in the present study were selected from JACC baseline study fields, and the biased distributions of their responses to Questions 1 and 2 were similar to those found among almost 80,000 people 40-79 years old in the JACC baseline survey²⁹⁾. The validity and reliability observed in the present study is considered generalizable to those people who participated in the JACC baseline survey.

In summary, the validity of Questions 1 and 3 was moderate, and the reliability of Questions 1, 2 and 3 at a 1-year interval was fair to moderate. We suggest that measuring physical activity level with these single-item questions may be appropriate for establishing baseline data that reflects long-term physical activity in a large-scale cohort study targeting lifestyle-related diseases.