Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2026 Sep 26.
Published in final edited form as: Neurotoxicol Teratol. 2025 Sep 26;112:107562. doi: 10.1016/j.ntt.2025.107562

Assessing Replicability and Power Estimates of Behavioral Performance of Control Rats Across Standardized Pre-clinical and Toxicology Studies

Kathryn S Konrad 1, Laura Betz 1, Sandra McBride 1, Keith R Shockley 2, Georgia Roberts 3, Helen Cunny 3, G Jean Harry 4,*
PMCID: PMC12588254  NIHMSID: NIHMS2115940  PMID: 41016607

Abstract

Behavioral assays are critical in evaluating impacts on nervous system function in rodents due to genetic or environmental factors and are frequently incorporated into regulatory decision-making studies. Despite numerous sources of guidance for such studies, results across behavioral assays are reputed to be highly variable with questionable replicability. Behavioral data obtained from control rats within four contract laboratory studies were used to evaluate replicability across studies, calculate the level of statistical power, and estimate the number of animals required for a specific effect size. For the three behaviors evaluated here (motor activity, acoustic startle response, and learning and memory), control rats from all studies showed the expected pattern of behavior, e.g., open field acclimation, startle habituation, % prepulse inhibition (PPI) over pre-pulse intensities, and acquisition and goal quadrant preference in the Morris Water Maze (MWM). For selected representative individual endpoints, power analyses were conducted to evaluate sample size requirements. Across all endpoints, a drop in power occurred as differences between two groups became smaller. Power analysis of multiple representative endpoints suggested that a sample size of 20 may detect a 30% effect with 80% power. Sample size requirements changed with the effect size, and achieving 80% power with a 20% effect size generally required a sample size of 30 rats. While the behavioral performance was replicated over the Study Cohorts, power analyses suggested a need for moderation of expectations regarding detectable differences if decisions relied on single endpoints or small effect sizes. Reporting results from a low powered study can have significant and wide-ranging impacts, including undermining confidence in data interpretation, misleading future research, and failing to adhere to the ethical framework of the 3 R’s.

Keywords: Startle, Motor Activity, Morris Water Maze, Replicability, Statistical Power, Sample Size

1. Introduction

Behavioral assessments of apical endpoints such as motor, sensory, and learning/memory represent integrated functioning of the nervous system and have high value as a surrogate index in clinical settings and across species in experimental studies. They serve as cornerstones for detection of altered nervous system function and contribute to the identification of beneficial or harmful therapeutic interventions. As such, behavioral assessments in experimental animals provide critical information on adverse and beneficial effects of various experimental manipulations. However, the integrated nature of apical responses arising from individual biological variability is also vulnerable to external influences. While assessments of the human population are unable to control for the broad array of external influences on observational measurements, experimental animal studies can be designed to minimize or standardize many factors. Phenotypic variation is a common factor in all levels of organization from molecular mechanisms to behavior. A robust experimental design offers a step towards precision in quantifying effects by accounting for environmental factors and testing paradigms (such as temperature, lighting, equipment, bedding, chow, and time of day) while considering individual biological factors (such as age, sex, weight, hunger level, general health, and stress level) that contribute to response variability (Gulinello et al., 2019; Kafkafi et al., 2018; Saré et al., 2021; Lewejohann et al., 2006). The goal of such efforts is not to disregard biological variation but rather to address external factors that may serve as experimental confounders and to balance experimental tractability and underlying biological complexity (Voelkl et al., 2020).

In experimental animal research, the effect of some form of manipulation is normally measured at the level of phenotype, allowing for the plasticity of the response. Because phenotypes are complex, the direction and magnitude of any such effects depend not only on the experimental manipulation but also on the current phenotype and the experimental context and external factors (Chesler et al., 2002). The impact of any experimental manipulation can vary based on the internal state of the animal as well as the environment, factors which can then additively or synergistically interact to shape the outcome. The ability to detect a relevant shift in the phenotype with manipulation requires knowledge of the baseline performance. The direction and magnitude of an effect depend on the nature, duration, and intensity of the manipulation, as well as the animal’s current phenotype and the experimental context. The magnitude of the effect size, or the anticipated percentage change between an experimental group mean and its control group mean, can inform the sample size and thus increase confidence in the data and interpretation.

A robust study design balances the use of a sufficient number of animals or samples with an adequately powered, biologically important, and defensible interpretation (Cao et al., 2024; Faber and Fonseca, 2014; Peterson and Foley, 2021; Serdar et al., 2020). The prevalence and impact of studies with too few samples to detect statistically significant differences in the experimental question have become increasingly evident (Carnahan and Brown, 2024; Jennions and Møller, 2003; Loken and Gelman, 2017). Such studies offer limited value to the field of study other than as preliminary, pilot, or hypothesis-generating datasets. Unless designed as such, they likely do not adhere to the principles of the National Centre for the Replacement, Refinement, and Reduction of Animals in Research (3 Rs), compromise efforts to replicate and thus, have confidence in findings and potentially misguide future research and regulatory decisions. Knowing the smallest effect size that a researcher could reliably detect using any given assay facilitates the balancing of sample size with power.

In addition to variability within an endpoint and study, variability across studies and laboratories contributes to the perceived validity and reliability of any findings. Replicability, or the ability to repeat an experiment in the same or similar conditions to achieve the same or similar results, is critical for research (Loken and Gelman, 2017; National Academies of Sciences et al., 2019). The inability to replicate findings across studies may be reduced by using and reporting comparable experimental paradigms and sufficient details on the apparatus, paradigm, and protocols. Various recommendations for inclusion of certain quality measures to address the experimental design and reporting guidelines such as the TOP (Transparency and Openness Promotion), ARRIVE, or PREPARE guidelines have been developed to overcome inadequate reporting of this information (Michel et al., 2020; Nosek et al., 2015; Sert et al., 2020; Smith et al., 2018). Additional consideration for replication of the underlying biological phenotype by incorporating varying factors such as genotype, sex, age, environmental conditions, and housing has been proposed. This approach is often referred to as “systematic heterogenization,” and incorporates biological variation in study designs to provide stronger evidence for reliability and translation (Voelkl et al., 2020).

Statistical power analysis estimates the requisite sample size necessary to answer a specified experimental question, given known experimental input based on certain assumptions and available prior information about endpoint variability. The power to detect a statistically significant difference in any specific endpoint and thus, reject the null hypothesis, is intrinsically linked to several factors: the chosen statistical test, magnitude of the effect size, sample size, estimated mean and variance of the endpoint, and the statistical significance threshold (Cohen, 2013; Jeffers et al., 2024; Serdar et al., 2020). Larger effect sizes enhance power for a given level of background variability, while subtle or moderate effects may be more difficult to detect. Increasing the sample size tends to increase power by reducing the standard error and increasing precision of parameter estimates such as the mean. Higher variance or absolute variability can obscure detection of an effect, requiring a larger sample size to achieve the same power. Increases in the relative variability across studies as measured by the coefficient of variation (CV; 100 times the ratio of the standard deviation to the mean) can reduce power. An adequately powered study maximizes the chance of detecting true effects by minimizing false negatives (Type II errors) while maintaining an acceptable level of false positives (Type I errors) through appropriate statistical significance thresholds (Serdar et al., 2020). Typically, a study is considered sufficiently powered when the statistical models and tests have at least 80% power on the study’s endpoints, effectively balancing the risks of Type I and Type II errors (Serdar et al., 2020). A robust experimental design and statistical power are interdependent in that a well-planned design reduces variability in results and enhances precision of parameter estimates, increasing the statistical power to detect true effects and facilitating replication of findings across studies.

Several points of consideration regarding variability and replication of studies utilizing apical behavioral endpoints in rodents have been raised in the literature as they apply to basic science, pre-clinical, or toxicological studies (Ankeny et al., 2023; Gulinello et al., 2019; Kafkafi et al., 2018; Vorhees and Makris, 2015). Many of these arguments simultaneously examine the contribution of multiple features of the experimental design, including individual variability, the testing paradigm, the conduct of the assay, and the simple screening approach to complex biological systems as well as research expectations placed on any screening approach to detect more moderate to subtle differences, over and above innate biological variability. To explore these questions and their implications for statistical power, the current investigation used datasets obtained from studies following recommended regulatory guidelines for behavioral assessments. Data of male and female rats within the control groups of four independent behavioral studies from contract laboratory facilities that followed specific husbandry, and experimental procedures were used to assess cross-study replication of multiple representative behavioral endpoints in control rats. Data obtained from motor, sensory, and learning/memory tasks, as characterized by ambulatory and rearing activity within an open field arena (OF), acoustic startle response (ASR) and pre-pulse startle inhibition (PPI), and the Morris Water Maze (MWM) test paradigms were assessed. With the appreciation of the complex components of each of these behavioral assessments and the expanded parameters that can be obtained using automated commercial data capture systems, the conducted power analyses focused on specific representative endpoints commonly reported in the literature. These were used to represent and compare the variability across rats, endpoints, assays, studies, and laboratories and to demonstrate the impact of environmental conditions. The findings provided sample size calculations across a variety of effect sizes and sample sizes. In general, the data suggested that studies looking for an effect size of 30% would be appropriately powered within 20 to 30 rats; however, this was found to be influenced by aspects of handling and housing conditions. The comparisons demonstrate the value of considering the patterns of behavior and the need to restrain from placing unrealistic expectations of effect size when designing or evaluating studies.

2. Materials and Methods

2.1. Animals

Four individual neurobehavioral studies of male and female rats (Hsd:Sprague Dawley, Envigo, Haslett, MI) were conducted during a 5-year window across two contract laboratory facilities. The studies selected were all conducted according to general guidelines as established by the Division of Translational Toxicology in the National Toxicology Program for animal husbandry, handling, testing paradigms, data collection, etc. (Roberts and Stout, 2023) using identical equipment and testing paradigms. Rats were housed in dual-corridor barrier facilities maintained at 70–78°F and relative humidity of 45–67% under a 12-hr light/dark cycle (6:00–18:00 hr). Food (irradiated NIH-07 chow (Study Cohorts 2,3,4) or meal (Study Cohort 1), Zeigler Bros, Inc., Gardners, PA) and municipal tap-water (quality tested for contaminants) or deionized water (Study Cohort 2) were available ad libitum. All animal procedures were conducted in compliance with the NRC Guide for Animal Care and Use of Laboratory Animals [https://policymanual.nih.gov/3040-2] under Institutional Animal Care and Use Committee approved protocols. Sentinel rats were health-monitored over the study duration.

For Study Cohorts 1, 2, and 3, time-mated female rats were received within the first two days post-implantation. Dams (n=20–22) were randomly assigned (using computer assisted random number generator) to each control group. Rats were housed in solid bottom polycarbonate cages with irradiated Sani-Chips hardwood bedding (P.J. Murphy Forest Products Corp., Montville, NJ). Nesting material was provided for dams. Day of birth was designated as postnatal day (PND)0. On PND 3–4, litters were standardized to 10 pups (5/sex/litter) and pups individually identified by tail tattoo. At weaning, PND28, 1 male and 1 female pup from each litter was randomly assigned for subsequent behavioral assessments. Male offspring were housed 2 per cage and females at 4 per cage to maintain housing cohorts throughout the testing period. In Study Cohort 3, dams received vehicle aqueous (0.2% methylcellulose /0.1% Tween 80) via gavage (5ml/kg vol) from GD6 until pup weaning at PND28. Study Cohort 4’s environmental and handling conditions were notably different from the others, but within the inhalation chambers, they were only exposed to ambient air. Study Cohort 4 litter sizes were standardized to 4 males and 4 females at PND4. Pups were identified with a limb tattoo on PND0 and tail tattoo at weaning (PND21). Dams (GD4 – GD20) and post-weaning pups (PND24 – PND35) were transferred daily from the solid polycarbonate home-cage cages to wire mesh R24 cages and inhalation chambers for 6 hr ambient air exposure and then returned to the home-cage. Upon reaching PND36, pups remained singly housed in wire mesh R24 cage and underwent the air control inhalation exposure daily for 6 hr until end of study. Lab affiliations, sample sizes, experimental handling factors, and PND testing ages for open field (OF), acoustic startle (ASR)/pre-pulse inhibition (PPI), and MWM are summarized in Table 1.

Table 1.

Overview of Behavioral Study Cohorts and Experimental Conditions.

Study Cohort 1 2 3 4
Lab X X Y X
Number/sex 20 20 (19 for juveniles in ASR/PPI due to outliers) 22 20
Experimental
Handling		 Dams: None
Pups: None		 Dams: None
Pups: None		 Dams: (GD6-PND28)
Oral Gavage - vehicle
Pups: None		 Dams: GD4-GD20
Pups: PND35–105
Inhalation-6hr daily
Home Cage polycarbonate
group
housing		 polycarbonate group housing polycarbonate group housing Dam (litters)
Pups: PND 24–35 polycarbonate
PND 35–103
R24 wire mesh (single)
OF PND 29–32
PND 59–62
(red-light)		 PND 29–32
PND 59–62
(red-light)		 PND 90
(dim house light)		 PND 42–52
PND 84–90
(red-light)
ASR/PPI PND 32–36
PND 62–65		 PND 32–35
PND 62–65		 Not conducted PND 41–51
PND 83–90
MWM PND 66–70 PND 67–70 Not conducted PND 91–101
Open in a new tab

OF: Open Field. ASR: Auditory Startle Response. PPI: Pre-pulse inhibition. MWM. Morris Water Maze. GD: Gestational Day. PND: Post-natal Day.

2.2. Behavioral Assessments

Handling of the control rats and general conduct for the behavioral assessments in general followed recommendations outlined for the Division of Translational Toxicology, NIEHS (Behl et al., 2023) and followed established protocols for each of the modalities tested. For each behavioral test, rats were randomly assigned to apparatus and time of testing. The same rats were assessed as juveniles and adults following the same sequence (OF, ASR/PPI) occurring with a 24hr test interval, the same apparatus, and counterbalanced for session. Counterbalancing for sex was not conducted. For OF and ASR, males were tested prior to females. MWM was conducted only in adults and after the ASR/PPI. Males were assigned to one room and females to another each with an assigned investigator. This allowed for standardization of time-of-day testing across sexes. For Study Cohorts 1, 2, and 3, rats were transported from the home-cage room to a defined holding area near the testing room and allowed to acclimate prior to testing. Food and water were available. All testing occurred within 1 hr relative to homeroom lighting change. For the MWM, males were tested in one room containing 4 tanks shielded by white partitions and females tested in a separate identical room to allow for concurrent testing across the day. For Study Cohort 4, to accommodate completion of behavioral testing prior to daily inhalation exposure, acclimation following transport was shorter with the first test session starting within 30 min of lights on.

2.2.1. Open Field (OF) Motor Activity

Motor activity was captured in an open field chamber (40 × 40cm MotorMonitor II system; Kinder Scientific, Poway, CA) outfitted with photocell detectors (16 × 16) located at a height of 2” above the floor to capture horizontal movements and at 4” above the floor to capture vertical (rearing) movements. Photocell breaks were recorded in 5-min epochs over a 30-min (juveniles) or 45-min (adults) session under red light (~20 lux using commercial red-light bulbs); Study Cohorts 1, 2, and 4) or slightly dimmed house-lighting (Study Cohort 3). The rat was gently placed into the center of the arena and allowed to freely explore. Data was captured in 5-min epochs representing basic movements, rearing, distance travelled, and total activity (basic movements + rearing).

2.2.2. Acoustic Startle Response (ASR) and Pre-Pulse Startle Inhibition (PPI)

Rats were assessed for ASR, short-term habituation, and PPI using a load cell design computer-assisted StartleMonitor II (Kinder Scientific, Poway, CA). Each unit was calibrated for sound and the software algorithm recorded the weight of the rat at the beginning of each session and tared-out the weight from the response measure, allowing for a recording of response independent of body weight (personal communication, Dr. Curt Kinghorn, Kinder Scientific). Background noise level was set at 65dB and pre-pulse intensities (3, 6, 12, 15dB) were set relative to background for each box. Prior to the test day, rats acclimated to the holder and sound attenuated chamber for 5 min. A test session included a 5 min acclimation to the apparatus followed by a sequence of one 120 dB trial; five 120 dB trials (40-msec null period followed by 40-msec 120 dB pulse); two blocks of 31 trials each comprised of two no-stimulus trials, six 120dB trials, and 18 pre-pulse stimulus trials (40-msec null period followed by 20-msec pre-pulse, 100-msec null period, and a 40-msec 120dB pulse) at 15 sec variable inter-trial interval (ITI). Maximum force, as measured in Newtons (Nmax), was collected over a 100-msec window within the total 500-msec recording window. For each rat, habituation was determined based on 15 120-dB response trials (trials 1,2,3,4,5,6,11,17,18,25,32,34,39,49). PPI was calculated as the mean percent change in pre-pulse trial response relative to the median of that rat’s 120-dB response in the two blocks of 31 trials.

2.2.3. Morris Water Maze (MWM)

A circular plastic tank (180 cm × 62 cm) was filled with water (24–27°C) to within 10 cm distance from the lip of tank and encircled by white curtains. The lighting was confirmed sufficient for video detection with an absence of reflections or shadows. Rats were allowed two days (4 trials/day, ITI ~60 sec) to demonstrate ability to find a visible 10 cm plexiglass platform located 36 cm from tank wall within 90 sec. For spatial learning, four geometrical cues (1 × 1m) were placed on curtain walls at tank rim level. The platform was submerged by 1 cm and located in one defined quadrant. Rats were placed into the start quadrant and allowed 90 secs to find the hidden platform. If the rat had not reached the hidden platform by that time, the trial was ended and the rat guided to the escape platform, though this occurred rarely. Sessions consisted of four trials per day, at a 60-sec ITI, conducted over seven consecutive days using a defined sequence of alternating start locations. On day eight, a probe test was conducted. The platform was removed, rats were placed within the start quadrant opposite to the prior hidden platform location, identified as the goal quadrant (GQ), and allowed free swimming for 90 sec. Reference memory was represented by the percentage of time spent within each quadrant across the total 90-sec session and in 30-sec epochs. Behavior was captured using computer assisted video tracking software (ANY-MAZE, ver. 4.99; Stoelting Co, Wood Dale, IL). Acquisition data representing latency or distance travelled to reach the platform (visual or hidden) was censored at 90 seconds. For the probe test, data representing time spend within each quadrant was captured in 30-sec epochs.

2.3. Statistical Analysis

All analyses were completed in R version 4.3.1 (R Core Team, 2021). Following review of datasets for implausible values and outlying observations, all recorded responses were retained for analysis. Given that the experiments were not fully counterbalanced for sex, independent analyses were conducted for males and for females.

2.3.1. Across Study Cohort Analysis for Pattern of Behaviors

For motor activity endpoints (session total distance, session total activity), an analysis of variance (ANOVA) followed by Tukey post-hoc tests was performed to determine differences across Study Cohorts. To examine differences across Study Cohorts for acclimation to a novel environment, repeated measures ANOVA (RMANOVA) models were fit to activity and distance responses in 5-min epochs with Study Cohort, epoch, and their interaction as factors, an autoregressive (AR(1)) autocorrelation structure (Stroup, 2016) to account for temporal correlation of statistical error terms, followed by Tukey post-hoc tests. Poisson mixed models were fit to the skewed rearing counts with AR(1) autocorrelation and Tukey post-hoc tests.

RMANOVA was used to analyze habituation to 120dB startle stimuli as a function of study cohort, trial number and their interaction, with a spatial power autocorrelation structure to account for unequal spacings between trials and Tukey post-hoc tests. Naïve Nmax response to the initial 120 dB stimuli and each %PPI were analyzed by a Kruskal-Wallis Test (Nussbaum, 2024). Dunn’s post-hoc tests with a Bonferroni multiple testing comparison correction were performed for pairwise study differences.

MWM spatial learning (hidden platform) was represented by daily session (four trials/day) endpoints. Distance travelled and latency to reach the platform were analyzed as median responses per day to provide a measure of the center that is more appropriate for right-censored data introduced by the experimental design of ending trials at 90 seconds if a rat had not reached the platform. Histograms of both endpoints showed a right skew and decreasing variance with session number. A log transformation was used to meet the parametric RMANOVA assumptions of normality and variance homogeneity. To examine the pattern of activity over sessions (acquisition) and studies, a RMANOVA was used to model responses as a function of day, Study Cohort, and their interaction, with an AR(1) correlation structure and Tukey post-hoc tests. For the probe test of reference memory, the percentage of time spent in each quadrant for each of the three 30-sec intervals and for the full 90-sec session displayed a non-normal distribution and was analyzed by Kruskal-Wallis test followed by Dunn’s pairwise tests with Bonferroni multiple testing corrections.

Model selection for all RMANOVA models and Poisson mixed models (for rearing) began with an interaction model for each endpoint, fit with Study Cohort, time factor (5-min epoch for motor activity, trial number for startle response, and day for MWM), and their interaction. If the interaction term was significant at the α=0.05 level, pairwise study comparisons were assessed at each time point. If the interaction was not significant, an additive model was fit with Study Cohort and time factors, followed by Tukey post-hoc tests. For these models, Study Cohort effects were assessed via F-test for all endpoints except Open Field Rearing, which, due to technical restraints, was assessed with the Wald Chi-squared test. While the F-test is generally preferred to the Wald at sample sizes less than 30, the R function used for testing (function Anova, package car v3.1) could not provide an F-test for a Poisson repeated measures model with correlation structures (function glmmTMB, package glmmTMB v1.1.11), but it could provide a Wald. For all ANOVA and RMANOVA models, normality and homogeneity of variance assumptions were evaluated using visual examination of model residual plots and Shapiro-Wilks tests of residuals.

2.3.2. Individual Study Cohort Power Analyses

Endpoints selected for power analyses reflected those that are well-established, often reported within the literature, as well as recommended within regulatory studies (NAFTA Technical Working Group on Pesticides (TWG), 2016; Slikker and Bowyer, 2005; Swerdlow et al., 2001; Vorhees and Makris, 2015; Harry et al., 2022). For OF, power analyses were conducted on session totals for rearing, distance travelled, and total activity. For ASR, Nmax of the initial naïve response to120-dB stimuli and %PPI at each pre-pulse level were selected. For the MWM, the latency and distance travelled acquisition curves were selected to best reflect learning, and the time spent in each of the quadrants over the total 90-sec session and in 30-sec epochs were selected for the probe test.

For each Study Cohort, sex, and age, the power of a two-sided two-sample t-test was calculated based on each study endpoint’s sample mean and standard deviation for sample sizes up to 30 rats per cohort, at a significance level of α=0.05, and for three pre-specified effect sizes: 10%, 20%, and 30% increases from the control mean. The effect size percentages were chosen to represent for subtle changes (10%) through a moderate change (30%). Whether the effect size in an experiment is biologically relevant will depend on the experimental question and conditions. The previous analyses used ANOVA and RMANOVA models to compare multiple studies, but for simplification of calculations, the power analyses on all but the Water Maze acquisition endpoints use the t-test, which compares two groups on a single endpoint. The two-sided test was selected to account for possible bidirectionality of a response. Due to the symmetry of the t-test’s underlying distribution, the p-values from testing an increasing effect size are equivalent to those from a decreasing effect size. However, if the experimental design is such that directionality is anticipated or known, a one-sided approach could be considered. Because power calculations rely on the noncentral t distribution and cannot be expressed in a closed-form solution, numerical methods implemented in the pwr package in R were used for estimation (Champely et al., 2020). Power analyses for a RMANOVA model were conducted to assess the learning and acquisition endpoints from the MWM test. The latency and distance travelled within the hidden platform sequence were measured over multiple daily sessions to generate a learning curve to demonstrate acquisition of the task. This required a more complicated statistical model. To find the power under an RMANOVA model, the approach described in Jeffers et al. (2024) was followed, and simulation studies with n = 2,500 simulations were conducted per endpoint, sex, and Study Cohort. Two simulated groups were included in the model. The RMANOVA used the session and group to predict the endpoints with an autoregressive correlation structure to account for dependency across the session. Power was defined as frequency with which the test correctly identified a group effect across the 2,500 simulations divided by the total number of simulations.

3. Results

For each of the assessments, similar patterns of behavior were observed across Study Cohorts, indicating replicability. Individual power analyses were conducted for each Study Cohort on defined endpoints under three different behavioral assessment scenarios.

3.1. Open Field Motor Activity

General features of rodent activity and acclimation within an open field arena were replicated across studies (Fig. 1, 2). When cumulative session totals were compared, significant differences between Study Cohorts were observed in juvenile females for rearing (F(2,57)=18.3, p<0.001), distance travelled (F(2,57)=48.5, p<0.001), and total activity (F(2,57)=59.2, p<0.001). In juvenile males, significant differences were observed for distance travelled (F(2,57)=74.9, p<0.001) and total activity (F(2,57)=81.3, p<0.001) with no differences in rearing observed between Study Cohorts. These differences were associated with elevated activity in Study Cohort 4. In adult rats, cumulative session totals were significantly different for rearing (F(3,78)=4.3, p<0.01), distance travelled (F(3,78)=12.1, p<0.001), and total activity (F(3,78)=69.5, p<0.001) in females. In males, a significant overall ANOVA was observed for rearing (F(3,78)=4.41, p<0.01), distance travelled (F(3,78)=7.4, p<0.001), and total activity (F(3,78)=19.9, p<0.001). These differences were primarily associated with the elevated behavior of rats in Study Cohorts 3 and 4, except in males, where the differences in rearing were associated with elevated behavior in Study Cohorts 1 and 4.

Figure 1.

Figure 1.

Open in a new tab

Open Field Motor Activity - Juveniles. Female (F) and male (M) SD rats across Study Cohorts were examined as juveniles (30 min session) for activity within an open field photocell-detection arena. Total number of rears, total distance travelled (cm), and the total activity counts across the full session are represented in bar graphs. Activity captured in 5-min epochs across are represented in line graphs. Data represents means +/− SEM (n=20–22). (*) represents a statistically significant Study Cohort or Study and Minute interaction effect at the 0.05 level; (**) at the 0.01 level.

Figure 2.

Figure 2.

Open in a new tab

Open Field Motor Activity - Adults. Female (F) and male (M) SD rats across Study Cohorts were examined as adults (45 min session) for activity within an open field photocell-detection arena. Total number of rears, total distance travelled (cm), and the total activity counts across the full session are represented in bar graphs. Activity captured in 5-min epochs across are represented in line graphs. Data represents means +/− SEM (n=20–22). (*) represents a statistically significant Study Cohort or Study and Minute interaction effect at the 0.05 level; (**) at the 0.01 level.

A general pattern of acclimation to the novel environment of the open field arena was observed in all Study Cohorts. In juvenile rats (Fig. 1), the females showed significant study cohort differences for rearing (χ2(10)=30.6, p<0.001), distance travelled (F(10, 342)=2.8, p<0.01), and total activity (F(10, 342)=2.2, p<0.05) while males showed differences in distance travelled (F(10, 342)=6.4, p<0.001) and total activity (F(10, 342)=5.2, p<0.001). Differences in both males and females were driven primarily by higher activity levels in Study Cohort 4. In adult rats (Fig. 2), females showed significant study cohort differences in rearing (χ2(24)=44.2, p<0.01), distance travelled (F(24, 702)=1.6, p<0.05), and total activity (F(24, 702)=4.0, p<0.001), and males showed differences for rearing (χ2(3)=22.3, p<0.001), total activity (F(24, 702)=2.6, p<0.001), and distance travelled (F(3, 702)=13.7, p<0.001). The primary differences observed were elevated total activity measures for Study Cohort 3 and elevated distance travelled measures for Study Cohort 4 over the first 15–20 minutes of the session, with spurious differences observed for rearing.

3.2. ASR/PPI

The pattern of startle habituation to 120dB startle stimuli trials was similar across Study Cohorts; a habituation plateau was reached within the first ~15 trials, consistent with previous reports (Schmid and Valsamis, 2011; Pilz et al., 2014). The strength of the pre-habituation responses (Nmax) to the 120dB stimuli appeared higher in the adult rats as compared to their response as juveniles and in males as compared to females (Fig. 3). A comparison across study cohorts showed significant differences in Nmax levels across trials in juvenile females (F(2, 784)=26.8, p<0.001) and trial-dependent study differences in males (F(26, 783)=1.61, p<0.05). As a general pattern, juvenile females showed higher levels in Study Cohort 4, and juvenile males showed lower response levels in Study Cohort 1, as compared to the other study cohorts. Differences were seen primarily over the initial ~8 trials. In adults, Nmax responses throughout the sequence were similar across the Study Cohorts for both males and females (Fig. 3B).

Figure 3.

Figure 3.

Open in a new tab

Acoustic Startle Response (ASR) and Pre-pulse Inhibition (PPI). Female (F) and male (M) SD rats across Study Cohorts were examined as juveniles and adults for startle response to 120 dB acoustic stimulus over a sequence of trials (1,2,3,4,5,6,11,17,18,25,32,34,39,49). ASR is the maximum startle response, Nmax, measured in Newtons. Trials 1–6 represent the consecutive presentation of the 120 dB stimulus and intermittent trials, 11–49, represent the 120 dB only trials that occurred within the blocks of trials containing pre-pulse stimuli. Data represents means +/− SEM (n=20–22). (*) represents a statistically significant Study Cohort or Study and Trial interaction effect at the 0.05 level; (**) at the 0.01 level.

The naive Nmax response to the initial 120dB stimuli (Trial 1) showed high individual variability. Significance differences were found in juvenile males (χ2(2)=20.1, p<0.001) and females (χ2(2)=9.5, p<0.01) primarily due to lower responses in Study Cohort 1 (Fig. 3A). In adults (Fig. 3B) males showed significant Study Cohort differences for naïve Nmax (χ2(2)=14.3, p<0.001), reflected in higher levels in Study Cohort 2 and lower levels in Study Cohort 4. No differences were seen in the adult females.

The 120dB baseline values for calculating PPI were not found to be different between Study Cohorts, confirming appropriate background level for conducting PPI trials. For PPI, all Study Cohorts displayed increased inhibition of the startle response with increasing intensities of the pre-pulse stimuli (Fig. 4). In juvenile rats, Study Cohort differences were significant in both sexes at PPI12 (females: χ2(2)=9.8, p<0.01; males: χ2(2)=9.0, p<0.01) and at PPI15 (females: χ2(2)=14.3, p<0.001; males: χ2(2)=7.5, p<0.05). Inhibition at PPI 3 and 6 was similar across juvenile Study Cohorts. In adults, significant differences were observed at PPI12 (females: χ2(2)=21.5, p<0.001; males: χ2(2)=27.5, p<0.001) and at PPI15 (females: χ2(2)=14.4, p < 0.001; males: χ2(2)=31.1, p<0.001).

Figure 4.

Figure 4.

Open in a new tab

Pre-pulse Inhibition (PPI). Female (F) and male (M) SD rats across Study Cohorts were examined as juveniles and adults for startle response to 120 dB acoustic stimulus over a sequence of trials. Pre-pulse inhibition (% PPI) was calculated as the mean % change in each pre-pulse trial response (pre-pulse intensities 3, 6, 12, 15 dB over 65 dB background) relative to the median of that rat’s 120-dB response in blocks 2 and 3 (trials 6–50). Negative PPI values were set to 0. Data represents means +/− SEM (n=20–22). (*) represents a statistically significant Study Cohort effect at the 0.05 level; (**) at the 0.01 level.

3.3. Morris Water Maze

Learning was demonstrated by acquisition of the task with a progressive decrease in latency and distance travelled over the sessions (Fig. 5). Similar acquisition profiles were demonstrated across all Study Cohorts. During the first 2 days of training, Study Cohort 4 rats showed a longer latency (females: F(2, 399)=19.4, p<0.001; males: F(2, 399)=6.0, p<0.01) and greater distance travelled (females: F(2, 399)=14.7, p<0.001), and, by days 3–4, activity levels were within those shown in the other cohorts.

Figure 5.

Figure 5.

Open in a new tab

Morris Water Maze – Hidden Platform. Adult female and male rats in Study Cohorts 1, 2 and 4 were assessed for acquisition of a spatial learning hidden platform task. Spatial learning data represents the daily session median of 4 trials per day over 7 consecutive days for latency to find platform (log) and distance travelled (cm). Data are represented as mean +/− SEM (n=20–22). (*) represents a statistically significant Study Cohort or Study and Session interaction effect at the 0.05 level; (**) at the 0.01 level.

In the probe test, a preference for the goal quadrant (GQ) was demonstrated, with the start quadrant (NW) showing a secondary level of preference (Fig. 6). Study Cohort 1 females and males both spent more time in the goal quadrant during the 2nd epoch (females: χ2(2)=6.1, p<0.05; males: χ2(2)=7.2, p<0.05). While the actual % time spent in the GQ differed slightly (5–10%) between the Study Cohorts, the pattern of preference relative to the other quadrants was demonstrated. Examination across the epochs suggested less of a clear preference for the GQ over the start quadrant (NW) in Cohort 4 females during the 2nd and 3rd epoch. In all Study Cohorts, both males and females showed a clear preference for the GQ over the SE and SW quadrants.

Figure 6.

Figure 6.

Open in a new tab

Morris Water Maze –Probe Trial. Adult female and male rats in Study cohorts 1, 2 and 4 were assessed for reference memory using a probe trial. The percent of time spent in each quadrant was analyzed within 30 sec sequential epochs and for the entire 90 sec session. GQ: Goal Quadrant. NW: Northwest (starting) quadrant. SE: Southeast quadrant. SW: Southwest quadrant. Data are represented as mean % of total session time +/− SEM (n=20–22). (*) represents a statistically significant Study Cohort effect at the 0.05 level; (**) at the 0.01 level.

3.4. Individual Study Cohort Power Analyses

Representative single endpoints across the 3 behavioral tests were selected for power analyses with a t-test. For OF, cumulative measures of rearing, distance travelled, and total activity (basic activity + rearing) were used. For ASR, the first response to a 120dB auditory stimulus representing a naïve response and the individual PPI at pre-pulse levels of 3, 6, 12, or 15 were used. For MWM, measures of performance on the probe trial for each of the 30-sec intervals and the full 90-sec session were used. In addition, considering the nature of learning, a power analysis was conducted based on the acquisition over training sessions with a RMANOVA. Differences in power for Study Cohorts were driven by the relative sizes of sample means, standard deviations, and resultant differences in coefficients of variation. This was demonstrated in the slope of the curves across the effect sizes for sex, age, and endpoint of each Study Cohort (Fig. 7–12). For example, power curves for a 10% effect size were flatter than those for a 30% effect size. The evaluation considered sufficiently powered studies as those with at least 80% power to detect an effect.

Figure 7.

Figure 7.

Open in a new tab

Open Field Motor Activity Power Analysis - Juveniles. Session activity totals for rearing, distance travelled, and total activity (rearing + basic movements) were used to calculate the power of a two-sided two-sample t-test with a significance level of α = 0.05 for each Study Cohort over a range of group sample sizes up to 30 and for each % level of difference (10, 20, or 30%). Points highlight the power for a sample size of n=20 rats per group for each of the study cohorts (●: Study Cohort 1, ■: Study Cohort 2, ▲: Study Cohort 3, ▼: Study Cohort 4). The horizontal line indicates 80% power.

Figure 12.

Figure 12.

Open in a new tab

Morris Water Maze Probe Trial Power Analysis - Adults. In adult male and female SD rats across Study cohorts 1,2, and 4, the % time spent in the goal quadrant (GQ) relative to the total session time interval (0–30 sec; 30–60 sec; 60–90 sec; 0–90 sec) was used to calculate the power of a two-sided two-sample t-test with a significance level of α = 0.05 for each Study cohort over a range of sample sizes up to 30 and for each % level of difference (10, 20, or 30%). Points highlight the power for a sample size of n=20 rats per group for each sex and study cohorts (●: Study Cohort 1, ■: Study Cohort 2, ▼: Study Cohort 4). The horizonal line indicates 80% power.

3.4.1. Open Field

Power curves for each Study Cohort represented cumulative measures of rearing, distance travelled, and total activity of males or females for sample sizes up to 30 and effect sizes of 10%, 20%, and 30%. For the t-test to detect a 10% effect size in either juveniles (Fig. 7) or adults (Fig. 8), 30 rats per group were not sufficient to achieve 80% power as demonstrated by the slopes of the power curves. An increase in power was found with effects sizes of 20% or 30%. In juveniles, rearing required an effect size of at least 30% at 80% power within 30 rats while distance travelled and total activity showed at least 80% power to detect a 20% effect within 20 rats. So, excluding rearing, 20 rats were sufficient for 80% power at a 20% effect size in juveniles. In adults, a 30% effect size in rearing, distance travelled, and total activity was detected within ~30, ~20, and ~20 rats, respectively.

Figure 8.

Figure 8.

Open in a new tab

Open Field Motor Activity Power Analysis - Adults. Session activity totals for rearing, distance travelled, and total activity (rearing + basic movements) were used to calculate the power of a two-sided two-sample t-test with a significance level of α = 0.05 for each Study Cohort over a range of group sample sizes up to 30 and for each % level of difference (10, 20, or 30%). Points highlight the power for a sample size of n=20 rats per group for each of the study cohorts (●: Study Cohort 1, ■: Study Cohort 2, ▲: Study Cohort 3, ▼: Study Cohort 4). The horizontal line indicates 80% power.

The generally lower power across sample sizes for rearing was linked to higher relative variability likely related to the low incidence. Conversely, total activity exhibited lower relative variability and correspondingly higher power. The reader is reminded that total activity is defined as a count of all photocell breaks within the arena (and includes the rearing component as well as any horizontal non-ambulatory activity), rather than a targeted activity measure. As expected, the response levels of total activity were similar to those representing ambulatory activity as that was the predominant input for calculating total activity.

The differences in power were reflective of the relative sizes of the sample means, standard deviations, and resultant relative variability of the studies as measured by the coefficients of variation (CV; 100 times the ratio of the standard deviation to the mean) (Supplemental Table 1).

3.4.2. ASR/PPI

While the first recorded startle response has been considered to represent the naïve response of an organism, using a commercial system that does not ensure immobility of the rat at the time of auditory stimuli delivery, the data obtained was highly variable and difficult to determine if it reflected biological processes or other confounders. In juveniles, the naïve startle response required at least 30 for a 30% effect size to be detected with 80% power (Fig. 9). In adults, a greater number of rats would be required to reach the same level of power (Fig. 10). The pattern of the PPI response showed the expected increasing inhibition with increasing pre-pulse stimuli dB level regardless of age or sex of the rats. This PPI pattern was associated with fewer rats required to reach 80% power (Juveniles - Fig. 9, Adults - Fig. 10). At the lowest pre-pulse level of 3dB above background (PPI3), sufficient power to detect a 30% effect size was generally not attained within 30 rats for juveniles but was attained for adults. However, the 3dB pre-pulse level remains close to the general background dB level and thus, the data may be representative more of the 120dB response alone which was stronger for the adults. At PPI6, a 30% effect size could be detected at 80% power with 30 rats in juveniles and adults. At PPI12, a 30% effect size could be detected at 80% power with 20 juvenile rats and a 20% effect size could be detected at 80% power with 20 adult rats. At the highest pre-pulse intensity, PPI15, a more robust inhibition was detected and, with 20 rats, at 80% power, a 30% effect size could be detected in juveniles and a 20% effect size in adults.

Figure 9.

Figure 9.

Open in a new tab

Acoustic Startle Response Power Analysis - Juveniles. Maximum force (Nmax) of the initial naïve ASR, %PPI 3, and % PPI 15 were used to calculate the power of a two-sided two-sample t-test with a significance level of α = 0.05 for each Study cohort over a range of group sample sizes up to 30 and for each % level of difference (10, 20, or 30%). Points highlight the power for a sample size of n=20 rats per group for each of the study cohorts (●: Study Cohort 1, ■: Study Cohort 2, ▼: Study Cohort 4). The horizontal line indicates 80% power.

Figure 10.

Figure 10.

Open in a new tab

Acoustic Startle Response Power Analysis - Adults. Maximum force (Nmax) of the initial naïve ASR, %PPI 3, and % PPI 15 were used to calculate the power of a two-sided two-sample t-test with a significance level of α = 0.05 for each Study cohort over a range of group sample sizes up to 30 and for each % level of difference (10, 20, or 30%). Points highlight the power for a sample size of n=20 rats per group for each of the study cohorts (●: Study Cohort 1, ■: Study Cohort 2, ▼: Study Cohort 4). The horizontal line indicates 80% power.

3.4.3. MWM

Acquisition in the MWM was assessed with a RMANOVA to best capture the ability of the rat to learn to perform the task over the multiple training days. Power curves were based on the latency and distance travelled to the hidden platform (Fig. 11). In general, similar patterns were observed across sexes and Study Cohorts. For latency, the Study Cohorts reached 80% power at 15 rats for females and 10 rats for males for a 10% effect size. At a 20% or 30% effect size, 80% was reached with 10 rats. For the distance travelled, a 20% effect size difference with 80% power could be obtained with 20 rats. Study 4 females showed slightly higher power on this endpoint, requiring approximately 10 rats to detect a 20% effect size with 80% power. At a 30% effect size, approximately 10 rats were needed across the sexes and Study Cohorts to reach 80% power.

Figure 11.

Figure 11.

Open in a new tab

Morris Water Maze Hidden Trial Power Analysis - Adults. In adult male and female SD rats across Study cohorts 1,2, and 4, the log of the median latency to the platform over 4 trials per day and the median distance travelled were used to calculate the power of a Repeated Measures Analysis of Variance with two groups and a significance level of α = 0.05 for each Study Cohort over a range of sample sizes up to 30 and for each % level of difference (10, 20, or 30%). Points highlight the power for a sample size of n=20 rats per group for each sex and study cohorts (●: Study Cohort 1, ■: Study Cohort 2, ▼: Study Cohort 4). The horizonal line indicates 80% power. Power curves are non-smooth due to the simulation process needed to generated the power estimates.

For the MWM Probe test, the raw data was assessed to confirm that the rats displayed a clear preference for the goal quadrant as compared to other quadrants prior to a power analysis. The different epochs provide some level of information on the strategy used by the test rat in searching for an escape over the session. Given the predominance of reliance on time spent in the GQ in the published literature, power curves were based on the % time spent in the GQ within each of the consecutive 30-sec intervals and the full 90-sec session (Fig. 12). In general, similar patterns were observed across sexes and Study Cohorts. For the individual 30-sec intervals, no Study Cohort reached 80% power at either a 10% or 20% effect size within 30 rats. A 30% effect size required a minimum of 20 – 30 rats. For the full 90-sec session, all Study Cohorts showed near to 80% power between 20 and 30 rats for a 20% effect size and within 20 rats for a 30% effect size. The exception was Study Cohort 4. Females showed an overall lower power across all sample sizes for all time intervals while in males, lower power levels were observed only within the initial 0–30 interval.

3.4.4. Study Cohort Distribution of Power Across Endpoints

The distribution of power of each Study Cohort across the various endpoints with 20% effect size and a sample size of 20 showed general trends and few consistent study differences (Fig. 13). The strength of the individual response impacted power and replicability across Study Cohorts. Reduced power was seen for OF rearing, with lower counts and relatively higher variability than other activity measures. Reduced power was also seen in the marginal %PPI at the lower pre-pulse levels with a smaller change from the 120dB response. Motor activity was relatively similar for juvenile and adult females and for juvenile males. Adult males showed lower activity levels and displayed greater differences across Study Cohorts. For PPI, overall higher power across sample sizes was observed for the adults as compared with the juveniles, with little difference observed between sexes. This is likely reflective of the maturation of the neural circuitry necessary for the response inhibition. The RMANOVA test for the acquisition pattern of power for the MWM showed high power, reaching over 80% power at a 20% effect size with 20 rats. Note that this is a different statistical model than used for the other endpoints, so comparisons across endpoints are not recommended. The power of the t-test for the MWM probe test using the % time spent in the GQ was similar for males and females across Study Cohorts. The cumulative 0–90 sec period showed the highest power likely due to lessened relative variability; however, it did not capture the different shifts in search pattern reflected in the 30-sec intervals.

Figure 13.

Figure 13.

Open in a new tab

Statistical Power of a two-sided t-test for a fixed sample size (n=20), significance level of α = 0.05, and fixed effect size (20%) across all present studies, cohorts, ages, and assays. Panels for female adults, male adults, female juveniles, and male juveniles. Each symbol represents a study (●: Study Cohort 1, ■: Study Cohort 2, ▲: Study Cohort 3, ▼: Study Cohort 4). RR: Rearing. DT: Distance Travelled. TA: Total Activity. T1: Trial 1 Maximum Force. PPI3: Pre-pulse inhibition level 3. PPI6: Pre-pulse inhibition level 6. PPI12: Pre-pulse inhibition level 12. PPI15: Pre-pulse inhibition level 15. 0–30: Percentage of time spent in the goal quadrant for the first 30 seconds. 30–60: Percentage of time spent in the goal quadrant for the second 30 seconds. 60–90: Percentage of time spent in the goal quadrant for the third 30 seconds. 0–90: Percentage of time spent in the goal quadrant for all 90 seconds. Dash line indicated 80% power.

For the power analyses conducted, there was agreement between Study Cohorts for the number of rats needed to detect mid-range effects (20%−30% effect sizes). The evaluation of only 10 rats per group would have resulted in underpowered tests for most endpoints at a 30% effect size. Increasing the sample size to 20 rats per group would be adequate for most endpoints with a 30% effect size.

4. Discussion

The current study used control rats to assess replicability of studies evaluating performance on standardized behavioral rodent assessments. With the relatively large individual group sample size of 20 – 22 rats per group, we were able to evaluate replicability across studies, calculate the level of power, and estimate the number of rats for a specific effect size. Of the three behaviors evaluated (motor activity, acoustic startle response, and learning and memory in the MWM), all studies showed the expected pattern of behavior, e.g., acclimation to the open field over time, habituation of startle response, %PPI over pre-pulse intensities, acquisition of the spatial MWM task, and preference for goal quadrant in the probe trial. Thus, replicability of longitudinal patterns was demonstrated across the behavioral assays. While the patterns of behavior were replicated, examining the use of single endpoints raised questions as to expectations placed on such data for the level of effect they can detect. Expectations of detecting a 20% effect with 80% power generally would require an increase in the number of rats tested. However, with 20 rats per group, it was determined that one could anticipate detecting a 30% effect with 80% power across the behavioral battery. Thus, while the behavioral performance was replicated over the Study Cohorts, power analyses of individual data points encourage any investigator or user of such data to display a level of restraint regarding the level of differences one might detect when examining alterations outside the range of normal controls.

The interpretation and utility of computer-generated endpoints depends on being able to correctly define and understand the endpoint. Open Field total activity may be a summation of all horizontal and vertical photocell breaks or only the horizontal, depending on the experimental setup. In this way, the total activity may be misleadingly high if, for example, a rat was stationary but grooming, thus repeatedly breaking the photocells. Testing history of the animal can also play a role in how behaviors are expressed. In this case, adult rats had been handled and tested as juveniles, which must be taken into consideration when interpreting results.

The differences in power that appeared between Study Cohorts could be explained by relative variability of endpoints. With all other factors being held constant (statistical test, sample size, effect size, endpoint, sex, age), each Study Cohort’s unique mean and standard deviation influenced power via the coefficient of variation (CV), which has an inverse relationship with power: the greater the CV, the lower the power. For example, for rearing, juvenile females in Study Cohort 4 showed a higher power across sample sizes and effect sizes due to a lower coefficient of variation (CV=21.7) than Study Cohorts 1 and 2 (29.7 and 29.2, respectively). Juvenile male rearing in Study Cohort 1 (CV=33.3) showed lower power as compared with Study Cohorts 2 and 4 (CV=25.8 and 24.9 respectively). A full list of endpoint summary statistics is provided (Supplemental Table 1). Endpoints not presented here would have their own relative variability and sensitivity and thus, require targeted power analyses.

Although the CV plays a large role in determining power, the researcher is advised to use the most biologically relevant endpoints to answer their experimental question within a framework of the interconnected and interdependency of different modalities. As in all types of assays, one needs to examine data within a linear portion of the response curve rather than within the detection plateaus at the extreme low or high ends. An inaccurate assessment of sensitivity could be driven by a compression of the variability at either end, thus raising the CV and increasing power, even though it may not be biologically relevant.

Both the data and the chosen statistical model can affect the power. While it may seem that a more complex model will have more power, this is not necessarily the case, and power across model types should not be compared. The most appropriate model will depend on the experimental design and the experimental questions, and its power should be calculated. An assessment of behavioral performance as reflective of the nervous system is complicated. Simply relying on a single behavioral task that shows the greatest power can fail to capture the complexities and interactions across various behaviors. Using a single assessment in one behavioral domain to infer overall behavioral performance ignores the fact that behavior is influenced by multiple factors and may not necessarily be reflective of the behavioral domain being tested (Cory-Slechta et al., 2021). The interrelated and interdependent nature of behavior requires the consideration of multiple assays and endpoints. An example of the interconnectivity of behavior modalities is evident in the OF paradigm. While the standard OF paradigm assesses a limited aspect of exploratory behavior, the response is influenced by motor components that impact rearing and distance travelled or attention and anxiety components that influence exploratory activities. Thus, the underlying biology associated with each endpoint and mitigating factors that influence exploration need to be considered in any data interpretation. For similar reasons, relying on any single endpoint as an assessment of behavior rather than examining patterns of behavior and different modalities may result in inaccurate conclusions. Replicable response patterns across tests serve to provide a more accurate assessment. In addition, with the availability of computer assisted instruments that allow for a greater exploration of relevant endpoints, there exist an opportunity to expand upon the standard basic endpoints collected to further explore aspects of the behavior and contribute to data interpretation (Harry et al., 2022). Thus, with the same level of effort, a more detailed and complex pattern of the behavior can be generated for evaluation.

In comparing the behavioral assessments, it was clear that, while differences could be observed across Study Cohorts when single endpoints were examined, the patterns of response were similar. For example, in MWM hidden platform procedure, while a difference was observed on the first couple of days in Study Cohort 4, acquisition of the task was clearly demonstrated over the 7 sessions, with all Study Cohorts reaching a similar latency or distance travelled by day 3. More differences were evident in Study Cohorts that were tested either in different laboratories or under different experimental conditions. In open field motor activity, the elevated levels of activity observed in Study Cohort 3 may likely be related to the fact that these rats, unlike those in the other Study Cohorts, were only assessed as adults. Thus, the elevated level of open field activity within the first epochs of the session likely reflects the novelty of the task and the exploration of the novel environment.

The most prominent differences were observed in Study Cohort 4. In contrast to the normal home-cage group housing of the other study cohorts, Study Cohort 4 rats were subjected to different gestational conditions for the dam and in the offspring during the early juvenile age to adulthood. This included differences related to single housing in wire mesh cages within inhalation chamber conditions of noise level, lighting, air flow, all of which can influence stress levels of laboratory rodents (Castelhano-Carlos et al., 2021). Additionally, testing commenced earlier in the day to accommodate the associated study design. It is likely that these inhalation-related environmental conditions contributed to the increased motor activity, the diminished strength in of the adult startle response, and slightly blunted pre-pulse startle inhibition pattern seen in Study Cohort 4 adult rats. The impact of these conditions and the higher level of variability translated over to the power analyses. These results underscore the importance of considering context and environmental conditions when examining treatment effects (Voelkl et al., 2020). Despite the rather severe difference in animal husbandry and handling between Study Cohort 4 and the others, the pattern in each of the behavioral tests was replicated across all four study cohorts.

A critical component of determining the replicability of an assay is to understand the variance of the behavioral performance in normal animals under study to account for innate biological variability. As the variability of an endpoint increases, the power of a test will decrease, and it will be more difficult to confidently determine any effect. Identifying and understanding variance serves as a first step in determining the number of experimental animals required to reach an effect of a certain size according to an experimental criterion for determining a rejection of the null hypothesis. While many studies report statistical significance as an estimate of such a rejection for a specific endpoint, the level of significance alone does not inform of the magnitude of the effect size nor does it confirm biological importance (Mellis, 2018; Michel et al., 2020). For those endpoints that show a statistically significant effect, one needs to consider: if those effects are meaningful; if the number of events captured were sufficient to reflect the endpoint; and, perhaps more importantly, if those effects are real, as reaching significance without adequate power may yield a misleading estimate and interpretation (Button et al., 2013). In the case of null effects, it may be that the experimental manipulation truly had no effect or that the absence of a detected effect was due to experimental design and/or power issues. These issues also arise for the possibility of false discoveries where statistically significant results may be scientifically incorrect. This is of concern especially when the effect sizes are small and are reported in the absence of independent replication.

The current study was based on the premise that, to estimate an appropriate sample size for any study evaluating the effect of an experimental manipulation on a behavioral phenotype, knowledge of the range in a behavioral performance in the control group is a critical first step. The premise was supported by the datasets with demonstrated differences in power across endpoints depending on effect size. The current evaluation of control data sets has shown that, across behavioral studies designed to meet various neurotoxicology regulatory guideline recommendations, there is a great degree of replicability, though it is moderated by environmental and handling factors. Expectations of effect size placed on pre-clinical data should fit within the validity and sensitivity of the endpoint assessed. While most endpoints were too variable to detect a small (10%) effect size, moderate effect sizes may be detected with the recommended number of 20 rats.

The current attention to minimizing animals in research and/or the costs associated with increasing sample size must consider the need for studies to be adequately powered to address the question at hand and to provide a level of confidence for data interpretation and replication. Conducting and reporting results from a low powered study can have significant, multi-faceted impacts, including undermining confidence in any interpretation of the relevance of the findings, potentially misguiding future research or regulatory decisions, and failing to adhere to the ethical framework of the 3R’s. While multiple approaches have been presented for minimizing the number of animals within a study (Bodden et al., 2019; Kimmelman et al., 2014; von Kortzfleisch et al., 2020; 2022; von Kortzfleisch and Richter, 2024), efforts should not lose sight of the focus to maximize the amount of knowledge obtained per animal, per endpoint, or per study, to provide findings that can be reproduced in subsequent experiments or in different laboratories. Inclusion of relevant additional information provides biologically important data to advance any interpretation or decision-making use of the findings. The consideration of power and effect size in the design of any experiment supports data interpretation and potential for reproducibility of the findings. This in turn provides useful information that contributes to the literature and advances the field.

Supplementary Material

1

Highlights:

  • Data from control rats from four behavioral studies were collected, examining motor activity, auditory startle response, and learning and memory.

  • Studies were assessed for replicability across longitudinal and single endpoints.

  • Power calculations were estimated for sample sizes up to 30 rats per group.

  • Most endpoints were too variable to detect a small (10%) effect size, but moderate effect sizes may be detected with 20 rats per group.

Acknowledgements.

The authors would like to acknowledge Dr. Katherine Allen-Moyer for her guidance and interactions on this work. Drs Cynthia Rider, Kristen Ryan, and Kelly Shipkowski for allowing us to use data obtained from control rats in the various studies.

Funding Sources.

This work was supported by the National Institute of Environmental Health Sciences under contract GS-00F-173CA / 75N96022F00055 to Social & Scientific Systems, Inc., A DLH Holdings Corp. Company, and CRO contracts: HHSN273201400015C; HHSN273201300010C. This research was supported in part by the Intramural Research Program of the National Institutes of Health, National Institute of Environmental Health Sciences [ZIA ES103387; ZIC ES103380].

Footnotes

Conflicts of Interest.

The authors declare no conflicts of interest.

Declaration of interests

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Declaration of generative AI and AI-assisted technologies in the writing process.

During the preparation of this work, the author(s) did not use generative AI or AI-assisted technologies.

Institutional Review Board Statement.

Data provided from rodent neurobehavioral studies were obtained according to an approved animal protocol at the participating commercial contract laboratories adhering to all guidelines of NIH for the humane use and care of laboratory animals. Approved protocol numbers can be provided at request.

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

References

  1. Ankeny RA, Whittaker AL, Ryan M, Boer J, Plebanski M, Tuke J, & Spencer SJ (2023). The power of effective study design in animal experimentation: Exploring the statistical and ethical implications of asking multiple questions of a data set. Brain, Behavior, and Immunity, 112, 163–172. 10.1016/j.bbi.2023.06.012 [DOI] [PubMed] [Google Scholar]
  2. Behl MV, Harry GJ, Roberts GK, Stout MD, & Witchey SK (2023). Neurobehavioral Testing. In Specifications for the Conduct of Toxicity Studies by the Division of Translational Toxicology at the National Institute of Environmental Health Sciences. National Institute of Environmental Health Sciences. https://www.ncbi.nlm.nih.gov/books/NBK591139/ [PubMed] [Google Scholar]
  3. Bodden C, von Kortzfleisch VT, Karwinkel F, Kaiser S, Sachser N, & Richter SH (2019). Heterogenising study samples across testing time improves reproducibility of behavioural data. Sci. Rep, 9(1), 8247. 10.1038/s41598-019-44705-2 [DOI] [PMC free article] [PubMed] [Google Scholar]
  4. Button KS, Ioannidis JPA, Mokrysz C, Nosek BA, Flint J, Robinson ESJ, & Munafò MR (2013). Power failure: Why small sample size undermines the reliability of neuroscience. Nat Rev Neurosci. 14(5), 365–376. 10.1038/nrn3475 [DOI] [PubMed] [Google Scholar]
  5. Cao Y, Chen RC, & Katz AJ (2024). Why is a small sample size not enough? The Oncologist, 29(9), 761–763. 10.1093/oncolo/oyae162 [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Carnahan RM, & Brown GD (2024). The power and pitfalls of underpowered studies. Pharmacotherapy, 44(9), 698–700. 10.1002/phar.4605 [DOI] [PubMed] [Google Scholar]
  7. Castelhano-Carlos MJ, Aslani S, & Sousa N (2021). The impact of physical enrichment in the structure of the medial prefrontal cortex and nucleus accumbens of the adult male rat Brain. Neurosci 454, 51–60. 10.1016/j.neuroscience.2020.01.040 [DOI] [PubMed] [Google Scholar]
  8. Champely S, Ekstrom C, Dalgaard P, Gill J, Weibelzahl S, Anandkumar A, Ford C, Volcic R, & Rosario HD (2020). pwr: Basic Functions for Power Analysis (Version 1.3–0) [Computer software; ]. https://cran.r-project.org/web/packages/pwr/index.html [Google Scholar]
  9. Cohen J (2013). Statistical Power Analysis for the Behavioral Sciences (2nd ed.). Routledge. 10.4324/9780203771587 [DOI] [Google Scholar]
  10. Cory-Slechta DA, Harvey K, & Sobolewski M (2021). Assessment of Neurotoxicant-Induced Changes in Behavior: Issues Related to Interpretation of Outcomes and Experimental Design. In Llorens J & Barenys M (Eds.), Exper. Neurotox. Meth. (pp. 239–263). Springer US. 10.1007/978-1-0716-1637-6_11 [DOI] [Google Scholar]
  11. Faber J, & Fonseca LM (2014). How sample size influences research outcomes. Dental Press J Orthodons, 19(4), 27–29. 10.1590/2176-9451.19.4.027-029.ebo [DOI] [PMC free article] [PubMed] [Google Scholar]
  12. Geyer MA, Krebs-Thomson K, Braff DL, & Swerdlow NR (2001). Pharmacological studies of prepulse inhibition models of sensorimotor gating deficits in schizophrenia: A decade in review. Psychopharm. 156(2), 117–154. 10.1007/s002130100811 [DOI] [PubMed] [Google Scholar]
  13. Gulinello M, Mitchell HA, Chang Q, Timothy O’Brien W, Zhou Z, Abel T, Wang L, Corbin JG, Veeraragavan S, Samaco RC, Andrews NA, Fagiolini M, Cole TB, Burbacher TM, & Crawley JN (2019). Rigor and reproducibility in rodent behavioral research. Neurobiology of Learning and Memory, 165, 106780. 10.1016/j.nlm.2018.01.001 [DOI] [PMC free article] [PubMed] [Google Scholar]
  14. Harry GJ, McBride S, Witchey SK, Mhaouty-Kodja S, Trembleau A, Bridge M, & Bencsik A (2022). Roadbumps at the Crossroads of Integrating Behavioral and In Vitro Approaches for Neurotoxicity Assessment. Frontiers in Toxicology, 4. 10.3389/ftox.2022.812863 [DOI] [PMC free article] [PubMed] [Google Scholar]
  15. Jennions MD, & Møller AP (2003). A survey of the statistical power of research in behavioral ecology and animal behavior. Behavioral Ecology, 14(3), 438–445. 10.1093/beheco/14.3.438 [DOI] [Google Scholar]
  16. Jeffers A, Konrad K, Larson G, Allen-Moyer K, Cunny H, & Shockley S (2024). Simulation methodologies to determine statistical power in laboratory animal research studies. Lab Animals, 58(5). 10.1177/00236772241273002 [DOI] [PMC free article] [PubMed] [Google Scholar]
  17. Kafkafi N, Agassi J, Chesler EJ, Crabbe JC, Crusio WE, Eilam D, Gerlai, et al. Reproducibility and replicability of rodent phenotyping in preclinical studies. Neurosci. Biobehav. Rev., 87, 218–232. 10.1016/j.neubiorev.2018.01.003 [DOI] [PMC free article] [PubMed] [Google Scholar]
  18. Kimmelman J, Mogil JS, & Dirnagl U (2014). Distinguishing between Exploratory and Confirmatory Preclinical Research Will Improve Translation. PLOS Biol., 12(5), e1001863. 10.1371/journal.pbio.1001863 [DOI] [PMC free article] [PubMed] [Google Scholar]
  19. Loken E, & Gelman A (2017). Measurement error and the replication crisis. Science, 355(6325), 584–585. 10.1126/science.aal3618 [DOI] [PubMed] [Google Scholar]
  20. Mellis C (2018). Lies, damned lies and statistics: Clinical importance versus statistical significance in research. Paediat. Respir. Rev., 25, 88–93. 10.1016/j.prrv.2017.02.002 [DOI] [PubMed] [Google Scholar]
  21. Michel MC, Murphy TJ, & Motulsky HJ (2020). New Author Guidelines for Displaying Data and Reporting Data Analysis and Statistical Methods in Experimental Biology. J. Pharmacol. Exper. Therap. 372(1), 136–147. 10.1124/jpet.119.264143 [DOI] [PubMed] [Google Scholar]
  22. NAFTA Technical Working Group on Pesticides (TWG). (2016). Developmental Neurotoxicity Study (DNT) Guidance Document. https://www.epa.gov/sites/default/files/2017-02/documents/developmental_neurotoxicity_study_internal_guidance_document_final_0.pdf
  23. National Academies of Sciences, E., Affairs, P. and G., Committee on Science, E., Information, B. on R. D. and, Sciences, D. on E. and P., Statistics, C. on A. and T., Analytics, B. on M. S. and, Studies, D. on E. and L., Board, N. and R. S., Education, D. of B. and S. S. and, Statistics, C. on N., Board on Behavioral, C., & Science, C. on R. and R. in. (2019). Understanding Reproducibility and Replicability. In Reproducibility and Replicability in Science. National Academies Press (US). https://www.ncbi.nlm.nih.gov/books/NBK547546/ [PubMed] [Google Scholar]
  24. National Centre for the Replacement, Refinement, and Reduction of Animals in Research. The 3Rs | NC3Rs. Retrieved February 20, 2025, from https://nc3rs.org.uk/who-we-are/3rs
  25. Nosek BA, Alter G, Banks GC, Borsboom D, Bowman SD, Breckler SJ, et al. (2015). SCIENTIFIC STANDARDS. Promoting an open research culture. Science, 348(6242), 1422–1425. 10.1126/science.aab2374 [DOI] [PMC free article] [PubMed] [Google Scholar]
  26. Nussbaum EM (2024). Categorical and Nonparametric Data Analysis: Choosing the Best Statistical Technique (2nd ed.). Routledge. 10.4324/9781003145264 [DOI] [Google Scholar]
  27. Peterson SJ, & Foley S (2021). Clinician’s Guide to Understanding Effect Size, Alpha Level, Power, and Sample Size. Nutri. Clin. Pract., 36(3), 598–605. 10.1002/ncp.10674 [DOI] [PubMed] [Google Scholar]
  28. Pilz PKD, Arnold SW, Rischawy AT, & Plappert CF (2014). Longterm-habituation of the startle response in mice is stimulus modality, but not context specific. Front. Integr. Neurosci. 7. 10.3389/fnint.2013.00103 [DOI] [PMC free article] [PubMed] [Google Scholar]
  29. R Core Team. (2021). R version 4.3.1. https://cran.rstudio.com/bin/windows/base/old/4.3.1/
  30. Roberts GK, & Stout MD (Eds.). (2023). Specifications for the Conduct of Toxicity Studies by the Division of Translational Toxicology at the National Institute of Environmental Health Sciences. National Institute of Environmental Health Sciences. http://www.ncbi.nlm.nih.gov/books/NBK591148/ [PubMed] [Google Scholar]
  31. Schmid S, & Valsamis B (2011). Habituation and Prepulse Inhibition of Acoustic Startle in Rodents. JOVE https://www.academia.edu/17249786/Habituation_and_Prepulse_Inhibition_of_Acoustic_Startle_in_Rodents [DOI] [PMC free article] [PubMed]
  32. Serdar CC, Cihan M, Yücel D, & Serdar MA (2020). Sample size, power and effect size revisited: Simplified and practical approaches in pre-clinical, clinical and laboratory studies. Biochemia Medica, 31(1), 010502. 10.11613/BM.2021.010502 [DOI] [PMC free article] [PubMed] [Google Scholar]
  33. Sert N. P. du, Hurst V, Ahluwalia A, Alam S, Avey MT, Baker M, Browne WJ, et al. (2020). The ARRIVE guidelines 2.0: Updated guidelines for reporting animal research. PLOS Biol. 18(7), e3000410. 10.1371/journal.pbio.3000410 [DOI] [PMC free article] [PubMed] [Google Scholar]
  34. Slikker W, & Bowyer JF (2005). Biomarkers of adult and developmental neurotoxicity. Toxicol. Appl. Pharmacol., 206(2).255–260. 10.1016/j.taap.2004.09.022 [DOI] [PubMed] [Google Scholar]
  35. Smith AJ, Clutton RE, Lilley E, Hansen KEA, & Brattelid T (2018). PREPARE: Guidelines for planning animal research and testing. Lab. Animal 52(2), 135–141. 10.1177/0023677217724823 [DOI] [PMC free article] [PubMed] [Google Scholar]
  36. Stroup WW (2016). Generalized Linear Mixed Models: Modern Concepts, Methods and Applications. CRC Press. 10.1201/b13151 [DOI] [Google Scholar]
  37. Swerdlow NR, Geyer MA, & Braff DL (2001). Neural circuit regulation of prepulse inhibition of startle in the rat: Current knowledge and future challenges. Psychopharm., 156(2–3), 194–215. 10.1007/s002130100799 [DOI] [PubMed] [Google Scholar]
  38. van den Buuse M (2010). Modeling the positive symptoms of schizophrenia in genetically modified mice: Pharmacology and methodology aspects. Schizophrenia Bull., 36(2), 246–270. 10.1093/schbul/sbp132 [DOI] [PMC free article] [PubMed] [Google Scholar]
  39. Voelkl B, Altman NS, Forsman A, Forstmeier W, Gurevitch J, Jaric I, Karp NA, Kas MJ, Schielzeth H, Van de Casteele T, & Würbel H (2020). Reproducibility of animal research in light of biological variation. Nat. Rev. Neurosci. 21(7), 384–393. 10.1038/s41583-020-0313-3 [DOI] [PubMed] [Google Scholar]
  40. von Kortzfleisch VT, Karp NA, Palme R, Kaiser S, Sachser N, & Richter SH (2020). Improving reproducibility in animal research by splitting the study population into several “mini-experiments.” Sci. Rep., 10(1), 16579. 10.1038/s41598-020-73503-4 [DOI] [PMC free article] [PubMed] [Google Scholar]
  41. von Kortzfleisch VT, & Richter SH (2024). Systematic heterogenization revisited: Increasing variation in animal experiments to improve reproducibility? Journal of Neurosci. Meth. 401, 109992. 10.1016/j.jneumeth.2023.109992 [DOI] [PubMed] [Google Scholar]
  42. Vorhees CV, & Makris SL (2015). Assessment of learning, memory, and attention in developmental neurotoxicity regulatory studies: Synthesis, commentary, and recommendations. Neurotoxicol.Teratol, 52, 109–115. 10.1016/j.ntt.2015.10.004 [DOI] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

1
NIHMS2115940-supplement-1.xlsx (22.4KB, xlsx)

RESOURCES