Figure B.2. Discovery-validation scheme for the estimation of replicability.
To estimate the performance of a harmonization procedure at improving the replicability of the voxels associated with age, we use the harmonized dataset as a discovery cohort, and an independent dataset (different participants) as a validation cohort. For each cohort separately, we perform a mass-univariate analysis for age to obtain a t-statistic at each voxel. This yields two vectors of t-statistics, tdis and tval, for the discovery and validation cohorts respectively. We calculate the agreement between tdis and tval using the concordance at the top (CAT) curve, described in the Methods section. A harmonization method that performs better will yield a vector tdis more similar to tval, that is a CAT curve closer to 1.