ABIDE Autism Classification with Site-Effect Analysis

What this research found

Autism classifiers built on pooled multi-site brain imaging are widely suspected of borrowing accuracy from scanner and cohort differences rather than biology. K-Dense tested this directly on 1,035 resting-state functional MRI scans from the Autism Brain Imaging Data Exchange, training regularized linear classifiers on functional connectivity and evaluating them two ways: with every site represented in both training and test data, and with an entire site withheld. Pooled discrimination barely moved between the two schemes, holding at roughly 0.76 area under the curve, but performance on individual withheld sites ranged from chance to 0.89.

• The best configuration, a tangent-space connectivity embedding with L2 logistic regression, reached 0.760 plus or minus 0.014 area under the curve under subject-stratified five-fold cross-validation and a pooled 0.761 when every prediction came from a model blind to that subject's site.
• Across all four model configurations the pooled area under the curve changed by less than 0.5% between schemes, and balanced accuracy and macro-averaged F1 were actually one to two percentage points higher under the strict site hold-out.
• The real cost of site confounding shows up as variance. Site-wise area under the curve for the best model spanned 0.50 at one adult sample with a wide age range up to 0.89, with a standard deviation of 0.10 — roughly seven times the fold-to-fold variation seen under stratification.
• That heterogeneity was unrelated to how much data a site contributed: the correlation between a withheld site's sample size and its area under the curve was weak and not significant (r = 0.23, p = 0.32). The largest site, with 175 subjects, reached 0.82, while two sites of 26 subjects each landed at 0.86 and 0.64.
• Tangent-space embedding modestly outperformed raw correlations, 0.760 against 0.747 under stratification, matching prior benchmarking. Absolute performance sits within the established range for this dataset and is far from clinically usable, with balanced accuracy around 0.68.
• The demographic structure makes the confound concrete: site sample sizes span an order of magnitude, mean age ranges from about 10 to 35 years, five sites are entirely male, and the case-to-control ratio shifts from control-heavy to case-heavy across sites.

How it was done

A single explicitly named preprocessed derivative was used to remove ambiguity: one standardized connectome pipeline with band-pass filtering and no global signal regression, parcellated into 200 functional regions. Subjects were retained only where the regional time series was present, non-empty, and correctly shaped, giving 1,035 participants across 20 site labels. Connectivity was represented two ways, as the vectorized upper triangle of the Pearson correlation matrix giving 19,900 features, and as a tangent-space projection of each covariance matrix at the group geometric mean giving 20,100 features. Each was classified with L2-regularized logistic regression and a linear support-vector classifier, with feature standardization fit on training partitions only. The two evaluation schemes were stratified five-fold cross-validation preserving the case-control ratio, and leave-one-site-out cross-validation producing 20 folds. Out-of-fold predictions were retained for every subject under both schemes.

Data sources

• ABIDE I preprocessed release from the Preprocessed Connectomes Project — 1,035 subjects (505 autism, 530 controls) across 20 site labels
• Di Martino et al., Molecular Psychiatry 19:659-667 (2014) — the original data exchange spanning roughly 17 international laboratories
• Craddock et al., Human Brain Mapping 33:1914-1928 (2012) — the 200-region functional parcellation used here
• Dadi et al., NeuroImage 192:115-134 (2019) — benchmarking of connectome-based predictive models
• Abraham et al., NeuroImage 147:736-745 (2017) — reproducible-biomarker benchmark reporting 66 to 67% accuracy on this dataset