ATLAS Higgs ML Challenge Classifier

What this research found

The 2014 ATLAS Higgs Boson Machine Learning Challenge asked entrants to separate Higgs decays into tau-lepton pairs from a far larger, kinematically similar background, scored not on accuracy but on Approximate Median Significance, a proxy for the discovery significance of a counting experiment. That protocol was reproduced on the permanent CERN Open Data release of 818,238 simulated collision events, with a gradient-boosted-tree classifier trained only on the official 250,000-event training split. It reached a weighted area under the ROC curve of 0.9329 and 0.9340 and a significance of 3.540 and 3.539 on the public and private leaderboard sets, at a decision threshold fixed before any test evaluation.

• At the pre-committed threshold of 0.058 the classifier scored 3.540 on the public leaderboard set and 3.539 on the private set, with weighted ROC areas of 0.9329 and 0.9340. Scanning the threshold against the test labels, which is an oracle procedure and only an upper bound, raises these to just 3.584 and 3.575, so choosing the threshold blind cost roughly 0.04 in significance.
• The competition weighting scheme is the trap in reproducing this benchmark. Each split is independently renormalised to carry the full physical yield, about 692 in summed signal weight and 411,000 in background, so pooling the public and private sets doubles the yield and inflates the significance to 5.009. The per-split figure near 3.54 is the physically comparable one.
• Ten of the eleven features carrying the -999 sentinel are not corrupted data but structurally undefined, following deterministically from reconstructed jet multiplicity: 40.0% of events have no jets and 70.9% have fewer than two. The exception is the missing-mass Higgs estimator, absent in 15.2% of events because its fit can fail to converge, making its absence informative rather than structural.
• Three ways of handling those sentinels proved statistically indistinguishable under five-fold cross-validation: 3.640 ± 0.154 for letting the trees learn a default direction for missing values, 3.640 ± 0.168 for median imputation, and 3.618 ± 0.175 for a jet-multiplicity-stratified ensemble, with ROC areas differing by less than 0.001. The simplest option was selected on parsimony.
• Test performance landed within one cross-validation standard deviation of the training estimate of 3.640 ± 0.154, and the test ROC area of about 0.9335 matched the cross-validated 0.9339, indicating no meaningful generalisation gap. The near-identical significance on two splits that differ 4.5-fold in raw event count independently confirms the per-split weighting behaves as designed.
• Physics-motivated derived variables dominate the model. The transverse mass of the missing-energy and lepton system leads by average gain, followed by the missing-mass Higgs estimate and the visible invariant mass, matching the quantities that distinguish the tau-pair Higgs signal from Z-boson and top-quark-pair backgrounds.

How it was done

The release was read from CERN Open Data Record 328 and partitioned using the official split field, recovering 250,000 training, 100,000 public leaderboard, 450,000 private leaderboard, and 18,238 withheld events; fitting used the designated training split alone. The eleven sentinel-carrying features were audited against jet multiplicity, then mapped to floating-point NaN so the boosted trees could learn a default branch direction for absent values and treat undefinedness as a learnable signal. Five-fold stratified cross-validation compared that against median imputation and a jet-multiplicity-stratified ensemble under the challenge significance objective, with validation-fold weights rescaled to reproduce the full physical normalisation and the threshold scanned across 197 grid points. The selected strategy was retrained on all 250,000 events at the mean of the per-fold optimal thresholds, held at identical hyperparameters throughout so the comparison isolated missing-value handling, and only then evaluated on the leaderboard sets.

Data sources

• ATLAS Higgs Boson Machine Learning Challenge dataset, CERN Open Data Record 328 (2014) - 818,238 simulated proton-proton collision events, 30 kinematic features, 279,560 signal and 538,678 background
• Adam-Bourdarios, Cowan, Germain, Guyon, Kégl and Rousseau, Proceedings of Machine Learning Research 42:19 (2015) - challenge design and the significance metric
• Cowan, Cranmer, Gross and Vitells, European Physical Journal C 71:1554 (2011) - asymptotic formulae underlying the significance approximation
• Chen and Guestrin, XGBoost: A scalable tree boosting system, ACM SIGKDD (2016)