This article explains a statistical test and reports our own results with it on our equity research. Nothing here is investment advice. Decisions are yours.
The probability of backtest overfitting asks the question every impressive backtest dodges: given all the variants you tried, how likely is it that your chosen winner is just the luckiest of them? It is a number, computable from your own trial history, and refusing to compute it is a choice with a meaning.
The problem it formalizes
Overfitting in crypto starts somewhere mundane: strategy development is search, dozens or hundreds of variants tried, one selected. Selection alone guarantees the winner looks good in-sample, even if every variant is noise: rank enough coin-flippers and the top one has a hot streak. The honest question is therefore never "how good is the winner's backtest" but "how much of that goodness is the act of choosing," and answering it requires the full trial archive, which is why the derivation journal keeps every variant our research has ever run, rejected ones included.
The method, from zero
The test, combinatorially symmetric cross-validation, comes from Bailey, Borwein, Lopez de Prado, and Zhu's work on backtest overfitting, and its logic fits in a paragraph. Slice the backtest period into S equal blocks. Form every way of choosing half the blocks as "training" and half as "testing": with S=16, that is 12,870 combinations. For each combination, rank all trial variants on the training half, take the training winner, and look up its rank on the testing half. If the training winner routinely ranks in the testing half's top ranks, selection is finding something real; if it routinely lands in the bottom half, selection is finding luck. PBO is the fraction of combinations where the in-sample winner finishes below-median out-of-sample: near zero is comforting, near 50% means your selection process is a coin flip, and above it, worse.
Our own number, and the gate
We ran exactly this during our own equity audit, on the full trial archive of 128 deduplicated strategy variants across the system's development history, S=16, all 12,870 splits, and the result was a PBO of 2.6%: in fewer than three percent of the combinations did the in-sample winner degrade to below-median out-of-sample. That result says the equity system's selection was overwhelmingly finding signal rather than luck; it says nothing about crypto, which is why the derivation program pre-registers the same test as a gate rather than citing the equity result as cover. The public roadmap says no crypto score ships unless its own PBO, computed on its own trial archive, comes in below 20%. The threshold is deliberately forgiving relative to our equity result and deliberately fatal to the industry norm, which is not to run the test at all.
What PBO cannot catch
The test audits selection, not data. A trial archive built on survivorship-biased data can pass PBO beautifully while every variant enjoys the same corpse-free flattery that point-in-time universes exist to strip out: the winner is genuinely the best of its cohort, and the whole cohort is playing a rigged game. It also cannot see the trials you did not log: a researcher who explores informally and only archives the polished finalists has laundered the search before the test arrives, which is why the derivation journal commits to logging every run, failures first, on the principle that losses are the product.
And it is silent on regime fragility: a low-PBO strategy validated across one era can still die when structure migrates, and the cycle that broke the script is the case in point. Probing that separately is the job of walk-forward validation and the deflated Sharpe ratio. PBO is one gate in a battery, and its real product is cultural: a research process that knows its trial count will be audited searches differently from one that knows it won't. Decisions are yours.