The audit described below was run against our own equity sibling's flagship backtest, and its unflattering findings were published in full at the time. Numbers are quoted from that audit. Nothing here is investment advice. Decisions are yours.
Before this site published a single number, our house pointed the tools institutions use for catching self-deception, probability-of-overfitting tests, deflated Sharpe ratios, survivorship bounds, at its own flagship equity backtest. Half the audit came back clean. The other half found a flaw large enough to bound the headline result down by as much as ninety percent in the worst scenario. We published both halves, and the damage report became the constitution this crypto site was built under.
Why audit yourself at all
Every quantitative shop faces the same temptation: your backtest is your marketing, and auditing it can only make it smaller. The reasons to do it anyway are practical, not noble. First, reality arrives eventually; a live track record converges on the truth whatever the backtest said, and the gap between promise and delivery is paid in reputation at the worst possible time. Second, an audit you run yourself, on your own terms, published on your own site, is worth more than the same flaw discovered by a critic. Third, and decisive for us: you cannot demand verification from an industry, as our evaluation cluster does in seven checks, while exempting your own numbers from the knife.
The setup
The subject was a five-year, four-engine systematic equity strategy whose locked backtest showed a 75x multiple, a Sharpe ratio near 2.4, and a maximum drawdown under 16%, numbers good enough to deserve suspicion. The audit asked the two questions every strong backtest must answer. One: is this result an artifact of SELECTION, the winner plucked from so many tried variants that something was bound to look this good by chance? Two: is it an artifact of DATA, a universe that quietly excludes the failures the strategy would have traded? Different tools answer each question, and they returned opposite verdicts.
The selection audit came back clean
The selection question has proper statistics. The probability of backtest overfitting, computed by combinatorially splitting the sample into in-sample and out-of-sample halves thousands of ways and checking whether the in-sample winner keeps winning, came back at 2.6% against a danger threshold of fifty: the variant that won in-sample stayed near the top out-of-sample in essentially every split. The deflated Sharpe ratio, which asks whether the headline Sharpe survives after accounting for how many variants were ever tried, survived even under the assumption of three thousand historical trials, several times more than the program's documented history. And a detail we treasure: the single highest-Sharpe variant in the entire research archive was one the house had REJECTED before the audit, for being a lone spike rather than a stable plateau. The discipline of preferring robust parameter regions over lucky points, it turns out, is measurable, and it measured well.
The data audit did not
Then the second question. The strategy's universe held 6,134 symbols with five years of history, assembled the industry-standard way from a broker's current listings. The audit asked one crude query: how many of those symbols STOPPED trading during the window? The answer was zero. Not one death in five years, in an asset class where several percent of small companies delist annually. The universe was survivor-only, the backtest had never been forced to trade through a single corporate failure, and every result it produced inherited that flattery. The mechanism, and why momentum strategies suffer it worst, is unpacked in the graveyard problem.
Bounding the damage
A flaw you cannot remove you must bound, so the audit modeled the missing graveyard adversarially: assume dead names would have taken candidate slots in proportion to an annual delisting rate, with a propensity boost because dying momentum names over-represent on breakout boards, and assume each such slot earned the year's average losing return while displacing a real trade. Under a 4% annual delisting rate, the 75x multiple bounds down to about 30x. Under 8%, to 15x. Under a pessimistic 12%, to 9x. The bound is deliberately harsh, it ignores that short holding periods blunt the bias and that dead names still had to pass quality gates, but its direction is not negotiable, and the honest summary we published was exactly that: the true multiple lives somewhere well below the headline, and the exact floor is unknowable without better data.
What the audit changed
Findings without consequences are theater, so here is what the damage report bought. On the equity side: the published marketing already leaned on risk-adjusted quality rather than the headline multiple, and the audit was published alongside the track record it criticized. On this side, the consequences are structural. The crypto research universe must be point-in-time and graveyard-inclusive from day one, and any data vendor that cannot produce a dead coin's final week is disqualified. Every backtest we publish states its universe construction and its trial count. The validation gates any strategy must pass, including the PBO and deflated Sharpe tests our equity work passed and the survivorship standard it failed, are pre-registered in measurement vs advice. And our founding crypto experiment inherited the humility directly: its results were labeled upper bounds on a survivor-biased universe in the same breath as their publication, per the transplant experiment.
What this means for you
Two portable lessons. First, the tools exist: PBO, deflated Sharpe ratios, and survivorship bounds are published methods, not house secrets, and any shop with real results can run them in a day. When you meet a spectacular backtest, ask whether its owner has, and whether they will show you the report. The answer sorts the industry faster than any returns figure. Second, the two failure modes are independent: a strategy can be honestly selected on rotten data, or luckily selected on clean data, and only testing BOTH questions closes the account. Our own report card, selection clean, data flawed and bounded, is public because that is what we would demand of anyone else. The knife cuts both ways here, permanently. Decisions are yours.