Why most backtests lie — and what in-sample vs out-of-sample actually fixes

Every strategy looks profitable on the data it was built on. That is not a sign of an edge — it is a near-mathematical certainty. The moment you tune a parameter to past prices, you have begun fitting the strategy to noise as much as to signal, and the backtest curve stops being a forecast and starts being a memory.

This is not a warning about carelessness. Careful people produce overfitted backtests constantly, because the mechanism that produces them is not sloppiness — it is the ordinary, sensible act of improving a strategy until it works. The improvement and the overfitting are the same activity viewed from two angles.

The standard defence is the in-sample / out-of-sample split, and it is worth understanding precisely what it does and does not buy you.

The split, plainly

You divide your history into two parts. The in-sample period is where you are allowed to look, tune, and optimise. The out-of-sample period is sealed off — you do not touch it until your parameters are locked. When you finally run the locked strategy on that untouched data, you are asking a single honest question: does the edge survive on prices it was never fitted to?

If performance holds up, you have weak evidence of a real effect. If it collapses, you have strong evidence that you were trading your own curve-fit.

The asymmetry there is the whole point, and it is worth sitting with, because it is the opposite of how most people use the test. A failed out-of-sample run is close to conclusive. A passed one is barely suggestive.

Why "weak evidence" is the best you get

A single out-of-sample pass is not proof. You can still get lucky on one slice of history, and a strategy that survives one split can fail on another regime entirely. This is why a clean OOS result should lower your confidence in failure more than it raises your confidence in success.

What it reliably catches is the worst case: the strategy that exists only because of overfitting. Those die immediately out of sample, and killing them early is most of the value. A validation process that mostly eliminates candidates is doing its job — the alternative is a process that approves everything, which protects you from nothing.

The problem the split does not solve

Here is the failure mode that survives a textbook-clean split, and it is the one that quietly ruins most retail research.

Suppose you test one strategy, honestly, on sealed out-of-sample data. A good result means something. Now suppose you test two hundred variants — different lookbacks, different filters, different stop placements — and report the one that survived. You have run the same test two hundred times and published the winner.

Some of those two hundred will clear the bar on luck alone. That is not a possibility; it is arithmetic. The more configurations you try, the more certain you become of finding one that looks excellent out of sample and is worth nothing forward.

The split assumes you get one shot. Most research quietly takes hundreds, because each failed attempt sends you back to adjust something and try again. The out-of-sample period erodes a little with every look, and after enough iterations it is functionally in-sample — you have simply laundered the fitting through a slower process.

There is no clean fix, but there are honest mitigations:

• Count your attempts. Write down how many variants you tested before this one. A strategy that survived on attempt 3 and one that survived on attempt 300 are not comparable pieces of evidence, even with identical statistics.
• Demand more from later attempts. If you are deep into a search, the bar should rise, not stay flat.
• Hold back a third slice that you touch exactly once, at the very end, and accept the result whatever it says.
• Prefer fewer parameters. Each free parameter multiplies the space you are searching, and the search is what does the damage.

The biases that outlive a clean split

Even with disciplined sampling, a backtest can be wrong for reasons that have nothing to do with fitting. These are data problems, and they pass straight through an out-of-sample test because they contaminate both halves equally.

Survivorship. If your universe is "stocks currently in the index," you have quietly excluded every company that fell out of it — usually by performing badly. The dataset has been curated by hindsight. Any long-only result on such a universe is flattered by construction, and the effect is largest exactly where people most want to test: small and mid caps.

Look-ahead. Using a closing price to trigger an entry at that same close. Using quarterly results on the quarter-end date rather than the date they were actually published. Using an index constituent list as it stands today for a period when it stood differently. Each of these hands your strategy information it could not have had, and each is easy to introduce without noticing.

Restated data. Fundamentals get revised. Adjusted price series get re-adjusted after corporate actions. A backtest run today on today's version of history is not the history a trader faced. Point-in-time data solves this properly; where you cannot get it, at least know that you are exposed.

None of these produce an obviously broken equity curve. They produce a slightly-too-good one, which is far more dangerous.

Costs are not a detail

A strategy is only real after costs, and Indian market microstructure has enough of them to turn a decent edge negative.

Brokerage is the smallest part. Securities transaction tax, exchange charges, stamp duty and GST all apply, and the effective rate depends on segment — cash delivery, intraday, futures and options are all treated differently, and the treatment changes with regulation. Any backtest carrying a single flat cost assumption across segments is approximating something that is not flat.

Then there is what the ledger never records:

• Slippage and impact. Backtests fill at the printed price. Real orders move the book. In liquid index futures this is small; in a mid-cap or a far out-of-the-money option it can exceed the entire edge.
• Circuit limits. A backtest will cheerfully buy a stock on the day it locks upper circuit. Nobody filled that order.
• Liquidity that exists on the chart but not in the book. An options strike can print a last-traded price on a handful of contracts. A strategy that assumes you can trade size there is trading a fiction.

The practical test is blunt: re-run with costs at two or three times your estimate. An edge that survives that is worth pursuing. One that inverts was never an edge — it was a rounding error with a good story.

What walk-forward adds

Rolling the split forward — optimise on a window, trade the next window, advance, repeat — is a genuine improvement on the single split. It tests the strategy repeatedly across changing conditions and, more usefully, tests whether your re-optimisation procedure is stable rather than just whether one parameter set was lucky.

Two cautions. First, it multiplies the number of decisions you are making, so it can accelerate the multiple-comparisons problem rather than contain it. Second, wildly unstable optimal parameters between windows are themselves the finding — if the best lookback swings from 8 to 60 and back, the strategy has no stable structure to exploit, no matter how the aggregate curve looks.

Regime is not a footnote

A strategy validated only across a trending stretch has not been tested against chop. Indian markets have delivered long grinding uptrends, sharp global-driven drawdowns and extended sideways ranges within the span of most people's available data — and a system tuned to one of those will often invert in another.

Wherever possible, your out-of-sample window should contain conditions your in-sample period did not. If you cannot arrange that, say so explicitly in your own notes, because the alternative is quietly believing a result that has only ever been tested in one weather.

A practical bar

Before a strategy earns real capital, it should clear:

1. A locked parameter set — decided before the out-of-sample run, not adjusted after seeing it.
2. An out-of-sample result that is directionally consistent with in-sample, not merely positive. Halving is normal and survivable; inverting is disqualifying.
3. Enough trades on both sides that the numbers are not hostage to a handful of outliers. If removing your three best trades destroys the result, you have three data points, not a strategy.
4. Survival under stressed costs, at a multiple of your estimate.
5. An honest count of how many variants you tried to get here.

To make that concrete: a system that shows a profit factor of 2.1 in-sample and 1.1 out-of-sample has not passed. It has shown you where most of its apparent edge came from. The honest reading of that pair is not "still profitable" — it is "thin, and possibly nothing."

None of this guarantees forward performance. Nothing does. What it removes is the population of strategies that never had a chance, and that population is the overwhelming majority of what any search produces.

The uncomfortable truth is that good validation mostly produces rejections. That is the point. A process that rarely says no is not protecting you from anything.