Why random splits lie
A shuffled train/test split assumes each row is independent. Market data does not care about that assumption. Regimes change, volatility clusters, and feature distributions move. If you let future rows leak into the training mix, the model gets to see the answer key.
That is why walk-forward exists. It keeps the past in front of the future and makes the model retrain as time advances. You trade a prettier score for a more honest one.
Walk-forward vs train-test split
A single train-test split gives you one answer from one cut in time. Walk-forward gives you a sequence of answers from a rolling cut, which is closer to how a live system behaves. The point is not just chronology. It is how the model reacts when the market state changes under it.
- Chronology stays intact. No future leakage through the split.
- Retraining is explicit. The model gets judged as the data moves.
- Thresholds move too. Signal magnitude drifts, so the cutoff should not be frozen.
What to look at
Do not stop at the aggregate score. Check the phase breakdown, the trade count, and whether the edge survives after costs. A model that prints one good backtest and then falls apart in later windows is not robust. It was lucky.
Walk-forward is also a good way to expose overfitting that hides in a single validation split. If the best round keeps changing wildly or the threshold only works once, the signal is probably thin.
The blunt rule
If time matters, use walk-forward. If it does not, you are probably not trading a market yet.
A small worked split
On a 10,000-bar series, a simple walk-forward setup might train on the first 4,000 bars, validate on the next 500, then roll forward by 500 bars at a time. The point is not the exact ratio. The point is that every decision is made with only past data in view.
If the score collapses in later windows, you learned something useful: the edge was tied to one regime, not the full sample.
Common mistakes
- Reusing the validation window. Once you tune on it, it stops being a test.
- Freezing the threshold forever. Score distributions drift with the market.
- Ignoring the later windows. A strong first fold can hide a weak live pattern.