The backtest that’s too good: five ways it tells you a fairy tale
Hi trader,
yesterday I posted a short reel (in Italian) on my Instagram about something that happened to me a few weeks ago.
A follower sends me his equity curve.
A backtest across several markets, with a line climbing cleanly from left to right. The kind that makes you think “ok, my work here is done”.
Looking closer, on the one-minute candle there were often stop and target inside the same candle.
And what decided which of the two came first wasn’t the market.
It was the backtest.
In the reel there was room for just one reason. Here I’ll give you five, because that clean line usually doesn’t have just one.
1. Stop and target in the same candle
Let’s start with the one from the reel.
A one-minute candle tells you four things: open, high, low, close.
It doesn’t tell you in which order the high and the low came.
If that candle’s low reaches your stop and its high reaches your target, both were touched within that minute.
But which one first?
The candle doesn’t know. And if the candle doesn’t know, neither does the backtest.
So the software has to guess.
MultiCharts and TradingView, for example, by default assume price goes first toward the extreme closest to the open. MetaTrader 5, if you don’t give it real ticks, makes them up.
Each in its own way, but it’s always a hypothesis, and you don’t necessarily know which one yours used.
Now let’s do the math, with round numbers made up on purpose.
One hundred trades, stop and target at the same distance, so every trade is worth roughly 1 R.
Eighty are clear-cut: 42 wins and 38 losses. That’s +4 R. A system that barely gets by.
The other twenty are ambiguous, stop and target in the same candle.
If the backtest counts them all as wins, you close at +24 R. Cover-worthy equity.
If they were actually all losses, you close at -16 R.
Same system, same trades. The difference comes from a hypothesis you didn’t choose and that you may not even know you made.
The tighter your stop and target are relative to the timeframe, the more ambiguous cases you get. And the more ambiguous cases you get, the more the result is decided by the software’s hypothesis instead of the market.
There’s only one real fix: where stop and target fall in the same candle, you go and look at the ticks. Almost every platform has it: Bar Magnifier on MultiCharts and TradingView, “High” order fill resolution on NinjaTrader, “Every tick based on real ticks” on MetaTrader 5.
If you don’t have the ticks, at least count them as losses. If the system holds up even like that, it starts to get interesting.
2. The gifted touch
Second gift, a close relative of the first.
Your limit order is at 100. Price reaches 100 and turns back.
Many backtests, with their default settings, write: filled.
In the real market, when price touches your level and leaves, you’re not necessarily in. There can be other orders ahead of you at the same price, and a single touch isn’t enough to clear them all.
And notice the nasty detail: the touches the backtest gifts you most willingly are exactly the ones where price just grazes you and immediately turns in your favour. In other words, the best trades.
Simple rule so you don’t get fooled: in backtest you count a limit as filled only if price trades through it by at least one tick. You lose a few trades on paper, but the ones left are real.
3. Costs left out “to keep it simple”
I see it all the time, even from good people.
“I’ll add the costs later.”
Then later never comes.
Broker commissions and exchange fees, and for stops and market orders slippage too, meaning the difference between the price you wanted and the one you get. A limit that’s actually sitting in the book won’t fill you worse than your price, at most it won’t fill you, as we saw above. A stop, on the other hand, becomes a market order when it triggers, and that’s where you pay slippage.
On a system that makes two trades a week with a thirty-point target, costs are noise.
On a system that makes ten trades a day with a three-tick target, costs are the system. An average profit per trade of a tick and a half, if you pay one tick between commissions and slippage, loses two thirds along the way.
The test is trivial: take the average profit per trade and put it next to the cost of one round-turn trade. If the two numbers are close, you don’t have a system. You have an elegant way of paying your broker.
4. Peeking into the future
This one is hard to see, because in the code everything looks fine.
The system uses a piece of information or a price that, at the moment it should have made the decision, it didn’t have.
A few examples I’ve seen:
- the signal is calculated on the candle’s close, but the entry is placed at the open of that same candle;
- the high or low of the day is used to decide a trade taken mid-morning;
- an indicator that repaints, meaning it changes past values when new data comes in, and in the backtest always shows you its “hindsight” version;
- Heikin Ashi candles used as if they were real prices. They’re averages: the open of a Heikin Ashi candle isn’t the price you could have entered at in that moment. If the backtest gets you in there, you’re entering at a price that, in the market, at that instant, didn’t exist.
The alarm bell is always the same: a win rate that’s too high, too steady, with very few bad days. The market isn’t that kind to anyone.
5. Too many knobs
Last one, and maybe the most human.
You have a system with five parameters. Moving average length, stop distance, target distance, start time, end time.
You run the optimizer on two years of data and find the perfect combination.
The problem is you haven’t found the system that works. You’ve found the one that best fits those two years, noise included. The more knobs you have and the more combinations you try, the easier it is for the best one to be best by chance.
This isn’t just my impression. There’s a serious line of research on it.
David Bailey, Marcos López de Prado and two fellow mathematicians showed that, by trying enough variants, it’s relatively easy to find a beautiful backtest even for a strategy with no real edge at all.
And Campbell Harvey and Yan Liu suggest “haircutting” a backtest’s results more, the more tests you ran to get there.
What you do in practice:
- set aside a chunk of data that you don’t touch while optimizing, and look at it only at the end, once;
- check whether the parameters close to the best one work almost as well. If the 20-period average is fantastic and the 19 and 21 are a disaster, you haven’t found a system, you’ve found an accident;
- fewer knobs is better. Every extra parameter is one more place to hide luck.
It happened to me too
I’m not telling you this from up high.
More than once I’ve looked at stunning equity curves, incredible tests, and then redone the math with the right data and found out the system, in reality, didn’t stand up.
The last time was not long ago. I took some backtests done on bars and redid them on ticks, with real costs and honest fills. Some systems that looked great on paper dropped to a profit factor below one.
In other words, they were losing.
It wasn’t pleasant. But finding out with my account would have been a lot less pleasant.
The five questions to ask your backtest
Before you trust a line that goes up, ask it these questions.
- How often do stop and target fall in the same candle? And how did the software count them?
- Does it count a touched limit as filled? Or does it require price to trade through it?
- Are the costs in there? Commissions, and slippage on stops and market orders.
- Does the system use something it couldn’t have known at decision time? Closes, daily highs, repainting indicators, candles that aren’t prices.
- How many variants did you try before this one? And how does it do on data you never used to optimize?
If after these five the line still goes up, good. But it’s not over.
The last test is run by the market
The most honest backtest in the world is still a hypothesis.
The next step is to run it forward, on demo or with minimum size, and compare trade by trade what the system does with what the backtest said. How many fills are missing, how much slippage you really pay, how many “backtest” trades never actually existed in reality.
For this comparison you need somewhere to log your real trades. If you don’t have one, use the TradingBlog Diary: it’s free, and at the end of the month you have the numbers in front of you instead of having to remember them.
If the gap between paper and reality is small, you’ve got something.
If it’s big, the backtest told you a fairy tale.
Better to know now.
A backtest is a hypothesis, not a result. The result is written by the market.
Suerte Amigo!
Tiziano Brunno
Tradingblog
Sources
- David H. Bailey, Jonathan M. Borwein, Marcos López de Prado, Qiji Jim Zhu, “Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance”, Notices of the American Mathematical Society, 2014.
- Campbell R. Harvey, Yan Liu, “Backtesting”, The Journal of Portfolio Management, 2015.
Want to put your market reading to the test?
Compete and challenge other traders inside an Arena in a demo environment, with no real capital at risk. Discover the Performance Arena Events by The Thunder Trader.
Disclaimer: purely informational and educational content. It does not constitute financial advice or an invitation to trade. Trading involves the risk of capital loss.
