How Many Trades Prove an Expert Advisor Works? Sample Size, Luck and the Statistics Behind a Good-Looking Backtest

The question a backtest cannot answer by itself

When you look at a backtest report, you are looking at one history of one set of rules. The report is accurate about that history. The question you care about is different: does the rule set have a genuine edge that will persist, or did it happen to fit the past? A backtest cannot settle this alone, because a rule set with no edge will also produce a backtest, and some of those backtests look excellent. The job of statistics is to tell you how often that happens and how to recognise it.

There are two distinct threats. The first is plain sampling noise: with few trades, even a genuinely profitable system can show a loss and a worthless one can show a profit. The second is selection: if many rule sets, or many settings of one rule set, were tried and the best was kept, the winner's result is inflated by the selection itself. Most weak EAs fail on one or both. This guide takes them in turn, then turns them into questions you can ask.

How many trades before the average trade means something

Take an EA whose trades are, on average, profitable. Its t-statistic after n trades is the average result divided by its standard error, which equals the average divided by the standard deviation of trade results, times the square root of n. The table does this for four typical shapes of system, each with a positive expectancy. Notice how quickly noise swamps a small edge.

Read the first table by columns. After 30 trades, none of the four systems reaches even a t-statistic of 1.2. A 30-trade equity curve, which is what plenty of marketing screenshots show, cannot tell a +0.25R system from a coin flip. After 100 trades only the strongest edge reaches the usual bar. A system earning a modest +0.10R per trade, which is a realistic size for a real edge after costs, needs close to 400 trades for t = 2 and nearly 900 for t = 3. If its trades come about 10 a week, that is most of a year of live data to reach 400 and nearly two years to reach 900.

The practical rule is that the smaller the edge, the more evidence it needs, and real edges in liquid markets are small. This is also why the number of trades in a backtest matters more than the number of years. A system that trades twice a month for ten years has about 240 trades, while one that trades twenty times a day for six months has thousands. The second can be evaluated sooner, though its edge is also usually thinner relative to costs, which brings its own risks, as our guide on scalping brokers and minimum holding times discusses.

How many years a Sharpe ratio needs to prove itself

Traders often quote a Sharpe ratio, the average excess return divided by its volatility, annualised. A simple relationship connects it to time: the t-statistic of a strategy's average return equals its annual Sharpe ratio times the square root of the number of years observed. Setting that to 2 gives the years needed as (2 divided by the Sharpe ratio) squared.

Two consequences follow. First, a six-month live record cannot separate a good EA from a lucky one unless its Sharpe ratio is above 4, which should itself be a red flag; the better explanation for a very high short-term Sharpe is luck, a hidden tail risk such as martingale or an unrealised floating loss, or selection of the account being shown. Second, a vendor who says an EA has run for three years at a Sharpe of 1.0 has shown a result that is still under the usual evidence bar. That does not mean it is worthless. It means you must lean on other evidence too: the logic of the strategy, the out-of-sample behaviour and the size of the sample in trades.

The win rate is the noisiest number on the page

Vendors love win rates because they are easy to read and easy to inflate. A win rate is also one of the least informative numbers, for two reasons. It ignores the size of wins and losses, so a 90% win rate with small winners and rare large losers can have a negative expectancy. And it is itself noisy: the 95% range around an observed win rate of 55% runs from about 37% to 73% on 30 trades, 45% to 65% on 100 trades, 51% to 59% on 500 trades and 53% to 57% on 2,000 trades, using the standard binomial approximation. A reported win rate on a short sample says little. Ask for the average winner, the average loser and the number of trades, and compute the expectancy yourself with the calculator above.

The search problem: what optimising does to a result

Now the second threat. Suppose a developer builds a strategy with a handful of adjustable inputs, tests many combinations on the same five years of data and publishes the best. Even if every combination is worthless, the best of many worthless results will look good. This is not fraud. It is the arithmetic of the maximum of many random numbers, and it is the reason a strategy 'optimised' on history is expected to do worse afterwards.

To see the size of the effect, take zero-edge strategies whose annual Sharpe ratio on any sample is pure noise. With N independent tries, the expected best in-sample Sharpe is the expected maximum of N draws from a normal distribution divided by the square root of the years in the sample. The table shows what that produces. We computed it analytically and checked it by simulation: the simulated best-of-45 zero-edge strategy on five years had an average in-sample Sharpe of 0.99, best-of-100 gave 1.12, best-of-10 gave 0.70, matching the table.

A Sharpe ratio of about 1.0 is what an investor would call a good strategy, and the table says you can find one from 45 tries on five years of pure noise. This is the result Bailey, Borwein, López de Prado and Zhu stated in their work on backtest overfitting: testing more than about 45 variations of a strategy on a five-year sample is almost guaranteed to find one with an in-sample Sharpe ratio near 1 and an out-of-sample expectation of zero. Their point was that most published backtests do not disclose how many trials were run, so a reader cannot tell. The same logic explains why long samples help: a decade of data pulls the same 45 tries back to 0.70.

How many tries is an ordinary optimisation?

Many more than 45. A modest EA has five optimisable inputs, and the Strategy Tester will happily run ten values of each. That is ten to the fifth power, or 100,000 combinations, and the last row of the table shows what the best of 100,000 noise strategies looks like on five years: an in-sample Sharpe near 2.0, which would be spectacular if it were real. Genetic optimisation does not escape the problem. It visits fewer points, but it picks the best of those it visits, and the more generations you run the closer you get to the same maximum. The guide on optimisation without overfitting and the shorter overfitting guide show what to do about it in practice.

Statisticians handle this with a stricter threshold. If you test N hypotheses and want the same 5% overall chance of a false positive, the simplest correction (Bonferroni) raises the required t-statistic from about 1.96 for one test to about 2.81 for 10, 3.48 for 100, 4.06 for 1,000, 4.56 for 10,000 and 5.03 for 100,000. Look back at the sample-size table. A +0.10R system needs about 900 trades to reach t = 3 and far more to reach t = 4. A backtest produced by a thousand-combination search that shows a t-statistic of 3 has not cleared the corrected bar. This is why the best defence is not a more careful look at the winner, but evidence the search never saw.

Evidence the search never touched

A result is trustworthy to the extent that it comes from data that was not used to choose the strategy. There are four layers, in rising order of strength.

The guides on walk-forward analysis in MT5 and forward testing on demo show how to do the third and fourth layers. For a buyer, the practical advice is to give most weight to the last layer and to ask any vendor for a record that began after the product was frozen. Our guide on verifying an EA track record lists the account types and badges worth trusting.

Data quality: the problem that is not statistical

Statistics assume the data are right. In MetaTrader 5 they are right only if the tester used good price history. The platform's own documentation separates real ticks, recorded from the broker, from generated ticks, which the tester builds from bar data, and explains how missing history is downloaded. Strategies that depend on precise entries, short targets or spread conditions can look far better on generated ticks than they would have in life, and a high 'history quality' percentage in the report does not by itself say the ticks were real. The guide on historical tick data for forex in MT5 covers the sources, and the guide on how backtests can be manipulated lists the settings that flatter a result without any dishonesty in the code.

Costs are the second silent assumption. A backtest with a fixed low spread and no commission, run on a strategy that trades often, can turn a marginal loser into a winner on paper. For a system earning +0.10R per trade, an extra cost of a tenth of a stop distance removes the entire edge. Always re-run the test with a spread and commission at least as bad as your broker's worst regular conditions, and compare against the live record when you have one, as our guide on backtest versus live results explains.

Markets change: why an edge is not a constant

Everything above assumes the process generating trades stays the same. Markets do not oblige. A trend-following EA does well in sustained moves and poorly in ranges, a mean-reversion EA the reverse, and the mix of conditions in the test period decides which one looks good. A ten-year test that happened to contain two strong trending years and eight quiet ones tells you about that mix. Look at the year-by-year or month-by-month results rather than the total: a strategy whose entire profit comes from one or two periods is concentrated risk, not a smooth edge. Our guide on adaptive trading strategies discusses EAs that try to respond to regime change.

Edges also decay. A pattern that was real can be competed away, an exchange or broker can change execution, and volatility regimes shift. That is a reason for the monthly review in the first 30 days guide and for the sizing limits in our guide on risk of ruin and position sizing. Statistics can say that a result is unlikely to be luck; they cannot say it will last.

What the headline numbers in a tester report really tell you

The MetaTrader 5 Strategy Tester report offers a column of statistics, and each has a statistical meaning worth knowing before you lean on it. The platform's help explains the fields; what follows is how to read them with sample size in mind.

Take profit factor, the number sales pages like best. A profit factor of 1.5 on 150 trades can result from one or two outsized winners. Remove the best five trades and recompute: if it falls close to 1.0, the edge was those trades. A genuine edge usually survives that test, because it is spread across many trades. The guide on reading the MT5 backtest report walks through every field, and the expectancy calculator earlier on this page shows the arithmetic behind them.

Trades are not independent: the effective sample is smaller

All the tables assume each trade is a fresh draw. EAs break that assumption in several ways. A grid or basket EA opens many positions that all depend on the same price move, so ten trades may be one event. A trend EA that re-enters in the same direction after each stop is exposed to the same trend ten times. Two trades opened in the same minute on correlated symbols share most of their outcome. Counting each as a separate observation overstates the sample.

A rough remedy is to count independent events instead of trades: for a basket, count baskets; for a trend system, count trends or weeks rather than orders; for a multi-symbol EA, group by day. If the independent count is a tenth of the trade count, divide the t-statistics in the tables by about the square root of ten, roughly 3.2. That can turn a convincing 3.0 into a weak 0.95. It is also a reason to prefer EAs that trade a single idea at a time with a clear stop, whose trade count means what it says, and to be careful about those whose large trade counts come from many overlapping positions. The guide on grid trading shows how a grid's results cluster into events.

Serial correlation causes a related problem. If winners tend to follow winners, which happens when a strategy rides a regime, the effective sample is smaller and drawdowns are deeper than independent-trade formulas suggest. The Monte Carlo analysis described in our risk of ruin guide can be run on blocks of consecutive trades instead of single trades to keep that clustering, and gives a more honest picture of drawdown risk.

Look at rolling windows, not only the total

A single total hides how results were distributed. Split the history into equal windows, such as quarters, and compute the average trade and the drawdown in each. A genuine edge usually shows up as positive in most windows with a few losing ones. A result carried by one or two windows, with the rest flat or negative, is concentrated and is the signature of a regime-dependent system or a lucky stretch. A useful rule of thumb is to ask how many windows are profitable and whether removing the best window turns the total negative. If it does, the total is one event, not a record.

A protocol for testing an EA in MT5 without fooling yourself

1. 1

   Step 1: Fix the rules before you look

   Write down the symbol, timeframe, the inputs you will allow to vary and the single measure you will judge on. Anything chosen after seeing results is part of the search.

2. 2

   Step 2: Split the history once, in advance

   Reserve the most recent third of the data as a hold-out and do not open it until the settings are frozen. Looking twice turns the hold-out into training data.

3. 3

   Step 3: Count your tries

   Keep a log of every optimisation run and how many combinations it covered. The number matters more than anything else you will write down, because it sets the bar the result must clear.

4. 4

   Step 4: Test costs at their worst regular level

   Use real ticks, your broker's commission and a spread at the upper end of its normal range. A system that only works at the best spread is not a system.

5. 5

   Step 5: Check stability, not just the peak

   Move each input up and down by 10% to 20% and re-run. Prefer a broad plateau of acceptable results to a single sharp peak.

6. 6

   Step 6: Open the hold-out once

   If the result there is far worse than in-sample, you have measured the overfitting. If it is similar, you have your first piece of real evidence.

7. 7

   Step 7: Forward test and scale slowly

   Run on demo, then at the smallest live size, and compare each month with the backtest. Raise the size only as the live sample grows, as the scaling ladder in the risk guide describes.

Following the protocol will not make an EA profitable, and it will usually make the first backtest look worse, which is the point. What it gives you is a number you can believe, with a record of how it was produced. That is rarer than a spectacular chart and far more useful when real money is on the line.

Twelve questions to put to any EA vendor

A vendor who can answer most of these has either done the work or knows what the work is. A vendor who answers with a better-looking chart has not. None of the questions require access to the source code, and all of them can be asked politely before payment. Pair them with the EA buying checklist and the warning signs in our scam guide.

A worked example: reading two sales pages

Take two imaginary EAs, described below with round numbers, and apply what you have learned. The figures are invented for the exercise and do not describe any product.

EA A looks stronger on the headline: a higher average per trade, a higher win rate and ten years of history. Apply the tests. With 260 trades and an average of +0.12R, the t-statistic is roughly 0.12 divided by about 1.2 times the square root of 260, which is 1.6 and short of the bar even before correcting for a search of probably tens of thousands of combinations. There is no out-of-sample result and the live record is six weeks, which is a handful of trades. EA B has a smaller edge, but 2,400 trades give a t-statistic near 2.9 on the same arithmetic, the search was small, a hold-out period supports it and a long live record with a named broker exists. EA B is by far the better-supported claim even though its sales page is less exciting. Whether it is worth buying also depends on costs, drawdown and fit with your account, but the evidence for it is not in question and the evidence for A is.

Mistakes to avoid when you test an EA yourself

• Re-optimising until the out-of-sample period also looks good. Each peek spends the sample. Decide the hold-out in advance and look once.
• Judging by the equity curve instead of the number of trades. A curve from 40 trades is a drawing, not a result.
• Optimising for net profit. It rewards a few lucky large trades. Prefer a measure that includes risk and trade count, and check stability across neighbouring settings.
• Accepting a sharp optimum. If a small change in an input collapses the result, the optimum is probably noise. A broad plateau of good results is more believable.
• Forgetting that spreads vary. Test with the worst regular spread at your broker, not the best.
• Mixing in-sample and live trades in one number. Report them separately.
• Stopping the forward test when it disappoints. Decide the length and the pass mark first.

What to take away

Evidence for an EA is a matter of degree, and you can measure how much you have. Count the trades, convert the edge into a t-statistic, ask how many tries produced the winner, and look for results that the search never saw. A modest edge needs hundreds of trades, a strong Sharpe needs years, an optimiser inflates what it finds, and a vendor's willingness to answer plain questions is itself a data point. None of this tells you whether an EA will make money. It tells you whether the evidence offered is strong enough to risk money on, and it lets you size your risk to match: small while the evidence is thin, larger only as live results, after costs, accumulate in line with the claim.