More trials, smaller error, with diminishing returns
Every simulation estimate comes with some error, because it is built from random results. Run it again and you will get a slightly different number. The question is how different, and the answer depends on how many trials you run.
The key fact is that the typical size of the error shrinks with the square root of the number of trials. Quadruple the trials and the error roughly halves. Multiply the trials by 100 and the error shrinks by a factor of ten. Improvement never stops, but it gets expensive.
The formula, in plain words
When you estimate a probability p by counting successes in n trials, the standard error is approximately the square root of p(1 - p) / n. Roughly 95 percent of the time, a result will land within about 1.96 standard errors of the true value. That range is the basis of what statisticians call confidence intervals.
The product p(1 - p) is largest when p = 0.5, so a coin-flip-like event is the hardest case for a given n. Here are the numbers for p near 0.5.
| Trials | Standard error | Approximate 95% margin |
|---|---|---|
| 100 | 0.05 | about 9.8 percentage points |
| 1,000 | about 0.016 | about 3.1 percentage points |
| 10,000 | 0.005 | about 1 percentage point |
| 1,000,000 | 0.0005 | about 0.1 percentage point |
So 100 trials can only tell you something like “somewhere between 40 and 60 percent,” while 10,000 trials narrows it to about a percentage point either side. (The 1,000-trial row: the square root of 0.25 / 1,000 is about 0.0158, and 1.96 times that is about 0.031.)
Working backwards from the precision you want
You can also pick a target margin m and ask how many trials it needs. In the worst case of p = 0.5, the count is about 0.96 / m², since 1.96² × 0.25 is 0.9604.
- A margin of 0.01 (one percentage point): about 9,604 trials.
- A margin of 0.005: about 38,416 trials.
- A margin of 0.001: about 960,400 trials.
If you have a rough idea that p is far from 0.5, you need fewer trials, because p(1 - p) is smaller. But when you do not know p, the worst case is a safe planning figure.
Rare events need far more trials
Precision is not only about the absolute margin. If a result happens one time in a thousand, an error of one percentage point is useless, because it is ten times the quantity being measured. What matters is the error relative to p.
Take p = 0.001 and n = 10,000 trials. You expect about 10 successes. The standard error is the square root of 0.001 × 0.999 / 10,000, roughly 0.0003, which is about 32 percent of the true value. A result of 0.0007 or 0.0013 would not be surprising. To get the relative error down to about 10 percent for such an event you would need around a hundred thousand trials, since the relative error is roughly one over the square root of the expected number of successes: 100 successes gives about 10 percent.
A practical rule: aim to see at least a few dozen successes before trusting an estimate of a rare event. If your simulation shows two or three, it is mostly telling you the event is rare, not how rare.
Practical checks that do not need formulas
- Rerun with different seeds. If five runs of the same simulation give estimates of 0.51, 0.53, 0.49, 0.52 and 0.50, you can see the spread directly. If you can live with that spread, you have enough trials.
- Watch the running average. Plot or print the estimate after every thousand trials. It should wander early and then settle. If it is still drifting, keep going.
- Compare with a known answer. Test the same code on a case with an exact answer, such as a fair die, before trusting it on a harder question.
- Count successes, not just trials. The number of times the event happened is often a better guide to reliability than the total number of runs.
What more trials cannot fix
Trial count controls random error only. It does nothing about a flawed model. If the simulation shuffles incorrectly, ignores a rule, or uses a poor random-number routine, a million trials will deliver a very precise wrong number. Larger samples converge on what the code actually does, which is guaranteed to match the real game only if the code is correct.
Dependence is another issue. The formulas above assume trials are independent. If each trial reuses part of the previous one, effective sample size is smaller than it looks.
The same logic helps with real data too. Win rates from a short run of real games are noisy for the same reason, and the law of large numbers only helps once the sample is big enough. A hot streak across fifty games says little, in the same way that fifty simulation runs say little.
A simple procedure
- Decide the margin you can accept, such as plus or minus one percentage point.
- Use 0.96 / m² as a starting trial count for a mid-range probability, or use the success-count rule for rare events.
- Run the simulation several times and look at the spread.
- If the spread is too large, increase trials by a factor of four to halve it.
- Report the estimate with its range, for instance “about 52 percent, give or take one point.”
Reliability is a matter of degree. No count of trials gives certainty, but a sensible number gives an estimate whose uncertainty you can state, and that is what makes a simulation useful for understanding odds.