What the test is for
Suppose you have two simple ways to play a game and want to know which tends to do better. Playing a handful of real games will not settle it, because luck dominates small samples. A simulation lets you play both approaches thousands of times against opponents you control. The most basic opponent to control is one that chooses at random.
The method is useful for learning and for design work, but it is easy to set up unfairly. Most of this article is about avoiding that.
Step 1: define the game and the metric
Write down, in rules a computer could follow, exactly what the game is and what you are measuring. Typical metrics include:
- Win rate: wins divided by games played.
- Average score: useful if margins matter, not just who won.
- Average game length: useful if speed is the point.
Pick one main metric before you run anything. If you check five metrics afterwards and report the one that looks best, you are selecting luck. That is the mechanism behind many cherry-picked statistics.
Step 2: define each strategy as a precise rule
A strategy must be something you can code without judgment calls. “Hold when ahead” is vague. “Stop rolling once the turn total reaches 15” is precise. Keep strategies simple at first, with one parameter each, so you can see what is driving any difference.
Step 3: use the same randomness for both
The fairest comparison runs both strategies through the same random events. If you are comparing two ways of playing a dice game, feed each strategy the same sequence of rolls in each trial, or reuse the same shuffled decks. This is called pairing, or using common random numbers. It cancels out much of the luck, so a smaller number of trials can reveal a real difference.
Without pairing, strategy A may simply have been given luckier dice than strategy B in your run. With pairing, any difference per trial comes from the decisions, not the draws. Some games make exact pairing awkward because choices change which random events occur; in that case, give both strategies the same random seed per trial and add more trials to compensate.
Step 4: seat strategies symmetrically
If the game has a first-player advantage, always putting strategy A first biases the result. Alternate the seats, so each strategy plays half its games first and half second. The same principle applies to any asymmetry: board positions, starting hands, colors.
Step 5: choose the opponent carefully
A random opponent picks legal moves without any plan. It is a useful floor, because a strategy that cannot beat random play has a problem. But it has weaknesses as a yardstick:
- It is easy to beat. Nearly any sensible strategy wins against random play, so the test may show little difference between a good plan and a mediocre one.
- It may not reward the right things. A strategy tuned against random mistakes can do worse against a thoughtful opponent. The bigger question of how opponents respond is the territory of game theory.
- Sometimes it hides everything. Consider rock-paper-scissors. Against an opponent choosing uniformly at random, every strategy you can name wins one third of the time, loses one third and ties one third, because the opponent’s choice is independent of yours. A simulation will show roughly equal win rates for all strategies, which is correct and shows that no plan helps against a perfectly random player. Only a predictable opponent gives a strategy something to exploit, which is where studying opponent patterns comes in.
A reasonable test ladder is: random opponent as a baseline, then a simple fixed-rule opponent, then another strategy you are comparing.
Step 6: run enough trials and report the uncertainty
A win rate from a simulation is an estimate. For a win rate near 50 percent, 1,000 games gives a margin of roughly plus or minus 3 percentage points; 10,000 games gives roughly plus or minus 1 point. If strategy A wins 51 percent and strategy B 49 percent over 1,000 games each, the gap is inside the noise and you cannot say which is better. Increase trials until the confidence intervals for the two win rates stop overlapping, or until you accept that the difference is too small to matter. The question of how large a sample size is enough has a short rule: halving the margin needs about four times as many games.
With paired comparisons, look at the difference in outcomes per trial and its margin of error, rather than comparing two separate rates.
Step 7: look past the average
Two strategies can share an average but differ in how widely results swing. A cautious plan might win 50 percent of the time with small margins, while an aggressive one wins 50 percent with big swings. Record variability as well, for example the spread of scores. Volatile approaches are riskier to rely on, which matters whenever something is at stake.
What a simulated test does and does not prove
A well-built test tells you how a strategy behaves in the simulated game against the chosen opponents. It does not show that the same approach would work against real opponents who adapt, and it cannot be treated as a guaranteed way to win anything. A strategy that looks good in simulation may exploit a quirk in your model.
This applies with extra force to gambling. For casino games where the rules give the house an edge, no strategy of choosing among the bets removes that edge, and a simulation that appears to show otherwise should be treated as a sign of an error in the code. Past simulated results do not change the odds of real future rounds, and gambling can cause real harm. Anyone struggling should contact a local support service.
A short checklist
- State the game, the metric and the strategies precisely.
- Use identical random draws for compared strategies.
- Alternate seats or positions.
- Include a random baseline, but do not rely on it alone.
- Run enough trials to separate the strategies, and say how uncertain the gap is.
- Check the code on a case with a known answer before trusting it.
- Describe the result as “in this model,” not as a proven truth.
Handled this way, a simulated comparison is a disciplined way to learn how simple strategies behave, and a strong guard against the human habit of crediting skill for what luck produced.