A player wrote to us about a haulage company whose corporation tax charge looked wrong in the nineteenth year of a save. Not wildly wrong. About £2,400 out on a bill of roughly £70,000, which is the kind of discrepancy that is easier to dismiss than to investigate.
It was real. It only appeared if a company had made a loss in a particular year, carried it forward, and then acquired a second business in a later year that changed its accounting period. Roughly one game in four hundred touched all three conditions, and none of them in the first decade of play.
No human tester was ever going to find that by playing. This article is about what we do instead.
You cannot test a forty year game by playing it
A Business Baron career can start in any year from 1930 and never has to end. A long playthrough is a serious commitment for a person, and a run from 1930 to 2040 takes a machine about a second.
That ratio is the entire testing strategy. Because the engine is a pure library with no graphics, no input and no clock, we can run complete games in a headless harness, thousands at a time, in parallel. A test run on the build server plays more simulated years before lunch than a person could play in a lifetime.
So we do not ask "does this look right". We ask "does this property hold in every one of these ten thousand games", and we let the machine answer.
Properties are assertions that must be true in every game
A property test does not check a specific expected output. It generates inputs, runs the system, and asserts something that must hold regardless of what was generated. When it fails, it shrinks the failing case down to the smallest version that still fails, which is usually small enough to read.
The properties we hold the simulation to look like this.
- The books balance at the close of every month of every game: assets equal liabilities plus equity, and the cash flow statement reconciles to the movement in the bank account.
- Every monetary value in every account is a whole number of pence, and no account holds a value the engine cannot explain by reference to a posted transaction.
- Tax charged in any year matches an independent recalculation from the rate table for that year, performed by different code from the code under test.
- A player who takes no actions at all can reach 2040 without the engine entering a state it considers impossible.
- Every pound that leaves one account arrives in another, so the sum of all balances moves only by the amounts the simulation says entered or left the world.
The third one is how the nineteenth-year bug was caught after it was reported. We wrote the independent recalculation, ran it over generated companies with generated histories, and the failing case shrank to a company with one loss year, one acquisition and a changed accounting period.
Property tests are unusually good value in a simulation, because the interactions are the product. Nobody writes a unit test for "loss relief interacts with an accounting period change during an acquisition", because nobody thinks of it. A generator thinks of it by accident, several thousand times an hour.
Golden saves and recorded replays catch what properties do not
A property says a rule was not broken. It does not say the game is the same game.
For that we keep golden runs. Each one is a seed plus a recorded list of decisions, taken from real play or constructed deliberately: a corner shop that never expands, a haulier that grows too fast, a company that goes into administration in 1987, a player who does nothing for seventy years.
On every build, every golden run is replayed and the resulting digest is compared with the stored one. The digest is computed over the whole state at the end of every month, folded forward, so a single pence different in a single account in a single month produces a different digest at the end.
That comparison is blunt and extremely useful. A refactor that is supposed to change nothing must produce identical digests, and if it does not we have found an accidental behaviour change in seconds rather than in a review. A deliberate change produces different digests, and then the work is to look at which runs moved, at which month they first diverged, and whether that is the change we intended.
Because the engine is deterministic and the runs are stored as decisions rather than snapshots, we can also bisect in time. When two digests diverge, a binary search over the months finds the first month where the states differ, and then a diff of the two states names the account. Debugging becomes reading rather than guessing.
When a property test finds a failing seed, that seed becomes a permanent regression run. The suite grows by accumulating the specific ways the simulation has previously been wrong.
The bugs that survive all of this
We would rather be honest about the limits.
The first category is the plausible wrong number. A test can prove that the books balance and that VAT is arithmetically correct. It cannot prove that our model of demand for a hardware shop in a market town in 1984 resembles the actual demand for a hardware shop in a market town in 1984. That is a judgement, and it is checked by reading, by comparing against what is publicly known about how those trades worked, and by playing. It is the part of the game most likely to be wrong, and it is wrong in a way no test will catch.
The second is the slow drift. A system that is correct every month but accumulates a small bias will look fine for a decade and absurd by 2035. We catch some of these with long-horizon assertions: bounds on how far a quantity may drift, checks that a market with no player in it does not spiral, checks that a competent competitor does not die of arithmetic. These bounds are themselves judgements, so they catch the obvious cases and not the subtle ones.
The third is the interface. The engine can be right and the screen can still be misleading. The rule that the front end holds no rules removes the worst version of this, where a screen computes its own answer, but it does not remove a badly chosen label.
The fourth is us. A test asserts what we thought to assert. The nineteenth-year bug existed for a while because nobody had thought of that combination, and it was found by a player with more patience than any of our generated companies.
Why we care about this
A game about money is a promise that the money is right. If the corporation tax charge in simulated 1993 is wrong, the whole exercise is decoration, and the player has no way to tell from the inside. Determinism, digests, golden replays and property tests are how we make that promise checkable rather than asserted. It is also why a bug report with a seed attached is worth more to us than almost anything else a player can send.