Trusting code you didn't write

On 16 September I changed how the artificial intelligence in ExaWar behaves, the turn-based strategy game I build. Then I ran the usual batch of simulated matches to see how it had gone.

Captures were up. Games ended in fewer turns. Almost every number I normally look at had moved the right way.

Then I put the new version against the old one, face to face. It lost eight to sixteen.

The batch was not measuring strength

In a normal batch the matches are machine against machine, and both sides have the change. If I make the attack more decisive, I become more decisive on both sides: the win rate does not move, because it describes the matchup and not the change.

Meanwhile the absolute metrics can all rise together while the team gets weaker. More captures also means more units thrown away, and the shorter game is the one you lose sooner.

The only question that counts is whether the new version beats the old one, and to answer it the change has to go to one side only.

That lesson went into the project instruction file, the one the machine I work with and I both open at the start of every session: changes to AI behaviour are validated in a direct match. I did not write it down because I had forgotten it. I wrote it down because in six months I will.

This is not a story about a game’s AI

I had written that code together with a machine, which is how I work almost always now. And the question you hear around, «does AI write good code?», would have been no use to me that day.

It was good. It did exactly what I had asked for. The thing I asked for was wrong, and no careful re-read told me so: a comparison did, one I could easily have skipped.

A different question is needed. What does a project have to look like for you to accept code you didn’t write? It holds for the machine’s code, for a colleague’s, and for what you wrote yourself in March, which by now belongs to someone else.

Impossible, rather than discouraged

Advice gets forgotten. A constraint written into the code does not.

In the ExaWar engine hit points cannot go below zero: the function that removes them stops there. Resources never go negative. Action points do not either. None of those three things is left to whoever writes the next line.

When you save, the existing file is never rewritten: a temporary one is written, forced to disk, and only then takes the old one’s place. If the machine dies halfway, the good save is still there.

Inside the game there is a console with its own filesystem, which lives in memory and never touches the computer’s real files. Its language has no way to execute arbitrary text. That is not a sensible rule to respect: it is something that cannot be done in there.

In multiplayer the server decides. The client does not predict the outcome of moves, it asks for it, and every line after the handshake is signed. A modified client does not become stronger: it becomes a client saying things the server will not accept.

When you work with someone who produces far faster than you do, this is the difference between a project that holds and one that crumbles under its own weight. The machine does not need to be careful. The dangerous things need to be out of reach.

One gate

Before every commit a single command runs, putting four things in a row: the linter, parity across the five languages, validation of the campaign missions, and the test suite. A hundred and forty-one test files, one thousand six hundred and ninety tests, two minutes and sixteen seconds.

An editor hook runs the linter on every Python file as soon as it changes, so errors show up at once instead of at the end of the day.

And one line, at the top of the instruction file, which holds for me as much as for the machine: never carry on with red tests.

Two minutes and sixteen seconds is the whole price. That is less than it takes me to re-read a three hundred line diff, and it finds things re-reading would not.

Code I didn’t write does not worry me. Code nobody can reject does.