Like everyone else, we are now using a lot of AI at work. Early on we found all the human reviewers overwhelmed. We went from a couple of deep reviews a week to a couple of really long reviews a day.
Using AI has become the standard in most workplaces, and with it has come the constant complaining about AI slop. Agents can do great things, but we’ve all also seen some real shockers.
I started a checklist of common mistakes that I looked out for in an AI review. After running through them, I would then start with obvious bugs and eventually get enough context to understand the bigger picture. Proud of my effort, we worked together to get it approved. Then, about to get back to my own work, I would look at my inbox again and there would be another one or two reviews waiting. It was relentless.
The big problem other than volume was that I spent so much time checking that it followed our rules. A human coder would build up trust over time and so I could slowly make assumptions about them doing the right thing. But each time with AI I had to check like they were new to the team!
I’m sure this sounds familiar to many people. I want to say it doesn’t have to be this way!
I wanted to write about a series of experiments I’ve run that I think will greatly reduce this issue for us. Ways to take a non-deterministic thing like an AI agent writing code and shape it so that we get a much more deterministic result.
None of these checks have been invented recently. Many people have been using some of the techniques for years. They just had an implementation cost that stopped them from being time-efficient for most teams. That has changed.
The idea is simple and there are many versions of it. How do we get the agent to do more without having to wait for a human?
A human should still read the output and own what is pushed to prod. I just don’t think we have to babysit the agent into meeting our standards and writing correct code.
This series is me exploring the checks we can add to the agent loop that it can run itself. With each check we get confidence that the result will be better, more correct code.
What makes a check worth having
What are the properties that make one of these checks worth adding?
- Deterministic
- Quick enough to run often
- Clear feedback to the agent on what failed
Nine hard checks
| Technique |
Catches |
| Property-based testing |
Agree what properties your code should have. Then throw random inputs to it to see what breaks them. i.e. have we thought of all the edge cases? |
| Mutation testing |
Having lots of tests doesn’t mean your code is covered. Automate changing the code and see if tests break. |
| Expect tests |
Lock in what the inputs and outputs of the system are. Just print them out. Is it what we expect? |
| Architecture contracts |
Can we enforce team standards in a deterministic way? |
| Linters and custom rules |
Let’s make sure the code all looks the same, and in a way we agree on. |
| Model checking/Formal methods |
Are there concurrency/ordering issues in your implementation? |
| Metamorphic testing |
How do we test something when the output isn’t a simple number? Maybe let’s run tests and check if it behaves the same way across similar inputs. |
| Observability with metrics |
When it’s running in prod or a harness, is it doing what we think it is? |
| Deterministic simulation |
Can we find rare ordering bugs and reproduce them every time? |
None of these give us everything we want but each can help in a different way. So why stop at one?
We can run as many as add value to a project as layers in the build process but more importantly as an agent hook.
Combined they can give a really strong signal on how correct your code is. The agent can test itself and only pass it back to you if it’s happy it’s correct.
Then you can spend more time thinking about if this should be done, and not whether it’s got some subtle bug.