In the intro post I spoke about my experiments with checks we can give AI to get better validation of our changes.
Mutation testing is a way to get confidence that all the tests our agent has created actually detect changes to the implementation. Having 20 tests on a module is great, but if they aren’t checking the behaviour properly then they’re just tech debt.
By adding mutation testing to your AI loop, you gain confidence that your test coverage is real.
How does it work?
Let’s start with an example.
I have a simple module to check an order in the market. It has some basic validation to make sure that the quantity and notional aren’t above a limit. In this system a defect means we are sending orders outside an agreed range. That can have real consequences.
# risk_limits.py
from __future__ import annotations
from dataclasses import dataclass
from decimal import Decimal
@dataclass(frozen=True)
class Decision:
accepted: bool
reason: str = ""
@classmethod
def accept(cls) -> Decision:
return cls(accepted=True)
@classmethod
def reject(cls, reason: str) -> Decision:
return cls(accepted=False, reason=reason)
MAX_ORDER_QTY = 10_000
MAX_ORDER_NOTIONAL = Decimal("1000000")
def check_order(
qty: int,
price: Decimal,
current_position: int,
position_limit: int,
) -> Decision:
"""Decide whether a single order may be sent.
qty is signed: positive to buy, negative to sell.
Quantity, notional, and position limits are inclusive.
Price must be strictly positive.
"""
if qty == 0:
return Decision.reject("qty must not be zero")
if price <= 0:
return Decision.reject("price must be positive")
if abs(qty) > MAX_ORDER_QTY:
return Decision.reject("order quantity above limit")
notional = abs(qty) * price
if notional > MAX_ORDER_NOTIONAL:
return Decision.reject("order notional above limit")
resulting_position = current_position + qty
if abs(resulting_position) > position_limit:
return Decision.reject("resulting position above limit")
return Decision.accept()
The code looks reasonable. Would most probably pass a review. Now let’s look at some generated tests.
# test_risk_limits_weak.py
from decimal import Decimal
from risk_limits import check_order
def test_accepts_a_normal_buy() -> None:
assert check_order(100, Decimal("10"), 0, 50_000).accepted
def test_accepts_a_normal_sell() -> None:
assert check_order(-100, Decimal("10"), 0, 50_000).accepted
def test_rejects_zero_quantity() -> None:
assert not check_order(0, Decimal("10"), 0, 50_000).accepted
def test_rejects_negative_price() -> None:
assert not check_order(100, Decimal("-5"), 0, 50_000).accepted
def test_rejects_an_enormous_order() -> None:
# 50,000 units against a 10,000 limit - nowhere near the boundary.
assert not check_order(50_000, Decimal("1"), 0, 10_000_000).accepted
def test_rejects_an_enormous_notional() -> None:
# 9,999 * 10,000 = 99,990,000 against a 1,000,000 limit.
assert not check_order(9_999, Decimal("10000"), 0, 10_000_000).accepted
def test_rejects_breaching_the_position_limit() -> None:
# Resulting position 9,000 against a limit of 100.
assert not check_order(9_000, Decimal("1"), 0, 100).accepted
def test_rejection_carries_a_reason() -> None:
assert check_order(0, Decimal("10"), 0, 50_000).reason
Not too long ago, if I saw eight tests against a simple module I would agree that it had great coverage. On the surface they also seem to cover all our cases.
All good?
Mutation tests
Let’s run our mutation tests with mutmut.
Mutmut is a Python mutation-testing library. It makes small changes to your implementation, runs your tests against those changes and reports which mutations weren’t detected.
Configure it in pyproject.toml:
[tool.mutmut]
source_paths = ["risk_limits.py"]
pytest_add_cli_args_test_selection = ["test_risk_limits_weak.py"]
And run it:
The important thing here is that we’re no longer just measuring which lines of code were executed. We’re checking whether the tests would notice if the implementation changed.
To do this a mutation might change a comparison, alter an arithmetic operation or change a value used by the implementation.
For example:
could become:
Or:
if abs(qty) > MAX_ORDER_QTY:
could become:
if abs(qty) >= MAX_ORDER_QTY:
If the tests still pass, the mutation has survived. That means the tests didn’t notice the change.
If a test fails, the mutation has been killed. Meaning that it caught the change.
You can inspect surviving mutations with:
This gives you an interactive view of the mutants, and you can inspect individual changes and retest them as you improve your tests.
What it found
Looking at the notional calculation:
notional = abs(qty) * price
We want buying and selling to have the same notional limit. In this case a buy or sell depends on the sign of the quantity.
If the abs() is removed, a negative quantity produces a negative notional, which never exceeds the positive notional limit. Sells would effectively stop being checked.
Where this fits in an agent loop
An AI agent can generate a set of tests, run them and report that everything passes. It can even report 100% line coverage. Adding mutation testing gives a stronger signal about what the tests actually detect.
Agents are great working to a structure. Get this in the loop and all this can happen before you even get review it!