In the intro post I spoke about my experiments with checks we can give AI to get better validation of our changes.
The previous post looked at mutation testing: deliberately changing the implementation to see whether our tests notice.
This post looks at how to solve a different problem with expect tests. What happens when the agent changes something you weren’t even thinking about testing?
This is about validating what is the full output of a module. For every change you make, you get the whole picture of what the output is.
How do you get that? Just print it out. No really, you just print it out to text. And save the result. No magic steps. That’s it.
It sounds simple when you first think about it. But it’s a contract. You can look at the output and agree that is what you want out of that module. If the agent changes the code and the output is now different, you can visually see if that change makes sense.
An example
Let’s start with a simple exposure report. We have a collection of fills and want to net them by desk and symbol.
For each combination we want:
- Net quantity.
- Gross notional traded.
- Number of fills.
We also want to see every symbol that traded, even if it ended the day with a zero net position.
# exposure_report.py
def build_report(fills: list[Fill]) -> list[Row]:
"""Net the fills by desk and symbol.
Every (desk, symbol) pair that saw a fill appears in the output,
including pairs that netted flat.
"""
net: dict[tuple[str, str], int] = defaultdict(int)
gross: dict[tuple[str, str], Decimal] = defaultdict(Decimal)
counts: dict[tuple[str, str], int] = defaultdict(int)
for fill in fills:
key = (fill.desk, fill.symbol)
net[key] += fill.qty
gross[key] += abs(fill.qty) * fill.price
counts[key] += 1
return [
Row(
desk=desk,
symbol=symbol,
net_qty=net[(desk, symbol)],
gross_notional=_money(gross[(desk, symbol)]),
fill_count=counts[(desk, symbol)],
)
for desk, symbol in sorted(net)
]
Now let’s say the agent is asked to improve the performance and simplify the report generation.
It comes back with this:
def build_report_fast(fills: list[Fill]) -> list[Row]:
...
return [
Row(
desk=desk,
symbol=symbol,
net_qty=net[(desk, symbol)],
gross_notional=Decimal(int(gross[(desk, symbol)])),
fill_count=counts[(desk, symbol)],
)
for desk, symbol in sorted(net)
if net[(desk, symbol)] != 0
]
Two changes:
- Monetary values are converted to integers, truncating rather than rounding to the expected precision.
- Rows with zero net positions are removed.
I’m sure it would look reasonable in a code review.
But now the output is wrong and it takes thinking about it to see how. Often that’s much further down the line when you are generating the report in a test environment and wonder why it’s so much shorter than the production one.
Compared with unit tests
With fine-grained unit tests, I choose which parts of the result to check. That choice depends on what I think might go wrong.
An expect test makes the whole output explicit. I can see every row, value, and total, then agree that this is the result I want. When it changes, the diff shows me what changed, including things I never thought to test individually.
A way to catch this
Let’s create a small test case with a set of fills. Definitely making money here!
# fixture.py
def sample_fills() -> list[Fill]:
return [
# Equities: a straightforward long.
Fill("equities", "ABC", 1_000, Decimal("12.50")),
Fill("equities", "ABC", 500, Decimal("12.75")),
# Equities: traded hard, ended flat.
Fill("equities", "WYZ", 4_000, Decimal("3.10")),
Fill("equities", "WYZ", -1_500, Decimal("3.15")),
Fill("equities", "WYZ", -2_500, Decimal("3.05")),
Fill("equities", "WYZ", 0, Decimal("3.05")),
# Equities: a short.
Fill("equities", "QRS", -800, Decimal("45.20")),
# Futures: a long and a partial reduction.
Fill("futures", "ESZ6", 25, Decimal("5150.25")),
Fill("futures", "ESZ6", -10, Decimal("5151.00")),
# Futures: netted flat across two fills.
Fill("futures", "NQZ6", 12, Decimal("18200.50")),
Fill("futures", "NQZ6", -12, Decimal("18205.75")),
# Options: awkward notional for rounding.
Fill("options", "ABC_C13", 300, Decimal("0.335")),
]
The important rows are WYZ and NQZ6. Both have a net position of zero, but they represent real trading activity.
What unit tests are missing
An AI agent might generate a suite like this:
# test_exposure_report_examples.py
from collections.abc import Callable
from decimal import Decimal
from typing import TYPE_CHECKING, cast
import pytest
from exposure_report import Row, build_report, build_report_fast
from fixture import sample_fills
if TYPE_CHECKING:
from exposure_report import Fill
Builder = Callable[[list["Fill"]], list[Row]]
IMPLEMENTATIONS = [build_report, build_report_fast]
@pytest.fixture(params=IMPLEMENTATIONS, ids=lambda f: f.__name__)
def build(request: pytest.FixtureRequest) -> Builder:
return cast("Builder", request.param)
def _row(rows: list[Row], desk: str, symbol: str) -> Row:
for row in rows:
if row.desk == desk and row.symbol == symbol:
return row
raise AssertionError(f"no row for {desk}/{symbol}")
def test_nets_the_long_equity_position(build: Builder) -> None:
assert _row(build(sample_fills()), "equities", "ABC").net_qty == 1_500
def test_nets_the_short_equity_position(build: Builder) -> None:
assert _row(build(sample_fills()), "equities", "QRS").net_qty == -800
def test_nets_the_futures_position(build: Builder) -> None:
assert _row(build(sample_fills()), "futures", "ESZ6").net_qty == 15
def test_counts_fills_per_symbol(build: Builder) -> None:
assert _row(build(sample_fills()), "equities", "ABC").fill_count == 2
def test_gross_notional_is_unsigned(build: Builder) -> None:
assert _row(build(sample_fills()), "equities", "QRS").gross_notional > 0
def test_rows_are_sorted_by_desk_then_symbol(build: Builder) -> None:
rows = build(sample_fills())
keys = [(row.desk, row.symbol) for row in rows]
assert keys == sorted(keys)
def test_an_empty_fill_list_produces_no_rows(build: Builder) -> None:
assert build([]) == []
That’s a lot of tests. We can even throw in some mutation testing to make sure they cover our current implementation. But in this case there is a problem with our implementation.
Now with expect tests
An expect test takes a different approach.
Instead of writing assertions for individual fields and scenarios, we print out the expected output and compare the entire result against it.
I’m using inline-snapshot for this.
An interesting part is that the expected output is stored directly in the test source. You don’t need to maintain a separate collection of snapshot files.
Let’s start with a test that renders the complete report.
# test_exposure_report_expect.py
from exposure_report import build_report, build_report_fast, render
from fixture import sample_fills
from inline_snapshot import snapshot
BUILD = build_report
def test_rendered_report() -> None:
assert render(BUILD(sample_fills())) == snapshot("""\
DESK SYMBOL NET GROSS NOTIONAL FILLS
------------------------------------------------------
equities ABC 1,500 18,875.00 2
equities QRS -800 36,160.00 1
equities WYZ 0 24,750.00 4
futures ESZ6 15 180,266.25 2
futures NQZ6 0 436,875.00 2
options ABC_C13 300 100.50 1
------------------------------------------------------
TOTAL 6 697,026.75 12\
""")
The important thing here is how the expected output gets into the test.
Initially, I write:
def test_rendered_report() -> None:
assert render(BUILD(sample_fills())) == snapshot()
Then I run:
pytest test_exposure_report_expect.py --inline-snapshot=create
The library records the actual output and writes it back into the test file.
The code above is what the test looks like after that recording step.
In this implementation the expected values become part of the test source itself.
What the expect test sees
Now set BUILD = build_report_fast and run the test without recording updates. The expected output remains the one recorded from build_report.
The output is quite different:
DESK SYMBOL NET GROSS NOTIONAL FILLS
------------------------------------------------------
-equities ABC 1,500 18,875.00 2
-equities QRS -800 36,160.00 1
-equities WYZ 0 24,750.00 4
-futures ESZ6 15 180,266.25 2
-futures NQZ6 0 436,875.00 2
-options ABC_C13 300 100.50 1
+equities ABC 1,500 18,875 2
+equities QRS -800 36,160 1
+futures ESZ6 15 180,266 2
+options ABC_C13 300 100 1
------------------------------------------------------
-TOTAL 6 697,026.75 12
+TOTAL 4 235,401.00 6
You can just read it and ask why did your output change when improving the performance. More importantly the agent can also just read it and it gets the feedback that something has broken.
Terms and conditions
One thing to get right with an AI agent.
If a snapshot changes, the tool can help you update the expected value. That’s useful when you’re deliberately changing behaviour. Unless the agent sees a snapshot failure and simply records the new output.
I want recording to be a deliberate action, not something that happens as a side effect of running tests.
In pyproject.toml:
[tool.inline-snapshot]
default-flags = ["report"]
This makes the normal non-interactive default report changes rather than automatically accepting them.
For CI, you should explicitly disable snapshot updates:
- name: Expect tests must not need rewriting
run: pytest test_exposure_report_expect.py -q --inline-snapshot=disable
Agentic loop
Adding expect tests forces your agent and you to be explicit about the changes in output of your code. It can guide it to keep on track when making wide ranging changes.