← Writing · · 4 min read

Why my AI agent isn't allowed to check its own work

In Casefile, an AI writes anti-money-laundering case notes, but plain code checks every number against the ledger before anyone sees them. Here is how it works, and the bug that got past it first.

An investigation note with one wrong amount in it is worse than no note at all. An analyst who trusts it can escalate the wrong case. An analyst who catches it stops trusting the tool.

A language model will happily write "$12,400 moved through 14 accounts" whether or not that's true. So when I built Casefile, an assistant that drafts the first pass of an anti-money-laundering (AML) investigation, I set one rule: the AI writes, but it never gets the last word on a fact.

Splitting the job

Most of an AML first pass doesn't need a language model at all. I gave each step to the simplest thing that can do it well:

  • Finding alerts and pulling transactions: SQL, through read-only, parameterized tools. The model never writes SQL.
  • Scoring how risky an alert is: a small tabular model.
  • Matching behaviour to red-flag indicators: retrieval over 9 real FINTRAC indicator sentences, which I mapped to patterns, plus a rule-based support check.
  • Writing the narrative: the language model. This is the only AI step.
  • Checking every number in that narrative: a deterministic verifier, which is plain Python.
  • Deciding: a human analyst. Casefile never files anything.

The writer produces the note one sentence at a time. Each sentence carries the transaction IDs it relies on, like a citation.

What the verifier checks

For every sentence, the verifier looks up the cited transactions in the ledger and checks:

  1. Every cited transaction exists, and every transaction ID in the text is cited.
  2. Every dollar amount equals a cited transfer, or the sum of the cited inflows or outflows.
  3. Every date is the date of a cited transfer, and "between D1 and D2" spans the cited transfers exactly.
  4. Every account is a party to a cited transfer.
  5. Every count ("12 transfers", "10 distinct senders") is right.
  6. No other number is allowed. A number the verifier doesn't recognise is rejected, not ignored.
  7. Every cited red-flag indicator really holds on those transfers. For example, "fan-in" needs at least 10 distinct senders (my threshold, not FINTRAC's).

Rule 6 matters most. A checker that ignores what it doesn't understand lets the scariest mistakes through.

When a sentence fails

A failed check isn't a dead end. The verifier sends back specific, machine-readable errors that say which check failed on which claim, such as an amount that matches no cited transfer. The agent then repairs the claim. It's a loop, built with LangGraph:

investigate -> match indicators -> write -> verify
     ^                                         |
     +--------------- errors (max 3) ----------+
                                               |
               still failing after 3 -> manual review (bad claims withheld)

After three repair attempts, the case goes to manual review, and the sentences that never passed are withheld. A person always sees which parts were proven.

Testing it by breaking it on purpose

To check that the verifier really catches mistakes, I made the writer lie. A fault-injection mode corrupts claims on purpose: wrong amounts, wrong dates, wrong accounts, wrong counts, fake transaction IDs, invented numbers, and indicators the data doesn't support.

Across two eval runs (30% and 60% fault rates):

  • 417 of 417 deliberately corrupted claims were flagged.
  • 0 of 994 clean claims were falsely flagged.

To see what the verifier is worth, I compared against the same pipeline with no verifier. At a 30% fault rate, the no-verifier version had 70% citation accuracy, and only 28% of its notes were fully correct. With the verifier, everything Casefile emitted checked out: 94% of cases passed, and 6% went to manual review.

The bug that got through first

The first version didn't score 100%. It caught only about 95% of the corrupted claims.

The gap was in date ranges. Each date in a sentence only had to match one of the cited transfers. So in "between March 3 and March 19", a wrong endpoint passed as long as it was the date of some cited transfer, even if it wasn't the first or the last one.

The fix was a stricter rule: a "between" range must span the cited transfers exactly, plus a test so it stays fixed.

The same run caught a bug in the fault injector too. One indicator it labelled "unsupported" was actually supported by the data. The verifier was right and my test label was wrong, so now the injector only picks indicators that really fail.

Honest limits

I'd rather say these myself than have someone find them:

  • The data is synthetic. I generated the ledger with planted laundering patterns, so the patterns are cleaner than real data.
  • The "AI writer" in these tests is a template stand-in, not a real language model. The numbers above measure how well the verifier catches and repairs injected mistakes. They say nothing about how often a real model gets things wrong. Testing against a real model is next.
  • "100% of what Casefile emits is correct" is partly true by construction. Casefile only emits claims that passed the verifier. The meaningful numbers are the catch rate, the false-flag rate and the gap against the baseline.
  • The verifier proves that numbers match the ledger. It doesn't prove a note is insightful, or that an indicator is legally the right one. That's the analyst's job.

What I took from it

I stopped trying to make the model careful and made the pipeline strict instead. A claim reaches a person only after code has checked it against the ledger. And when the checker has a hole, it gets a rule and a test, so it stays closed.

Casefile is built with Python, LangGraph, FastAPI, SQLite, React and TypeScript, and has 36 automated tests in CI. The repo is private while I finish it.

← All writing · Portfolio