A language model will happily write "$12,400 moved through 14 accounts" whether or not that's true. So when I built Casefile, an assistant that drafts the first pass of an anti-money-laundering (AML) investigation, I set one rule: the AI writes, but it never gets the last word on a fact.

Splitting the job

Most of an AML first pass doesn't need a language model at all. I gave each step to the simplest thing that can do it well:

• Finding alerts and pulling transactions: SQL, through read-only, parameterized tools. The model never writes SQL.
• Scoring how risky an alert is: a small tabular model.
• Matching behaviour to red-flag indicators: retrieval over 9 real FINTRAC indicator sentences, which I mapped to patterns, plus a rule-based support check.
• Writing the narrative: the language model. This is the only AI step.
• Checking every number in that narrative: a deterministic verifier, which is plain Python.
• Deciding: a human analyst. Casefile never files anything.

The writer produces the note one sentence at a time. Each sentence carries the transaction IDs it relies on, like a citation.

What the verifier checks

For every sentence, the verifier looks up the cited transactions in the ledger and checks:

1. Every cited transaction exists, and every transaction ID in the text is cited.
2. Every dollar amount equals a cited transfer, or the sum of the cited inflows or outflows.
3. Every date is the date of a cited transfer, and "between D1 and D2" spans the cited transfers exactly.
4. Every account is a party to a cited transfer.
5. Every count ("12 transfers", "10 distinct senders") is right.
6. No other number is allowed. A number the verifier doesn't recognise is rejected, not ignored.
7. Every cited red-flag indicator really holds on those transfers. For example, "fan-in" needs at least 10 distinct senders (my threshold, not FINTRAC's).

Rule 6 matters most. A checker that ignores what it doesn't understand lets the scariest mistakes through.

When a sentence fails

A failed check isn't a dead end. The verifier sends back specific, machine-readable errors that say which check failed on which claim, such as an amount that matches no cited transfer. The agent then repairs the claim. It's a loop, built with LangGraph:

investigate -> match indicators -> write -> verify
     ^                                         |
     +--------------- errors (max 3) ----------+
                                               |
               still failing after 3 -> manual review (bad claims withheld)

After three repair attempts, the case goes to manual review, and the sentences that never passed are withheld. A person always sees which parts were proven.

Testing it by breaking it on purpose

To check that the verifier really catches mistakes, I made the writer lie. A fault-injection mode corrupts claims on purpose: wrong amounts, wrong dates, wrong accounts, wrong counts, fake transaction IDs, invented numbers, and indicators the data doesn't support.

Across two eval runs (30% and 60% fault rates):

• 417 of 417 deliberately corrupted claims were flagged.
• 0 of 994 clean claims were falsely flagged.

To see what the verifier is worth, I compared against the same pipeline with no verifier. At a 30% fault rate, the no-verifier version had 70% citation accuracy, and only 28% of its notes were fully correct. With the verifier, everything Casefile emitted checked out: 94% of cases passed, and 6% went to manual review.

The bug that got through first

The first version didn't score 100%. It caught only about 95% of the corrupted claims.

The gap was in date ranges. Each date in a sentence only had to match one of the cited transfers. So in "between March 3 and March 19", a wrong endpoint passed as long as it was the date of some cited transfer, even if it wasn't the first or the last one.

The fix was a stricter rule: a "between" range must span the cited transfers exactly, plus a test so it stays fixed.

The same run caught a bug in the fault injector too. One indicator it labelled "unsupported" was actually supported by the data. The verifier was right and my test label was wrong, so now the injector only picks indicators that really fail.

Honest limits

I'd rather say these myself than have someone find them:

• The data is synthetic. I generated the ledger with planted laundering patterns, so the patterns are cleaner than real data.
• The "AI writer" in these tests is a template stand-in, not a real language model. The numbers above measure how well the verifier catches and repairs injected mistakes. They say nothing about how often a real model gets things wrong. Testing against a real model is next.
• "100% of what Casefile emits is correct" is partly true by construction. Casefile only emits claims that passed the verifier. The meaningful numbers are the catch rate, the false-flag rate and the gap against the baseline.
• The verifier proves that numbers match the ledger. It doesn't prove a note is insightful, or that an indicator is legally the right one. That's the analyst's job.

What I took from it

I stopped trying to make the model careful and made the pipeline strict instead. A claim reaches a person only after code has checked it against the ledger. And when the checker has a hole, it gets a rule and a test, so it stays closed.

Casefile is built with Python, LangGraph, FastAPI, SQLite, React and TypeScript, and has 36 automated tests in CI. The repo is private while I finish it.

A confident wrong diagnosis is expensive. It sends people to fix the wrong thing while the outage keeps going. So when I built RootCause, an on-call assistant, I wanted its answers to be checkable, not just plausible.

An answer key

You can't score a diagnosis without knowing the right answer. The OpenTelemetry Demo, a sample online store made of many microservices, ships with failure switches like paymentFailure and emailMemoryLeak. Flip one, and you know exactly what broke.

I built a simulator of a 15-service store that follows the demo's shape and uses 10 of its failure switches. That gave me incidents with a known culprit, so I could measure top-1 accuracy: did the agent name the right service and the right kind of failure?

How the agent works

RootCause is a LangGraph agent, and it is strictly read-only. It never changes anything; a person acts on what it finds.

1. Detect: a forecast band on each metric flags when something drifts outside normal. No AI here.
2. Triage: rank the suspect services.
3. Gather evidence: query metrics, traces and logs for the top suspects.
4. Check the runbook: retrieve the matching runbook notes.
5. Hypothesise: propose a root cause.
6. Verify: prove it, or loop back.

The verifier re-queries

The verifier doesn't just reread the evidence the agent already used. It asks fresh, independent questions: a later time window, with stricter thresholds. For a crash, for example, the suspect needs at least 10% errors on its own spans (at least 3 spans), in a window that adds 4 fresh minutes the evidence step never saw.

If the hypothesis survives, RootCause reports it. If it doesn't, the agent widens its search and tries again, up to 3 cycles. After that, it reports "inconclusive" instead of guessing.

The red herring

My simulator includes a deliberately noisy currency service that logs ERROR lines all the time, whether anything is wrong or not. I added it on purpose, because a noisy-but-healthy service is exactly what fools a quick look at the logs.

I compared RootCause against a naive baseline: blame the service with the most ERROR lines. The baseline blamed currency in 28 of 30 incidents and got 6.7% right. RootCause got 96.7% (29 of 30, on simulated data with a rule-based stand-in for the LLM), because it looks at signals that separate a culprit from its victims: errors on a service's own spans, its own processing time, and its CPU and memory use.

These numbers are on simulated data, with a rule-based stand-in where the language model would go (more under Honest limits). Over 5 random seeds (150 incidents and 50 healthy runs), RootCause scored 98.0%. When it confirmed an answer, it was right every time (147 of 147), and it raised 0 false alarms on the healthy runs.

"Inconclusive" is a feature

All of the misses were the same case: a weak payment failure (paymentFailure at low intensity). The verifier couldn't find enough errors in the fresh window, so the agent said "inconclusive". It never gave a wrong answer with confidence.

I built it that way on purpose. It costs some recall, but on call, "I can't prove this yet" is far more useful than a confident wrong answer.

The fancy model didn't win

I also tried a real pretrained forecasting model (Chronos-Bolt, the tiny version) in place of my simple robust linear forecaster. On this data it made no measurable difference (98.0% either way) and was about 7 times slower. So the simple model stayed as the default.

That says more about my simulated data, which is close to linear plus noise, than about Chronos. On real, seasonal production metrics a learned forecaster may well win. But I only know which one to use because I measured both.

Honest limits

• The data is simulated. The service names and failure switches follow the OpenTelemetry Demo, but the effect sizes and noise are my own choices. I also tuned the thresholds on this same simulator. Real telemetry will be harder.
• The reasoning step in these tests is a rule-based stand-in, not a real language model. So 96.7% is a result for the whole pipeline, not a measure of how good an LLM is. Testing against a real model is next.
• It names one culprit per incident. When two things break at once, it reliably names one real culprit and never blames a victim, but it can't report both yet.

What I took from it

Make the agent prove its answer with data it hasn't already seen. Give it a way to say "I don't know". And measure the fancy option before you adopt it.

RootCause is built with Python, LangGraph, FastAPI, React and TypeScript, with parsers for Prometheus, Jaeger and JSON-lines logs, and it has 52 automated tests in CI. The repo is private while I finish it.