Proving a root cause before reporting it

When something breaks in a system of microservices, the first stretch of an incident is spent jumping between dashboards, trying to work out which service actually failed. And the loudest symptom is usually not the cause. When the payment service breaks, checkout and the front end light up too, because they're waiting on it.

A confident wrong diagnosis is expensive. It sends people to fix the wrong thing while the outage keeps going. So when I built RootCause, an on-call assistant, I wanted its answers to be checkable, not just plausible.

An answer key

You can't score a diagnosis without knowing the right answer. The OpenTelemetry Demo, a sample online store made of many microservices, ships with failure switches like paymentFailure and emailMemoryLeak. Flip one, and you know exactly what broke.

I built a simulator of a 15-service store that follows the demo's shape and uses 10 of its failure switches. That gave me incidents with a known culprit, so I could measure top-1 accuracy: did the agent name the right service and the right kind of failure?

How the agent works

RootCause is a LangGraph agent, and it is strictly read-only. It never changes anything; a person acts on what it finds.

1. Detect: a forecast band on each metric flags when something drifts outside normal. No AI here.
2. Triage: rank the suspect services.
3. Gather evidence: query metrics, traces and logs for the top suspects.
4. Check the runbook: retrieve the matching runbook notes.
5. Hypothesise: propose a root cause.
6. Verify: prove it, or loop back.

The verifier re-queries

The verifier doesn't just reread the evidence the agent already used. It asks fresh, independent questions: a later time window, with stricter thresholds. For a crash, for example, the suspect needs at least 10% errors on its own spans (at least 3 spans), in a window that adds 4 fresh minutes the evidence step never saw.

If the hypothesis survives, RootCause reports it. If it doesn't, the agent widens its search and tries again, up to 3 cycles. After that, it reports "inconclusive" instead of guessing.

The red herring

My simulator includes a deliberately noisy currency service that logs ERROR lines all the time, whether anything is wrong or not. I added it on purpose, because a noisy-but-healthy service is exactly what fools a quick look at the logs.

I compared RootCause against a naive baseline: blame the service with the most ERROR lines. The baseline blamed currency in 28 of 30 incidents and got 6.7% right. RootCause got 96.7% (29 of 30, on simulated data with a rule-based stand-in for the LLM), because it looks at signals that separate a culprit from its victims: errors on a service's own spans, its own processing time, and its CPU and memory use.

These numbers are on simulated data, with a rule-based stand-in where the language model would go (more under Honest limits). Over 5 random seeds (150 incidents and 50 healthy runs), RootCause scored 98.0%. When it confirmed an answer, it was right every time (147 of 147), and it raised 0 false alarms on the healthy runs.

"Inconclusive" is a feature

All of the misses were the same case: a weak payment failure (paymentFailure at low intensity). The verifier couldn't find enough errors in the fresh window, so the agent said "inconclusive". It never gave a wrong answer with confidence.

I built it that way on purpose. It costs some recall, but on call, "I can't prove this yet" is far more useful than a confident wrong answer.

The fancy model didn't win

I also tried a real pretrained forecasting model (Chronos-Bolt, the tiny version) in place of my simple robust linear forecaster. On this data it made no measurable difference (98.0% either way) and was about 7 times slower. So the simple model stayed as the default.

That says more about my simulated data, which is close to linear plus noise, than about Chronos. On real, seasonal production metrics a learned forecaster may well win. But I only know which one to use because I measured both.

Honest limits

• The data is simulated. The service names and failure switches follow the OpenTelemetry Demo, but the effect sizes and noise are my own choices. I also tuned the thresholds on this same simulator. Real telemetry will be harder.
• The reasoning step in these tests is a rule-based stand-in, not a real language model. So 96.7% is a result for the whole pipeline, not a measure of how good an LLM is. Testing against a real model is next.
• It names one culprit per incident. When two things break at once, it reliably names one real culprit and never blames a victim, but it can't report both yet.

What I took from it

Make the agent prove its answer with data it hasn't already seen. Give it a way to say "I don't know". And measure the fancy option before you adopt it.

RootCause is built with Python, LangGraph, FastAPI, React and TypeScript, with parsers for Prometheus, Jaeger and JSON-lines logs, and it has 52 automated tests in CI. The repo is private while I finish it.