RootCause

Can an LLM agent find what broke, better than a simple rule?

Loading the result.

RootCause is an LLM agent that reads an incident’s metrics, traces and logs and names the service that broke. It is scored against a 28-line rule that names the service whose fault signal rose most, on incidents recorded from a 28-service demo shop, with every run shown, failures included.

What moved accuracy

Each row changes one thing and keeps the rest. Swapping the model moves accuracy most, until the false alarms on no-fault controls are counted. The tools barely matter.

Every run

One mark per run, for every incident. Open an incident to replay what the agent looked at and why it answered as it did.

Loading results.

By incident type

Does stated confidence help?

Method

How an incident is scored

  1. A fault is injected into the OpenTelemetry Demo, a shop made of 28 services, and its metrics, traces and logs are recorded. The true cause is stored separately.
  2. Anything that could give the fault away, such as feature-flag events, log wording or delay markers, is stripped, and a scan checks every recording for leaks.
  3. Each solver gets the same recording and the same alert. The rule names the service whose fault signal rose most. Agent v0 reads raw metric series, traces and logs; agent v1 adds a ranking of what changed, per-service health and a service map.
  4. Answers are scored by rules fixed before the first run. The top-ranked cause must be right; on two-fault incidents each cause found earns half. A no-fault control is right only with “no incident”.
  5. The agent’s cited evidence is checked against the telemetry: does it exist, and does it point at something unusual?

Honest limits

  • Small. Twenty incidents from one demo app, recorded on one laptop. Category results rest on two to six incidents each.
  • Generous inputs. Every solver is told when the fault began, and several alerts hint at the area of the cause. With one generic alert instead, false alarms double.
  • Free models. Gemini 3.5 Flash-Lite and Nemotron 3 Super on free tiers, at temperature 0. The spread across runs is API nondeterminism, not sampling.
  • One author. The same person built the incidents, the rule, the agent and the scoring. The methodology FAQ lists every known threat to validity.

Reproduce it

The recorded incidents and every run behind this page are in the data release. Three commands rescore the rule from the raw telemetry; the agents need a free API key.

pip install -e .
rootcause fetch-data
rootcause eval heuristic