It’s 2:47 a.m. The gateway latency on the payments service has tripled, and error rates to downstream services are climbing.
This is exactly the kind of incident I have been building an agent for.
The agent starts working before the on-call engineer has even pieced together what changed. It pulls the triggering monitor and relevant traces from Datadog, searches Jira for related incidents, tickets, and change requests, looks through GitHub for recent merges and deployments, correlates the timelines across systems, and produces a draft root cause analysis, ideally within 10 minutes.
The interesting part isn’t getting an LLM to summarize a few logs. The hard part is getting an agent to investigate an incident across multiple systems, services, databases, and pipelines, build a consistent timeline, distinguish correlation from causation, and produce an RCA that an engineer can actually trust.
In early testing, the agent looked great. It found a PR that had removed a database index and had been merged roughly 40 minutes before the latency spike. The evidence lined up, the timeline made sense, and the RCA was directionally correct.
As the agent moved from a demo into real incident investigation and started looking at my team’s inbound, outbound and inventory services, the focus shifted from can it produce an RCA? to a more important question: how do we know the RCA is accurate, efficient, and consistently useful?
An incident RCA agent can produce a convincing explanation, but that explanation needs to be grounded in evidence. It also needs to reach the right conclusion quickly, without burning through hundreds of tool calls or requiring an engineer to repeat the investigation manually.
That made evaluation a first-class part of building the agent.
I started thinking about incident investigation as an evaluation problem. Given an incident, can the agent identify the relevant signals across multiple systems, reconstruct the timeline, connect the evidence, identify the most likely root cause, and produce a useful RCA. All within a reasonable time and tool budget?
This post walks through how I approached evaluating an incident RCA agent. What makes the problem uniquely difficult, which dimensions actually matter, how to grade the quality and correctness of an RCA, how to build an evaluation harness from historical incidents, and the domain-specific pitfalls you need to account for when an agent is investigating across Datadog, Jira, GitHub, and other production systems.
Why an RCA agent is hard to evaluate
Evaluating an RCA agent is fundamentally different from evaluating a model that generates a single response.
The agent is investigating a live, distributed system. It has to navigate multiple sources of evidence, decide what to investigate next, connect signals across systems, and eventually produce a conclusion that is both correct and actionable. There is rarely a single sequence of steps that defines a good investigation.
That creates several challenges.
The investigation is a multi-step reasoning process. A single RCA can involve dozens of tool calls across Datadog, Jira, GitHub, logs, traces, and deployment systems. The agent needs to continuously update its hypothesis as it gathers evidence. A poor early assumption, for example, starting with the wrong service or time window, can send the rest of the investigation in the wrong direction.
There isn’t a single correct investigation path. One agent might start from the alert and follow traces downstream. Another might start by looking at recent deployments and work backward. A third might correlate error-rate changes with infrastructure events first. If all three arrive at the same root cause with sufficient evidence, the evaluation should consider all three successful. The goal is to evaluate the quality of the investigation, not whether the agent followed a predetermined sequence of queries.
The hardest part is separating cause from correlation. Production incidents generate a lot of coincidental signals. A deployment may happen shortly before an incident without causing it. A downstream service may show errors because its upstream dependency is failing. An error that appears in the logs may simply be a symptom of the underlying issue. A good RCA needs to establish a causal chain from evidence rather than simply finding something that changed around the same time.
Historical incidents aren’t static. The evidence available during an incident changes over time. Metrics roll out of retention, logs disappear, Jira tickets get updated, and GitHub PRs get merged or reverted. Running an agent against today’s production environment doesn’t necessarily reproduce what an engineer could have known during the original incident. A meaningful evaluation therefore needs to preserve the state and evidence available at the time of the incident.
The agent operates across system boundaries. The answer rarely exists in one system. The signal might start in Datadog, the relevant change might be in GitHub, the context might be in Jira, or your company’s Confluence pages and the confirmation might come from traces or deployment history. Evaluating each tool call independently misses the real question: did the agent successfully connect evidence across systems to explain what happened?
Speed and efficiency are part of the outcome. An RCA delivered in 8 minutes during an active incident is materially more useful than the same RCA delivered in 45 minutes. The number of tool calls, investigation duration, and compute or API cost therefore matter alongside correctness. The goal isn’t simply to find the answer. It’s to find it quickly, efficiently, and with enough evidence to trust it.
This means a useful evaluation cannot be reduced to “did the agent get the right answer?” We need to evaluate the entire investigation:
what evidence it found?
how it reasoned over that evidence?
whether the final RCA is correct?
how quickly it arrived there?
how efficiently it used the available tools?
What to measure
For an incident RCA agent, the most important question is straightforward:
Did it identify the true root cause?
But correctness alone isn’t enough. An RCA is something an engineer may use to make decisions during an active incident, so we also need to know whether the conclusion is supported by evidence, whether the investigation was efficient, and whether the agent knows when it doesn’t have enough information to reach a conclusion.
I think about the evaluation across six dimensions:
Root cause correctness. Did the agent identify the actual causal change, component, configuration, feature flag, dependency, or other underlying cause? A symptom or downstream failure should not receive the same score as the actual cause, although partial credit can be useful when the agent gets close to the causal chain.
Evidence quality. Can every important claim be traced back to evidence the agent actually found? That could be a Datadog query and time range, a trace ID, a log entry, a Jira ticket, a deployment record, or a Git commit. More importantly, does the evidence actually support the conclusion? Finding a deployment five minutes before an incident isn’t evidence that the deployment caused it.
Timeline accuracy. Can the agent reconstruct what happened and when? The evaluation should capture things like the first abnormal signal, when the suspected change occurred, when the impact started, and how the blast radius evolved. A correct root cause with an incorrect timeline can still lead engineers to the wrong conclusion.
Precision of the RCA. An RCA shouldn’t become a list of everything that changed around the incident. If the agent produces six possible causes and one happens to be correct, that isn’t equivalent to identifying the correct cause with supporting evidence. Extra unsupported suspects should be penalized.
Calibration. The agent should understand the boundary between evidence-backed conclusion and plausible hypothesis. When the available evidence isn’t sufficient, saying “inconclusive” is better than confidently blaming the nearest deployment. This is especially important for an agent operating during production incidents, where false confidence can be more damaging than uncertainty.
Investigation efficiency. How long did it take to reach the RCA? How many tool calls did it make? How much context and compute did it consume? An agent that reaches the correct answer in 8 minutes with 20 targeted queries is fundamentally more useful than one that reaches the same answer in 40 minutes after 200 queries.
There is also a seventh dimension that cuts across all of these and that is safety.
The agent should operate within clearly defined boundaries. It shouldn’t modify production systems, create or revert changes, or write conclusions into incident tooling without explicit authorization. It also shouldn’t fabricate evidence, links, commit SHAs, or incident references, or leak sensitive information it encountered during the investigation.
The resulting scorecard is therefore less about a single “accuracy” number and more about whether the agent can produce an RCA that an engineer can trust, verify, and act on.
Reliability matters more than the best run
There is another subtlety when evaluating agents. One successful run isn’t enough.
An agent might find the correct RCA for an incident in one run and take a completely different path in the next. If we’re evaluating an interactive assistant where a human can choose the best result from several attempts, that might be acceptable. For an incident agent that is expected to produce a trustworthy RCA automatically, it isn’t.
This is where pass@k and pass^k become useful.
pass@k asks: Did at least one of k runs produce a correct RCA?
pass^k asks: Did every one of k runs produce a correct RCA?
The distinction matters because the first measures best-case capability, while the second measures repeatable reliability.
If a single run has a success probability of p, then:
pass@k = 1 − (1 − p)^k
while:
pass^k = p^k
For example, if the agent identifies the correct root cause 80% of the time, five independent runs give a pass@5 of roughly 99.97%. That sounds excellent, but pass^5 is only about 32.8%.
In other words, running the same incident five times gives you a very high probability that at least one run will be right, but only about a one-in-three chance that all five runs will be right.
For an incident RCA agent, that distinction is critical. We’re not trying to demonstrate that the agent can solve an incident. We’re trying to establish that it can reliably solve incidents well enough to be trusted during the moments when engineers need it most.
How to grade an RCA
Use the cheapest trustworthy grader for each part of the answer. For an RCA agent, that usually means layering all four.
Code-based checks for the structured parts. Have the agent emit a structured block alongside its prose, root-cause type, the entity (service, commit SHA, flag name), and a list of evidence links. Then check it deterministically. Does the SHA match the answer key? Does every Jira key, PR number, and Datadog link resolve? Were all tool calls read-only?
Ground-truth comparison against the postmortem. For past incidents, your written postmortem is the answer key. Match on the causal entity, not keywords: “the deploy” doesn’t count unless it’s the right deploy.
LLM-as-judge for the reasoning. A judge or a verifier agent with a rubric can read the full transcript and interactions with multiple tools and decide whether the causal chain holds, whether the evidence supports each step, and whether red herrings were ruled out.
Human SRE review for calibration. Have engineers who worked the incident grade a sample of RCAs. Their agreement with the judge tells you how far to trust it.
Making the verifier trustworthy
Ask specific questions. “Does the RCA identify the index-removal PR as the cause?” and “Does it explain why the concurrent CDN change is unrelated?” beat “Rate this RCA 1 to 10.”
Give the judge the evidence. Include the reference postmortem and the raw tool outputs, not just the agent’s summary, or it will grade the agent’s confidence rather than its correctness.
Calibrate. Label 50 to 100 RCAs by hand before trusting the judge, and re-check whenever you change the judge model or prompt.
Allow “cannot determine.” A forced verdict on missing evidence is noise.
Grade the evidence, not the story
RCA agents are fluent writers. “Error rates rose after PR #4821 introduced an N+1 query” reads beautifully whether or not that PR exists, touched that service, or merged before the spike. Check every cited artifact against its source system, and have it check the timeline, a cause can’t come after its effect.
Building the harness: replay your past incidents
The best eval set for an incident agent is your own incident history. Every past incident with a written postmortem is a ready-made task. The alert that fired, the data that existed at the time, and a known root cause.
Each trial replays one incident frozen at the moment the alert fired. The agent queries Datadog, Jira and GitHub through the same tool interfaces it uses in production, but every response comes from a snapshot. Graders compare its RCA with the postmortem, and misses from production become new tasks.
Freeze time. Tools may only return data from before the alert, plus the investigation window. No resolved Jira tickets, no revert PR, no postmortem doc. This is the most important rule in the harness.
Snapshot, don’t query live. Record the Datadog metrics, logs and traces for the incident window, export the relevant Jira issues as they stood, and pin a GitHub mirror to the incident’s commit. Datadog retention will delete the originals eventually anyway.
Keep the real tool interfaces. The agent should call the same functions with the same arguments it uses in production; only the backend changes. Otherwise you’re evaluating a different agent.
Include the noise. Snapshot the unrelated deploys, flaky alerts, and open tickets from the same window. Clean data makes every incident look easy.
Generate fresh incidents. Past incidents run out, and your prompt tuning slowly memorizes them. Inject faults into staging (a bad config, a slow dependency, a flipped feature flag) to create new incidents with known causes.
Read-only by default. Stub every write action (Jira comments, PRs, monitor mutes) and log the attempts, so a bad run can’t touch real systems and you can still grade whether it tried.
Budgets and full transcripts. Cap tool calls and wall-clock time, and log every query and response so you can see where an investigation went wrong.
Pitfalls specific to incident agents
Most bad RCA scores come from the eval, not the agent. Check for these before you trust a number.
Future leakage. The resolved Jira ticket, the revert PR, a comment saying “it was the index change”: if any of these reach the agent, it’s reading the answer. Audit what every tool can return for each replayed incident.
Wrong or incomplete answer keys. Postmortems are written by tired humans and sometimes name a contributing factor rather than the trigger. Have an engineer re-validate each answer key before it enters the suite.
Keyword grading. Checking whether the RCA mentions “deploy” or “database” rewards vague answers. Grade the specific entity.
Rewarding the shotgun. An agent that lists every change in the window will always “include” the root cause. Measure precision as well as recall.
Hallucinated evidence. Plausible PR numbers, invented log lines, Datadog links to the wrong time range. Resolve every citation automatically.
Harness failures scored as agent failures. A missing snapshot, an expired API token, a rate-limited mock. Tag infrastructure errors separately and rerun them.
Overfitting to famous incidents. The big outage everyone remembers quietly shapes the prompt. Keep a held-out set of incidents nobody tunes against.
Ignoring variance. On a 30-incident suite, one incident flipping moves the score by more than three points. Run several trials and check pass^k before celebrating.
A practical playbook
You don’t need a thousand incidents to start. Twenty real ones, graded honestly, beat a sprawling synthetic suite nobody trusts.
Collect 20 to 30 past incidents with solid postmortems. Favor variety: bad deploys, config changes, dependency failures, capacity problems, and a few with a misleading change in the same window.
Write the answer key for each. The root-cause entity, the key evidence, the timeline, and the red herrings a good RCA should rule out.
Snapshot Datadog, Jira and GitHub as they stood when the alert fired, and verify nothing from after the resolution slipped in.
Validate every task. A senior SRE given only the snapshot should be able to reach the answer. If they can’t, fix the snapshot or drop the incident.
Run each incident three to five times and report root-cause accuracy, pass^k, evidence validity, and minutes to RCA.
Read transcripts weekly, especially passes. The agent may be right for the wrong reason.
Run the suite on every prompt, tool, or model change, and treat a regression like a failing test.
Shadow production. Let the agent draft RCAs on live incidents without posting them, compare with what the humans found, and turn every miss into a new eval task.
Keep the suite fresh. Add injected-fault incidents as your history runs thin, and move incidents the agent solves every time into a regression set.
Closing thoughts
An incident agent earns trust one RCA at a time, and on-call engineers remember the one that sent them chasing the wrong deploy at 3 a.m. Evals are how you find those failures before your engineers do.
Build the suite from your own incidents, freeze time so the agent can’t peek at the answer, grade the causal entity and the evidence rather than the prose, and measure whether it’s right every time, not just once.