Fifty gigabytes of metrics, logs, and traces. Six days of them, streaming out of a running microservice system wired with OpenTelemetry instrumentation, piped through the same tools a human engineer would open at two in the morning: Prometheus for metrics, Jaeger for traces, OpenSearch through Grafana for logs. Somewhere inside all of it is the reason a customer said the checkout page felt slow. That is the setup Albert Gong, Kyuseong Choi, and six colleagues built to ask a question that a lot of engineering teams are quietly asking already: can a language model agent take an oncall shift?
The short answer from their benchmark, called ORCA-bench, is not yet, and not close.
Oncall root cause analysis is a specific kind of work, and the authors are careful to separate it from what coding models are already good at. Writing code, patching a bug, searching a repository: those tasks start from a reasonably clear statement of what is wrong. An incident does not. It starts with an ambiguous, user-facing report, often arriving hours after the underlying failure began, and the investigator has to reason across noisy signals of several different types at once while also reading the source code to understand what those signals mean.
So the team assembled 1,079 root cause analysis tasks against their live system, and they varied the tasks along axes that mirror how real incidents differ from one another. Some reports are specific, some vague. Some are caught quickly, some long after the fact. Some incidents involve a single fault, others involve faults that overlap and confuse each other. Ground-truth symptoms for each task were curated and signed off by expert site reliability engineers, the people who do this job for a living.
Grading an open-ended investigation is its own problem, and the authors addressed it by having a language model judge the agents' answers and then having humans independently re-score the same outputs. Agreement between the two came out at a weighted Cohen's kappa of 0.90, a statistic where 1.0 would mean perfect agreement and 0 would mean no better than chance. That is high, and it matters: it means the scores below are not an artifact of a lenient automated grader.
What the agents actually did
Five frontier agents ran the benchmark. On Medium-difficulty tasks, which the authors describe as the realistic-input setting, the best RCA accuracy was 25.3 percent. On Hard tasks it fell to 10.0 percent. The gap between the two persisted even with Claude Fable 5, the authors note, which is their way of saying this is not a problem that the newest model in the set quietly solved.
Two further results are worth sitting with. The weakest of the five models produced an implausible root cause, a hallucinated explanation, in 40 percent of incident reports. Not a wrong-but-reasonable diagnosis; an implausible one. For a task where the output is a claim about why production broke, a confidently invented answer is arguably worse than no answer at all.
The other result concerns what the agents were allowed to see. When the team removed access to the source code and left the agents with telemetry alone, every metric got worse. Reading the signals was not enough. The agents needed to read the program that generated them.
Why it matters
The most useful thing about this paper is the direction of its own caveat. Most benchmark papers argue that their numbers understate real-world performance, or at least leave the question open. Gong and colleagues argue the opposite, and they argue it firmly.
Everything above was measured on a curated 50 GB, six-day testbed. Each task was investigated in isolation, with no other incident competing for attention. And the system under investigation is public: its code and its instrumentation are open, which means the models may well have encountered material about it. Real production systems, the authors write, are orders of magnitude larger, more dynamic, and more idiosyncratic than this. The 25.3 percent, on their reading, is a ceiling dressed up as a score. The gap they report is a lower bound on the engineering work still needed before frontier coding agents can be trusted with production reliability.
That framing is worth taking seriously precisely because it cuts against the incentive to oversell. It also gives teams something concrete: the source-code ablation suggests that the tooling wrapped around an agent, specifically what context it can reach, changes outcomes measurably, which is a lever available today rather than a capability to wait for.
The paper is a preprint posted to arXiv in July 2026 and has not yet been through peer review. The authors have released the public portion of the benchmark, so the numbers here are, at least in principle, something other groups can check and push on.