SRE-1: Dominating the Pareto Frontier of Production Incident Investigation with a 27B Model
November 18, 2026
Abstract
We introduce SRE-1, a 27B open-weight model trained with reinforcement learning to find the root cause of production alerts. On 49 difficult live production cases, our configurations dominate the Pareto frontier of pass rate against cost. An ensemble of five SRE-1 investigations ties Fable 5.2, the strongest frontier model, at a 71% pass rate for 29% lower cost. RL raises the pass rate of a single investigation from 6.9% for the base model to 38.8%. We train on real and synthetic incidents through the tools used in production, with every tool pinned to the alert time to prevent information leakage. Starting from Qwen3.8-27B, we train with a modified CISPO objective for 200 steps on up to 512 GPUs, over 1,549 unique incidents, and select SRE-1 from the resulting checkpoints. The model returns a causal chain with citations to raw tool output.
Introduction
Finding the root cause of a production alert is a hard and ambiguous search problem [7, 21]. One investigation can span thousands of codebases and petabytes of telemetry, and take hundreds of tool calls. An incident can have several contributing causes, or none that can be established, and unlike a coding task there is no test to run. An agent must connect evidence from metrics, logs, traces, code, and tickets into a causal chain back to the fault (Figure 2). Two properties make this hard to train for. First, there is no ready answer to grade against: most alerts receive no postmortem, so we score a run by a graded comparison with a reference chain that we build, judged by LLMs. Second, the incidents that could supply training data are rare and short-lived: telemetry ages out within weeks, and by the time we train on an incident that does survive, its fix is recorded in commits, tickets, and chat, which can give away the answer. Most public work on SRE agents has focused on evaluation [9, 11, 22, 23, 51], and most public reinforcement learning (RL) recipes train coding agents, which tests can grade [8, 12, 13, 27].
In this technical report we present SRE-1,1 a 27B dense model trained with RL to investigate production incidents. We train SRE-1 from the base weights of Qwen3.8-27B [39] on real and synthetic incidents surfaced through the same tools and integrations the agent uses in production. Every tool call in training reaches a real query engine or captured production data. After 200 steps of RL training at a batch size of 2,048 (64 prompts × 32 rollouts), we select SRE-1 as the best checkpoint on a validation set of held-out incidents, at the 1M-token window we serve. We reach the 1M context with 4× YaRN scaling [35]; training used windows of at most 262k tokens.
We evaluate on 49 challenging cases from live production, each run five times in our production harness. LLM judges [55] score each run’s causal chain against the reference chain, and a run passes if its score reaches a threshold that we calibrated with on-call engineers (Section 3). On these cases, SRE-1’s configurations dominate the Pareto frontier of pass rate against cost (Figure 1, Table 8). We run SRE-1 alone at several context windows, which trade cost for accuracy without retraining, and in ensembles of up to five investigations. We compare against 25 other configurations: four frontier models at every tested effort level [3, 5, 32, 33], four other models [2, 4, 16, 31], the base model, and ensembles. From our cheapest configuration upward, the highest pass rate at any budget comes from SRE-1, often by a wide margin: an ensemble of three passes 60%, against 47.8% for the costlier Opus 5.5 at xhigh effort.
Fable 5.2 is the strongest frontier model, ranked first on the Artificial Analysis Intelligence Index [6] and first among frontier models on our live production cases. SRE-1, a 27B model, ties it by combining RL with ensembling. RL alone lifts the base model from 6.9% to 38.8%, and an ensemble of five SRE-1 investigations passes 71%, the same as Fable 5.2 at xhigh effort, while costing 29% less. On an offline task of re-investigating saved evidence, SRE-1 outperforms Fable 5.2 at xhigh effort (Section 7.2).
We draw two conclusions. First, RL in realistic environments is enough to make a small open-weight model competitive on this task. Second, the gain is economic as well as technical. A single SRE-1 investigation passes as often as Fable 5.2 at medium effort for roughly 5× lower cost. With high alert volumes, cost per investigation sets how many alerts get investigated and how large an ensemble each one can afford.
1 SRE-1 is being rolled out gradually inside Resolve’s agent and will be available soon.
Related Work
SRE benchmarks. Most SRE benchmarks inject faults into a few open-source demo applications and check the answer against a label (ITBench [23], AIOpsLab [9], ORCA-bench [22]). OpenRCA [51] draws 335 failures from three enterprise systems, and SREGym [11] injects faults into live clusters. Earlier systems fine-tuned or prompted LLMs to recommend root causes from incident reports [1, 10]. In contrast, we train on real and synthetic incidents through production tools, graded by LLM judges.
Agentic RL. We build on recent work in agentic RL. We keep reasoning and token ids across turns, as SkyRL-Agent does [8], so that we train on exactly what the model produced, and we train fully asynchronously in a harness that matches production, as Composer 2 does [12]. Composer 2.5 [13] localizes credit with textual feedback; we assign credit per turn and per token (Section 5.3). Inkling [45] randomizes tool sets and schemas during training; we vary the tool interface and its formatting across groups, so that the policy learns the task and not one harness (Section 5.4). DeepSWE [27] masks truncated rollouts from the loss; our harness asks for the answer before the context runs out, so 99.9% of the rollouts we trained on end with one. Scout [14] trains an SRE model on simulated investigations. Our most realistic training environment is our own production environment, which the agent queries partly through read-only live backends (Section 4).
Algorithms. Our recipe is inspired by ScaleRL [24], a study of which RL design choices continue to help as compute scales. Following it, our loss is a modified CISPO [30], which clips the importance weight rather than the update so that every token keeps its gradient; we additionally mask the rare tokens whose ratio leaves a wide band (Section 5.1). We also overlap sampling and training with in-flight weight updates, as in PipelineRL [36] and AReaL [18], and we keep the LM head in fp32, which ScaleRL, following MiniMax-M1 [30], shows significantly improves RL performance. We depart from ScaleRL in two places: we bound staleness at 2 steps rather than 8 for stability (Section 6.1), and we divide advantages by a running estimate of each environment’s reward standard deviation. We chose a running estimate because the usual divisors were unstable or biased: ScaleRL’s, the batch’s standard deviation, moved 11% to 14% per step in our runs, and the step size with it, while GRPO’s [44], each group’s own, is small on very easy or very hard prompts and so inflates their advantages, the difficulty bias that Dr. GRPO [26] identifies. Finally, as in DAPO [53], we drop groups whose rollouts all score alike, since they carry no signal.
Problem Setting
An investigation starts when an alert fires. The alert reports a symptom downstream of the fault, and some alerts fire on a healthy system because the rule was misconfigured.
SRE-1 starts from the alert and a knowledge graph of the environment, and queries the organization’s observability, code, and ticketing systems, through more than 400 tools across about 60 integrations (Figure 2). We train and evaluate SRE-1 as a single agent; like the base model it can delegate parts of an investigation to sub-agents, but we do not study that here.
Each chain is scored from 0 to 1 against a reference chain, averaging the scores of frontier-model judges from three model families [47] to offset a judge’s tendency to favour chains written by its own family [34]. We ask each how similar the submitted chain is to the reference, and we tuned the judging prompt to match the scores of on-call engineers on 87 labelled chains (mean squared error 0.024 on the 0 to 1 scale). We count a score of 0.8 or more as a pass: when we calibrated the judge with on-call engineers, they judged a chain scoring 0.8 or more to be correct. A passing chain names the same trigger, mechanism, and root cause as the reference, though it may miss one or two minor details or supporting pieces of evidence. This three-judge average is the score we report: for the 49 live cases, for offline synthesis, and for the held-out score that we track during training on 101 incidents (Section 5.6). The training reward instead uses one randomly drawn judge per group (Section 5.2).
Formally, an investigation starts from an alert \(q\) in an environment that holds the incident’s telemetry, tools, and code at the alert’s revision. Each action \(a_k\) is one model response, with reasoning and zero or more concurrent tool calls that return observations \(y_k\). Actions are sampled from the policy
and the final action submits an answer \(c\), an ordered causal chain whose links cite evidence. A judge scores \(c\) against a reference chain \(c^\star\) and returns a score \(s \in [0, 1]\). We adopt standard terminology: a prompt is one incident as the policy receives it (its alert, tools, and harness), a rollout is one sampled investigation of a prompt, a group is the \(G\) rollouts sampled for one prompt, and a step is one update of the policy. A rollout’s reward \(R\) is built from its score \(s\) (Section 5.2), and its advantage is its reward minus the mean reward of its group.
Training Environments
Real incidents are difficult to train on for two reasons. First, most alerts never receive a postmortem, so there is no answer to grade against (Section 4.2). Second, an incident’s telemetry ages out within weeks, and by the time we train on it, its fix and cause are recorded in commits, tickets, and chat, where the agent could look them up (Section 4.3). We train SRE-1 on two tasks (Figure 3): (a) live-tool investigation, where it investigates by issuing live queries to production tools, and (b) offline synthesis, where it writes the causal chain from evidence an earlier investigation already gathered, with no access to production tools (Section 4.4). Our dataset of 1,503 live-tool incidents exposes the production tool surface as it stood when the alert fired. No evidence is gathered in advance, so the model must decide what to query. The training run started on an earlier 904-incident dataset and later upgraded to the 1,503-incident dataset after step 125 (Table 4 in Appendix B). Offline synthesis targets a quarter of each training batch and draws on 7,716 problems built from 1,774 saved investigations (Section 4.4). SRE-1 is the best checkpoint of the run on the validation set, at step 110.
Every tool call runs a real query against real or captured data. We also tried simulation, in which an LLM wrote a response to each query we had not recorded, based on the responses we had. We discontinued it because, in our tests, its fidelity was poor and hard to improve. Tool calls in our environments also fail the way they do in production, for infrastructure reasons such as timeouts, rate limits, and unhealthy backends, and we keep those failures: 18.1% of training investigations had at least one failed tool call (Appendix B). We inject no tool call failures, and we never drop a rollout because a production tool failed.
Sources of live-tool incidents
We combine four sources of incidents (Table 1), and Appendix B breaks the datamix down by type of fault (Figure 31). Live incidents and incident replays are real and the hardest for the base model (Figure 4), but they come without answers: the root cause must be established before we can train on them (Section 4.2), and they are slow to collect. Synthetic companies have known answers, since we plant their faults ourselves, but planted faults are easier to detect. Licensed environments from external vendors (8% of incidents) add systems that we did not build. RL raises the held-out score on all four, by 0.13 to 0.22 from the base model to SRE-1 (Figure 5).
| Live incidents | Incident replays | Synthetic companies | Licensed environments | Offline synthesis | |
|---|---|---|---|---|---|
| Real fault | ✓ | ✓ | ✗ | ✗ | ✓ |
| Closeness to production | High | Medium | Low | Low | High |
| Answer known by construction | ✗ | ✗ | ✓ | ✓ | ✗ |
| Make telemetry tool calls | ✓ | ✓ | ✓ | ✓ | ✗ |
| Diversity | Low | High | Medium | Low | High |
| Volume | Medium | Medium | Medium | Low | High |
| Cost | Low | Medium | Medium | High | Low |
| Setup effort | Medium | High | Medium | Medium | Low |
Live incidents (23%). Our own production environment is a rich source of incidents and our most realistic environment. We use our agent heavily there, and we keep its investigations together with the feedback that engineers gave while working each incident with it in chat. Because we own this environment, we can snapshot all of the telemetry behind each incident and re-investigate it with live tools long after it happened; this is what separates live incidents from incident replays, which serve a partial capture.
Incident replays (34%). Production telemetry ages out within weeks, so we capture each incident while its data can still be queried and serve it from a replay. Capture is driven by exploration: agents from several model families, including one conditioned on the known root cause, investigate the incident, and each query they issue is generalized into a broader view of the data around it (Figure 6). The capture is therefore a neighbourhood of the evidence around the explored paths rather than the set of answers the investigators received.
A rollout rarely repeats a captured query verbatim; exact-match lookup against earlier answers covered only about 20% of agents’ queries. The replay instead executes each query in the tool’s own query language over the captured data, so it answers queries that no exploring agent issued, and one capture serves thousands of rollouts. On new investigations of captured incidents, 94% of tool calls are served.
Synthetic companies (35%). In a synthetic company the fault is planted, so the answer is known by construction and no reference chain has to be reconstructed. Agent swarms build each company, with services, simulated traffic, and vendors, and the same agents then plant a fault, investigate it, and write it up. The companies are designed to be as complex as real production systems: the one in Figure 32 (Appendix B) runs 22 services and 28 scheduled jobs over several data stores and queues, and any of them can be made to fail. The agents that planted the faults knew the answer, and some accidentally left it in commit messages, file names, code, and even metric names (Figure 7), so most of the leaks that our quality checks excluded came from synthetic companies.
Reference chains
Expert annotation is slow and difficult to scale. Production evidence expires faster than experts can write reference chains, leaving many incidents unusable while they await annotation. We build reference chains with a team of frontier agents. For each production incident, agents from two model families debate the raw evidence until they reach agreement [17], and reviewer models audit each link against primary records [40]. This process requires no expert time and labels most of the incidents it is run on. We compared its chains with expert-written chains on 27 incidents. Our evaluation team of expert SREs judged the agents’ chain as good as an expert’s or better on 26. Scoring the same rollouts against either reference barely changes their mean score (Table 7). Synthetic incidents have reference chains known by construction and do not require this process.
Leakage and time pinning
Training on historical incidents introduces a risk of information leakage from records created after the incident. By the time the model investigates an incident, often the fix has been deployed and the cause documented in commits, tickets, and chat. Access to these records trains the model to retrieve documented answers in place of investigating the incident. We use time pinning to prevent leakage by making every tool return results consistent with the environment at the time of the alert (Figure 8). Implementing time pinning is difficult because each data source represents time differently and provides indirect access to future states. For example, in one replay an agent found the hash of a later commit and requested its contents. It retrieved the fix for its own incident, reported that change as the root cause, and scored high against our reference chain. Out of our production tool set (hundreds of tools), we either added time pinning or dropped the tool for training. Information leakage can also occur through record contents, including commit messages and metric names that identify the root cause (Figure 7). We apply several quality checks to every incident in the dataset (Figure 9; Table 5 in Appendix B). These checks retained 1,503 of 1,663 candidates, and leakage was the most common reason for exclusion.
Auxiliary task: offline synthesis
Offline synthesis is a second task that we train alongside live-tool investigation. The model receives an alert and a read-only folder containing what an earlier investigation saved: its tool calls, their full outputs, and the runbooks it consulted (Figure 10). From that evidence alone, it must construct the best causal chain it can, citing the files. We train this task for three reasons: (a) it is plentiful, since every saved investigation can be posed as a problem in up to six variants, so 1,774 investigations gave 7,716 problems (Table 2); (b) reasoning over gathered evidence is crucial to every live investigation; and (c) it powers ensembling, in which a synthesizer combines several independent investigations into one answer that is more accurate than a single investigation [48, 49] (Section 7.1).
| Component | Setting |
|---|---|
| Input | The alert and a README of the saved evidence; in half of rollouts also a budgeted index of the original agents' tool calls (10K–30K tokens) |
| Tool | bash in a sandbox with no network; /workspace read-only, /tmp writable; 30 s per command; 12,000-character previews with full output saved to a file |
| Output | A causal chain whose evidence entries each have a file reference, a verbatim quote, and the reasoning; 1–5 suggested next investigations; a confidence that predicts the chain's own score |
| Termination | A valid final answer, or an exhausted context of about 154K–162K tokens |
| Corpus | 1,774 training investigations, rendered in up to six variants each (7,716 problems), of which the run served 1,295; 73 held-out investigations, one problem each. The rendered corpus holds 371,820 reference citations, none dangling |
Reinforcement Learning
We train for 200 steps from the base weights of Qwen3.8-27B. SRE-1 is the checkpoint at step 110, which we chose on the held-out score (Section 5.6). We later changed the recipe to trade some accuracy for faster investigations, and the checkpoint at step 200 scores the same within its interval. Each step trains on one batch of 2,048 rollouts, 32 rollouts for each of 64 prompts; 64 rollouts per prompt gave no held-out gain (Table 7). Appendix A shows one such batch interactively.
Objective
ScaleRL [24] compares RL objectives for language models at scale, and we adopt its choice, CISPO [30], which weights each token’s gradient by a clipped, stop-gradient importance ratio instead of clipping the update. However, the token probabilities computed by the trainer and the sampler are rarely exactly equal, and recent systems report that a few tokens with extreme ratios destabilize training unless they are masked [25, 50]. Inspired by these results, and by IcePop’s double-sided mask in particular, we add a mask to CISPO. With \(\pi_{\theta_{\text{old}}}\) the sampling policy that generated token \(o_{i,t}\) of rollout \(o_i\), and \(r_{i,t}(\theta) = \pi_\theta(o_{i,t} \mid q, o_{i,<t})\,/\,\pi_{\theta_{\text{old}}}(o_{i,t} \mid q, o_{i,<t})\) the importance ratio, we maximize
Here \(\mathrm{sg}\) stops the gradient, \(o_{i,<t}\) is the rollout’s context before token \(t\), \(\hat{A}_{i,t}\) is the advantage, centred within each group of \(G = 32\) rollouts and scaled per environment, \(w_i\) weights each rollout, \(\epsilon_{\max}\) is CISPO’s cap, and \(M_{i,t}\) masks tokens whose ratio leaves the band \([\alpha, \beta] = [0.2, 5]\). Since the trainer steps once per batch, \(r_{i,t}\) reflects only trainer–sampler mismatch and at most two steps of staleness. We set \(\epsilon_{\max} = 5\), the band’s upper edge, so inside the band a token keeps its full weight \(r_{i,t}\), and outside the band it is dropped.
The mask improves RL stability: during a spike, the KL divergence between trainer and sampler (\(k_3\) estimator [42]) was 38 times lower with the mask than without it (Figure 11b). The band has been well studied [25, 50] and should be set as wide as possible to avoid unnecessarily discarding informative tokens (Figure 11a). Following recent work [53] we don’t use a KL term, and an earlier ablation showed that adding one harms performance (Table 7).
Reward
The reward is a weighted sum dominated by an LLM judge’s score of the submitted causal chain against a reference chain. We use a continuous judge score rather than a binary pass or fail, because in our tests a binary reward gave only about a third of hard incidents a trainable group, against more than half under the continuous score (row 9 of Table 7), and gave near-misses the most negative advantage. We add smaller-weighted terms to encourage faster investigations, and to discourage specific problematic tool-call behaviours that we observed in training.
Reward hacking. A well-known problem in RL is that a policy learns to score better with the judge without getting better at the task [20]. We saw this in early runs. Trained against a single judge, an earlier policy learned to restate the alert as the first link of its chain and to list ruled-out causes that nobody had proposed. Its reward rose and its diagnoses did not improve. A study of candidate judges pointed to the remedy: the measured gain from training depended on the judge’s model family, and two judges from one provider made correlated errors where judges from different providers did not. We therefore score each group with one judge drawn at random from a set of 13, which spans several API providers and effort settings. Across groups, the policy trains against the average of the set and not against one judge’s biases. Within a group, one judge scores every rollout, so their rewards are comparable even though the judges differ widely in strictness (Figure 12).
In earlier testing, the policy exploited the grader in five ways, such as reversing the causal order or citing a README it never opened. We fixed each in the citation check or the judge. An earlier scorer also rewarded hedged answers, so the judge now caps the score of a chain that offers several candidate causes or marks its own mechanism as unconfirmed. We resolve every citation against its source: of the 3.8 million citations in the rollouts we trained on, 98.2% resolved.
Dense credit assignment
Standard policy-gradient RL gives every token of a rollout the same advantage, which is too coarse for long investigations. A rollout of 26 turns and 63 tool calls can reach the right root cause despite a turn that issues many inefficient queries, or a repeated block, wrong-language text, or a malformed tool call. Under a single advantage, those tokens are rewarded along with the answer, and the mistakes persisted. In an earlier run, tool mistakes persisted long into training: high- and low-reward rollouts made them at the same rate, and a small reward penalty on the mistake rate changed little. Each call is a query that a backend must execute, so the harness allows at most ten in a turn. In an earlier run, up to 24% of rollouts broke this limit, one of them with 1,458 calls in a single turn. Up to 12% produced repetition loops or incoherent text (Figure 13).
We therefore assign credit densely (Figure 14a). When a turn or a span of tokens contains a mistake, we reduce the credit of that turn or span alone. Two systems inspired this: (a) Composer 2.5 [13], which inserts a hint at the point in a rollout where the model could have done better and distills the policy toward the hinted distribution for that turn only, and (b) MiMo-V2.6 [50], which masks or reshapes the advantages of flagged tokens such as malformed tool calls.
SRE-1 trained with credit reductions for turns that exceed the stated call limit, for repetition, and for malformed or wrong-language tokens. At step 200, 23.9% of rollouts received one of these reductions. Over the run, the three targeted mistakes became 12 to 343 times rarer (Figure 14b). Other settings changed over the same steps, so we do not attribute these reductions to credit assignment alone.
Harness
We design the training harness to match deployment.
Tool-result cap. We limit how much of a tool result enters the context, and keep the rest of a long result in a file that the model can read or search with later bash tool calls. These limits are critical: raising the cap from 25k to 100k characters caused the model to frequently run out of context, and the mean score fell from 0.53 to 0.37 (Figure 15).
Tokens in, tokens out (TITO). Most sampling frameworks (e.g. vLLM, SGLang) de-tokenize their output, returning text instead of the raw tokens. In RL, re-tokenizing that text before sending it back to the sampler would subtly change ~94% of rollouts (Figure 16). Most RL frameworks have fixed or avoid this bug [37], and similarly our framework guarantees that we operate only with append-only token id streams. Figure 16 illustrates an example of how one string can tokenize in multiple different ways.
Sampled decisions. Each time we train on an incident, we sample a new harness configuration for it, so that the policy cannot overfit to any one harness (Figure 17). Half of all groups run the evaluation harness unchanged. The other half draw from about 70 decisions, covering the tool interface, the budgets and whether they are stated, the form of the prompt, and the names and schemas of the tools, which allows more than 1050 combinations. All rollouts in a group share its draw, so advantages compare rollouts under the same conditions. A setting that every group shares can collapse the policy onto one behaviour, as happened with parallel calls (below). In related work, Inkling [45] randomizes tool sets and schemas, and Scout [14] drops optional tools and randomizes turn budgets. Domain randomization is standard for moving policies out of simulation [46] and prompt formatting alone can change a model’s accuracy substantially [43].
Budgets. Frontier models expose one effort level, such as “low”, “medium”, or “xhigh”. SRE-1 instead takes two budgets, one for turns and one for context, which gives finer control over latency. Randomized groups draw the two independently, so that one policy learns to work under any budget. Each tool response includes the remaining context budget, allowing the policy to condition its subsequent actions on available capacity. RL raised the judge’s score at every budget that we draw (Figure 18), and at deployment the two budgets move SRE-1 along the frontier of Figure 1 without retraining.
Parallel calls. All calls in a turn run concurrently, so calling tools in parallel shortens an investigation. However, RL alone did not preserve parallel calling. In four earlier training jobs the policy collapsed to one call per turn; in the first attempt at this run, calls per turn fell from 1.52 to 1.02 within 40 steps. Once every rollout makes one call per turn, RL cannot bring the behaviour back because there is no variation left to reinforce. We therefore forked the run from before the collapse, with a statement in the prompt that calls in one response run concurrently and a temporary penalty on turns that made too few parallel calls. Each randomized group draws one of four wordings of that statement, from neutral to emphatic. Calls per turn reached 2.4 by step 60, with the held-out score unchanged (Figure 19).
Batches
Batch composition. Each batch targets equal shares of four kinds of environment (Figure 3). Three are live and differ only in their interface to the same production tools: a CLI interface (one shell, with each tool run through a command-line client), a search interface (the model searches the catalog, reads a tool’s documentation, and runs it through a wrapper), and a direct interface (each tool is its own function). The fourth is offline synthesis (Section 4.4). Some systems combine tasks by distilling task-specific teachers into one student after RL, as MiMo-V2.6 [50] and DeepSeek-V4 [15] do. We instead train one policy on all four at once and balance them by sampling.
Curriculum. In reinforcement learning, the learning signal depends on the reward variance within each group of 32 rollouts, so any group with either all-solved or no-solved problems has no gradient to learn from. ScaleRL [24] permanently removes a prompt once its pass rate is high, since a prompt that has become easy tends to stay easy. Unlike permanent filtering, our curriculum resamples both easy and hard prompts to detect performance regressions and reconsider previously unsolved tasks as training progresses (Figure 20). When all 32 rollouts pass, the prompt moves to an easy pool, and when none passes, to a hard pool. Prompts return at random, with a half-life of about 245 steps for easy prompts and about 25 for hard ones in the live environments. The next group determines the pool again. Most returning prompts repeat their earlier outcome: 56% of returning easy prompts were again solved by every rollout, against 14% of ordinary draws, and 65% of returning hard incidents were again solved by none, against 19%.
Batch construction. A group trains the policy only through the differences among its rollouts, and PODS [52] shows that training on the highest-variance part of a larger sample learns faster than training on all of it. We apply the idea to groups. We initially sample approximately twice the target number of groups per training batch. We drop a group whose rollouts score almost alike or all score low, and rank the rest within each environment by the spread of their rewards, discounted by age. The top groups fill that environment’s share of the 64, and a group not chosen waits for a later batch until it is more than two steps old. Over the run, 53% of the groups that finished sampling were trained on; the rest were dropped for lack of signal or exceeded the age limit. Selected groups exhibited greater reward variance than groups excluded by the staleness limit, consistent with the objective of prioritizing higher-variance groups for training.
Training dynamics
We follow training with a validation score: 101 incidents held out of training, 57 of them from environments held out of training, run in the training harness at every fifth checkpoint with eight rollouts each. The score rises from 0.45 for the base model to 0.60 at SRE-1’s checkpoint, a paired gain of 0.149 [0.125, 0.173] (Figure 21a). The gain is broad: SRE-1 scores higher than the base model on 95 of the 101. Most of the gain came in the first 20 steps. A longer context window helps at almost every checkpoint: at 1M SRE-1 scores 0.66, a paired gain of 0.064 over 262k [0.042, 0.087]. We chose step 110 of the 200 as SRE-1, balancing this score against speed, so it benefits from that selection. The live cases of Section 7.1 are independent of the selection.
With RL, the model solved incidents it could not solve before (Figure 22). On math and code, a base model matches RL performance with many more samples [54]; we test eight rollouts.
Behaviour. Investigations grow longer through training: SRE-1 takes an estimated 1.7 times the base model’s time at 262k and 5.0 times at 1M (Figure 21b). Calls per turn rose after we added support for parallel calls and then partly fell back (Figure 23).
Infrastructure
The run used up to 512 GPUs across two clouds for 7.4 days. It ran up to 8,736 investigations concurrently and ran 812,695 rollouts to completion, in which the model made 51.5 million tool calls, and it trained on 409,600 of those rollouts, holding 76.2 billion tokens (Table 6 in Appendix D).
Training system
Training runs on Cortex Training from Snowflake, which abstracts the trainer and the samplers behind low-level APIs (Figure 24). Our training code is a heavily modified fork of the open-source PRIME-RL [38]. The low-level work (forward passes, backward passes, and sampling) runs on Cortex, with 192 GPUs for the trainer and up to 320 for the samplers. Doubling both over our earlier runs kept a step about as long while the batch grew from 1,280 to 2,048 rollouts. The orchestration and the advantage logic run on a separate cloud, next to every environment, tool, and judge. There, an orchestrator drives every rollout: it sends prompts to the samplers, executes tool calls in the environments, and assembles batches for the trainer. We monitor training live in a web UI that we built for this work (Appendix H).
We overlap sampling and training to avoid idle compute caused by differences in their execution times (Figure 25). Following PipelineRL [36], we use asynchronous training, allowing sampling to continue during gradient updates and updating the samplers’ weights mid-rollout. In this run the trainer spent 95% of a median 47-minute step in its forward and backward passes and waited a median of 80 seconds for its next batch. A batch admits rollouts at most two steps behind the trainer, and about half of the trained rollouts were at this bound. We keep the bound for stability, because the trainer–sampler mismatch grows with lag: in a small reproduction, tokens sampled with weights two steps old had 3.0 times the mismatch of tokens sampled with current weights.
Trainer numerics are critical
Numerical precision was a significant source of training instability. We run the forward and backward passes in bf16 for speed, and keep gradient accumulation, the LM head, the master weights, and the optimizer state in fp32 [28]. We identified a bug in which gradients were accumulated in bf16 rather than fp32, causing small contributions to be lost through rounding. In a controlled probe, a bf16 accumulator lost all 31 small terms that an fp32 accumulator kept (Figure 26a), and on one recorded training step an estimated 12.6% of the additions were lost. After the fix, a fresh run of the same recipe scored 0.026 higher on held-out incidents at step 25 [0.012, 0.040], a 24% larger gain over the base model (Figure 26b). The same release included other, minor fixes, so we cannot isolate this one. We tested about twenty reward and advantage changes that month, and none changed the score as much.
Numerical consistency is particularly important for our objective, which weights token gradients using trainer-to-sampler importance ratios, so any disagreement between the two becomes error in the gradient (Figure 26c shows this mismatch over the run). We addressed other such problems, for example master weights stored in bf16, which rounded away most updates at our learning rate.
Restarts and recipe changes
The run restarted 14 times, for recipe changes, preemption, and platform reclaims, and changed its recipe at 11 of them (Figure 27). On a graceful stop, we save each rollout in flight at a turn boundary and the next job continues it, because regenerating interrupted rollouts would bias training toward short ones [15] (Appendix D). We change the recipe by resuming the same run with a new configuration rather than starting a new one. Most changes adjusted how strongly the reward valued speed and cost. Others changed the datamix and added harness variations. Other long RL runs are also resumed after failures: MAI-Thinking-1 [29] recovers from collapsed runs by self-distillation, reporting that a checkpoint from before a numerical collapse often already carries the instability. The collapse of parallel calling (Section 5.4) was behavioural, and forking from an earlier checkpoint recovered it.
Results
Challenging live production cases
Setup. The live production cases are our most realistic evaluation: recent, real incidents, investigated live. We began with 175 live production alerts with agent-team reference chains (Section 4.2). After excluding alerts with telemetry expiring within days of evaluation or inaccessible environments, 67 remained; a frontier model and an earlier SRE-1 checkpoint each investigated them twice. Auditing these rollouts removed 12 alerts where a rollout read post-alert records or two rollouts lacked evidence to answer, then 4 without a scored checkpoint rollout and 2 solved by neither model, leaving 49. Training alerts were excluded, and the pool over-represented less common environments and alerts our production system had scored lower on. We run each case five times in our production harness, which uses the training harness’s prompt, answer format, and context management on live integrations, and score rollouts as in Section 3. Unless stated otherwise, runs use a 150-turn cap and a 1M-token context. Figure 1 and Table 8 (Appendix G) show the results for our configurations and each other model at its strongest setting, and Table 9 lists every configuration.
Pareto frontier. SRE-1’s configurations dominate the Pareto frontier of pass rate against cost (Figure 1). Each passes more often or costs less than every one of the 25 other configurations we measured. From our cheapest configuration upward, the highest pass rate at any budget comes from SRE-1: any other configuration that costs at least as much as our cheapest is matched or beaten by one of ours that costs no more. We run SRE-1 alone at four context windows, with harness guidance to work faster, and in ensembles that combine several investigations by synthesis. The ensemble of five is a tree: two investigations and three investigations are each synthesized, and the two syntheses are combined into the final answer (Figure 10). This ensemble ties Fable 5.2 at xhigh effort, the strongest frontier model, at 0.71 of its cost (71.4% [57.1, 83.7] against 71.0% [60.4, 80.4]).
A single investigation. RL raises the pass rate of a single investigation from 6.9% for the base model to 38.8% for SRE-1, a paired gain of 31.8 points [22.4, 41.6] (Table 8). SRE-1 solves 32 of the 49 cases at least once, against 6 for the base model. One SRE-1 investigation passes as often as Astra at xhigh effort or Fable 5.2 at medium effort, for about a fifth of their cost or less, and more often than Opus 5.5 at high effort at 36% of its cost. Only Opus 5.5 at xhigh effort and Fable 5.2 at high and xhigh effort pass significantly more often, at 4.5, 8, and 10 times the cost.
Context window. The pass rate rises with the context window, from 15.5% at 262k, where we also cap an investigation at 60 turns, to 38.8% at 1M (the points along our line in Figure 1).
Ensembles. The synthesis skill of Section 4.4 also combines several investigations into one answer. When SRE-1 combines its own investigations of a live case, its pass rate climbs from 38.8% for one investigation to 71% for an ensemble of five (Figure 28). That ensemble passes more often than any Opus 5.5 (xhigh) configuration we measured, including Opus’s own ensemble of five, which passes 65% at 4.1 times the cost of ours. Synthesis can recover correct causal hypotheses from individual investigations, improving ensemble performance when the constituent investigations produce different conclusions.
Offline synthesis
Setup. We measure offline synthesis on its own held-out evaluation set of 73 problems, 40 of them from environments held out of training (Appendix G describes how the set was built). Our checkpoints run in the training harness with eight rollouts per problem. The frontier models run through their APIs with two rollouts per problem, to reduce API cost. The same three judges score every rollout.
| Model | Mean score | Pass rate | SRE-1’s lead in pass rate |
|---|---|---|---|
| SRE-1 | 0.749 [0.708, 0.786] | 51.7% | |
| Opus 5.5 (xhigh) | 0.739 [0.699, 0.775] | 45.9% | +5.8 points [−0.7, 12.2] |
| Fable 5.2 (xhigh) | 0.731 [0.692, 0.764] | 39.0% | +12.7 points [5.3, 20.5] |
| Base model | 0.650 [0.613, 0.684] | 23.5% |
On offline synthesis, SRE-1 passes 12.7 points more often than Fable 5.2 (xhigh) [5.3, 20.5] and does not differ significantly from Opus 5.5 (xhigh) (+5.8 points [−0.7, 12.2]; Table 3, Figure 29). RL raises the mean score by 0.099 [0.075, 0.123] and the base model’s pass rate from 23.5% to 51.7% (+28.3 points [21.6, 34.9]). The gain is smaller but still significant on the 40 problems from environments never seen in training (+0.07 [0.04, 0.10]). Section 4.4 describes the task and how this skill carries over to ensembles.
Ablations and qualitative behaviour
We report 34 ablations in Appendix E, including the 23 that were neutral, negative, or not adopted. The 28 with per-incident records took 221,000 GPU-hours of training, 2.5 times SRE-1’s own run, and 3.5 million judge calls. Harness and sampling configurations produced the largest changes in held-out performance: at a fixed checkpoint, changing them moved the score by 0.12 to 0.16. Changes to training moved it far less. In every A/B pair of training runs, the two runs’ held-out scores after the same number of steps (usually 15) differed by less than 0.05; the largest gap, 0.049, was for the wide importance-ratio band (row 4 of Table 7). RL also changed how the model investigates (Appendix F): SRE-1 calls tools in parallel in most of its turns and answers in 39% fewer turns than the base model, which mostly makes one call at a time.
Limitations
- Holistic scores. Each judge gives one score for a whole chain, so a score does not say which link or citation is wrong.
- Training environments. On replays, 6% of tool calls fall outside what was captured and 38% return partial results, and synthetic companies still look synthetic to a blind reviewer. These limits apply to training only.
- Case selection. The pool over-represents alerts on which our production system, backed by frontier models, scored lower, which can favour SRE-1 against those models (Section 7.1).
- Scope. We train and evaluate one task, finding a root cause from an alert in a single pass. On-call engineers do much more than this, and an ensemble takes longer than one investigation, which matters when an answer is needed in real time.
- Statistics. Bracketed intervals are 95% confidence intervals and are not adjusted for multiple comparisons.
Conclusion
We trained SRE-1 with RL to investigate production incidents, and on challenging live production cases its configurations dominate the Pareto frontier of pass rate against cost. Each passes more often or costs less than every one of the 25 alternatives we measured. On held-out incidents, RL raises the mean score from 0.45 to 0.60 (±0.04), with a higher score on 95 of 101 incidents. An SRE-1 investigation costs a small fraction of the strongest frontier models’, allowing several investigations per alert. An ensemble of five ties the strongest frontier model at lower cost. We obtained this result through months of work on realistic captured and synthetic environments, evaluations, a new reference-chain pipeline, scaling reinforcement learning infrastructure, and careful ablations.
Future work. We are scaling up to larger models, adding tasks, and improving the realism and diversity of our environments.
Contributors
Ordered alphabetically within each group.
- Michael Alverson
- Kshitij Alwadhi
- Sean Bell (Lead Researcher)
- Charlie Ji
- Xuchao Jia
- Dhruv Mahajan (Lead Researcher)
- Henry Maier
- Harsha Polavaram
- Abhishek Sharma
- Harry Shin
- Alvin Zhu
- Vasanth Balakrishnan
- Madhu CS
- Deep Desai
- Michael Preibisch
- Vishwa Sheth
Acknowledgements
SRE-1 was trained on Cortex Training from Snowflake. We were an early design partner for the platform, and we thank the Snowflake team for collaborating closely with us to support this work. We also thank Annalise Krueger and Claire Yin. Finally, we thank Spiros Xanthos (CEO), Mayank Agarwal (CTO), and Steven Karis (Chief Architect) for funding and supporting this work.
References
- Toufique Ahmed, Supriyo Ghosh, Chetan Bansal, Thomas Zimmermann, Xuchao Zhang, and Saravan Rajmohan. Recommending root-cause and mitigation steps for cloud incidents using large language models. In ICSE, 2023. arXiv:2301.03797.
- Anthropic. Introducing Claude Fable 5.1 and Claude Mythos 5.1. Blog post, www.anthropic.com/claude-fable-and-mythos-5-1, 2026.
- Anthropic. Introducing Claude Fable 5.2 and Claude Mythos 5.2. Blog post, www.anthropic.com/claude-fable-and-mythos-5-2, 2026.
- Anthropic. Introducing Claude Haiku 5.5. Blog post, www.anthropic.com/claude-haiku-5-5, 2026.
- Anthropic. Introducing Claude Opus 5.5. Blog post, www.anthropic.com/claude-opus-5-5, 2026.
- Artificial Analysis. Artificial Analysis Intelligence Index v5.0.1. Leaderboard, artificialanalysis.ai/evaluations/artificial-analysis-intelligence-index, accessed November 2026.
- Betsy Beyer, Chris Jones, Jennifer Petoff, and Niall Richard Murphy, editors. Site Reliability Engineering: How Google Runs Production Systems. O’Reilly Media, 2016.
- Shiyi Cao, Dacheng Li, Fangzhou Zhao, Shuo Yuan, Sumanth R. Hegde, Connor Chen, Charlie Ruan, Tyler Griggs, Shu Liu, Eric Tang, et al. SkyRL-Agent: Efficient RL training for multi-turn LLM agent. arXiv preprint arXiv:2511.16108, 2025.
- Yinfang Chen, Manish Shetty, Gagan Somashekar, Minghua Ma, Yogesh Simmhan, Jonathan Mace, Chetan Bansal, Rujia Wang, and Saravan Rajmohan. AIOpsLab: A holistic framework to evaluate AI agents for enabling autonomous clouds. In MLSys, 2025. arXiv:2501.06706.
- Yinfang Chen, Huaibing Xie, Minghua Ma, Yu Kang, Xin Gao, Liu Shi, Yunjie Cao, Xuedong Gao, Hao Fan, Ming Wen, et al. Automatic root cause analysis via large language models for cloud incidents. In EuroSys, 2024. arXiv:2305.15778.
- Jackson Clark, Yiming Su, Saad Mohammad Rafid Pial, Yifang Tian, Lily Gniedziejko, Hans-Arno Jacobsen, Yinfang Chen, and Tianyin Xu. SREGym: A live benchmark for AI SRE agents with high-fidelity failure scenarios. arXiv preprint arXiv:2605.07161, 2026.
- Cursor Research. Composer 2 technical report. arXiv preprint arXiv:2603.24477, 2026.
- Cursor Team. Introducing Composer 2.5. Blog post, cursor.com/blog/composer-2-5, 2026.
- Datadog AI Research. Scout 1.0: Post-training a model for SRE investigations. Technical report, datadoghq.com/blog/ai/introducing-scout-sre-specialized-model-bits-ai, 2026.
- DeepSeek-AI. DeepSeek-V4: Towards highly efficient million-token context intelligence. arXiv preprint arXiv:2606.19348, 2026.
- DeepSeek-AI. DeepSeek-V4.1-Flash: Pushing the limits of KV cache compression. arXiv preprint arXiv:2609.19969, 2026.
- Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate. In ICML, 2024. arXiv:2305.14325.
- Wei Fu, Jiaxuan Gao, Xujie Shen, Chen Zhu, Zhiyu Mei, Chuyi He, Shusheng Xu, Guo Wei, Jun Mei, Jiashu Wang, et al. AReaL: A large-scale asynchronous reinforcement learning system for language reasoning. In NeurIPS, 2025. arXiv:2505.24298.
- Chang Gao, Chujie Zheng, Xiong-Hui Chen, Kai Dang, Shixuan Liu, Bowen Yu, An Yang, Shuai Bai, Jingren Zhou, and Junyang Lin. Soft adaptive policy optimization. arXiv preprint arXiv:2511.20347, 2025.
- Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In ICML, 2023. arXiv:2210.10760.
- Supriyo Ghosh, Manish Shetty, Chetan Bansal, and Suman Nath. How to fight production incidents? An empirical study on a large-scale cloud service. In SoCC, 2022. doi:10.1145/3542929.3563482.
- Albert Gong, Kyuseong Choi, Abhineet Agarwal, Jason Schechner, Ryan Huang, Raj Agrawal, Anish Agarwal, and Raaz Dwivedi. ORCA-bench: How ready are language model agents for oncall? arXiv preprint arXiv:2607.28545, 2026.
- Saurabh Jha, Rohan R. Arora, Yuji Watanabe, Takumi Yanagawa, Yinfang Chen, Jackson Clark, Bhavya Bhavya, Mudit Verma, Harshit Kumar, Hirokuni Kitahara, et al. ITBench: Evaluating AI agents across diverse real-world IT automation tasks. In ICML, 2025. arXiv:2502.05352.
- Devvrit Khatri, Lovish Madaan, Rishabh Tiwari, Rachit Bansal, Sai Surya Duvvuri, Manzil Zaheer, Inderjit S. Dhillon, David Brandfonbrener, and Rishabh Agarwal. The art of scaling reinforcement learning compute for LLMs. In ICLR, 2026. arXiv:2510.13786.
- Ling Team, Anqi Shen, Baihui Li, Bin Hu, Bin Jing, Cai Chen, Chao Huang, Chao Zhang, Chaokun Yang, Cheng Lin, et al. Every step evolves: Scaling reinforcement learning for trillion-scale thinking model. arXiv preprint arXiv:2510.18855, 2025.
- Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding R1-Zero-like training: A critical perspective. In COLM, 2025. arXiv:2503.20783.
- Michael Luo, Naman Jain, Jaskirat Singh, Sijun Tan, Ameen Patel, Qingyang Wu, Alpay Ariyak, Colin Cai, Tarun Venkat, Shang Zhu, et al. DeepSWE: Training a fully open-sourced, state-of-the-art coding agent by scaling RL. Blog post, Agentica and Together AI, together.ai/blog/deepswe, 2025.
- Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al. Mixed precision training. In ICLR, 2018. arXiv:1710.03740.
- The Microsoft AI Team. MAI-Thinking-1: Building a hill-climbing machine. Technical report, Microsoft AI, microsoft.ai/pdf/mai-thinking-1.pdf, 2026.
- MiniMax. MiniMax-M1: Scaling test-time compute efficiently with lightning attention. arXiv preprint arXiv:2506.13585, 2025.
- OpenAI. GPT-5.6 Luna. Model documentation, developers.openai.com/api/docs/models/gpt-5.6-luna, 2026.
- OpenAI. GPT-6 Astra. Model documentation, developers.openai.com/api/docs/models/gpt-6-astra, 2026.
- OpenAI. GPT-6.1 Sol. Model documentation, developers.openai.com/api/docs/models/gpt-6.1-sol, 2026.
- Arjun Panickssery, Samuel R. Bowman, and Shi Feng. LLM evaluators recognize and favor their own generations. In NeurIPS, 2024. arXiv:2404.13076.
- Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. YaRN: Efficient context window extension of large language models. In ICLR, 2024. arXiv:2309.00071.
- Alexandre Piché, Ehsan Kamalloo, Rafael Pardinas, Xiaoyin Chen, and Dzmitry Bahdanau. PipelineRL: Faster on-policy reinforcement learning for long sequence generation. TMLR, 2026. arXiv:2509.19128.
- Prime Intellect Team. renderers: Token-level templating for agentic RL. Blog post, primeintellect.ai/blog/renderers, 2026.
- Prime Intellect. PRIME-RL. Software, github.com/PrimeIntellect-ai/prime-rl, 2025.
- Qwen Team. Qwen3.8-27B. Model card, huggingface.co/Qwen/Qwen3.8-27B, 2026.
- ResolveAI Labs. Inside the Resolve Agentic System. Companion report, claude.ai/artifact/26KLrqsTanWZkCKWPGngZL, 2026.
- John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
- John Schulman. Approximating KL divergence. Blog post, joschu.net/blog/kl-approx.html, 2020.
- Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. Quantifying language models’ sensitivity to spurious features in prompt design or: How I learned to start worrying about prompt formatting. In ICLR, 2024. arXiv:2310.11324.
- Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, et al. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
- Thinking Machines Lab. Inkling: Our open-weights model. Blog post, thinkingmachines.ai/news/introducing-inkling, 2026.
- Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In IROS, 2017. arXiv:1703.06907.
- Pat Verga, Sebastian Hofstatter, Sophia Althammer, Yixuan Su, Aleksandra Piktus, Arkady Arkhangorodsky, Minjie Xu, Naomi White, and Patrick Lewis. Replacing judges with juries: Evaluating LLM generations with a panel of diverse models. arXiv preprint arXiv:2404.18796, 2024.
- Junlin Wang, Jue Wang, Ben Athiwaratkun, Ce Zhang, and James Zou. Mixture-of-Agents enhances large language model capabilities. In ICLR, 2025. arXiv:2406.04692.
- Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In ICLR, 2023. arXiv:2203.11171.
- Xiaomi LLM-Core Team. MiMo-V2.6: Scaling reinforcement learning towards self-improvement. arXiv preprint arXiv:2610.11959, 2026.
- Junjielong Xu, Qinan Zhang, Zhiqing Zhong, Shilin He, Chaoyun Zhang, Qingwei Lin, Dan Pei, Pinjia He, Dongmei Zhang, and Qi Zhang. OpenRCA: Can large language models locate the root cause of software failures? In ICLR, 2025.
- Yixuan Even Xu, Yash Savani, Fei Fang, and J. Zico Kolter. Not all rollouts are useful: Down-sampling rollouts in LLM reinforcement learning. TMLR, 2026. arXiv:2504.13818.
- Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Juncai Liu, et al. DAPO: An open-source LLM reinforcement learning system at scale. In NeurIPS, 2025. arXiv:2503.14476.
- Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in LLMs beyond the base model? In NeurIPS, 2025. arXiv:2504.13837.
- Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, et al. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. In NeurIPS Datasets and Benchmarks Track, 2023. arXiv:2306.05685.
Appendix
A. Explore one training batch (interactive)
This figure is interactive: it opens one training batch of 2,048 rollouts for reading, down to single turns.
B. Training environments
Incident sources
| Source | Share of 1,503-incident dataset | Share of 904-incident dataset | Served from | Reference chain | Base model never solves | Strongest agent solves |
|---|---|---|---|---|---|---|
| Live incidents | 23% | 31% | Live read-only backends and restored snapshots | Cross-family debate | 91% | 69% |
| Incident replays | 34% | 17% | Replay fleet | Cross-family debate | 88% | 75% |
| Synthetic companies | 35% | 44% | Restored snapshots | By construction | 67% | 85% |
| Licensed environments | 8% | 8% | Restored snapshots | By construction | 64% | 82% |
Incident taxonomy
To check that the datamix resembles real on-call work, we used LLMs to categorize every live-tool incident across 24 categories on four dimensions: fix location, fault trigger, fault type, and distance from root cause to symptom (Figure 31). They agree with human reviewers on 89% of labels. The four sources are complementary. Incident replays have the largest shares of roots in third-party systems and platforms and of outages. Live incidents have the largest share of workload-driven failures; licensed environments, faults from the team’s own recent changes; and synthetic companies, failures with no identifiable trigger. Compared with a sample of 400 production investigations labelled identically, the datamix has more bugs in an organization’s own code (50% against 40%), fewer monitor fixes (12% against 19%), and fewer alerts where nothing was wrong (13% against 20%).
An example synthetic company
Quality checks
| Check | An incident is dropped when | Flagged | Excluded |
|---|---|---|---|
| Reference-chain quality | A link restates a symptom or a mitigation instead of following from cited evidence, or pre-alert telemetry does not support the chain | 32 | 28 |
| Solvability | No evidence path reaches the reference chain: the strongest agent never solves it, its best rollout scores below 0.6, and a blind tracer fails | 7 | 4 |
| Open findings | A reviewer has flagged an unresolved problem | 76 | 48 |
| Leakage | Code history, tickets, chat, documents, metric names, or recorded rollouts give away the answer in a way a real investigation would not see | 140 | 63 |
| Deduplication | It is not the earliest alert of its outage | 38 | 9 |
| Decontamination | It shares an incident, or a template whose answer carries over, with a held-out evaluation set | 3 | 4† |
| Difficulty | The strongest agent solves it on every rollout from an ordinary change record | 4 | 4 |
Environment types
Tool use differs by environment type (Figure 33). Call shares are significantly larger than elsewhere for code reads in live environments, metrics in snapshots, and alerts, tickets, and events in replays. Logs account for similar shares in all three.
Production noise
Production tools fail during investigations: backends time out, gateways return errors, and queries hit rate limits. Training should face the same failures, so we keep the real ones and inject none. Of the tool calls that training sent to an environment backend, 1.1% failed there, and 18.1% of training investigations had at least one such failure (Figure 34). All of them stayed in their batches.
The failure rate shows no trend across the run. Most failed calls found their system unreachable or timed out (Figure 34b). Nearly all failures came from backends read live, where 3.5% of calls failed, against 0.32% for replayed telemetry and 0.20% for restored snapshots, which we serve from our own stores (Figure 34c).
An investigation with a failed call still finished with an answer 99.8% of the time. We track these rates at every step and by kind of backend, aiming to match production’s failure rate.
Environment serving
Snapshots. We copy raw storage blocks for logs, metrics, and traces into object storage, together with dashboards, the service graph, alert history, and code. We restore the blocks into open-source query engines on the training cluster, allowing real queries against frozen data. We serve synthetic incidents entirely from snapshots, live incidents partly through read-only live backends, and incident replays through a fleet implementing each vendor’s query language.
Fidelity. To measure how faithful these environments are, we rerun the tool calls that each reference investigation used to reach its answer and compare the results with what production returned. On average, 81% of an incident’s calls return the same result. Most of the other 19% are calls the environment cannot serve at all, because it does not offer that tool or the query exceeds a backend’s scan limit. Of the calls it can serve, 94.5% return the same result as production.
Requests. Every remote tool call in training goes through one service, the dispatcher, which handled 29.9 million backend calls over the run; about 1.1% of them failed the way production calls do. It clamps every windowed call to the alert time and gives a call that names no range the hour before the alert. It rejects SQL clock functions such as now(), which would read present-day data, and clips post-alert dates written into query text; 6%–8% of training rollouts write such dates.
Responses. It rolls alerts and tickets back to their state at alert time, withholds reports edited afterwards, and strips resolution fields so an open ticket reads as an ordinary in-progress record.
C. Reinforcement learning
Training details
We use no KL penalty and no entropy bonus (Section 5.1). Over this run entropy fell from 0.74 to 0.37 nats per token.
- Group selection. We drop groups whose scores have a standard deviation below 0.05, and groups in which no rollout scores 0.6 or more, where differences mostly reflect judge noise.
- Staleness. Generation overlaps training with a staleness bound of two steps. Of trained rollouts, 2% were zero steps stale, 47% one, and 51% two, measured from their starting weights.
- Advantages and weighting. Advantages are centred within each group and scaled by a running estimate of each environment’s reward standard deviation; every rollout gets equal mass, and no token weighs more than 10 times a token of the batch’s longest rollout.
- Why these settings. We bound staleness at two steps because mismatch grows with lag. We use 32 rollouts per prompt because 64 gave no held-out gain at 1.3× the rollout time. We start from the base weights because a run initialized from supervised fine-tuning started higher but gained no more under RL.
- Budgets. Each randomized group draws 196,608–260,096 tokens of context, since in earlier draws the solve rate fell by 3 to 6 points below about 197k, and 30–160 turns, with a 16,642-token answer reserve.
- Numerics. Master weights and Adam moments are fp32. For 93.6% of the base model's weights one step at our learning rate is smaller than half a bf16 spacing, so updates reach the served policy only after several steps.
How the recipe was chosen
Recipe search. The recipe came from a six-week search in which most runs differed from their parent in one setting, because an earlier run that changed many settings at once stalled and did not show which change was responsible.
Stacking. Effects were not additive: four combined changes scored below their baseline (row 18 of Table 7).
Reproduction. Five rounds of screening never beat an earlier run, so we reproduced that run on the newer stack to learn why, and the reproduction exposed the collapse to one call per turn (Section 5.4). The run that we forked from before the collapse, with the hint and the temporary penalty, became SRE-1.
D. Infrastructure and operations
| Run | Value |
|---|---|
| Base model | Qwen3.8-27B, dense, RL from the base weights |
| Hardware | Up to 512 GPUs: 192 for the trainer, 320 for samplers (160 for the last three steps) |
| Duration | 7.4 days, restarts included; 86.6k GPU-hours; median step 47 min |
| Rollouts | 812,695 run to completion; 409,600 trained: 311,680 investigations, 97,920 syntheses |
| Tool use | 51.5 million tool calls by the model and 81B tokens of tool output read, over all completed rollouts |
| Tokens | 76.2B trained, 20.8B of them generated by the policy; 2.5T prompt tokens processed |
| Median rollout | 195k tokens of context, 26 turns, 63 tool calls |
Fault tolerance
Rollouts interrupted by a stop resume after the restart (Figure 35). In early runs, each restart discarded about 1,500 unfinished investigations and an hour or more of generation. Therefore, on a graceful stop we save each rollout at a turn boundary, and the next job continues it if nothing it depends on changed. Where a resume left the environment unchanged, 99.9% of suspended rollouts continued. At the three restarts that changed the environment, we regenerated them instead of risking a mismatched continuation.
Agent-operated development
The training system was built and operated entirely by coding agents with human operators. Human operators direct Claude Code and Codex sessions, which start subagents, write the code, launch and resume the training job, and review each other’s work in writing. One lead researcher’s sessions account for 16,182 agents over 19 weeks (Figure 36). Model judges read the run as well: a “qualitative judge” (Appendix F) wrote 33,908 notes on sampled rollouts, and further model calls summarized those notes for each step and for the whole run.
E. Ablations
Most design decisions in this report rest on an A/B run or a measurement. We report all 34 of them here, including the 23 that were neutral, negative, or not adopted, because negative results are rarely published and we hope they save others the compute. The 28 with per-incident records took 221,000 GPU-hours of training, 2.5 times SRE-1’s own run, and 3.5 million judge calls. The whole recipe search behind this report took 60 training runs, more than 660,000 GPU-hours, and 11 million judge calls. Table 7 numbers them in descending order of impact, and Figure 37 draws the held-out curves behind the same numbers. Most compare one training run per arm for 10 to 20 steps, and several ran on an earlier model, so each supports the decision it tested and may not generalize. Harness and sampling settings moved the score most, by 0.12 to 0.16 at a fixed checkpoint, and two of them decide whether an investigation has room left to answer. In every A/B pair of training runs, the two runs’ held-out scores after the same number of steps (usually 15) differed by less than 0.05; the largest gap, 0.049, was for the wide importance-ratio band (row 4 of Table 7). The intervals resample incidents and do not include differences between repeated evaluations: two evaluations of the same weights differed by 0.008 [0.001, 0.015], so effects near 0.01 are within that difference.
| # | Ablation | Result |
|---|---|---|
| Changes that helped | ||
| 1 | Capping inline tool output at 25k characters (Figure 15) | A 100k cap reduced the score by 0.16 [0.13, 0.20] and left a third of rollouts out of context* |
| 2 | Reserving room for a full final turn and the answer | +0.14 [0.12, 0.16]* |
| 3 | Sampling at temperature 1 instead of 0.6 | +0.12 [0.10, 0.14]* |
| 4 | A wide importance-ratio band instead of a narrow one (Figure 11) | +0.049 after 15 steps [0.032, 0.066], in a pair that differed only in the band† |
| 5 | fp32 gradient accumulation (Figure 26) | +0.026 at step 25 [0.012, 0.040]‡ |
| 6 | Dense credit for tool mistakes | +0.017 [0.006, 0.028]; mistakes fell to 0.44 of their starting rate [0.37, 0.53], against 0.84 [0.69, 1.04] without it; scaled by the fraction of the turn to keep parallel calls |
| 7 | A newer datamix | +0.012 [0.004, 0.019] |
| 8 | One judge per group, from a set of 13 across providers (Figure 12) | A three-judge average tracked a blind reference more closely than a single judge (correlation 0.95 against 0.92; +0.028 [0.006, 0.054]) |
| 9 | A continuous judge score instead of a binary pass | A binary pass gave 27%–36% of hard incidents a trainable group, against 57%–60% under the continuous score; an early binary run scored 0.024 higher at step 10 [0.009, 0.040], and its later rounds are incomplete |
| 10 | Empty results for uncaptured replay queries (Section 4.1) | When a miss returned a connection error, agents abandoned the integration: metric calls fell from 86 to 16 |
| 11 | A hint and a temporary penalty for parallel calls (Figure 19) | Calls per turn recovered from 1.02 to 2.4 by step 60, with the held-out score unchanged |
| Neutral, negative, or not adopted | ||
| 12 | A KL anchor to the sampler | −0.039 [−0.055, −0.024]: it slowed learning |
| 13 | PPO’s clip [41] instead of CISPO | PPO scored 0.036 higher after 10 steps [0.013, 0.059], one run each; we kept CISPO, following ScaleRL* |
| 14 | Starting from supervised fine-tuning, first study | Started 0.08 higher and led by 0.034 after 15 steps [0.023, 0.045] |
| 15 | A budget-relative penalty in the reward | −0.027 [−0.036, −0.017] |
| 16 | Withholding credit for citation mistakes | −0.020 [−0.028, −0.011]: local credit does not always help |
| 17 | Supplying extra written knowledge about the environment | −0.016 [−0.024, −0.008] |
| 18 | Stacking four single changes | −0.012 against its own base [−0.021, −0.004]: effects did not add |
| 19 | Stronger speed incentives in the reward | −0.012 [−0.021, −0.003] |
| 20 | SAPO’s soft gate instead of CISPO [19] | No significant difference after 10 steps (+0.022 [−0.005, 0.047])* |
| 21 | An instruction-only nudge | No significant difference (+0.012 [−0.008, 0.030])* |
| 22 | 64 rollouts per incident instead of 32 | No significant difference (−0.009 [−0.027, 0.009])† |
| 23 | Synchronous sampling instead of asynchronous | No significant difference (+0.006 [−0.020, 0.033]); we keep asynchronous sampling because synchronous stepping idles whichever side is faster* |
| 24 | Starting from supervised fine-tuning, second study | Started 0.09 higher; no significant lead after 15 steps (+0.006 [−0.012, 0.025]) |
| 25 | A higher minimum context | No significant difference (+0.005 [−0.004, 0.014]) |
| 26 | A notice of the remaining budget | No significant difference (+0.003 [−0.017, 0.022])* |
| 27 | The newer datamix, on the next recipe | No significant difference (−0.002 [−0.010, 0.005]) |
| 28 | A cost on tool calls | No significant difference (+0.002 [−0.005, 0.010]) |
| 29 | Other precision changes | No significant difference (−0.001 [−0.011, 0.008]) |
| 30 | Prompt-balanced loss weighting | No significant difference (0.000 [−0.007, 0.008]) |
| 31 | A larger fixed budget | More turns, no more incidents solved* |
| 32 | A fork that reset the optimizer state | Weights moved less over the next 20 steps (one pair; score not compared) |
| 33 | Telemetry synthesized by a model | Picked the better of two models only 66% of the time against live runs [54%, 75%], so we removed it |
| 34 | Reference chains written by experts instead of the agent team | No significant change in score (0.012 [−0.013, 0.034]), so we stopped routine annotation |
F. Qualitative analysis
The “qualitative judge”. Reading the data is one of the most useful practices in machine learning, and this run produced far more than a person can read: 409,600 trained rollouts with a median of 195k tokens of context each. To read it at scale, we built a hierarchy of LLM readers that we call the “qualitative judge”. It escalates the anomalies that it finds, including problems we did not anticipate. It has four levels:
- Read. A reader goes through one whole rollout (the prompt, every turn with its reasoning, the tool calls and their results, and the reference chain) and writes typed notes: known failure modes, and open-ended observations of anomalies. Each note must quote its evidence verbatim, and a mechanical check drops any note whose quote is not in the transcript.
- Pool. A second level pools the notes of one training step into findings, with counts computed by code and not by the model.
- Compare. A third level compares steps and reports which findings improved, worsened, or persisted.
- Investigate. A fourth level takes a single finding and investigates it down to a proposed cause.
Over this run the first level wrote 33,908 notes, on a 20% sample of each evaluation round and on sampled training rollouts. We treat what it reports as leads to check: every rate below was recomputed from the typed notes.
Evidence retrieval. After RL, fewer rollouts fail because they never retrieved the deciding evidence. In the “qualitative judge”’s notes on the same incidents, the share of rollouts that failed because the deciding evidence was never retrieved fell from 66% for the base model to 46% for checkpoints 100 to 110, a drop of 20 points [13, 28] (inset of Figure 38b). Rollouts that retrieved the evidence but left it out of the answer rose by 10 points. The judge sees each rollout’s score, so this reading is not independent of the score gain.
Paired investigations. On the same held-out incidents at a 262k window, SRE-1 searches in parallel where its base model makes one call at a time (Figure 38). SRE-1 makes two or more tool calls in 89% of its turns and answers in 39% fewer turns; the base model makes exactly one call in 88% of its turns. Its investigations nonetheless use more calls and more tokens: 74 tool calls per rollout against 55, and a context that peaks at 224k tokens against 133k, since it also reasons more than twice as long. SRE-1 often runs to the end of its window: in 38% of rollouts it answers only when the harness asks, which the base model never did. Appendix G follows one incident through training.
G. Additional results
| Model | Setting | Mean score | Pass rate (%) | Cost (×) | |||
|---|---|---|---|---|---|---|---|
| SRE-1 ensemble | ensemble of 5 | 0.83 | [0.79, 0.87] | 71.4 | [57.1, 83.7] | 7.38 | [6.60, 8.18] |
| SRE-1 ensemble | ensemble of 3 | 0.80 | [0.76, 0.84] | 59.9 | [46.3, 72.8] | 3.77 | [3.34, 4.20] |
| SRE-1 ensemble | ensemble of 2 | 0.76 | [0.72, 0.80] | 50.2 | [37.6, 62.8] | 2.74 | [2.41, 3.06] |
| SRE-1 | 1M context | 0.70 | [0.66, 0.74] | 38.8 | [28.2, 49.8] | 1.00 | [0.88, 1.12] |
| SRE-1* | 1M context, guidance to work faster | 0.67 | [0.62, 0.72] | 36.2 | [25.2, 47.5] | 0.70 | [0.62, 0.78] |
| SRE-1 | 524k context | 0.66 | [0.62, 0.71] | 33.1 | [22.0, 44.1] | 0.49 | [0.45, 0.53] |
| SRE-1 | 393k context | 0.61 | [0.56, 0.66] | 24.1 | [15.5, 33.1] | 0.35 | [0.33, 0.38] |
| SRE-1 | 262k context, 60-turn cap | 0.54 | [0.48, 0.60] | 15.5 | [8.2, 24.1] | 0.21 | [0.20, 0.22] |
| Fable 5.2* | xhigh effort | 0.84 | [0.81, 0.87] | 71.0 | [60.4, 80.4] | 10.45 | [9.55, 11.38] |
| Opus 5.5 ensemble | ensemble of 5 | 0.83 | [0.78, 0.87] | 65.3 | [51.0, 77.5] | 30.01 | [27.25, 32.97] |
| Fable 5.1 | xhigh effort | 0.73 | [0.69, 0.78] | 48.2 | [38.0, 59.2] | 10.75 | [9.95, 11.63] |
| Opus 5.5 | xhigh effort | 0.74 | [0.70, 0.78] | 47.8 | [37.1, 58.8] | 4.51 | [4.04, 5.03] |
| Astra* | xhigh effort | 0.70 | [0.64, 0.75] | 38.4 | [26.9, 49.8] | 7.92 | [6.89, 9.06] |
| Sol 6.1 | xhigh effort | 0.64 | [0.57, 0.69] | 29.8 | [19.6, 40.4] | 1.18 | [1.03, 1.32] |
| DeepSeek V4.1 Flash 1017 | xhigh effort | 0.50 | [0.45, 0.55] | 8.6 | [2.9, 15.5] | 0.20 | [0.18, 0.23] |
| Haiku 5.5 | xhigh effort | 0.44 | [0.39, 0.49] | 8.2 | [2.9, 14.3] | 0.54 | [0.49, 0.58] |
| Qwen3.8-27B (base) | 1M context | 0.44 | [0.39, 0.49] | 6.9 | [1.6, 13.9] | 0.40 | [0.37, 0.44] |
| Luna* | xhigh effort | 0.41 | [0.35, 0.46] | 4.1 | [0.8, 9.0] | 0.14 | [0.11, 0.16] |
| Model | Setting | Mean score | Pass rate (%) | Cost (×) | |||
|---|---|---|---|---|---|---|---|
| SRE-1 ensemble | ensemble of 5 | 0.83 | [0.79, 0.87] | 71.4 | [57.1, 83.7] | 7.38 | [6.60, 8.18] |
| Fable 5.2* | xhigh effort | 0.84 | [0.81, 0.87] | 71.0 | [60.4, 80.4] | 10.45 | [9.55, 11.38] |
| Opus 5.5 ensemble | ensemble of 5 | 0.83 | [0.78, 0.87] | 65.3 | [51.0, 77.5] | 30.01 | [27.25, 32.97] |
| Fable 5.2 | high effort | 0.81 | [0.78, 0.84] | 64.9 | [54.7, 74.7] | 7.97 | [7.27, 8.72] |
| Opus 5.5 ensemble | ensemble of 2 | 0.79 | [0.75, 0.84] | 61.2 | [46.9, 73.5] | 11.07 | [10.05, 12.17] |
| SRE-1 ensemble | ensemble of 3 | 0.80 | [0.76, 0.84] | 59.9 | [46.3, 72.8] | 3.77 | [3.34, 4.20] |
| Opus 5.5 ensemble | ensemble of 3 | 0.80 | [0.76, 0.85] | 59.2 | [44.9, 73.5] | 16.28 | [14.50, 18.15] |
| SRE-1 ensemble | ensemble of 2 | 0.76 | [0.72, 0.80] | 50.2 | [37.6, 62.8] | 2.74 | [2.41, 3.06] |
| Fable 5.1 | xhigh effort | 0.73 | [0.69, 0.78] | 48.2 | [38.0, 59.2] | 10.75 | [9.95, 11.63] |
| Opus 5.5 | xhigh effort | 0.74 | [0.70, 0.78] | 47.8 | [37.1, 58.8] | 4.51 | [4.04, 5.03] |
| SRE-1 | 1M context | 0.70 | [0.66, 0.74] | 38.8 | [28.2, 49.8] | 1.00 | [0.88, 1.12] |
| Astra* | xhigh effort | 0.70 | [0.64, 0.75] | 38.4 | [26.9, 49.8] | 7.92 | [6.89, 9.06] |
| Fable 5.2* | medium effort | 0.72 | [0.69, 0.76] | 38.3 | [28.1, 49.3] | 4.74 | [4.28, 5.24] |
| SRE-1* | 1M context, guidance to work faster | 0.67 | [0.62, 0.72] | 36.2 | [25.2, 47.5] | 0.70 | [0.62, 0.78] |
| SRE-1 | 524k context | 0.66 | [0.62, 0.71] | 33.1 | [22.0, 44.1] | 0.49 | [0.45, 0.53] |
| Sol 6.1 | xhigh effort | 0.64 | [0.57, 0.69] | 29.8 | [19.6, 40.4] | 1.18 | [1.03, 1.32] |
| Fable 5.2 | low effort | 0.67 | [0.63, 0.71] | 29.4 | [20.4, 38.8] | 3.96 | [3.53, 4.42] |
| Astra* | high effort | 0.66 | [0.60, 0.71] | 29.2 | [18.4, 40.2] | 7.01 | [6.14, 7.95] |
| Opus 5.5 | high effort | 0.65 | [0.60, 0.69] | 26.5 | [18.4, 35.5] | 2.76 | [2.43, 3.11] |
| Astra* | medium effort | 0.61 | [0.55, 0.67] | 24.2 | [14.0, 34.7] | 5.26 | [4.65, 5.90] |
| SRE-1 | 393k context | 0.61 | [0.56, 0.66] | 24.1 | [15.5, 33.1] | 0.35 | [0.33, 0.38] |
| Sol 6.1 | high effort | 0.63 | [0.56, 0.69] | 24.1 | [14.7, 34.3] | 1.07 | [0.95, 1.19] |
| Sol 6.1 | medium effort | 0.57 | [0.51, 0.63] | 18.4 | [9.4, 27.8] | 0.78 | [0.68, 0.88] |
| SRE-1 | 262k context, 60-turn cap | 0.54 | [0.48, 0.60] | 15.5 | [8.2, 24.1] | 0.21 | [0.20, 0.22] |
| Opus 5.5 | medium effort | 0.59 | [0.54, 0.63] | 15.5 | [8.2, 24.1] | 2.18 | [1.95, 2.42] |
| DeepSeek V4.1 Flash 1017 | xhigh effort | 0.50 | [0.45, 0.55] | 8.6 | [2.9, 15.5] | 0.20 | [0.18, 0.23] |
| Haiku 5.5 | xhigh effort | 0.44 | [0.39, 0.49] | 8.2 | [2.9, 14.3] | 0.54 | [0.49, 0.58] |
| Astra | low effort | 0.48 | [0.42, 0.53] | 7.8 | [2.5, 14.3] | 2.94 | [2.59, 3.30] |
| Sol 6.1 | low effort | 0.45 | [0.38, 0.51] | 7.3 | [2.9, 13.1] | 0.45 | [0.40, 0.50] |
| Qwen3.8-27B (base) | 1M context | 0.44 | [0.39, 0.49] | 6.9 | [1.6, 13.9] | 0.40 | [0.37, 0.44] |
| Luna* | xhigh effort | 0.41 | [0.35, 0.46] | 4.1 | [0.8, 9.0] | 0.14 | [0.11, 0.16] |
| Qwen3.8-27B (base) | 1M context, low effort | 0.36 | [0.31, 0.41] | 4.1 | [0.8, 9.0] | 0.28 | [0.25, 0.31] |
| Opus 5.5 | low effort | 0.33 | [0.29, 0.39] | 1.6 | [0.0, 4.9] | 0.57 | [0.48, 0.67] |
Offline evaluation set
We built the offline evaluation set with 80 problems. Two audits after training found seven whose answer also appears in training. For most of them, the same root cause recurs in a training problem built from another firing of the alert, and one is the same incident as a live-tool training incident. We removed all seven, and every offline result in this report is on the remaining 73. On the original 80, SRE-1 passes 52.3% of rollouts, against 45.0% for Opus 5.5 and 38.1% for Fable 5.2, and its lead over Opus 5.5 is 7.3 points [1.1, 13.9], against 5.8 points [−0.7, 12.2] on the 73. Up to ten of the remaining problems share part of a causal mechanism with a training incident, and we kept them. We have not yet run this check on the live production cases.
Context length
The median training investigation ended with 127k tokens in context over the first twelve steps, and with 218k over the last twelve. The training window was at most 262k tokens. On held-out incidents at SRE-1’s checkpoint, the median investigation peaks at 238k tokens of context under a 262k window, 349k under 393k, and 441k under 1M (Figure 39).
We train SRE-1 at the base model’s native 262k-token window and serve it with up to 1M tokens by scaling its rotary position embeddings fourfold [35]. On held-out incidents, the score rises with the window at almost every checkpoint (Figure 21a): 1M scores above 262k at 39 of 40. At SRE-1’s checkpoint the score is 0.60 at 262k, 0.63 at 393k, and 0.66 at 1M.
Case study: a synthetic check that broke after a feature-flag rollout
Alert. A synthetic browser check on our own product paged: highlighting text in a report and asking a question about it failed in the test browser.
Reference chain. A configuration commit turned on a feature flag for threaded comments in two environments. With the flag on, a highlight-and-ask comment opens a thread panel, and the comment panel the check waits for never mounts, so its 15-second wait times out. The check failed continuously from a few hours after the commit and paged six days later.
Step 15. The rollout made 116 tool calls, one per turn. It read the test and the sidebar code, examined the check’s status series for many turns, and concluded that the failures began six days before the flag commit. At turn 108, it found the commit, noted the mismatch, and reexamined deployments from the earlier day. It attributed the failures to an unrelated frontend rollout. One scoring judge wrote: “Same symptom, wrong trigger and mechanism.” The answer scored 0.43.
Step 100. The rollout made 72 tool calls in 40 turns, two or three at a time in 28 of them. It read the test, the sidebar, and the panel code in paired calls, reached the flag definition and the same commit by turn 37, and based its chain on them: the flag routes a highlight-and-ask comment through the thread path, opening the thread panel instead of the comment panel. The answer scored 0.92.
Across eight rollouts each, this incident’s mean score was 0.23 for the base model, 0.54 at step 15, and 0.84 for SRE-1, ten steps after the rollout above. We chose this incident because the “qualitative judge” read a rollout from each step. It is one example and may not be typical.
H. Bespoke training dashboard
We built a web UI to monitor training as it runs. It records every rollout, turn, and token, places samplers, trainer, environments, and judges on a common time axis, and opens any element down to the token ids the sampler returned (Figure 40). We record at this granularity because several of the failures described above raised no error.