Loading the page…
Shaping the future of software engineering
Without a way to test your agents, you can't tell whether an incident got harder, the agent got worse, or you just got lucky when you checked. Evals settle it — and let a team qualify a new frontier model in days instead of weeks.
On τ-bench's retail tasks, an agent powered by GPT-4o succeeded about 61% of the time in a single trial, but solved the same task eight times in a row only about 25% of the time (Sierra, τ-bench).
Anthropic recommends starting an eval suite with 20 to 50 tasks drawn from real failures, and warns that evals get harder to build the longer you wait (Demystifying evals for AI agents).
Strong LLM judges agree with human preferences over 80% of the time, roughly the rate at which humans agree with each other, but show position, verbosity, and self-enhancement biases (MT-Bench).
An agent eval is a repeatable test for an AI agent: give it a task, let it run, and grade its output. Each attempt is a trial, the full record of a trial is a transcript, and a grader is the logic that scores some part of it.
What separates agent evals from ordinary LLM evals is the thing under test. A single-turn eval checks one prompt and one response. An agent eval checks a whole system: the model, the system prompt, the tool definitions, the memory, and an environment whose state changes as the agent works. Change any one of those and the score can move.
The grade also comes from the environment, not from what the agent said. It is about the entire outcome. A flight-booking agent can say it booked a flight, but the eval has to check the database for the booking. For an agent that investigates alerts, the equivalent is whether it identified the deploy that actually caused the incident, not whether its summary sounded confident.
Agents are non-deterministic, which means they never behave the same way twice. Model outputs vary between runs, and in a multi-step task, a small early difference changes what the agent sees next, so the whole run goes differently. A task that passed on run 1 will be subtly different on run 2, run 3, or run k.
The fix is repeated trials, reported as a distribution. Two metrics carry most of the weight:
An alert investigation shows why pass^k matters. When a page fires at 3 am, the agent runs once, and the on-call engineer acts on whatever comes back. There's no picking the best of five attempts, because nobody knows which attempt was right without doing the investigation themselves. An agent that's right three times out of four sounds fine, but on-call, that's a wrong lead every fourth incident.
You grade with code where you can, with models where code can't judge, and with humans where the stakes are high:
| Grader | What it checks | Why it's used | Tradeoff |
|---|---|---|---|
| Code | Anything a script can verify: the tests pass, the database row exists, no forbidden tool was called | Fast, cheap, and reproducible, so it runs on everything first | Only works for things a script can check |
| Model (LLM-as-judge) | Open-ended qualities, like whether an investigation summary is grounded in the evidence it cites | Scales far past what humans can review | Drifts with prompt wording and carries the judge biases above, so it needs calibration against human labels |
| Human experts | Rubrics, calibration sets, high-risk failures, and cases where the judges disagree | Sets the standard the other graders are measured against | Slow and expensive, so best saved for where the stakes are high |
What gets graded matters as much as who grades it. Requiring one exact sequence of tool calls marks valid alternative approaches as failures. Hence, the safer default is to grade what the agent produced and keep trajectory checks for diagnosis and safety limits.
There are a couple of gotchas to think about when grading agents:
Agents optimize for the grade rather than for what you meant by it. So keep answer keys and hidden tests out of the agent's reach, and when a score jumps suddenly, read the transcripts before believing it.
Start small and treat it as a loop rather than a one-time project. The sequence Anthropic recommends, and the one most eval tooling assumes, looks like this.
The first three steps build the suite:
From there, it runs as a loop:
Over time, the suite splits in two. Capability tasks are the ones the agent still fails, and they give the team room to improve. Regression tasks must stay near a 100% pass rate, and capability tasks graduate into them once the agent passes reliably.
Ground truth is scarce. Coding agents can be graded mechanically. They run the test suite, and green means the change worked. SWE-bench Verified grades this way, and scores on it moved from around 40% to above 80% in a single year. Troubleshooting has nothing to run, and most organizations lack the raw material to build a substitute:
The result is that most organizations can't build these evals themselves: the incident data was never captured cleanly, and the expertise to reconstruct it sits with a handful of engineers who are busy running production.
Resolve AI's agents investigate alerts and incidents across code, infrastructure, and telemetry, which is exactly the work where ground truth is scarce and a wrong answer is expensive. Its eval suite tests whether an alert was correctly troubleshot and stays indifferent to how the answer was produced, so nothing in it is written against any one model's behavior.
That's what makes a frontier model release a normal week. The new model goes in, the same evals run, and the team has a decision within a day or two. The suite also catches what accuracy alone hides: one cost optimization looked great until evals showed it would take two to two and a half times longer to reach an answer. This regression would otherwise have shipped straight to customers.
Henry Maier, who works on evals at Resolve AI, walks through the approach in the video above. Engineering teams at Coinbase, DoorDash, Toast, and Zscaler use Resolve AI on their production systems. The quickest way to judge it is to have it run one of your real incidents.
Common questions about agent evals