What Pass Rates Hide About Long-Horizon Agents

A 0% pass rate can make two agents look equally useless. One may fail before opening the right file. The other may reach the final verification step on every run, then time out. Those systems need different fixes, but the headline score erases the difference.

Long-horizon agent evaluations need a failure curve. At a fixed budget, report the share of runs that reach each externally verified milestone. The curve shows where reliability collapses instead of compressing an entire trajectory into one pass rate.

Partial credit is still one number too few

End-to-end success remains the metric users care about. A support agent either applied the correct refund or it did not. A coding agent either produced an acceptable patch or it did not.

As a diagnostic, the final result is too coarse. A mean partial-credit score reveals more, but it can still hide the bottleneck. Two agents can average 60% while one reliably clears the first three stages and another alternates between immediate failure and complete success.

AgentBoard introduced a progress-rate metric for multi-turn agents because success rate missed incremental improvements. That metric instruments the states between start and finish, but averaging those states would hide the bottleneck again. Keep the reach rate for each state separate.

ExploitBench exposes the steps between crash and code execution

ExploitBench shows what this instrumentation looks like for software exploitation. A working exploit passes through several distinct capabilities: reaching vulnerable code, reproducing a crash, constructing useful read or write primitives, hijacking control flow, and achieving arbitrary code execution.

The benchmark turns that path into 16 measurable flags across five capability tiers. Each flag has a deterministic verifier, including challenge-response checks for exploit primitives and a signal-handler proof for code execution. A plausible explanation from the model earns nothing by itself.

That design distinguishes agents that can find a bug from agents that can turn it into control. A binary benchmark aimed at the crash would call both successful. A benchmark aimed only at code execution would call most current systems failures. The capability ladder preserves the transition between them, so evaluators can see the first capability an agent fails to demonstrate.

A failure curve reports reach at every milestone

For milestone $i$ and budget $B$, define:

Plot $S_i(B)$ across the ordered milestones. The result looks like a survival curve: every drop is a set of runs that reached one verified state but not the next.

Read the curve at three levels:

The conditional transition rate is $S_{i+1}(B) / S_i(B)$. Its largest drop is the first place to inspect. If retrieval passes but source verification collapses, buying a larger model may be less useful than fixing the evidence tool. If the drop moves later as the budget increases, the lower score may have been resource-censored rather than a hard capability boundary.

The curve is uninterpretable without its budget. The UK AI Security Institute found that fixed compute caps can hide agent capability, especially on longer tasks, and recommends reporting reach and reliability against budget. One score at one cutoff is a point sampled from a curve.

A budgeted curve still needs repeated runs. $\tau$-bench uses $pass^k$ to measure whether an agent succeeds consistently across trials. Pairing $pass^k$ with milestone reach shows whether repeated failures share a bottleneck or wander through different ones.

Kimi K3 separates completion, depth, and budget

In a NIST and UK AISI evaluation of Kimi K3, the model scored 32% overall on ExploitBench but reached arbitrary code execution on 0 of 41 samples. On The Last Ones, a 32-step simulated network attack, it reached step 17 on average within a 100-million-token limit and completed the range once in ten attempts.

Completion establishes whether the end state is attainable; average depth locates progress before failure; the token cap records how much opportunity the agent had. A full milestone curve would identify the transition where most attempts stopped.

Milestones break when the work has many valid paths

A ladder assumes ordered progress. Many workflows branch, revisit earlier states, and sometimes undo work. AgentBoard had to simplify a small share of its tasks so each goal followed a unique subgoal sequence. An arbitrary checklist can therefore mislabel valid paths.

For branching work, use a graph of acceptable state predicates instead of one hand-written path. Verify durable facts such as “tests cover the regression” or “the database matches the goal state,” not activity such as “the agent called the test tool.” Keep final acceptance tests alongside the progress instrumentation. Intermediate milestones diagnose a failed run; they do not make it acceptable.

Start by naming the four to eight externally verifiable states that separate a task's major failure modes. Run multiple trials under a stated budget, report the reach rate for every state, then investigate the largest conditional drop. Keep the pass rate as the acceptance metric and use the failure curve to decide what to change next.