Speculative Agents Need a Commit Boundary

The fastest agent architectures start work before they know whether they will need it. That can remove seconds from an interactive turn, but only if the system separates discardable computation from effects that cannot be taken back.

This is speculative agent execution: predict likely next actions, launch reversible work in parallel, and commit only the branch that an authoritative result confirms.

Serial tool loops put every delay on the critical path

A conventional agent generates a tool call, waits for it, observes the result, and generates again. The order is logically clean and painfully serial. Faster token generation helps, but it does not hide network round trips, tool execution, or the next model call.

PASTE attacks that exposed tool time directly. It learns recurring sequences such as search followed by visit, or edit followed by test, then starts likely tool work while the model is still generating. Speculative results stay isolated until the model confirms the call.

Interactive agents can also overlap work with the user. OpenAI notes that continuous WebRTC audio lets a system start transcribing, reasoning, or calling tools while someone is still speaking. The useful latency metric becomes time to the first safe action, not time to the complete response.

Speculation turns idle time into compute time

Speculative Interaction Agents decouples the agent from streaming user input and in-flight tools. Its authors report 1.3–1.7x speedups for cloud real-time APIs, with a minor accuracy loss, by letting reasoning and acting continue while information arrives.

An Ello engineering write-up describes the same pattern at application level: stream actions as they are generated, run slower planning outside the response path, and pre-generate likely branches before the user's next input arrives.

The mechanism is broader than voice or tutoring. Any workflow with predictable next steps can trade extra compute for shorter wall-clock time. The important question is which work may cross the boundary before the prediction is verified.

A commit boundary is not a rollback button

Speculation is safest when it produces local artifacts: candidate text, parsed data, cached reads, or a prepared payload. A missed prediction can delete those artifacts without affecting anyone outside the runtime.

External tools are different. The Ghost Tool Calls paper found that speculative requests can disclose inferred user intent to a third-party service even when the agent later abandons the branch. A read-only call may preserve database state while still leaking the query. No rollback can unsend it.

Irreversible effects need an even harder boundary. Speech can be heard, messages can be delivered, and payments can settle. Prepare those actions early if it saves time, but do not apply them until the authoritative model, policy checks, and current state all agree.

This is why prepare, commit, and cancel should be explicit runtime operations rather than comments inside a tool handler. The distinction must exist below the model, where the system can enforce it.

Build the commit point before adding branches

Start with four pieces:

1. Record user input, model output, tool results, and branch decisions in an append-only event log.
2. Classify every tool by external visibility and reversibility, not just whether it mutates state.
3. Trace prediction hit rate, wasted compute, time to first safe action, and stale-plan errors.
4. Add speculation first to local, idempotent work where a miss is cheap.

Only then speculate across network or state boundaries. The goal is not to make an agent act before it knows. It is to overlap knowing and acting without letting a guess escape as a consequence.