Treat Reasoning Controls Like API Migrations
On Claude Opus 5, disabling thinking can turn a structured tool call into visible prose. The request finishes successfully, the tool never runs, and the fake call stays in conversation history.
The failure crosses a protocol boundary. A setting that looked like a cost and latency control changed the interface between the model and its harness. Reasoning controls should therefore ship like API migrations: version the configuration with the adapter, test the wire behavior, and assume old conversation state may be incompatible.
The thinking switch can change the response envelope
Anthropic documents two occasional artifacts when Opus 5 runs with thinking disabled. The model may print a tool call instead of emitting a structured tool_use block, or expose internal XML tags in user-facing output. Its Opus 5 prompting guide recommends keeping thinking enabled at low effort rather than disabling it.
The first artifact is especially nasty in an agent loop. An HTTP success tells the harness nothing went wrong. No schema validator sees a tool call because the response is ordinary text. The model then reads its own unexecuted action on the next turn and may reason as though the search, write, or API request happened.
A task-level evaluation might record only a worse answer. The useful diagnosis is more specific: the response crossed from the action channel into the message channel.
Hidden reasoning has become continuation state
Other providers expose the same coupling from a different direction. Gemini attaches encrypted thought signatures to model responses so later steps can continue the reasoning behind a function call. For Gemini 3, the client must return each signature in the exact response part where it arrived. Omitting one from a current-turn function call produces a 400 error, according to Google's thought-signature documentation.
OpenAI's current model guidance similarly treats reasoning as state that can persist across turns. A manually managed history must preserve every output item and replay encrypted reasoning items when server-side storage is unavailable. Programmatic tool calls also add program and program_output items with their own linkage requirements. The migration guidance tells developers to test reasoning settings on representative work rather than assume more effort is always better.
The tool schema is only one layer of this interface. Response item types, call IDs, encrypted reasoning state, and replay rules now belong to the contract too.
A reasoning migration needs four contract tests
Treat the tuple model snapshot + reasoning mode + effort + API mode + history adapter as one deployable version. Changing any field should run the same conformance suite.
| Contract | Failure to catch | Minimum assertion |
|---|---|---|
| Action envelope | A tool call appears as text or malformed JSON | Every requested action arrives in the provider's structured action channel and validates against the declared schema |
| Continuation | A signature, reasoning item, or call link is lost | A sequential two-tool task completes when history is stored by the provider and when the client replays it |
| Recovery | A malformed action contaminates later turns | The harness quarantines invalid output and never records an unexecuted action as a completed event |
| Behavior | The envelope is valid but routing drifts | Repeated runs stay within budgets for tool choice, argument validity, completion, latency, and cost |
The recovery test deserves its own event model. Store proposed, validated, executed, and observed as different states. Visible text that resembles a function call remains text. It cannot become an executed event merely because the next model turn refers to it.
Test the paths your production traffic actually uses
A useful fixture set includes a no-tool answer, one required tool call, parallel calls, two sequential calls, and a request where no declared tool fits. Add a tool error and a refusal if those occur in the product. Run every fixture against the old and new configuration with the same prompt, schemas, adapter, and history policy.
One pass is insufficient because valid envelopes can still contain unstable choices. A 2026 study of multi-step tool-calling agents separated tool-sequence consistency from argument consistency and found that agents could follow a stable procedure while varying the parameters they sent. Its evaluation framework is a useful reminder to compare structure and arguments separately across repeated trials.
Gate the rollout on hard invariants first. No action-as-text, no missing linkage, no execution before validation, and no completed event without a tool result. Then compare distributions for success, tool sequence, arguments, tokens, latency, and cost. An average task score cannot explain which contract moved.
The migration analogy has a limit
A database schema migration has deterministic compatibility rules. A model is stochastic, so an exact golden trajectory will reject harmless variation and still miss consistently bad behavior.
Keep protocol assertions exact and behavioral assertions statistical. The response item must have the right type every time. The agent may choose either of two safe search tools if both satisfy the task, but its failure rate and cost still need an explicit budget.
When a reasoning setting changes response types, history requirements, or tool behavior, create a new adapter version and run the conformance matrix before sending production conversations through it. If the old and new versions cannot replay the same history safely, start a fresh conversation rather than hot-swapping the model underneath live state.
Bundle the reasoning configuration with the adapter and test both as one protocol.