An agent is only as reliable as its worst JSON. Sample a simulated token stream, watch small per-token slips compound into unparseable output, then switch on grammar masking and price the retries with real formulas.
Sign in to save progress
The contract: one schema, one token path
The task: emit a tool call for the weather in Oslo, in celsius, as one exact token path. The generator picks from a vocabulary of 24 tokens at every step, and a single grammar-illegal pick anywhere in the path makes the whole output unparseable. That is why per-token reliability compounds so brutally.
Intended token path (18 tokens)
{ "tool" : "get_weather" , "args" : { "city" : "Oslo" , "units" : "c" } } <end>Three positions are value slots (tool, city, units). Everything else is structure: exactly one token is legal there.
Token vocabulary (24 tokens)
Schema strictness: Strict enums
Each value slot has an enum: tool must be get_weather, city one of three known cities, units c or f.
tool slot accepts 1 of 24 tokens
city slot accepts 3 of 24 tokens
units slot accepts 2 of 24 tokens
This guide is the reliability practice layer: what schemas and masking buy you and what retries cost. How constrained decoding manipulates the logits under the hood is a topic for the LLMs section; here the mask is simply a fact you can toggle.
The sampling lab
Set a per-token error rate, pick a schema strictness, and sample. Unconstrained, the generator emits whatever it draws. Constrained, every draw is checked against the grammar: illegal tokens are masked and the generator redraws, which is renormalization by rejection sampling. All samples come from a seeded PRNG, so the same settings always reproduce the same numbers.
To earn completion, push the error rate to 20% or higher and run a 500-generation batch in each decoding mode. Watch what happens to the valid rate, and watch which failure class refuses to die.
Retry economics
A schema-invalid output is at least detectable: the parser rejects it and you retry. Attempts to the first valid output follow a geometric distribution, so the expected attempt count is 1 / p where p is the valid rate at your current settings. Multiply by your price per call and the retry bill is exact.
Unconstrained at 10% error, Strict enums
Constrained at the same settings
The mask guarantees every emission parses, so one call is one valid output. Masking itself adds some inference overhead per token; how it works under the hood is its own topic.
Retry table, unconstrained: P(first success at attempt k) = (1 - p)^(k-1) · p
| Attempt k | First success exactly here | Valid output by attempt k | Spend by k |
|---|---|---|---|
| 1 | 16.54% | 16.54% | $0.0100 |
| 2 | 13.80% | 30.34% | $0.0200 |
| 3 | 11.52% | 41.86% | $0.0300 |
| 4 | 9.61% | 51.47% | $0.0400 |
| 5 | 8.02% | 59.50% | $0.0500 |
| 6 | 6.70% | 66.20% | $0.0600 |
The failure no retry loop can see
At 10% error with strict enums strictness, constrained decoding still yields 1.37% valid-but-wrong outputs: perfectly parseable JSON that calls for the wrong city or the wrong units. The parser says yes, the schema says yes, and only an evaluation of the content catches it. Tighter enums shrink this class; only evals close it.