Structured Outputs: JSON You Can Actually Parse

Every system consuming model output faces the same moment of truth: the answer arrives as text, and the system needs it as structure — a JSON object, a list of fields, a value fitting an enum. When the text parses, the pipeline hums. When it does not — a trailing comma, a prose preamble, a field renamed by the model’s mood — the pipeline stops, and the failure is yours to handle.

This is not a rare edge. It is the single most common integration failure in production LLM systems, and it is expensive in a way the pricing page never shows: every malformed answer is either a retry (paid twice) or a dropped request (a user lost).

The fix is a discipline, not a hope: constrain the model, validate the result, and treat the parse gate as part of the product, not part of the prompt.

Because they are trained to be helpful in prose — and helpfulness includes helpful extras. Ask for JSON and the model may wrap it in a sentence, add a comment field it thinks you would appreciate, or rename a key to something it considers clearer. None of this is misbehaviour from the model’s perspective; it is the model doing what it was trained to do — producing text that helps.

Temperature makes it probabilistic: the same prompt produces the schema today and a near-miss tomorrow, and a near-miss is still a miss to a parser. Long outputs make it likelier: more fields, more chances for one to drift.

And the prompt is not the contract. “Respond in JSON” is a request, not a constraint — the model can and will decline the request in small ways. The constraint has to come from the API, not the prose.

The fix has two strengths, worth distinguishing. JSON mode constrains the output to valid JSON — no preamble, no trailing prose, no markdown fences. That alone eliminates the most common failure class. Schema enforcement goes further: the output must conform to a schema you supply, with the right fields, the right types, nothing else.

Where the platform supports schema enforcement, use it for anything a parser consumes. It converts “the model might” into “the API guarantees” — and a guarantee is what a pipeline needs.

Where only JSON mode exists, pair it with a validation gate on your side: parse, validate against the schema, and on failure retry with the error message appended. The resilience post covers the retry mechanics; the point here is that the gate is the contract and the retry is the enforcement.

Even with schema enforcement, keep validating. The schema guarantees shape, not sense: a field can be the right type and the wrong value; an enum can be respected and still be the wrong choice. The parse gate from the resilience post — “the response is not done until it parses and validates” — is the minimum, and business validation is the next layer.

Validation also decides the retry policy. A response failing the schema is worth retrying with the error attached; a response passing the schema but failing a business rule is a different failure, and retrying it blindly is pure spend. Classify before retrying.

And log the failure rate. Schema violations per hundred calls deserves a dashboard slot next to latency — it is the earliest symptom of prompt drift, a model change, or a temperature setting that stopped being cute.

It costs more than the retry. The failed attempt bills in full — prompt tokens, generated tokens, the parse error. The retry bills again, usually longer because the error message rode along. The latency of two round trips where one was promised. And, worst case, the user who left between attempt one and attempt two.

The cost-per-task framework is built for exactly this: the all-in cost of an accepted answer, retries included, is the number that matters, and a schema that fails often is a hidden multiplier on it. Teams that measure usually find the parse failure rate is the cheapest thing in the whole pipeline to fix.

Which is the optimistic reading: this is a failure class that responds to engineering. A schema, a gate, and a retry policy convert a probabilistic annoyance into a deterministic pipeline.

When a human reads the output and nothing parses it on the other end. Drafting, summarisation, chat — the places where prose is the product. Forcing JSON onto a drafting task buys nothing and costs the model’s best quality, because the constraint narrows the answer.

The mistake is the middle ground: output that is “mostly JSON” consumed by a parser that is “mostly strict”. Either the consumer is a parser — then constrain and validate — or the consumer is a person — then let the model write. The middle ground is where the retries live.