Structured outputs require structured thinking

JSON mode doesn't save you from an underspecified schema. It just makes the failure more consistent.


Structured output support in modern LLMs is genuinely useful, and also genuinely oversold. The pitch is that you get reliable JSON instead of prose that sometimes contains JSON. The reality is that reliability applies to the format, not the semantics.

If your schema has an 'analysis' field typed as a string, you will get a string. Whether that string contains an analysis, a hallucination, a hedge, or a confident-sounding non-answer is not a property of the structured output feature. That's still on you.

The discipline that structured outputs reward is schema design. A well-designed schema constrains the model's output space in ways that make evaluation tractable. Each field should have a clear definition, a finite set of valid values where possible, and a specified behaviour for edge cases. The schema is a contract, and like any contract, its value comes from its specificity.

Start with the evaluation. What would it look like to programmatically verify that the output is correct? Work backwards from that to the schema. If you can't write the evaluator, your schema isn't specific enough.