Asking nicely vs. making it impossible
You need JSON. You write "Respond only with valid JSON." It works. It works again. On the 300th call it returns something like this — a chatty preamble, then the JSON politely wrapped in a markdown code fence:
Sure! Here's the JSON you asked for:
<fenced code block>
{"sentiment": "positive"}
<end fence>
and your parser throws at 2am.
This isn't disobedience. "Sure! Here's..." is a genuinely plausible continuation of a request — the model has seen a million of them. A prompt makes the format likely, not certain, and "likely" is exactly the kind of promise that holds in testing and breaks in production.
Constrained decoding
Structured outputs change the mechanism instead of the wording. You supply a JSON Schema. At every generation step, the decoder masks out any token that could not legally continue a valid instance of that schema.
If the schema says the next character must be {, then the tokens spelling
"Sure" have their probability set to zero. Not discouraged. Not penalised.
Unreachable.
The difference is categorical, and it's worth being precise about:
- "Please return JSON" → JSON is probable. Validate, retry, hope, alert.
- Schema-constrained decoding → JSON is the only reachable output. Parsing cannot fail.
You stopped negotiating with a probability distribution and started editing it.
Function calling is the same idea
A tool definition is a schema, and the model's call is constrained to match its
signature. issue_refund(order_id: string, amount: number) cannot come back
with a missing amount or a string where a number goes. Same machinery,
different destination.