How do I stop the model wrapping JSON in a code fence?

State it explicitly — "no markdown code fences" — and use assistant prefill with an opening brace where available, which makes a fence impossible. Failing that, strip a wrapping fence in code before parsing, since it is a predictable and easily-handled failure.

Should the schema go before or after the source material?

Before. It is the one place where instruction-before-material beats material-before-instruction, because the model reads the target shape and then reads the content already knowing what to extract.

What is the difference between JSON mode and structured outputs?

JSON mode guarantees the response parses as JSON. Schema-constrained structured outputs guarantee it matches the schema you supplied — the right fields, the right types. The first prevents syntax errors; only the second prevents shape errors.

How do I stop the model inventing values for missing fields?

Include an explicit null rule and show it in an example: one worked case where a field is absent from the source and null in the output. Then validate in code that every extracted value actually appears in the source.

Can I get reasoning and structured output in the same response?

Yes. Ask for reasoning inside a delimited block, then the JSON in its own block, and extract only the JSON. You pay for the reasoning tokens, which is what buys the accuracy on harder extractions.

Getting structured output from AI

Published 2026-07-26

To get reliable structured output, put the schema in the prompt before the source material, instruct the model to return only the data with no prose or code fences, give an explicit rule for missing values so it returns null rather than guessing, and use schema enforcement or response prefill where the API offers it.

Why does asking for JSON not reliably produce JSON?

Because "return JSON" is an instruction, and instructions are requests rather than guarantees. Three failures dominate: a conversational preamble before the JSON, a wrapping markdown code fence, and a trailing explanation after the closing brace. All three produce output that is valid JSON surrounded by text that is not.

The fourth and worst failure is subtler: the JSON parses, but a field the source never mentioned has been filled with a plausible invention rather than left null.

Where should the schema go in the prompt?

Before the source material. This is the one ordering rule that reverses the usual advice, and it reverses it for a good reason: the model reads the target shape first, then reads the content knowing what it is looking for.

<output_format>
Return only a JSON object of this shape:

{
  "vendor": string | null,
  "invoice_number": string | null,
  "date": string | null,     // ISO 8601, YYYY-MM-DD
  "total": string | null     // decimal, no currency symbol
}
</output_format>

<rules>
- Return only the JSON. No preamble, no explanation, no code fences.
- If a field is not present in the source, return null. Never guess.
- Copy values verbatim except where a format is specified above.
</rules>

<source>
[the document]
</source>

Write the schema as an actual example rather than as prose describing the fields. "An object with a vendor field, an invoice number field..." is measurably weaker than showing the object.

What is the single most important rule to include?

The null rule: if a field is not present in the source, return null; never guess a value to fill a slot.

Without it, a model given a schema will fill every field, because a fully-populated object looks more like a correct answer than one full of nulls. The invented values are the most dangerous output a structured-extraction pipeline can produce, precisely because they parse cleanly and look right downstream.

A fabricated invoice number in a field you expected to be populated is far more expensive than an empty field you can see and handle.

How do you enforce the format rather than request it?

Mechanism	What it guarantees	Availability
Instruction only	Nothing. A request the model may not follow.	Everywhere
Assistant prefill with `{`	No preamble; output starts as an object.	Anthropic API, many open-weight stacks
JSON mode	The output parses as JSON.	Several major APIs
Schema-constrained decoding	The output matches your schema.	Several major APIs
Stop sequence	Generation halts at a known boundary.	Most APIs

Prefer enforcement over instruction wherever the API offers it. Note the gap between the middle rows: JSON mode guarantees the output parses, not that the fields are the ones you asked for. Only schema-constrained decoding gives you that.

How should you handle values the model gets wrong?

Validate in code, always. A prompt is not a guardrail: a model can fail to follow any instruction, so anything that must be true has to be checked by something that cannot be persuaded otherwise.

Parse and validate against the schema before using anything.
Check that extracted values actually appear in the source — this catches invented fields immediately.
Range-check numbers and dates. A total of 0 or a date in 1900 is usually a parse failure rather than a fact.
On failure, retry once with the error message included. A malformed response often self-corrects when shown its own mistake.

When is structured output the wrong choice?

When the task genuinely needs prose, forcing it into fields makes the answer worse. A model asked to summarise a complex argument into {"summary": "..."} will produce something flatter than one asked to write a summary, because the container implies brevity and neutrality.

A useful middle path is a hybrid: a <thinking> block for open reasoning, then a JSON block containing only the fields a program needs. You extract the JSON and discard the rest, and the model gets room to reason before committing.

Common questions

How do I stop the model wrapping JSON in a code fence?: State it explicitly — "no markdown code fences" — and use assistant prefill with an opening brace where available, which makes a fence impossible. Failing that, strip a wrapping fence in code before parsing, since it is a predictable and easily-handled failure.
Should the schema go before or after the source material?: Before. It is the one place where instruction-before-material beats material-before-instruction, because the model reads the target shape and then reads the content already knowing what to extract.
What is the difference between JSON mode and structured outputs?: JSON mode guarantees the response parses as JSON. Schema-constrained structured outputs guarantee it matches the schema you supplied — the right fields, the right types. The first prevents syntax errors; only the second prevents shape errors.
How do I stop the model inventing values for missing fields?: Include an explicit null rule and show it in an example: one worked case where a field is absent from the source and null in the output. Then validate in code that every extracted value actually appears in the source.
Can I get reasoning and structured output in the same response?: Yes. Ask for reasoning inside a delimited block, then the JSON in its own block, and extract only the JSON. You pay for the reasoning tokens, which is what buys the accuracy on harder extractions.