Structured Output: Reliable LLM Responses with JSON Schema

Telling an LLM to "return JSON" leaves the door open to two failure classes in production: the model sometimes produces invalid JSON (an escaped quote, a trailing comma, markdown code-fence wrapping), and sometimes produces valid JSON that still drifts from the schema's field names or types. This article covers forcing model output into a JSON Schema with OpenAI's Structured Outputs, why tool use is architecturally different on the Anthropic side, and how to guarantee reliable structured LLM output by deriving the schema from code with Zod/Pydantic.

💡 Pro Tip: Don't hand-write the schema as a string — derive it from your Pydantic/Zod model. The SDK's own helper (client.responses.parse or zodTextFormat) manages both the strict-mode constraints and the schema-to-code sync for you.

Table of Contents

• Why "return JSON" isn't enough
• What it guarantees, what it doesn't
• Structured outputs vs tool/function calling
• Don't confuse this with tool calling
• JSON Schema design: enum, required, additionalProperties
• Minimum working schema example
• End-to-end type safety with Zod and Pydantic
• Retry and repair pattern on validation failure
• Safely handling partial/streaming JSON
• Schema versioning and backward compatibility
• The Anthropic-side tool_use flow (2025-10-14 era)
• FAQ
• How do you get valid JSON from an LLM every single time?
• What's the difference between structured outputs and function calling?
• What should you do when schema validation fails?
• How do enums and required fields constrain the model's output?
• How do you use Zod together with OpenAI Structured Outputs?
• Does Anthropic offer a direct JSON Schema guarantee like OpenAI's?
• Update (September 2026)
• Conclusion
• Sources

Why "return JSON" isn't enough

Asking for structure with a plain prompt instruction ("please return JSON") is exposed to two failure classes: the model doesn't guarantee the generated text will parse as JSON, and even when it parses, there's no guarantee the field names and required fields will match the schema.

OpenAI splits this gap into two layers: "JSON mode" (guarantees only valid JSON, not schema match), then "Structured Outputs" (guarantees both valid JSON and conformance to the given JSON Schema). OpenAI's official definition is explicit: the model always generates a response matching the supplied schema — the risk of a skipped required field or an invented enum value is eliminated. The practical payoff: no re-validation/retry loop, safety refusals arrive in a separate, programmatically detectable field, and heavy prompt engineering like "answer in exactly this format" becomes unnecessary since the schema already enforces the type.

What it guarantees, what it doesn't

Structured Outputs guarantees the output MATCHES the schema; it doesn't guarantee the output is correct or meaningful. A response can be schema-valid but wrong in content — this distinction comes back in the retry/repair section below.

Structured outputs vs tool/function calling

These are two different architectures from two different ecosystems, and treating them as equivalent is a mistake. On OpenAI the distinction is clear: connect the model to tools/data/functions in your system with function calling; force the model's final answer into a specific schema with Structured Outputs via text.format — separate response shapes that share the same JSON Schema subset. It's enabled as text: { format: { type: "json_schema", name: "answer", strict: true, schema: ... } } in the Responses API.

On Anthropic (Claude), the official docs as of 2025-10-14 offer a single mechanism: tool use. Tools are defined with input_schema; the model doesn't directly force the final user-facing answer into a schema — instead it returns a tool_use content block with stop_reason: "tool_use", and the calling application runs that tool and sends the result back in a separate tool_result block; this step is optional only if you just want schema-conforming parameters (it's required for Claude to produce the final answer). To handle this inside a planner loop, see our agentic tool-use and planner loop article — that piece covers the tool call itself, while this one covers schema conformance of the call's PARAMETERS.

So on Claude, "schema conformance" operates on an intermediate step (tool-call parameters), not the final free-text answer the way it does on OpenAI. Tools split further into "client tools" (user-defined plus Anthropic-defined ones like computer/text-editor, running in your system) and "server tools" (like web_search/web_fetch, running on Anthropic's infrastructure).

Don't confuse this with tool calling

Connecting the model to a tool (function/tool calling) and forcing its FINAL answer into a schema (structured output) are different problems; conflating them misleads the reader.

JSON Schema design: enum, required, additionalProperties

OpenAI Structured Outputs does not support the full JSON Schema set — only a subset defined for "strict mode"; these constraints were still in effect as of the 2025-10-14 period:

• Root-object rule: the schema's top level must be type: "object", never anyOf. Zod's discriminated-union pattern (z.discriminatedUnion) produces anyOf at the root, which makes it invalid as a Structured Outputs schema (the counter-example in OpenAI's docs demonstrates this with the Chat Completions helper zodResponseFormat()).
• required is mandatory: in strict mode every property must appear in the required array; the notion of an "optional field" is simulated with a null-union like "type": ["string", "null"] — there is no true optionality, only permission to return null.
• additionalProperties: false: mandatory at every object level; it prevents the model from generating extra fields not defined in the schema and is a prerequisite for opting into strict mode.
• Size/nesting limits: a total of 5,000 object properties, 10 levels of nesting; the combined character length of property names, definition names, enum values and const values cannot exceed 120,000.
• Enum limits: 1,000 enum values in total; if a single string-enum property has more than 250 values, the combined character length of those values cannot exceed 15,000.
• Unsupported keywords: allOf, not, dependentRequired, dependentSchemas, if/then/else cannot be used in strict mode. Supported string formats are date-time, time, date, duration, email, hostname, ipv4, ipv6, uuid, plus pattern (regex).
• Key order is preserved: the output is produced in exactly the same order as the property definitions in the schema — this can be used to design field ordering, e.g. explanation first, then result.

Minimum working schema example

1{
2  "type": "object",
3  "properties": {
4    "answer": { "type": "string" },
5    "confidence": { "type": ["number", "null"] }
6  },
7  "required": ["answer", "confidence"],
8  "additionalProperties": false
9}

Here confidence looks "optional" but is actually inside required — it's only permitted to return null. This is the concrete form of strict mode's "no true optional field" rule.

End-to-end type safety with Zod and Pydantic

Instead of hand-writing the JSON Schema as a string kept separate from the app's actual type, the official recommendation is to derive the schema from the type via native SDK helpers: in Python, define a pydantic.BaseModel subclass and call client.responses.parse(model=..., input=[...], text_format=MyModel); in JS/TS, use z.object({...}) with zodTextFormat(). This closes off "JSON schema divergence" (schema and type drifting apart over time) at the SDK level — OpenAI's docs present this as a strong recommendation, with a schema/type sync check in CI as the alternative.

1from pydantic import BaseModel
2from openai import OpenAI
3 
4class Answer(BaseModel):
5    answer: str
6    confidence: float | None
7 
8client = OpenAI()
9resp = client.responses.parse(
10    model="gpt-4o-2024-08-06",
11    input=[{"role": "user", "content": "Answer the question."}],
12    text_format=Answer,
13)
14print(resp.output_parsed.answer)

On the Pydantic side, the infrastructure had already matured: the official PyPI package description describes Pydantic V2 as a ground-up rewrite compared to V1, bringing new features and performance improvements — V2 was stable well before 2025-10-14.

1import { z } from "zod";
2import { zodTextFormat } from "openai/helpers/zod";
3import OpenAI from "openai";
4 
5const Answer = z.object({
6  answer: z.string(),
7  confidence: z.number().nullable(),
8});
9 
10const client = new OpenAI();
11const resp = await client.responses.parse({
12  model: "gpt-4o-2024-08-06",
13  input: [{ role: "user", content: "Answer the question." }],
14  text: { format: zodTextFormat(Answer, "answer") },
15});

One point on the Zod side: because z.discriminatedUnion produces anyOf at the root, it can't be used directly as the Structured Outputs root schema — the SDK helper reduces schema-divergence risk but doesn't automatically work around OpenAI's strict-mode constraints (root-object requirement, anyOf ban); for union types, use nested anyOf inside a property, not at the root.

Retry and repair pattern on validation failure

Even though Structured Outputs "always conforms to the schema," there are two real exit paths outside the schema that still need separate handling:

1. Refusal: if the model refuses for safety reasons, the response returns a separate refusal field/type: "refusal" content block; it isn't required to conform to the schema and must be parsed by checking block.type == "refusal" in the message item's content blocks.
2. Incomplete/max_tokens: hitting the max_output_tokens limit returns status: "incomplete", incomplete_details.reason: "max_output_tokens"; throw an error and retry rather than use the unparsed/incomplete JSON.

1resp = client.responses.parse(
2    model="gpt-4o-2024-08-06",
3    input=[...],
4    text_format=Answer,
5    max_output_tokens=200,
6)
7if resp.status == "incomplete":
8    raise RuntimeError(resp.incomplete_details.reason)
9for item in resp.output:
10    for block in getattr(item, "content", []) or []:
11        if getattr(block, "type", None) == "refusal":
12            raise ValueError(block.refusal)

Content can still be wrong even when it conforms to the schema. The official recommendation is the classic prompt-engineering repertoire: clarify instructions, add examples to the system prompt, break the task into smaller subtasks. "Repair" isn't a retry the model performs itself — it's an escape hatch defined at the prompt level, not enforced by the schema.

Safely handling partial/streaming JSON

Structured Outputs works with streaming; for apps that want to display fields one at a time or process function-call arguments as they generate, the official advice is to rely on the SDK's stream helper rather than hand-parsing JSON piece by piece: in Python, client.responses.stream(model=..., input=[...], text_format=MyModel) opens a context manager you iterate over. This avoids the errors of manually json.loads-ing a partial/invalid JSON fragment — the SDK safely accumulates the partial object per the schema.

Schema versioning and backward compatibility

The official docs offer no dedicated "schema versioning" feature; instead two indirect mechanisms stand out: CI schema/type sync checks under the "JSON schema divergence" recommendation (a process control that keeps the type updated when the schema changes, not versioning itself), and simulating optional fields with a ["string","null"] union — making a newly added field null-union so existing consumers aren't broken. No official versioning API — you need discipline (CI checks + null-union).

The Anthropic-side tool_use flow (2025-10-14 era)

1import anthropic
2 
3client = anthropic.Anthropic()
4resp = client.messages.create(
5    model="claude-sonnet-4-5",
6    max_tokens=1024,
7    tools=[{
8        "name": "save_answer",
9        "input_schema": {
10            "type": "object",
11            "properties": {
12                "answer": {"type": "string"},
13            },
14            "required": ["answer"],
15        },
16    }],
17    messages=[{"role": "user", "content": "Answer the question and call the save_answer tool."}],
18)
19 
20if resp.stop_reason == "tool_use":
21    tool_call = next(b for b in resp.content if b.type == "tool_use")
22    print(tool_call.name, tool_call.input)

Note: resp.stop_reason == "tool_use" means the model produced NOT the final answer but a tool call; the app needs to run that tool and send the result back in a separate tool_result message for Claude to produce its final text answer — for parameter extraction only, you can stop after step 2.

FAQ

How do you get valid JSON from an LLM every single time?

Instead of relying on a prompt instruction, use Structured Outputs via text.format: { type: "json_schema", strict: true } on OpenAI; this guarantees the response always conforms to the given schema, with no separate validation/retry loop needed.

What's the difference between structured outputs and function calling?

Function/tool calling connects the model to tools/data in your system; structured outputs force the model's FINAL answer into a schema. On OpenAI these are separate response shapes sharing the same JSON Schema subset. On Anthropic, as of 2025-10-14, there was a single mechanism — tool use — with schema conformance applying only to tool-call parameters.

What should you do when schema validation fails?

Handle two cases separately: refusal (the model refused — parsed by checking block.type == "refusal" in the message item's content blocks) and incomplete/max_output_tokens (throw and retry). If content is wrong despite conforming to the schema, the fix is clarifying the prompt and splitting the task, not a model retry.

How do enums and required fields constrain the model's output?

The required array makes every field mandatory (optionality is simulated via ["string","null"]), additionalProperties: false blocks extra fields, and enum fixes the value set without letting the model produce free text — total limit 1,000 values, 15,000-character limit for any single property with more than 250 values.

How do you use Zod together with OpenAI Structured Outputs?

Pass a z.object({...}) schema directly to text.format via zodTextFormat(); but don't use z.discriminatedUnion at the root — it produces anyOf there, and strict mode rejects root-level anyOf.

Does Anthropic offer a direct JSON Schema guarantee like OpenAI's?

As of 2025-10-14, no — the guarantee was offered only via tool_use, as an intermediate step. This has since changed; see the update section below.

Update (September 2026)

As of 2025-10-14, schema enforcement on the Anthropic side worked only through tool_use parameters, as an intermediate step. After this article was published, Anthropic released a more direct mechanism similar to OpenAI's: the Claude API now officially has a feature named Structured Outputs, with two parts.

JSON outputs: passing type: "json_schema" to the output_config.format field forces Claude's text response directly into the given schema — no more wrapping it through tool_use.

Strict tool use: adding "strict": true to a tool definition guarantees the tool's input field conforms to the schema via grammar-constrained sampling (token sampling restricted by a compiled grammar).

The Python, TypeScript (Zod), Java, Ruby, PHP, C# and Go SDKs can now derive the schema from a native type definition (client.messages.parse(), zodOutputFormat(), etc.); the SDK also auto-moves unsupported constraints such as minimum, maximum and minLength into the description and adds additionalProperties: false.

Two things to watch: the first request compiles the schema into a grammar, cached for 24 hours (a schema change invalidates the cache — this is why the first call is slow in production); and built-in tool sets like computer_toolset_20260801 and browser_toolset_20260801 don't accept strict: true — the request is rejected if you set it. Strict tool use works in HIPAA-eligible environments, but schema definitions (enum, const, pattern, field names) must not contain PHI — compiled schemas are cached separately from message content, without the same protections.

On OpenAI, the core mechanics here (strict mode, key ordering, refusal/incomplete) are still described the same way in current docs. Model names have changed on both providers — always verify the model name in your code samples against current documentation.

Conclusion

Getting reliable JSON from an LLM is an architectural decision, not a prompt-engineering trick: on OpenAI it's text.format: json_schema plus a native SDK schema (Pydantic/Zod); on Anthropic (as of 2025-10-14) it's tool_use via input_schema, plus, for the final answer, the app feeding tool_result back in a round trip (optional for parameter extraction only). On both sides, handling schema-external exit paths like refusal/incomplete with separate code paths, and deriving the schema from the type instead of hand-writing it, is what prevents silent breakage in production.

If you want to explore this topic in a broader context, see these articles: Agentic AI: Tool Use, Planner Loops and Production Agent Architecture, MCP (Model Context Protocol): The AI Integration Standard, Claude Prompt Caching: The Guide to Cutting Costs 10x, RAG or Fine-tuning? The Definitive Guide to Production LLM Decisions, and LLM Benchmarks 2026: MMLU, HumanEval, SWE-bench and Real-World Performance.

Sources

• OpenAI Structured Outputs guide — the JSON Schema subset, strict-mode constraints, refusal/incomplete behavior, and SDK examples.
• Anthropic tool use overview (2025-09-30 Wayback archive) — the client/server tool distinction and the tool_use flow, the closest verified view to the 2025-10-14 period.
• Understanding JSON Schema — the official, evergreen reference document for JSON Schema.
• Pydantic — PyPI — the official package listing for Pydantic V2.
• Claude Structured Outputs (current) — Anthropic's output_config.format-based JSON outputs feature.
• Claude Strict Tool Use (current) — grammar-constrained sampling, the 24-hour cache, and unsupported tool-set exceptions.