Skip to main content

Structured Outputs (JSON Mode)

Quick Start

from litellm import completion
import os

os.environ["OPENAI_API_KEY"] = ""

response = completion(
model="gpt-4o-mini",
response_format={ "type": "json_object" },
messages=[
{"role": "system", "content": "You are a helpful assistant designed to output JSON."},
{"role": "user", "content": "Who won the world series in 2020?"}
]
)
print(response.choices[0].message.content)

Check Model Support

Call litellm.get_supported_openai_params to check if a model/provider supports response_format.

from litellm import get_supported_openai_params

params = get_supported_openai_params(model="anthropic.claude-3", custom_llm_provider="bedrock")

assert "response_format" in params

Pass in 'json_schema'

To use Structured Outputs, simply specify

response_format: { "type": "json_schema", "json_schema": … , "strict": true }

Works for:

  • OpenAI models
  • Google AI Studio - Gemini models
  • Vertex AI models (Gemini + Anthropic)
import os
from litellm import completion

# add to env var
os.environ["OPENAI_API_KEY"] = ""

messages = [{"role": "user", "content": "List 5 cookie recipes"}]

class CalendarEvent(BaseModel):
name: str
date: str
participants: list[str]

resp = completion(
model="gpt-4o-2024-08-06",
messages=messages,
response_format=CalendarEvent
)

print("Received={}".format(resp))

Validate JSON Schema

Not all vertex models support passing the json_schema to them (e.g. gemini-1.5-flash). To solve this, LiteLLM supports client-side validation of the json schema.

litellm.enable_json_schema_validation=True

If litellm.enable_json_schema_validation=True is set, LiteLLM will validate the json response using jsonvalidator.

See Code

# !gcloud auth application-default login - run this to add vertex credentials to your env
import litellm, os
from litellm import completion
from pydantic import BaseModel


messages=[
{"role": "system", "content": "Extract the event information."},
{"role": "user", "content": "Alice and Bob are going to a science fair on Friday."},
]

litellm.enable_json_schema_validation = True
litellm.set_verbose = True # see the raw request made by litellm

class CalendarEvent(BaseModel):
name: str
date: str
participants: list[str]

resp = completion(
model="gemini/gemini-1.5-pro",
messages=messages,
response_format=CalendarEvent,
)

print("Received={}".format(resp))