Get eval runs
GET
/evals/{eval_id}/runsGet a list of runs for an evaluation.
Get eval runs
curl https://api.openai.com/v1/evals/egroup_67abd54d9b0081909a86353f6fb9317a/runs \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json"
Response example
{
"object": "list",
"data": [
{
"object": "eval.run",
"id": "evalrun_67e0c7d31560819090d60c0780591042",
"eval_id": "eval_67e0c726d560819083f19a957c4c640b",
"report_url": "https://platform.openai.com/evaluations/eval_67e0c726d560819083f19a957c4c640b",
"status": "completed",
"model": "o3-mini",
"name": "bulk_with_negative_examples_o3-mini",
"created_at": 1742784467,
"result_counts": {
"total": 1,
"errored": 0,
"failed": 0,
"passed": 1
},
"per_model_usage": [
{
"model_name": "o3-mini",
"invocation_count": 1,
"prompt_tokens": 563,
"completion_tokens": 874,
"total_tokens": 1437,
"cached_tokens": 0
}
],
"per_testing_criteria_results": [
{
"testing_criteria": "Push Notification Summary Grader-1808cd0b-eeec-4e0b-a519-337e79f4f5d1",
"passed": 1,
"failed": 0
}
],
"data_source": {
"type": "completions",
"source": {
"type": "file_content",
"content": [
{
"item": {
"notifications": "\n- New message from Sarah: \"Can you call me later?\"\n- Your package has been delivered!\n- Flash sale: 20% off electronics for the next 2 hours!\n"
}
}
]
},
"input_messages": {
"type": "template",
"template": [
{
"type": "message",
"role": "developer",
"content": {
"type": "input_text",
"text": "\n\n\n\nYou are a helpful assistant that takes in an array of push notifications and returns a collapsed summary of them.\nThe push notification will be provided as follows:\n<push_notifications>\n...notificationlist...\n</push_notifications>\n\nYou should return just the summary and nothing else.\n\n\nYou should return a summary that is concise and snappy.\n\n\nHere is an example of a good summary:\n<push_notifications>\n- Traffic alert: Accident reported on Main Street.- Package out for delivery: Expected by 5 PM.- New friend suggestion: Connect with Emma.\n</push_notifications>\n<summary>\nTraffic alert, package expected by 5pm, suggestion for new friend (Emily).\n</summary>\n\n\nHere is an example of a bad summary:\n<push_notifications>\n- Traffic alert: Accident reported on Main Street.- Package out for delivery: Expected by 5 PM.- New friend suggestion: Connect with Emma.\n</push_notifications>\n<summary>\nTraffic alert reported on main street. You have a package that will arrive by 5pm, Emily is a new friend suggested for you.\n</summary>\n"
}
},
{
"type": "message",
"role": "user",
"content": {
"type": "input_text",
"text": "<push_notifications>{{item.notifications}}</push_notifications>"
}
}
]
},
"model": "o3-mini"
},
"error": null,
"metadata": {}
}
],
"first_id": "evalrun_67e0c7d31560819090d60c0780591042",
"last_id": "evalrun_67e0c7d31560819090d60c0780591042",
"has_more": true
}
Query Parameters
Returns
data: array of object { id, created_at, data_source, 11 more } An array of eval run objects.
data: array of object { id, created_at, data_source, 11 more }
An array of eval run objects.
id: string Unique identifier for the evaluation run.
created_at: integer Unix timestamp (in seconds) when the evaluation run was created.
formatunixtime data_source: CreateEvalJSONLRunDataSource or CreateEvalCompletionsRunDataSource or ResponsesRunDataSource Information about the run’s data source.
One of the following: CreateEvalJSONLRunDataSource { source, type } A JsonlRunDataSource object with that specifies a JSONL file that matches the eval
source: EvalJSONLFileContentSource or EvalJSONLFileIDSource Determines what populates the item namespace in the data source.
One of the following: EvalJSONLFileContentSource { content, type } content: array of { item, sample } The content of the jsonl file.
item: map[unknown] sample: optional map[unknown] type: "file_content" (default: "file_content") The type of jsonl source. Always file_content.
EvalJSONLFileIDSource { id, type } id: string The identifier of the file.
type: "file_id" (default: "file_id") The type of jsonl source. Always file_id.
type: "jsonl" (default: "jsonl") The type of data source. Always jsonl.
CreateEvalCompletionsRunDataSource { source, type, input_messages, 2 more } A CompletionsRunDataSource object describing a model sampling configuration.
source: EvalJSONLFileContentSource or EvalJSONLFileIDSource or StoredCompletionsRunDataSource Determines what populates the item namespace in this run’s data source.
One of the following: EvalJSONLFileContentSource { content, type } content: array of { item, sample } The content of the jsonl file.
item: map[unknown] sample: optional map[unknown] type: "file_content" (default: "file_content") The type of jsonl source. Always file_content.
EvalJSONLFileIDSource { id, type } id: string The identifier of the file.
type: "file_id" (default: "file_id") The type of jsonl source. Always file_id.
StoredCompletionsRunDataSource { type, created_after, created_before, 3 more } A StoredCompletionsRunDataSource configuration describing a set of filters
type: "stored_completions" (default: "stored_completions") The type of source. Always stored_completions.
created_after: optional integer or null An optional Unix timestamp to filter items created after this time.
created_before: optional integer or null An optional Unix timestamp to filter items created before this time.
limit: optional integer or null An optional maximum number of items to return.
metadata: optional Metadata or null Set of 16 key-value pairs that can be attached to an object. This can be
useful for storing additional information about the object in a structured
format, and querying for objects via API or the dashboard.
Keys are strings with a maximum length of 64 characters. Values are strings
with a maximum length of 512 characters.
model: optional string or null An optional model to filter by (e.g., ‘gpt-6-astra’).
type: "completions" (default: "completions") The type of run data source. Always completions.
input_messages: optional TemplateInputMessages or ItemReferenceInputMessages Used when sampling from a model. Dictates the structure of the messages passed into the model. Can either be a reference to a prebuilt trajectory (ie, item.input_trajectory), or a template with variable references to the item namespace.
One of the following: TemplateInputMessages { template, type } template: array of EasyInputMessage or EvalMessageObject A list of chat messages forming the prompt or context. May include variable references to the item namespace, ie {{item.name}}.
One of the following: EasyInputMessage { content, role, phase, type } A message input to the model with a role indicating instruction following
hierarchy. Instructions given with the developer or system role take
precedence over instructions given with the user role. Messages with the
assistant role are presumed to have been generated by the model in previous
interactions.
content: string or ResponseInputMessageContentList Text, image, or audio input to the model, used to generate a response.
Can also contain previous assistant responses.
One of the following: TextInput = string A text input to the model.
ResponseInputMessageContentList = array of ResponseInputContent A list of one or many input items to the model, containing different content
types.
One of the following: ResponseInputText { text, type, prompt_cache_breakpoint } A text input to the model.
text: string The text input to the model.
type: "input_text" (default: "input_text") The type of the input item. Always input_text.
prompt_cache_breakpoint: optional { mode, ttl } Marks the exact end of a reusable prompt prefix. When ttl is omitted, the breakpoint inherits the request’s prompt_cache_options.ttl; the boundary is not rounded to a token block.
mode: "explicit" (default: "explicit") The breakpoint mode. Always explicit.
ttl: optional "30m" or "12h" The minimum lifetime of this cache breakpoint. Defaults to the request TTL.
One of the following: "30m" "12h" ResponseInputImage { detail, type, file_id, 2 more } An image input to the model. Learn about image inputs.
detail: ImageDetail The detail level of the image to be sent to the model. One of high, low, auto, or original. Defaults to auto.
One of the following: "low" "high" "auto" "original" type: "input_image" (default: "input_image") The type of the input item. Always input_image.
file_id: optional string or null The ID of the file to be sent to the model.
image_url: optional string or null The URL of the image to be sent to the model. A fully qualified URL or base64 encoded image in a data URL.
formaturi prompt_cache_breakpoint: optional { mode, ttl } Marks the exact end of a reusable prompt prefix. When ttl is omitted, the breakpoint inherits the request’s prompt_cache_options.ttl; the boundary is not rounded to a token block.
mode: "explicit" (default: "explicit") The breakpoint mode. Always explicit.
ttl: optional "30m" or "12h" The minimum lifetime of this cache breakpoint. Defaults to the request TTL.
One of the following: "30m" "12h" ResponseInputFile { type, detail, file_data, 4 more } A file input to the model.
type: "input_file" (default: "input_file") The type of the input item. Always input_file.
detail: optional "auto" or "low" or "high" The detail level of the file to be sent to the model. Use auto to let the system select the detail level; for GPT-5.6 and later models, auto uses high-quality rendering, which may increase input token usage. Use low for lower-cost rendering, or high to render the file at higher quality. Defaults to auto.
One of the following: "auto" "low" "high" file_data: optional string The content of the file to be sent to the model.
file_id: optional string or null The ID of the file to be sent to the model.
file_url: optional string The URL of the file to be sent to the model.
formaturi filename: optional string The name of the file to be sent to the model.
prompt_cache_breakpoint: optional { mode, ttl } Marks the exact end of a reusable prompt prefix. When ttl is omitted, the breakpoint inherits the request’s prompt_cache_options.ttl; the boundary is not rounded to a token block.
mode: "explicit" (default: "explicit") The breakpoint mode. Always explicit.
ttl: optional "30m" or "12h" The minimum lifetime of this cache breakpoint. Defaults to the request TTL.
One of the following: "30m" "12h" role: "user" or "assistant" or "system" or "developer" The role of the message input. One of user, assistant, system, or
developer.
One of the following: "user" "assistant" "system" "developer" phase: optional "commentary" or "final_answer" or null Labels an assistant message as intermediate commentary (commentary) or the final answer (final_answer).
For models like gpt-5.3-codex and beyond, when sending follow-up requests, preserve and resend
phase on all assistant messages — dropping it can degrade performance. Not used for user messages.
One of the following: "commentary" "final_answer" type: optional "message" The type of the message input. Always message.
EvalMessageObject { content, role, type } A message input to the model with a role indicating instruction following
hierarchy. Instructions given with the developer or system role take
precedence over instructions given with the user role. Messages with the
assistant role are presumed to have been generated by the model in previous
interactions.
content: string or ResponseInputText or OutputText or 3 more Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
One of the following: TextInput = string A text input to the model.
ResponseInputText { text, type, prompt_cache_breakpoint } A text input to the model.
text: string The text input to the model.
type: "input_text" (default: "input_text") The type of the input item. Always input_text.
prompt_cache_breakpoint: optional { mode, ttl } Marks the exact end of a reusable prompt prefix. When ttl is omitted, the breakpoint inherits the request’s prompt_cache_options.ttl; the boundary is not rounded to a token block.
mode: "explicit" (default: "explicit") The breakpoint mode. Always explicit.
ttl: optional "30m" or "12h" The minimum lifetime of this cache breakpoint. Defaults to the request TTL.
One of the following: "30m" "12h" OutputText { text, type } A text output from the model.
text: string The text output from the model.
type: "output_text" The type of the output text. Always output_text.
InputImage { image_url, type, detail } An image input block used within EvalItem content arrays.
image_url: string The URL of the image input.
formaturi type: "input_image" The type of the image input. Always input_image.
detail: optional string The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.
ResponseInputAudio { input_audio, type } An audio input to the model.
input_audio: { data, format } data: string Base64-encoded audio data.
format: "mp3" or "wav" The format of the audio data. Currently supported formats are mp3 and
wav.
One of the following: "mp3" "wav" type: "input_audio" The type of the input item. Always input_audio.
GraderInputs = array of string or ResponseInputText or OutputText or 2 more A list of inputs, each of which may be either an input text, output text, input
image, or input audio object.
One of the following: TextInput = string A text input to the model.
ResponseInputText { text, type, prompt_cache_breakpoint } A text input to the model.
text: string The text input to the model.
type: "input_text" (default: "input_text") The type of the input item. Always input_text.
prompt_cache_breakpoint: optional { mode, ttl } Marks the exact end of a reusable prompt prefix. When ttl is omitted, the breakpoint inherits the request’s prompt_cache_options.ttl; the boundary is not rounded to a token block.
mode: "explicit" (default: "explicit") The breakpoint mode. Always explicit.
ttl: optional "30m" or "12h" The minimum lifetime of this cache breakpoint. Defaults to the request TTL.
One of the following: "30m" "12h" OutputText { text, type } A text output from the model.
text: string The text output from the model.
type: "output_text" The type of the output text. Always output_text.
InputImage { image_url, type, detail } An image input block used within EvalItem content arrays.
image_url: string The URL of the image input.
formaturi type: "input_image" The type of the image input. Always input_image.
detail: optional string The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.
ResponseInputAudio { input_audio, type } An audio input to the model.
input_audio: { data, format } data: string Base64-encoded audio data.
format: "mp3" or "wav" The format of the audio data. Currently supported formats are mp3 and
wav.
One of the following: "mp3" "wav" type: "input_audio" The type of the input item. Always input_audio.
role: "user" or "assistant" or "system" or "developer" The role of the message input. One of user, assistant, system, or
developer.
One of the following: "user" "assistant" "system" "developer" type: optional "message" The type of the message input. Always message.
type: "template" The type of input messages. Always template.
ItemReferenceInputMessages { item_reference, type } item_reference: string A reference to a variable in the item namespace. Ie, “item.input_trajectory”
type: "item_reference" The type of input messages. Always item_reference.
model: optional string The name of the model to use for generating completions (e.g. “o3-mini”).
sampling_params: optional { max_completion_tokens, reasoning_effort, response_format, 4 more } max_completion_tokens: optional integer The maximum number of tokens in the generated output.
reasoning_effort: optional ReasoningEffort or null (default: "medium") Constrains effort on reasoning for reasoning models. Currently supported
values are none, minimal, low, medium, high, xhigh, and max.
Reducing reasoning effort can result in faster responses and fewer tokens
used on reasoning in a response. Not all reasoning models support every
value. See the
reasoning guide
for model-specific support.
One of the following: "none" "minimal" "low" "medium" "high" "xhigh" "max" response_format: optional ResponseFormatText or ResponseFormatJSONSchema or ResponseFormatJSONObject An object specifying the format that the model must output.
Setting to { "type": "json_schema", "json_schema": {...} } enables
Structured Outputs which ensures the model will match your supplied JSON
schema. Learn more in the Structured Outputs
guide.
Setting to { "type": "json_object" } enables the older JSON mode, which
ensures the message the model generates is valid JSON. Using json_schema
is preferred for models that support it.
One of the following: ResponseFormatText { type } Default response format. Used to generate text responses.
type: "text" The type of response format being defined. Always text.
ResponseFormatJSONSchema { json_schema, type } JSON Schema response format. Used to generate structured JSON responses.
Learn more about Structured Outputs.
json_schema: { name, description, schema, strict } Structured Outputs configuration options, including a JSON Schema.
name: string The name of the response format. Must be a-z, A-Z, 0-9, or contain
underscores and dashes, with a maximum length of 64.
description: optional string A description of what the response format is for, used by the model to
determine how to respond in the format.
schema: optional map[unknown] The schema for the response format, described as a JSON Schema object.
Learn how to build JSON schemas here.
strict: optional boolean or null (default: false) Whether to enable strict schema adherence when generating the output.
If set to true, the model will always follow the exact schema defined
in the schema field. Only a subset of JSON Schema is supported when
strict is true. To learn more, read the Structured Outputs
guide.
type: "json_schema" The type of response format being defined. Always json_schema.
ResponseFormatJSONObject { type } JSON object response format. An older method of generating JSON responses.
Using json_schema is recommended for models that support it. Note that the
model will not generate JSON without a system or user message instructing it
to do so.
type: "json_object" The type of response format being defined. Always json_object.
seed: optional integer (default: 42) A seed value to initialize the randomness, during sampling.
temperature: optional number (default: 1) A higher temperature increases randomness in the outputs.
tools: optional array of ChatCompletionFunctionTool { function, type } A list of tools the model may call. Currently, only functions are supported as a tool. Use this to provide a list of functions the model may generate JSON inputs for. A max of 128 functions are supported.
function: FunctionDefinition { name, description, parameters, strict } name: string The name of the function to be called. Must be a-z, A-Z, 0-9, or contain underscores and dashes, with a maximum length of 64.
description: optional string A description of what the function does, used by the model to choose when and how to call the function.
parameters: optional FunctionParameters The parameters the functions accepts, described as a JSON Schema object. See the guide for examples, and the JSON Schema reference for documentation about the format.
Omitting parameters defines a function with an empty parameter list.
strict: optional boolean or null (default: false) Whether to enable strict schema adherence when generating the function call. If set to true, the model will follow the exact schema defined in the parameters field. Only a subset of JSON Schema is supported when strict is true. Learn more about Structured Outputs in the function calling guide.
type: "function" The type of the tool. Currently, only function is supported.
top_p: optional number (default: 1) An alternative to temperature for nucleus sampling; 1.0 includes all tokens.
ResponsesRunDataSource { source, type, input_messages, 2 more } A ResponsesRunDataSource object describing a model sampling configuration.
source: EvalJSONLFileContentSource or EvalJSONLFileIDSource or EvalResponsesSource Determines what populates the item namespace in this run’s data source.
One of the following: EvalJSONLFileContentSource { content, type } content: array of { item, sample } The content of the jsonl file.
item: map[unknown] sample: optional map[unknown] type: "file_content" (default: "file_content") The type of jsonl source. Always file_content.
EvalJSONLFileIDSource { id, type } id: string The identifier of the file.
type: "file_id" (default: "file_id") The type of jsonl source. Always file_id.
EvalResponsesSource { type, created_after, created_before, 8 more } A EvalResponsesSource object describing a run data source configuration.
type: "responses" The type of run data source. Always responses.
created_after: optional integer or null Only include items created after this timestamp (inclusive). This is a query parameter used to select responses.
minimum0 created_before: optional integer or null Only include items created before this timestamp (inclusive). This is a query parameter used to select responses.
minimum0 instructions_search: optional string or null Optional string to search the ‘instructions’ field. This is a query parameter used to select responses.
metadata: optional unknown or null Metadata filter for the responses. This is a query parameter used to select responses.
model: optional string or null The name of the model to find responses for. This is a query parameter used to select responses.
reasoning_effort: optional ReasoningEffort or null (default: "medium") Constrains effort on reasoning for reasoning models. Currently supported
values are none, minimal, low, medium, high, xhigh, and max.
Reducing reasoning effort can result in faster responses and fewer tokens
used on reasoning in a response. Not all reasoning models support every
value. See the
reasoning guide
for model-specific support.
One of the following: "none" "minimal" "low" "medium" "high" "xhigh" "max" temperature: optional number or null Sampling temperature. This is a query parameter used to select responses.
tools: optional array of string or null List of tool names. This is a query parameter used to select responses.
top_p: optional number or null Nucleus sampling parameter. This is a query parameter used to select responses.
users: optional array of string or null List of user identifiers. This is a query parameter used to select responses.
type: "responses" (default: "responses") The type of run data source. Always responses.
input_messages: optional InputMessagesTemplate or InputMessagesItemReference Used when sampling from a model. Dictates the structure of the messages passed into the model. Can either be a reference to a prebuilt trajectory (ie, item.input_trajectory), or a template with variable references to the item namespace.
One of the following: InputMessagesTemplate { template, type } template: array of ChatMessage or EvalMessageObject A list of chat messages forming the prompt or context. May include variable references to the item namespace, ie {{item.name}}.
One of the following: ChatMessage { content, role } content: string The content of the message.
role: string The role of the message (e.g. “system”, “assistant”, “user”).
EvalMessageObject { content, role, type } A message input to the model with a role indicating instruction following
hierarchy. Instructions given with the developer or system role take
precedence over instructions given with the user role. Messages with the
assistant role are presumed to have been generated by the model in previous
interactions.
content: string or ResponseInputText or OutputText or 3 more Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
One of the following: TextInput = string A text input to the model.
ResponseInputText { text, type, prompt_cache_breakpoint } A text input to the model.
text: string The text input to the model.
type: "input_text" (default: "input_text") The type of the input item. Always input_text.
prompt_cache_breakpoint: optional { mode, ttl } Marks the exact end of a reusable prompt prefix. When ttl is omitted, the breakpoint inherits the request’s prompt_cache_options.ttl; the boundary is not rounded to a token block.
mode: "explicit" (default: "explicit") The breakpoint mode. Always explicit.
ttl: optional "30m" or "12h" The minimum lifetime of this cache breakpoint. Defaults to the request TTL.
One of the following: "30m" "12h" OutputText { text, type } A text output from the model.
text: string The text output from the model.
type: "output_text" The type of the output text. Always output_text.
InputImage { image_url, type, detail } An image input block used within EvalItem content arrays.
image_url: string The URL of the image input.
formaturi type: "input_image" The type of the image input. Always input_image.
detail: optional string The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.
ResponseInputAudio { input_audio, type } An audio input to the model.
input_audio: { data, format } data: string Base64-encoded audio data.
format: "mp3" or "wav" The format of the audio data. Currently supported formats are mp3 and
wav.
One of the following: "mp3" "wav" type: "input_audio" The type of the input item. Always input_audio.
GraderInputs = array of string or ResponseInputText or OutputText or 2 more A list of inputs, each of which may be either an input text, output text, input
image, or input audio object.
One of the following: TextInput = string A text input to the model.
ResponseInputText { text, type, prompt_cache_breakpoint } A text input to the model.
text: string The text input to the model.
type: "input_text" (default: "input_text") The type of the input item. Always input_text.
prompt_cache_breakpoint: optional { mode, ttl } Marks the exact end of a reusable prompt prefix. When ttl is omitted, the breakpoint inherits the request’s prompt_cache_options.ttl; the boundary is not rounded to a token block.
mode: "explicit" (default: "explicit") The breakpoint mode. Always explicit.
ttl: optional "30m" or "12h" The minimum lifetime of this cache breakpoint. Defaults to the request TTL.
One of the following: "30m" "12h" OutputText { text, type } A text output from the model.
text: string The text output from the model.
type: "output_text" The type of the output text. Always output_text.
InputImage { image_url, type, detail } An image input block used within EvalItem content arrays.
image_url: string The URL of the image input.
formaturi type: "input_image" The type of the image input. Always input_image.
detail: optional string The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.
ResponseInputAudio { input_audio, type } An audio input to the model.
input_audio: { data, format } data: string Base64-encoded audio data.
format: "mp3" or "wav" The format of the audio data. Currently supported formats are mp3 and
wav.
One of the following: "mp3" "wav" type: "input_audio" The type of the input item. Always input_audio.
role: "user" or "assistant" or "system" or "developer" The role of the message input. One of user, assistant, system, or
developer.
One of the following: "user" "assistant" "system" "developer" type: optional "message" The type of the message input. Always message.
type: "template" The type of input messages. Always template.
InputMessagesItemReference { item_reference, type } item_reference: string A reference to a variable in the item namespace. Ie, “item.name”
type: "item_reference" The type of input messages. Always item_reference.
model: optional string The name of the model to use for generating completions (e.g. “o3-mini”).
sampling_params: optional { max_completion_tokens, reasoning_effort, seed, 4 more } max_completion_tokens: optional integer The maximum number of tokens in the generated output.
reasoning_effort: optional ReasoningEffort or null (default: "medium") Constrains effort on reasoning for reasoning models. Currently supported
values are none, minimal, low, medium, high, xhigh, and max.
Reducing reasoning effort can result in faster responses and fewer tokens
used on reasoning in a response. Not all reasoning models support every
value. See the
reasoning guide
for model-specific support.
One of the following: "none" "minimal" "low" "medium" "high" "xhigh" "max" seed: optional integer (default: 42) A seed value to initialize the randomness, during sampling.
temperature: optional number (default: 1) A higher temperature increases randomness in the outputs.
text: optional { format } Configuration options for a text response from the model. Can be plain
text or structured JSON data. Learn more:
Text inputs and outputs
Structured Outputs
format: optional ResponseFormatTextConfig An object specifying the format that the model must output.
Configuring { "type": "json_schema" } enables Structured Outputs,
which ensures the model will match your supplied JSON schema. Learn more in the
Structured Outputs guide.
The default format is { "type": "text" } with no additional options.
Not recommended for gpt-4o and newer models:
Setting to { "type": "json_object" } enables the older JSON mode, which
ensures the message the model generates is valid JSON. Using json_schema
is preferred for models that support it.
One of the following: ResponseFormatText { type } Default response format. Used to generate text responses.
type: "text" The type of response format being defined. Always text.
ResponseFormatTextJSONSchemaConfig { name, schema, type, 2 more } JSON Schema response format. Used to generate structured JSON responses.
Learn more about Structured Outputs.
name: string The name of the response format. Must be a-z, A-Z, 0-9, or contain
underscores and dashes, with a maximum length of 64.
schema: map[unknown] The schema for the response format, described as a JSON Schema object.
Learn how to build JSON schemas here.
type: "json_schema" The type of response format being defined. Always json_schema.
description: optional string A description of what the response format is for, used by the model to
determine how to respond in the format.
strict: optional boolean or null (default: false) Whether to enable strict schema adherence when generating the output.
If set to true, the model will always follow the exact schema defined
in the schema field. Only a subset of JSON Schema is supported when
strict is true. To learn more, read the Structured Outputs
guide.
ResponseFormatJSONObject { type } JSON object response format. An older method of generating JSON responses.
Using json_schema is recommended for models that support it. Note that the
model will not generate JSON without a system or user message instructing it
to do so.
type: "json_object" The type of response format being defined. Always json_object.
tools: optional array of Function or FileSearch or Computer or 13 more An array of tools the model may call while generating a response. You
can specify which tool to use by setting the tool_choice parameter.
The two categories of tools you can provide the model are:
Built-in tools: Tools that are provided by OpenAI that extend the
model’s capabilities, like web search
or file search. Learn more about
built-in tools.
Function calls (custom tools): Functions that are defined by you,
enabling the model to call your own code. Learn more about
function calling.
One of the following: Function { name, parameters, strict, 6 more } Defines a function in your own code the model can choose to call. Learn more about function calling.
name: string The name of the function to call.
parameters: map[unknown] or null A JSON schema object describing the parameters of the function.
strict: boolean or null Whether strict parameter validation is enforced for this function tool.
type: "function" (default: "function") The type of the function tool. Always function.
allowed_callers: optional array of "direct" or "programmatic" or null The tool invocation context(s).
One of the following: "direct" "programmatic" async: optional boolean defer_loading: optional boolean Whether this function is deferred and loaded via tool search.
description: optional string or null A description of the function. Used by the model to determine whether or not to call the function.
output_schema: optional map[unknown] or null A JSON schema object describing the JSON value encoded in string outputs for this function.
FileSearch { type, vector_store_ids, filters, 2 more } A tool that searches for relevant content from uploaded files. Learn more about the file search tool.
type: "file_search" (default: "file_search") The type of the file search tool. Always file_search.
vector_store_ids: array of string The IDs of the vector stores to search.
filters: optional ComparisonFilter or CompoundFilter or null A filter to apply.
One of the following: ComparisonFilter { key, type, value } A filter used to compare a specified attribute key to a given value using a defined comparison operation.
key: string The key to compare against the value.
type: "eq" or "ne" or "gt" or 5 more (default: "eq") Specifies the comparison operator: eq, ne, gt, gte, lt, lte, in, nin.
eq: equals
ne: not equal
gt: greater than
gte: greater than or equal
lt: less than
lte: less than or equal
in: in
nin: not in
One of the following: "eq" "ne" "gt" "gte" "lt" "lte" "in" "nin" value: string or number or boolean or array of string or number The value to compare against the attribute key; supports string, number, or boolean types.
One of the following: string number boolean array of string or number One of the following: string number CompoundFilter { filters, type } Combine multiple filters using and or or.
filters: array of ComparisonFilter or CompoundFilter Array of filters to combine. Items can be ComparisonFilter or CompoundFilter.
One of the following: ComparisonFilter { key, type, value } A filter used to compare a specified attribute key to a given value using a defined comparison operation.
key: string The key to compare against the value.
type: "eq" or "ne" or "gt" or 5 more (default: "eq") Specifies the comparison operator: eq, ne, gt, gte, lt, lte, in, nin.
eq: equals
ne: not equal
gt: greater than
gte: greater than or equal
lt: less than
lte: less than or equal
in: in
nin: not in
One of the following: "eq" "ne" "gt" "gte" "lt" "lte" "in" "nin" value: string or number or boolean or array of string or number The value to compare against the attribute key; supports string, number, or boolean types.
One of the following: string number boolean array of string or number One of the following: string number CompoundFilter { filters, type } This type repeats the enclosing schema. type: "and" or "or" Type of operation: and or or.
One of the following: "and" "or" max_num_results: optional integer The maximum number of results to return. This number should be between 1 and 50 inclusive.
ranking_options: optional { hybrid_search, ranker, score_threshold } Ranking options for search.
hybrid_search: optional { embedding_weight, text_weight } Weights that control how reciprocal rank fusion balances semantic embedding matches versus sparse keyword matches when hybrid search is enabled.
embedding_weight: number The weight of the embedding in the reciprocal ranking fusion.
text_weight: number The weight of the text in the reciprocal ranking fusion.
ranker: optional "auto" or "default-2024-11-15" The ranker to use for the file search.
One of the following: "auto" "default-2024-11-15" score_threshold: optional number The score threshold for the file search, a number between 0 and 1. Numbers closer to 1 will attempt to return only the most relevant results, but may return fewer results.
Computer { type } A tool that controls a virtual computer. Learn more about the computer tool.
type: "computer" (default: "computer") The type of the computer tool. Always computer.
ComputerUsePreview { display_height, display_width, environment, type } A tool that controls a virtual computer. Learn more about the computer tool.
display_height: integer The height of the computer display.
display_width: integer The width of the computer display.
environment: "windows" or "mac" or "linux" or 2 more The type of computer environment to control.
One of the following: "windows" "mac" "linux" "ubuntu" "browser" type: "computer_use_preview" (default: "computer_use_preview") The type of the computer use tool. Always computer_use_preview.
WebSearch { type, external_web_access, filters, 2 more } Search the Internet for sources related to the prompt. Learn more about the
web search tool.
type: "web_search" or "web_search_2025_08_26" (default: "web_search") The type of the web search tool. One of web_search or web_search_2025_08_26.
One of the following: "web_search" "web_search_2025_08_26" external_web_access: optional boolean (default: true) Allow live internet access for web search. Defaults to true when omitted. When false, the web search tool runs in offline/cache-only mode and will not fetch new external content.
filters: optional object or null Filters for the search.
allowed_domains: optional array of string or null (default: []) Allowed domains for the search. If not provided, all domains are allowed.
Subdomains of the provided domains are allowed as well.
Example: ["pubmed.ncbi.nlm.nih.gov"]
search_context_size: optional "low" or "medium" or "high" (default: "medium") High level guidance for the amount of context window space to use for the search. One of low, medium, or high. medium is the default.
One of the following: "low" "medium" "high" user_location: optional object or null The approximate location of the user. If omitted or null, defaults to the
United States. To avoid this fallback, pass {"type": "approximate"} without
location fields. To localize results, provide the relevant location fields.
city: optional string or null Free text input for the city of the user, e.g. San Francisco.
country: optional string or null The two-letter ISO country code of the user, e.g. US.
region: optional string or null Free text input for the region of the user, e.g. California.
timezone: optional string or null The IANA timezone of the user, e.g. America/Los_Angeles.
type: optional "approximate" (default: "approximate") The type of location approximation. Always approximate.
Mcp { server_label, type, allowed_callers, 9 more } Give the model access to additional tools via remote Model Context Protocol
(MCP) servers. Learn more about MCP.
server_label: string A label for this MCP server, used to identify it in tool calls.
type: "mcp" The type of the MCP tool. Always mcp.
allowed_callers: optional array of "direct" or "programmatic" or null The tool invocation context(s).
minItems1 One of the following: "direct" "programmatic" allowed_tools: optional array of string or McpToolFilter or null List of allowed tool names or a filter object.
One of the following: McpAllowedTools = array of string A string array of allowed tool names
McpToolFilter { read_only, tool_names } A filter object to specify which tools are allowed.
read_only: optional boolean Indicates whether or not a tool modifies data or is read-only. If an
MCP server is annotated with readOnlyHint,
it will match this filter.
tool_names: optional array of string List of allowed tool names.
authorization: optional string An OAuth access token that can be used with a remote MCP server, either
with a custom MCP server URL or a service connector. Your application
must handle the OAuth authorization flow and provide the token here.
connector_id: optional "connector_dropbox" or "connector_gmail" or "connector_googlecalendar" or 5 more Deprecated.
Identifier for service connectors, like those available in ChatGPT. One of
server_url, connector_id, or tunnel_id must be provided. Learn more
about service connectors here.
This field is deprecated for models released after September 1, 2026.
Use server_url to connect to a remote MCP server, or tunnel_id to
connect through a Secure MCP Tunnel.
Currently supported connector_id values are:
Dropbox: connector_dropbox
Gmail: connector_gmail
Google Calendar: connector_googlecalendar
Google Drive: connector_googledrive
Microsoft Teams: connector_microsoftteams
Outlook Calendar: connector_outlookcalendar
Outlook Email: connector_outlookemail
SharePoint: connector_sharepoint
One of the following: "connector_dropbox" "connector_gmail" "connector_googlecalendar" "connector_googledrive" "connector_microsoftteams" "connector_outlookcalendar" "connector_outlookemail" "connector_sharepoint" defer_loading: optional boolean Whether this MCP tool is deferred and discovered via tool search.
headers: optional map[string] or null Optional HTTP headers to send to the MCP server. Use for authentication
or other purposes.
require_approval: optional McpToolApprovalFilter or "always" or "never" or null (default: "always") Specify which of the MCP server’s tools require approval.
One of the following: McpToolApprovalFilter { always, never } Specify which of the MCP server’s tools require approval. Can be
always, never, or a filter object associated with tools
that require approval.
always: optional { read_only, tool_names } A filter object to specify which tools are allowed.
read_only: optional boolean Indicates whether or not a tool modifies data or is read-only. If an
MCP server is annotated with readOnlyHint,
it will match this filter.
tool_names: optional array of string List of allowed tool names.
never: optional { read_only, tool_names } A filter object to specify which tools are allowed.
read_only: optional boolean Indicates whether or not a tool modifies data or is read-only. If an
MCP server is annotated with readOnlyHint,
it will match this filter.
tool_names: optional array of string List of allowed tool names.
McpToolApprovalSetting = "always" or "never" Specify a single approval policy for all tools. One of always or
never. When set to always, all tools will require approval. When
set to never, all tools will not require approval.
One of the following: "always" "never" server_description: optional string Optional description of the MCP server, used to provide more context.
server_url: optional string The URL for the MCP server. One of server_url, connector_id, or
tunnel_id must be provided.
formaturi tunnel_id: optional string The Secure MCP Tunnel ID to use instead of a direct server URL. One of
server_url, connector_id, or tunnel_id must be provided.
pattern^tunnel_[a-z0-9]{32}$ CodeInterpreter { container, type, allowed_callers } A tool that runs Python code to help generate a response to a prompt.
container: string or CodeInterpreterToolAuto The code interpreter container. Can be a container ID or an object that
specifies uploaded file IDs to make available to your code, along with an
optional memory_limit setting.
One of the following: string The container ID.
CodeInterpreterToolAuto { type, file_ids, memory_limit, network_policy } Configuration for a code interpreter container. Optionally specify the IDs of the files to run the code on.
type: "auto" (default: "auto") Always auto.
file_ids: optional array of string An optional list of uploaded files to make available to your code.
maxItems50 memory_limit: optional "1g" or "4g" or "16g" or "64g" or null The memory limit for the code interpreter container.
One of the following: "1g" "4g" "16g" "64g" network_policy: optional ContainerNetworkPolicyDisabled or ContainerNetworkPolicyAllowlist Network access policy for the container.
One of the following: ContainerNetworkPolicyDisabled { type } type: "disabled" (default: "disabled") Disable outbound network access. Always disabled.
ContainerNetworkPolicyAllowlist { allowed_domains, type, domain_secrets } allowed_domains: array of string A list of allowed domains when type is allowlist.
minItems1 type: "allowlist" (default: "allowlist") Allow outbound network access only to specified domains. Always allowlist.
domain_secrets: optional array of ContainerNetworkPolicyDomainSecret { domain, name, value } Optional domain-scoped secrets for allowlisted domains.
minItems1 domain: string The domain associated with the secret.
minLength1 name: string The name of the secret to inject for the domain.
minLength1 value: string The secret value to inject for the domain.
minLength1 maxLength10485760 type: "code_interpreter" The type of the code interpreter tool. Always code_interpreter.
allowed_callers: optional array of "direct" or "programmatic" or null The tool invocation context(s).
minItems1 One of the following: "direct" "programmatic" ProgrammaticToolCalling { type } type: "programmatic_tool_calling" (default: "programmatic_tool_calling") The type of the tool. Always programmatic_tool_calling.
ImageGeneration { type, action, background, 13 more } A tool that generates images using the GPT image models.
type: "image_generation" The type of the image generation tool. Always image_generation.
action: optional "generate" or "edit" or "auto" Whether to generate a new image or edit an existing image. Default: auto.
One of the following: "generate" "edit" "auto" background: optional "transparent" or "opaque" or "auto" (default: "auto") Set the background of the generated image. One of transparent, opaque,
or auto. gpt-image-2.5-sunburst and gpt-image-2.5-flare, including
their 2026-09-08 snapshots, support opaque and transparent
backgrounds. Transparent backgrounds are available for supported GPT Image
models. For gpt-image-2 and gpt-image-2-2026-04-21, this support is in
preview. When using transparent, set the output format to png or webp.
Default: auto.
One of the following: "transparent" "opaque" "auto" complexity: optional "low" or "medium" or "high" Computation spent planning the image, independently of rendering quality.
Overrides the outer response’s reasoning.effort for image generation.
When omitted, inherits a supported outer reasoning effort or uses the
image model’s configured default.
For SVG output, controls the reference image, not the SVG author’s
reasoning.
One of the following: "low" "medium" "high" frame_count: optional integer (default: 9) Number of GIF frames. Requires output_format to be gif. Default: 9.
minimum2 maximum16 frame_duration_ms: optional integer (default: 200) Duration of each GIF frame in milliseconds, rounded to 10 ms.
Requires output_format to be gif. Default: 200.
minimum20 maximum2000 input_fidelity: optional "high" or "low" or null Control how much effort the model will exert to match the style and features, especially facial features, of input images. Supports high and low on gpt-image-1.5; gpt-image-1-mini supports only low. For gpt-image-2, omit this parameter. Defaults to low on supported models.
One of the following: "high" "low" input_image_mask: optional { file_id, image_url } Optional mask for inpainting. Contains image_url
(string, optional) and file_id (string, optional).
file_id: optional string File ID for the mask image.
image_url: optional string Base64-encoded mask image.
model: optional string or "gpt-image-1" or "gpt-image-1-mini" or "gpt-image-2" or 9 more The image generation model to use. One of gpt-image-1-mini,
gpt-image-1.5, gpt-image-2,
gpt-image-2-2026-04-21, gpt-image-2.5-sunburst,
gpt-image-2.5-sunburst-2026-09-08, gpt-image-2.5-flare,
gpt-image-2.5-flare-2026-09-08, gpt-image-2.6-sunburst,
gpt-image-2.6-flare, or chatgpt-image-latest. Default:
gpt-image-2.6-sunburst.
gpt-image-1 was shut down on October 23, 2026. Use
gpt-image-2.5-sunburst or gpt-image-2.5-flare instead.
One of the following: string "gpt-image-1" or "gpt-image-1-mini" or "gpt-image-2" or 9 more The image generation model to use. One of gpt-image-1-mini,
gpt-image-1.5, gpt-image-2,
gpt-image-2-2026-04-21, gpt-image-2.5-sunburst,
gpt-image-2.5-sunburst-2026-09-08, gpt-image-2.5-flare,
gpt-image-2.5-flare-2026-09-08, gpt-image-2.6-sunburst,
gpt-image-2.6-flare, or chatgpt-image-latest. Default:
gpt-image-2.6-sunburst.
gpt-image-1 was shut down on October 23, 2026. Use
gpt-image-2.5-sunburst or gpt-image-2.5-flare instead.
One of the following: "gpt-image-1-mini" "gpt-image-2" "gpt-image-2-2026-04-21" "gpt-image-2.5-sunburst" "gpt-image-2.5-sunburst-2026-09-08" "gpt-image-2.5-flare" "gpt-image-2.5-flare-2026-09-08" "gpt-image-2.6-sunburst" "gpt-image-2.6-flare" "gpt-image-1.5" "chatgpt-image-latest" "gpt-image-1" Deprecated. GPT Image 1 was shut down on October 23, 2026. Use gpt-image-2.5-sunburst or gpt-image-2.5-flare instead.
moderation: optional "auto" or "low" (default: "auto") Moderation level for the generated image. Default: auto.
One of the following: "auto" "low" output_compression: optional integer (default: 100) Compression level for the output image. Default: 100.
minimum0 maximum100 output_format: optional "png" or "webp" or "jpeg" or 2 more (default: "png") The output format of the generated image. One of png, webp, jpeg,
gif, or svg. Default: png. GIF and SVG output require a model
that supports the selected format. GIF output does not support
partial image streaming (partial_images greater than 0),
input_image_mask, or custom output_compression.
Responses lifecycle events can still be streamed for GIF output.
One of the following: "png" "webp" "jpeg" "gif" "svg" partial_images: optional integer (default: 0) Number of partial images to generate in streaming mode, from 0 (default value) to 3.
minimum0 maximum3 quality: optional "low" or "medium" or "high" or 3 more (default: "auto") The quality of the generated image. The GPT image models support low,
medium, and high. gpt-image-2.5-sunburst and gpt-image-2.5-flare,
including their 2026-09-08 snapshots, also support xhigh and max.
Default: auto.
One of the following: "low" "medium" "high" "xhigh" "max" "auto" size: optional string or "1024x1024" or "1024x1536" or "1536x1024" or "auto" (default: "auto") The size of the generated images. For gpt-image-2, gpt-image-2-2026-04-21, gpt-image-2.5-sunburst, gpt-image-2.5-sunburst-2026-09-08, gpt-image-2.5-flare, and gpt-image-2.5-flare-2026-09-08, arbitrary resolutions are supported as WIDTHxHEIGHT strings, for example 1536x864. Width and height must both be divisible by 16 and the requested aspect ratio must be between 1:3 and 3:1. Resolutions above 2560x1440 are experimental, and the maximum supported resolution is 3840x2160. The requested size must also satisfy the model’s current pixel and edge limits. The standard sizes 1024x1024, 1536x1024, and 1024x1536 are supported by the GPT image models; auto is supported for models that allow automatic sizing.
One of the following: string "1024x1024" or "1024x1536" or "1536x1024" or "auto" The size of the generated images. For gpt-image-2, gpt-image-2-2026-04-21, gpt-image-2.5-sunburst, gpt-image-2.5-sunburst-2026-09-08, gpt-image-2.5-flare, and gpt-image-2.5-flare-2026-09-08, arbitrary resolutions are supported as WIDTHxHEIGHT strings, for example 1536x864. Width and height must both be divisible by 16 and the requested aspect ratio must be between 1:3 and 3:1. Resolutions above 2560x1440 are experimental, and the maximum supported resolution is 3840x2160. The requested size must also satisfy the model’s current pixel and edge limits. The standard sizes 1024x1024, 1536x1024, and 1024x1536 are supported by the GPT image models; auto is supported for models that allow automatic sizing.
One of the following: "1024x1024" "1024x1536" "1536x1024" "auto" svg: optional { model, reasoning } SVG author model settings. Requires output_format to be svg.
When SVG model selection is enabled, omitting this object uses the
outer response model. Otherwise, the default SVG author is used and
explicit model overrides are unavailable.
model: optional string minLength1 reasoning: optional object or null Reasoning effort for SVG authoring. When omitted, uses the author model’s default in both Images and Responses.
effort: optional "none" or "minimal" or "low" or 4 more Controls how much effort the model should spend on reasoning for this request.
One of the following: "none" "minimal" "low" "medium" "high" "xhigh" "max" LocalShell { type } A tool that allows the model to execute shell commands in a local environment.
type: "local_shell" (default: "local_shell") The type of the local shell tool. Always local_shell.
Shell { type, allowed_callers, environment } A tool that allows the model to execute shell commands.
type: "shell" (default: "shell") The type of the shell tool. Always shell.
allowed_callers: optional array of "direct" or "programmatic" or null The tool invocation context(s).
minItems1 One of the following: "direct" "programmatic" environment: optional ContainerAuto or LocalEnvironment or ContainerReference or null One of the following: ContainerAuto { type, file_ids, memory_limit, 2 more } type: "container_auto" (default: "container_auto") Automatically creates a container for this request
file_ids: optional array of string An optional list of uploaded files to make available to your code.
maxItems50 memory_limit: optional "1g" or "4g" or "16g" or "64g" or null The memory limit for the container.
One of the following: "1g" "4g" "16g" "64g" network_policy: optional ContainerNetworkPolicyDisabled or ContainerNetworkPolicyAllowlist Network access policy for the container.
One of the following: ContainerNetworkPolicyDisabled { type } type: "disabled" (default: "disabled") Disable outbound network access. Always disabled.
ContainerNetworkPolicyAllowlist { allowed_domains, type, domain_secrets } allowed_domains: array of string A list of allowed domains when type is allowlist.
minItems1 type: "allowlist" (default: "allowlist") Allow outbound network access only to specified domains. Always allowlist.
domain_secrets: optional array of ContainerNetworkPolicyDomainSecret { domain, name, value } Optional domain-scoped secrets for allowlisted domains.
minItems1 domain: string The domain associated with the secret.
minLength1 name: string The name of the secret to inject for the domain.
minLength1 value: string The secret value to inject for the domain.
minLength1 maxLength10485760 skills: optional array of SkillReference or InlineSkill An optional list of skills referenced by id or inline data.
maxItems200 One of the following: SkillReference { skill_id, type, version } skill_id: string The ID of the referenced skill.
minLength1 maxLength64 type: "skill_reference" (default: "skill_reference") References a skill created with the /v1/skills endpoint.
version: optional string Optional skill version. Use a positive integer or ‘latest’. Omit for default.
InlineSkill { description, name, source, type } description: string The description of the skill.
name: string The name of the skill.
source: InlineSkillSource { data, media_type, type } Inline skill payload
data: string Base64-encoded skill zip bundle.
minLength1 maxLength70254592 media_type: "application/zip" (default: "application/zip") The media type of the inline skill payload. Must be application/zip.
type: "base64" (default: "base64") The type of the inline skill source. Must be base64.
type: "inline" (default: "inline") Defines an inline skill for this request.
LocalEnvironment { type, skills } type: "local" (default: "local") Use a local computer environment.
skills: optional array of LocalSkill { description, name, path } An optional list of skills.
maxItems200 description: string The description of the skill.
name: string The name of the skill.
path: string The path to the directory containing the skill.
ContainerReference { container_id, type } container_id: string The ID of the referenced container.
type: "container_reference" (default: "container_reference") References a container created with the /v1/containers endpoint
Custom { name, type, allowed_callers, 4 more } A custom tool that processes input using a specified format. Learn more about custom tools
name: string The name of the custom tool, used to identify it in tool calls.
type: "custom" (default: "custom") The type of the custom tool. Always custom.
allowed_callers: optional array of "direct" or "programmatic" or null The tool invocation context(s).
minItems1 One of the following: "direct" "programmatic" async: optional boolean Whether the tool response can be returned asynchronously versus immediately returned on next response creation.
defer_loading: optional boolean Whether this tool should be deferred and discovered via tool search.
description: optional string Optional description of the custom tool, used to provide more context.
format: optional CustomToolInputFormat The input format for the custom tool. Default is unconstrained text.
One of the following: Text { type } Unconstrained free-form text.
type: "text" (default: "text") Unconstrained text format. Always text.
Grammar { definition, syntax, type } A grammar defined by the user.
definition: string The grammar definition.
syntax: "lark" or "regex" The syntax of the grammar definition. One of lark or regex.
One of the following: "lark" "regex" type: "grammar" (default: "grammar") Grammar format. Always grammar.
Namespace { description, name, tools, type } Groups function/custom tools under a shared namespace.
description: string A description of the namespace shown to the model.
name: string The namespace name used in tool calls (for example, crm).
minLength1 tools: array of Function or Custom The function/custom tools available inside this namespace.
minItems1 One of the following: Function { name, type, allowed_callers, 6 more } name: string pattern^[a-zA-Z0-9_-]+$ minLength1 maxLength128 type: "function" (default: "function") allowed_callers: optional array of "direct" or "programmatic" or null The tool invocation context(s).
minItems1 One of the following: "direct" "programmatic" async: optional boolean Whether the tool response can be returned asynchronously versus immediately returned on next response creation.
defer_loading: optional boolean Whether this function should be deferred and discovered via tool search.
description: optional string or null output_schema: optional map[unknown] or null A JSON Schema describing the JSON value encoded in string outputs for this function tool. This does not describe content-array outputs.
parameters: optional unknown or null strict: optional boolean or null Whether to enforce strict parameter validation. If omitted, Responses attempts to use strict validation when the schema is compatible, and falls back to non-strict validation otherwise.
Custom { name, type, allowed_callers, 4 more } A custom tool that processes input using a specified format. Learn more about custom tools
name: string The name of the custom tool, used to identify it in tool calls.
type: "custom" (default: "custom") The type of the custom tool. Always custom.
allowed_callers: optional array of "direct" or "programmatic" or null The tool invocation context(s).
minItems1 One of the following: "direct" "programmatic" async: optional boolean Whether the tool response can be returned asynchronously versus immediately returned on next response creation.
defer_loading: optional boolean Whether this tool should be deferred and discovered via tool search.
description: optional string Optional description of the custom tool, used to provide more context.
format: optional CustomToolInputFormat The input format for the custom tool. Default is unconstrained text.
One of the following: Text { type } Unconstrained free-form text.
type: "text" (default: "text") Unconstrained text format. Always text.
Grammar { definition, syntax, type } A grammar defined by the user.
definition: string The grammar definition.
syntax: "lark" or "regex" The syntax of the grammar definition. One of lark or regex.
One of the following: "lark" "regex" type: "grammar" (default: "grammar") Grammar format. Always grammar.
type: "namespace" (default: "namespace") The type of the tool. Always namespace.
ToolSearch { type, description, execution, parameters } Hosted or BYOT tool search configuration for deferred tools.
type: "tool_search" (default: "tool_search") The type of the tool. Always tool_search.
description: optional string or null Description shown to the model for a client-executed tool search tool.
execution: optional "server" or "client" Whether tool search is executed by the server or by the client.
One of the following: "server" "client" parameters: optional unknown or null Parameter schema for a client-executed tool search tool.
WebSearchPreview { type, search_content_types, search_context_size, user_location } This tool searches the web for relevant results to use in a response. Learn more about the web search tool.
type: "web_search_preview" or "web_search_preview_2025_03_11" (default: "web_search_preview") The type of the web search tool. One of web_search_preview or web_search_preview_2025_03_11.
One of the following: "web_search_preview" "web_search_preview_2025_03_11" search_content_types: optional array of "text" or "image" One of the following: "text" "image" search_context_size: optional "low" or "medium" or "high" High level guidance for the amount of context window space to use for the search. One of low, medium, or high. medium is the default.
One of the following: "low" "medium" "high" user_location: optional object or null The approximate location of the user. If omitted or null, defaults to the United States. To avoid this fallback, pass {"type": "approximate"} without location fields. To localize results, provide the relevant location fields.
type: "approximate" (default: "approximate") The type of location approximation. Always approximate.
city: optional string or null Free text input for the city of the user, e.g. San Francisco.
country: optional string or null The two-letter ISO country code of the user, e.g. US.
region: optional string or null Free text input for the region of the user, e.g. California.
timezone: optional string or null The IANA timezone of the user, e.g. America/Los_Angeles.
ApplyPatch { type, allowed_callers } Allows the assistant to create, delete, or update files using unified diffs.
type: "apply_patch" (default: "apply_patch") The type of the tool. Always apply_patch.
allowed_callers: optional array of "direct" or "programmatic" or null The tool invocation context(s).
minItems1 One of the following: "direct" "programmatic" top_p: optional number (default: 1) An alternative to temperature for nucleus sampling; 1.0 includes all tokens.
error: EvalAPIError or null An object representing an error response from the Eval API.
code: string The error code.
message: string The error message.
eval_id: string The identifier of the associated evaluation.
metadata: Metadata or null Set of 16 key-value pairs that can be attached to an object. This can be
useful for storing additional information about the object in a structured
format, and querying for objects via API or the dashboard.
Keys are strings with a maximum length of 64 characters. Values are strings
with a maximum length of 512 characters.
model: string or null The model that is evaluated, if applicable.
name: string or null The name of the evaluation run.
object: "eval.run" (default: "eval.run") The type of the object. Always “eval.run”.
per_model_usage: array of object or null Usage statistics for each model during the evaluation run.
cached_tokens: integer The number of tokens retrieved from cache.
completion_tokens: integer The number of completion tokens generated.
invocation_count: integer The number of invocations.
model_name: string The name of the model.
prompt_tokens: integer The number of prompt tokens used.
total_tokens: integer The total number of tokens used.
per_testing_criteria_results: array of object or null Results per testing criteria applied during the evaluation run.
failed: integer Number of tests failed for this criteria.
passed: integer Number of tests passed for this criteria.
testing_criteria: string A description of the testing criteria.
report_url: string The URL to the rendered evaluation run report on the UI dashboard.
formaturi result_counts: { errored, failed, passed, total } Counters summarizing the outcomes of the evaluation run.
errored: integer Number of output items that resulted in an error.
failed: integer Number of output items that failed to pass the evaluation.
passed: integer Number of output items that passed the evaluation.
total: integer Total number of executed output items.
status: string The status of the evaluation run.
Returns Examples
Response example
{
"object": "list",
"data": [
{
"object": "eval.run",
"id": "evalrun_67e0c7d31560819090d60c0780591042",
"eval_id": "eval_67e0c726d560819083f19a957c4c640b",
"report_url": "https://platform.openai.com/evaluations/eval_67e0c726d560819083f19a957c4c640b",
"status": "completed",
"model": "o3-mini",
"name": "bulk_with_negative_examples_o3-mini",
"created_at": 1742784467,
"result_counts": {
"total": 1,
"errored": 0,
"failed": 0,
"passed": 1
},
"per_model_usage": [
{
"model_name": "o3-mini",
"invocation_count": 1,
"prompt_tokens": 563,
"completion_tokens": 874,
"total_tokens": 1437,
"cached_tokens": 0
}
],
"per_testing_criteria_results": [
{
"testing_criteria": "Push Notification Summary Grader-1808cd0b-eeec-4e0b-a519-337e79f4f5d1",
"passed": 1,
"failed": 0
}
],
"data_source": {
"type": "completions",
"source": {
"type": "file_content",
"content": [
{
"item": {
"notifications": "\n- New message from Sarah: \"Can you call me later?\"\n- Your package has been delivered!\n- Flash sale: 20% off electronics for the next 2 hours!\n"
}
}
]
},
"input_messages": {
"type": "template",
"template": [
{
"type": "message",
"role": "developer",
"content": {
"type": "input_text",
"text": "\n\n\n\nYou are a helpful assistant that takes in an array of push notifications and returns a collapsed summary of them.\nThe push notification will be provided as follows:\n<push_notifications>\n...notificationlist...\n</push_notifications>\n\nYou should return just the summary and nothing else.\n\n\nYou should return a summary that is concise and snappy.\n\n\nHere is an example of a good summary:\n<push_notifications>\n- Traffic alert: Accident reported on Main Street.- Package out for delivery: Expected by 5 PM.- New friend suggestion: Connect with Emma.\n</push_notifications>\n<summary>\nTraffic alert, package expected by 5pm, suggestion for new friend (Emily).\n</summary>\n\n\nHere is an example of a bad summary:\n<push_notifications>\n- Traffic alert: Accident reported on Main Street.- Package out for delivery: Expected by 5 PM.- New friend suggestion: Connect with Emma.\n</push_notifications>\n<summary>\nTraffic alert reported on main street. You have a package that will arrive by 5pm, Emily is a new friend suggested for you.\n</summary>\n"
}
},
{
"type": "message",
"role": "user",
"content": {
"type": "input_text",
"text": "<push_notifications>{{item.notifications}}</push_notifications>"
}
}
]
},
"model": "o3-mini"
},
"error": null,
"metadata": {}
}
],
"first_id": "evalrun_67e0c7d31560819090d60c0780591042",
"last_id": "evalrun_67e0c7d31560819090d60c0780591042",
"has_more": true
}