10. Calling the APIs

Back to overview · Full table of contents · Download full PDF

This chapter is for developers and technical owners who connect to the gateway. It explains how a business application calls models through the gateway — paths and authentication of the three protocols, streaming, embeddings, rerank, images, speech, batch decisions, protocol entry and ports, official SDKs and programming clients, idempotency keys, automatic parameter adaptation, request-body and token caps, and the unified error format.

The gateway finishes authentication, authorization, quota, and traffic-rule judgment at the entry, then converts the request into the upstream provider's format. Responses and errors are returned as-is in the protocol the caller used, so code already written for a vendor protocol only needs to change Base URL and key. Applying for credentials and managing their status are in chapters 8 and 9. Traffic-rule and protection configuration is in Chapter 15.

10.1 Entry URL and authentication

When to use: first connection, or switching existing code onto the gateway.

Prerequisites: you already hold an API key (see section 8.4), and the model you will call belongs to the model group this key is bound to.

Base URL is the gateway access URL plus a fixed prefix, and can be copied from the model detail Make your first API call. Below, <网关地址> is that access URL:

Note

• Each protocol only accepts its native request header. The OpenAI path does not accept x-api-key; use Authorization: Bearer. Do not carry two credentials with different values on the same request.
• Every model call must carry a credential. You cannot turn off authentication per route.
• The model name (the model field in the body for OpenAI and Anthropic, the model name in the path for Gemini) is the call name copied from the model detail. If it is not in the model group this key is bound to, the response is 404 LLM_MODEL_NOT_AVAILABLE.

Confirm the result: a successful call carries x-request-id in the response header. In Operations overview › Requests you can find this call by that value; initiator and access credential match the key holder (see section 9.9).

10.2 OpenAI-compatible APIs

The official OpenAI SDK only needs base_url and api_key changed. Paths are relative to the OpenAI Base URL:

curl https://<网关地址>/api/v1/agent-gateway/openai/v1/chat/completions \
  -H "Authorization: Bearer <YOUR_API_KEY>" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "<模型名>",
    "messages": [{"role": "user", "content": "用一句话介绍杭州。"}],
    "max_tokens": 200
  }'

from openai import OpenAI

client = OpenAI(
    base_url="https://<网关地址>/api/v1/agent-gateway/openai/v1",
    api_key="<YOUR_API_KEY>",
)

resp = client.chat.completions.create(
    model="<模型名>",
    messages=[{"role": "user", "content": "用一句话介绍杭州。"}],
    max_tokens=200,
)
print(resp.choices[0].message.content)

To list available models use client.models.list(), which returns each model's id.

What you will see

• chat/completions returns a standard chat.completion structure (choices, message, usage). Reasoning models additionally carry reasoning_content.
• responses returns a standard response object, status is completed, output contains both message and reasoning segments, and output_text is the final answer.
• completions is available only when the upstream model supports it. When it does not, the gateway returns 400 directly, reason is MODEL_ENDPOINT_UNSUPPORTED ("This model service does not support the requested endpoint or call style."). The request is not dispatched upstream and produces no cost.
• GET /models returns models in the platform model catalog, which may include models that are not enabled in the service this key belongs to. Calling them returns 404 LLM_MODEL_NOT_AVAILABLE.

Note: when max_tokens is very small on a reasoning model, reasoning may use up the whole budget, content is an empty string, and finish_reason is length. That is normal model behavior, not a gateway truncation. Increase max_tokens. Upstream-side errors (for example a product line not opened) return as 400 LLM_UPSTREAM_CLIENT_ERROR with the upstream original text; handle them on the vendor console.

10.3 Native Anthropic and Gemini APIs

Business already written against the Anthropic or Google official SDK only needs to change Base URL; you do not rewrite the request format.

Anthropic: POST /v1/messages (supports streaming, accepts anthropic-version and anthropic-beta headers), GET /v1/models, GET /v1/models/{model}.

import anthropic

client = anthropic.Anthropic(
    base_url="https://<网关地址>/api/v1/agent-gateway/anthropic",
    api_key="<YOUR_API_KEY>",
)
msg = client.messages.create(
    model="<模型名>",
    max_tokens=1024,
    messages=[{"role": "user", "content": "你好"}],
)

Gemini: POST /v1beta/models/{model}:generateContent, POST /v1beta/models/{model}:streamGenerateContent, also accepts /v1/models/... paths; GET /v1beta/models, GET /v1beta/models/{model}.

from google import genai

client = genai.Client(
    api_key="<YOUR_API_KEY>",
    http_options={
        "base_url": "https://<网关地址>/api/v1/agent-gateway/gemini",
        "api_version": "v1beta",
    },
)
resp = client.models.generate_content(model="<模型名>", contents="你好")

What you will see

• Anthropic returns type: message, stop_reason is end_turn. Reasoning models have content blocks [thinking, text]. On a tool call, stop_reason is tool_use, and the content blocks give the tool name and arguments.
• Gemini returns candidates, text in candidates[0].content.parts. Tool calls return as function_call, with structured-object arguments.
• Multi-turn conversation and tool round-trips close on both native protocols: Anthropic tool_use → return tool_result, Gemini functionCall → return functionResponse, both get a final answer.
• In the call record, Client request keeps the native structure you sent; Request sent upstream is the format the upstream actually uses, so you can check the conversion the gateway did.

Note

• On a Gemini tool round-trip, put the previous round's candidate.content back into contents as a whole, unchanged, then append function_response. Reasoning upstreams need the thinking content sent back; the gateway restores it. Trimming it yourself loses context.
• Native protocols and the OpenAI-compatible protocol share the same access pipeline. Authentication, quota pools, traffic rules, and metering are identical; you do not configure them separately per protocol.

10.4 Streaming

When to use: a chat UI that shows tokens as they arrive, or you need to hand content to downstream processing as early as possible during generation.

How to turn it on: OpenAI and Anthropic set "stream": true in the request body. OpenAI can also add "stream_options": {"include_usage": true} to get usage at the end. Gemini uses the :streamGenerateContent method.

curl https://<网关地址>/api/v1/agent-gateway/openai/v1/chat/completions \
  -H "Authorization: Bearer <YOUR_API_KEY>" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "<模型名>",
    "messages": [{"role": "user", "content": "写一段三百字的产品介绍。"}],
    "stream": true,
    "stream_options": {"include_usage": true}
  }'

What you will see

• Response headers are content-type: text/event-stream; charset=utf-8, transfer-encoding: chunked, cache-control: no-cache.
• Each data frame is an increment. Reasoning-model reasoning increments are in reasoning_content; body increments are in content.
• The OpenAI protocol ends with a last frame that has finish_reason and usage, plus data: [DONE]. The Anthropic protocol ends with message_delta and message_stop. Gemini ends when the stream ends.

Note

• The gateway only retries or switches target before the upstream returns the first content. Once content starts, it does not retry, to avoid duplicate content. Parameter errors (400) are not retried and do not switch; x-should-retry is false.
• If the client disconnects halfway, it will not receive [DONE]. That is the client abandoning the request, not an upstream truncation.
• Do not set idle timeouts on an intermediate proxy or load balancer shorter than the generation interval, or a long answer will be cut in the middle.
• When the upstream disconnects without an end signal, the gateway treats it as truncation: it appends a same-shape in-stream error frame, marks the call record as failed, and does not return it as a normal completion.

10.5 Embeddings, rerank, images, and speech

When to use: a knowledge base needs vectorization, search needs rerank, an application needs image generation or speech synthesis, or you need to transcribe recordings.

Prerequisites: these capabilities are only provided in OpenAI-compatible format; paths are relative to the OpenAI Base URL. Before you call, the target model's upstream must declare the matching purpose (embeddings, rerank, image generation, speech synthesis, speech transcription). When the upstream does not support that interface, the gateway returns 400 MODEL_ENDPOINT_UNSUPPORTED before dispatch. When the model does not belong to the group this key is bound to, it returns 404 LLM_MODEL_NOT_AVAILABLE.

{ "model": "<向量模型名>", "input": ["退款进度怎么查", "订单已发货", "发票如何开具"] }

{ "model": "<重排模型名>", "query": "退款进度", "documents": ["订单已发货。", "退款将在三个工作日内到账。"] }

What you will see: the embeddings API returns a data array, each item with embedding and index. The rerank API returns results sorted by relevance. Image generation returns an image URL or base64. Speech synthesis returns audio binary; speech transcription returns text. When the model has not declared the matching capability, speech endpoints are all stopped on the gateway (400); the request does not reach upstream.

Note

• The rerank API is only provided under the OpenAI Base URL (/openai/v1/rerank). Other protocol paths have no matching endpoint.
• Speech transcription only reads the audio file header to compute duration; it does not decode the whole audio. Unrecognized format returns 400; a file over 25 MiB returns 413.
• When the upstream provider is temporarily unavailable, it returns 503 LLM_PROVIDER_UNAVAILABLE with Retry-After; retry after that value.
• When the service only has chat models, these calls are denied with 400 or 404. That is a normal configuration result, not an API fault.

10.6 Batch decisions

When to use: answer several yes/no, choice, or scoring questions about one piece of content in one request, for example ticket classification, intent recognition, content-compliance pre-check, replacing several separate calls.

Prerequisites: the model's upstream declared the decisions purpose.

Request format: POST /decisions. The body contains model, input (the content being judged), and questions. Each question has name (question name), instructions (question description), and type:

{
  "model": "<判断模型名>",
  "input": "这笔订单被扣了两次款，我要求马上退款！",
  "questions": [
    { "name": "is_complaint", "instructions": "这是不是一条投诉？", "type": "predicate" },
    { "name": "department", "instructions": "应该交给哪个部门？", "type": "choice",
      "choices": [ { "value": "billing", "description": "付款、发票和退款。" },
                   { "value": "other", "description": "以上都不是。" } ] }
  ]
}

POST /systemone is another format for the same purpose: put the content being judged in state, and questions as an object indexed by question name. The gateway converts both formats into what the upstream needs, then sends.

What you will see: returns 200, with an answer per question by question name, independent of submit order. In the call record, this call's purpose shows as decisions.

Note

• One request at most 256 questions; question names cannot repeat. An illegal request structure returns 400 VALIDATION_FAILED. The error body carries request_id; attach that value when you ask an administrator to locate it.
• If the chosen model does not support decisions, it returns 400 MODEL_ENDPOINT_UNSUPPORTED. Images can only be put in content as data: URLs.
• Decision scores from different upstreams cannot be compared directly. When you compare across, first unify to one upstream. Billing counts both input and output tokens.

10.7 Protocol entry, ports, and transport

One entry, several protocols: the model gateway only exposes one entry URL. The three protocols are distinguished by path prefix:

The gateway recognizes the inbound protocol by prefix, then forwards in the target model's format. The same model and the same key can be called by all three protocols at once; metering and limits are counted together.

HTTP versions and ports: the entry supports HTTP/1.1, HTTP/2, and HTTP/3 (QUIC) at the same time. Response header alt-svc: h3=":443" advertises that HTTP/3 is available; the client negotiates by its own capability. Only 443 is open externally (port 80 redirects to 443 automatically). This product does not force TLS. For an intranet deployment, follow the deployment document for listen port and protocol. Do not put another proxy in front of the gateway that rewrites request headers.

Response compression: static page assets return gzip or zstd according to Accept-Encoding. Model-call APIs (non-streaming JSON and SSE streams) are designed not to compress, to lower time to first token, so a chat response without content-encoding is normal.

WebSocket: only for realtime speech sessions (/openai/v1/realtime), not a general WebSocket proxy. A handshake missing a credential or with a wrong credential returns 401. With a valid credential it returns 101, echoes the realtime subprotocol, then receives a session.created event. Browser clients can first call POST /realtime/client_secrets with a long-lived key to exchange a short-lived temporary token, then handshake. Do not put a long-lived key in the frontend.

10.8 Vendor-native paths

The gateway provides OpenAI, Anthropic, and Gemini inference-class native paths, re-encoded by the upstream protocol then forwarded, still going through authentication, metering, and call records.

That is, only model-inference paths are available. Management-class APIs such as content moderation, Assistants, files, and token counting are not provided. To forward an arbitrary HTTP path as-is to a backend, an administrator registers that backend as an API service and you call through its entry; see the API-service chapters.

Note: native paths are not byte-level pass-through. In the request detail, Client request is the native structure and Request sent upstream is the upstream's actual format. The gateway does protocol conversion in the middle.

10.9 Official SDKs and programming clients

10.9.1 Official SDKs per language

Any language can use each vendor's official SDK directly, changing only Base URL and key. Chat, streaming, and tool-call round-trips behave the same as calling the official API directly. Python samples are in 10.2 and 10.3. The product does not auto-generate SDKs in many languages; other languages you configure yourself from each vendor's official SDK:

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://<网关地址>/api/v1/agent-gateway/openai/v1",
  apiKey: process.env.ZENOVA_API_KEY,
});

const stream = await client.chat.completions.create({
  model: "<模型名>",
  messages: [{ role: "user", content: "从 1 数到 5" }],
  stream: true,
});
for await (const chunk of stream) {
  process.stdout.write(chunk.choices[0]?.delta?.content ?? "");
}

client := openai.NewClient(
    option.WithBaseURL("https://<网关地址>/api/v1/agent-gateway/openai/v1"),
    option.WithAPIKey(os.Getenv("ZENOVA_API_KEY")),
)

10.9.2 Programming clients

Make your first API call on the model detail gives copy-ready access configuration for three programming clients (<网关地址> is copied from the model detail):

export ANTHROPIC_BASE_URL="https://<网关地址>/api/v1/agent-gateway/anthropic"
export ANTHROPIC_API_KEY="<YOUR_API_KEY>"

model = "<模型名>"
model_provider = "zentrix"

[model_providers.zentrix]
name = "Zentrix Gateway"
base_url = "https://<网关地址>/api/v1/agent-gateway/openai/v1"
env_key = "OPENAI_API_KEY"
wire_api = "responses"

What you will see: the client can read files in the working directory, change code, and write changes to disk, meaning tool-call round-trips already work. The matching client name appears in the Operations overview Prompt cache · by client table.

Note

• Fill the URL only to the protocol prefix. Do not add /v1/messages or /v1beta/models/... yourself; the client appends them.
• Codex uses a read-only sandbox by default. For non-interactive runs you must configure Codex's own sandbox and approval policy, or the model only suggests and does not write to disk. That is not a gateway problem.
• Claude Code prefers ANTHROPIC_AUTH_TOKEN in the environment. If the terminal still has that variable from another service, clear it first, then use ANTHROPIC_API_KEY.
• To distinguish by client in records, set Attach to which client to the matching type when you create the key; otherwise records show Unknown client.
• OpenCode has no dedicated access-instructions page; write opencode.json yourself in the OpenAI-compatible way. Members who use ZenSwitch can have it pushed automatically.

10.10 Idempotency keys and safe retries

When to use: after a network interrupt or timeout, you cannot tell whether the previous request was already handled. Resending with an idempotency key avoids calling upstream twice and billing twice.

Usage: carry Idempotency-Key: <唯一键> in the request header. All retries of the same logical request use the same key.

curl https://<网关地址>/api/v1/agent-gateway/openai/v1/chat/completions \
  -H "Authorization: Bearer <YOUR_API_KEY>" \
  -H "Idempotency-Key: order-8842-retry-1" \
  -H "Content-Type: application/json" \
  -d '{"model": "<模型名>", "messages": [{"role": "user", "content": "只回复数字 9"}], "max_tokens": 30}'

Value requirements: 1–256 printable ASCII characters. The idempotency scope is credential + key + API. The same key used by a different secret or a different API does not affect each other.

What you will see

• An illegal key (for example over the length cap) returns 400 VALIDATION_FAILED before dispatching upstream, and produces no cost.
• The same key with a different request body may return 409 RESOURCE_CONFLICT.
• The gateway does not mark in the response header whether this is a replay or a new result. Whether it was a replay you check in the call record.

Note

• Idempotency keys fit write operations and billing-sensitive calls that can be retried safely. Generation requests themselves are random; do not assume a retry will get the same text.
• To confirm it was really billed only once, go to Operations overview › Requests and check the count of same-key requests and upstream attempts by time and credential. Do not only look at whether the response content is the same.
• Do not reuse one fixed key for all requests. The key means this one logical request. Reusing it on different requests introduces conflicts.

10.11 Automatic adaptation to a model's parameter spec

When to use: the same code is sent to several models, and models differ in which parameters they support (for example some reasoning models do not accept temperature or tool_choice).

What you will see

• Parameters the target model supports are forwarded as usual. Unsupported parameters are dropped automatically, and reasoning gears are downgraded to the model's capability. The gateway only drops and downgrades; it does not fill in parameters you did not provide.
• When fields are dropped, response header x-zenova-conversion-dropped lists them comma-separated, for example openai_chat.temperature,openai_chat.tool_choice. When everything is forwarded as usual, that header is absent.
• In Operations overview › request detail › Request › Request sent upstream › Diff you can see rewrites field by field, for example a tool definition had strict filled in, and other messages were unchanged.

Note

• Model parameter capabilities are maintained by the platform's built-in catalog. Administrators do not configure them, and custom rewrite rules are not supported.
• Business code should treat whether a parameter was dropped as an observable signal: a non-empty x-zenova-conversion-dropped means some parameter you depend on did not take effect on this model; change model or change how you call.
• When the Diff tab says Failed to load the message, click Retry to fetch again. That does not affect this call's result or metering.

10.12 Request-body and token caps

An illegal structure always returns 400 LLM_REQUEST_BODY_INVALID, with the text "The request body is not valid JSON or does not match the request structure of this API.", plus param (for example messages, messages[0].role, max_tokens) and details.field_violations explaining the concrete reason. Missing model, missing messages, messages written as a string, illegal role, wrong max_tokens type, illegal JSON, and a request body that is an array all belong here. Denied requests are still written to call records and are not billed.

Administrators can configure stricter caps in a traffic rule for a given combination of model, model group, caller, department, or request header. Exceeding Request-body size cap returns 413, detail request_body_limit_exceeded. Exceeding Per-request token cap on input returns 400 VALIDATION_FAILED, detail token_limit_input_exceeded. An output cap rewrites the request's max_tokens so it does not exceed that value; the excess ends with finish_reason: length. Requests denied by a rule carry deniedBy: routePolicy. The request-detail Routing and rewrite › Hit records shows the matched rule name.

Note

• The gateway does not validate numeric ranges of parameters (for example whether temperature is in an allowed interval). Those parameters are forwarded to the upstream; the upstream refuses and writes the reason in upstream_message. The business side should validate ranges first against the model documentation on the upstream side.
• Traffic-rule caps take effect by matching conditions. After you configure, first verify the actual effect with a request of known size, then put it in production.
• Token counts are estimated by the gateway and may differ slightly from the upstream bill. Limit judgment uses the gateway estimate.

10.13 Unified error format

Errors are returned in each protocol's native structure. All three protocols carry the same stable reason code reason:

Business systems should handle errors by reason, not by parsing message text (it is explanatory text that changes with versions, usually English).

Common situations and reason codes

Response headers

• x-request-id: gateway-side request ID; attach this value when you report a problem to an administrator.
• x-should-retry: true or false, whether a retry might change the result. 400, 401, 404, 413 are all false.
• Retry-After: returned only when the error is retryable and the gateway knows how long to wait.

Handling: first look at HTTP status and reason to decide the class, then look at x-should-retry — when it is false, retry will not change the result; change the request or the credential. Caller errors (400, 401, 403, 404, 413) change the request or credential. Quota errors (429 POOL_EXHAUSTED) ask an administrator to adjust quota. Rate limits wait Retry-After then retry. Upstream-class errors (502, 503, 504) retry later; if they persist, ask an administrator to check the provider.

Note

• Error messages do not contain upstream URLs, upstream keys, or internal stacks.
• Upstream business errors and out-of-range parameters put the upstream original text in the response (including the upstream request ID), so you can locate the upstream-side problem. If that should not appear on the client, record the error on the server and return only reason to the frontend.

Capability notes

Previous: 9. Callers and API keys · Next: 11. Configure and distribute MCP services and API services