Custom content-safety detector API

When connecting a self-built or third-party detector to the platform content-safety guardrail, the service must implement the HTTP contract described here. When adding a provider under Model Providers, choose the preset Custom content-safety detector, fill in Base URL and API key (that provider ships a fixed model content-safety-detector), then select it on the Content safety row under Global settings › Default models. The platform then sends content-safety detection requests to the endpoint in this page. How to choose between this path and a WASM gateway extension is in the Security vendor integration guide.

1. Endpoint and authentication

2. Request

{
  "direction": "output",
  "userText": "the user's input this turn",
  "responseText": "the model's output this turn",
  "history": [
    { "role": "user", "text": "previous user input" },
    { "role": "assistant", "text": "previous model output" }
  ]
}

• direction: input (request direction) or output (response direction).
• userText: required; the user input being checked. When multi-turn history is present, the platform prefixes it onto this field (separated by "user input this turn:"). A service that does not support multi-turn can detect this field as-is.
• responseText: present only when direction is output; the model output being checked.
• history: optional; present when the rule turns on multi-turn detection. Previous turns in time order; role is user or assistant. A service that supports multi-turn judgment can use it directly.

3. Response

{
  "categories": {
    "political_security": 0.93,
    "illegal_terror_rumor": 0.0,
    "porn_vulgar_values": 0.0,
    "discrimination_hate": 0.0,
    "privacy_compliance": 0.0,
    "high_risk_domain": 0.0
  },
  "jailbreak": 0.02,
  "labels": [{ "label": "political", "score": 0.93 }]
}

• categories: all 6 categories are required. Keys are the fixed tokens in the table below; values are scores 0–1 (higher is more dangerous). Missing any key, an unknown key, or a score out of range makes the platform treat the response as invalid.
• jailbreak: required; prompt-injection (jailbreak) score, 0–1.
• labels: optional raw-label list. Each item must contain label (string) and score (number). Used only for audit display; the platform judgment does not read it.
• The response top level may only have the three keys categories, jailbreak, and labels. Any other key is also invalid.

4. Timeout and behavior

• A single platform call times out in 8 seconds. Timeout, connection failure, or an invalid response is handled by the guardrail's detection-failure policy (the When detection cannot run action configured on that rule). It is not silently allowed.
• The service should be idempotent and stateless; the same text called several times returns a consistent result.
• Checking before saving the model actually calls this endpoint with one violating sample and one safe sample. If the service is not ready, save is refused and a reason is given.

5. Verification

After connecting, verify in this order:

1. Saving with that preset under Model Providers succeeds.
2. Selecting that service under Global settings › Default models and saving succeeds (save triggers the check).
3. In a traffic rule Try a sentence, send a safe sample and a real business-risk sample separately, and confirm the content-risk row returns the expected category.