6. Models and model groups

Back to overview · Full table of contents · Download full PDF

This chapter is for platform administrators. It explains the three objects on the platform and what each owns, and how to manage them. After this chapter you can: understand the relationship among upstreams, models, and model groups; see which upstreams serve a model, where calls go, and whether it is available; create a model, attach or re-attach upstreams, and delete a model; create a model group and change group order; use the prompt library to inject prompts into model calls and manage versions; and check references before you delete.

Each of the three objects owns one thing:

When callers call through the API, they write the model's call name in the request model field. The gateway first finds the model by call name, then confirms that a group the caller can use contains that model, then dispatches among that model's upstreams. Both "call name does not exist" and "the caller is not allowed to use this model" return 404 model_not_found. The platform does not leak whether the model exists through the error code.

Prerequisites: sign in as tenant administrator or a role that can manage models; at least one connected provider (Chapter 4).

6.1 Where models come from

When to use: first time you connect a provider, or you want to know why a model already appears in the list.

When you connect a provider, each upstream model you enable gets a destination in the Will become model column:

• An upstream whose name matches an existing model (case-insensitive) defaults to attaching to that model. If its deployment location differs from that model's existing upstreams, the default is to create a separate model and require a new call name.
• An upstream that the reference catalog judges to be the same model gets an attach suggestion with the evidence shown.
• Other upstreams default to creating a new same-name model, which you can rename. Call names are unique in the tenant.

A newly created model can also tick model groups to join. Any upstream attached to an existing model shows impact before confirm: which groups and which traffic rule will start dispatching calls to this new upstream, and how it differs from existing upstreams in deployment location, price, and capabilities. After confirm it takes effect.

Turning on the enable switch of an upstream that has never been attached to a model, in the provider Models and prices, opens the same Will become model confirmation. Cancel restores the switch.

Note: an enabled upstream must belong to a model. The platform does not provide Detach. To move an upstream to another model, re-attach it (Section 6.4).

6.2 Model list and detail

When to use: inventory models in the tenant, confirm which upstreams serve a model, where calls go, and whether it is currently available.

Procedure

1. Open Admin Console › Models › Models.
2. Search by call name or display name in the top search box.
3. Click a model to open its detail.

What you will see

• List columns are Model, Upstreams, Destination, Groups, Price, Status. Upstreams lists providers that serve this model. When prices differ across upstreams, a range is shown.
• The Destination column states where calls go when no other condition is specified: Default dispatch means among all enabled upstreams of this model; Routed by rule "X", Allowed by rule "X", Denied by rule "X" mean a rule that always applies to this model decides the destination. When other conditional rules sit in front, it also says N more conditional rules sit in front. This column is absent when traffic governance is not enabled.
• Status values:

The model-detail header shows display name and status. When a display name is set, the call name is shown under it. Merge into another model and delete are in the header More menu. Model detail has these tabs, in order:

• Upstreams: lists each upstream's status, accounts (how many, how many available), deployment location, whether redaction is required, and current price, plus the aggregated call destination. Deployment location and sensitive-data admission are read-only here; change them in the metadata dialog of the provider Models and prices row. Each row's Limits can set concurrency or rate limits for that upstream without leaving the model page.
• Traffic rules: rules that act on this model, plus a last-row Default, listed in effective order. From here you can create a rule with this model as a condition, or jump to the validation page with this model prefilled (Chapter 15). This tab is absent when traffic governance is not enabled.
• Limits: traffic limits that will act on calls to this model, including limits that constrain this model or its upstreams, and limits that do not constrain a model. Add limit prefills Model equals this model.
• Call activity: call counts, share, and cost taken by each upstream in the last 7, 30, or 90 days, used to confirm that split is as intended. View cost by upstream account jumps to Operations overview with this model selected.
• Basic information: external display (display name, vendor, description), capability comparison (actual upstream capabilities listed next to the declared ones; declarations that exceed the upstream are marked), and model categories. Fill from models.dev writes the reference model's information into the form; it takes effect after save. The call name cannot be changed after create.
• Groups: model groups this model is in. Adding and removing a model is done in the model group (Section 6.6).
• Try: send a message on the real call path to confirm the model can produce a result. Try is a real billed call. It is also written to call records and operation audit.

Note: when traffic governance is not enabled, the Upstreams tab states that calls are dispatched the product-default way: weighted round-robin among all enabled upstreams, the same session stays on the same upstream, and a failed upstream is replaced by another.

6.3 Create a model

When to use: you need a unified external call name whose backend is decided by a traffic rule; or you want to create a model by hand for a set of upstreams.

Procedure

1. On the Models page choose New model.
2. Fill in Call name: the name callers use in the request, case-insensitive, unique in the tenant, cannot be changed after create; only English letters, digits, and visible symbols, no spaces.
3. Fill in Display name (empty shows the call name). Optionally pick a Reference model to bring in that model's capability declaration and description.
4. Under Upstreams, add upstreams to attach. When traffic governance is enabled you can attach none; then the model can only get a backend from a traffic rule. When traffic governance is not enabled, attach at least one upstream.
5. Confirm create.

What you will see: a model with no upstreams is created with status No available backend, and calls return 503. After you create a rule on its Traffic rules tab that gives it a target, status becomes Available (provided by a rule). When the rule only applies to some callers, it is Available for some calls. After you add the model to a model group, members in that group's availability scope can see it on the model plaza and call it.

Note: a rule that provides a backend for a model with no upstreams must have a target that covers the capabilities the model declares, or save will say the capabilities do not match.

6.4 Attach, re-attach, and merge

When to use: two differently named models are actually the same model and need to be merged; or an upstream was attached to the wrong model.

Procedure

• Attach an existing upstream: on model A's Upstreams tab choose Attach existing upstream. You can pick an upstream currently hanging under another model, which re-attaches it to A.
• Re-attach to another model: in the upstream row's actions choose Re-attach to…, pick the target model, or choose Create a new model instead.

Both operations compute impact before confirm: upstream-count change on both models (Upstreams N → M) and groups. After confirm they finish in one step; there is no state where an upstream belongs to no model.

After a merge, if original model B has no upstreams left, you can keep it (backend provided by a traffic rule) or delete it.

Disable an upstream: use the enable switch in the provider Models and prices. After disable, the upstream still belongs to the original model. Disable is refused when that upstream is referenced as a target by a traffic rule; change the rule first.

Vendor-side retirement: when a sync finds that an upstream is no longer provided, the platform marks it Vendor retired without confirmation, the model status becomes Some upstreams failed, and configuration checks flag rules that use it as a target.

6.5 Delete a model

When to use: clean up a model that is no longer used.

Procedure: in the upper right of the model detail choose delete. The platform first checks delete impact, then shows a confirm box.

What you will see

• If enabled upstreams remain, delete is blocked: The following upstreams are still enabled; re-attach them to another model or disable them first.
• If the model is still in model groups, delete is blocked and the groups are listed: The model is still in the following groups; remove it first.
• If the model is still the default model for a purpose in Global settings, delete is blocked: The model is still the default model for the following purposes; change it first.
• The following are listed but do not block: a traffic-rule condition references this model (after delete the condition no longer matches); an evaluator, dataset run, or diagnostic review picked this model (after delete you must pick again); disabled upstreams are detached with the delete, and enabling them later opens the Will become model confirmation.
• When there is no impact it shows Delete does not affect other configuration.

Note: delete cannot be undone. After delete, calling that call name returns 404 model_not_found.

6.6 Model groups

When to use: open a set of models to members or business systems, and set availability scope and quota for them.

A model group is the unit of authorization, quotas, target-specific prices, and API keys: first put models into the group, then grant the group to members, departments, functional groups, or workspaces (Chapter 7). Only then can members call models in the group with an API key bound to that group. The same model can appear in more than one group. Each group's authorization, quotas, and billing are independent. The group only owns authorization and quota; it does not decide which upstream a call goes to.

Entry: Admin Console › Organization & Access › Model Groups. The page has Model Groups and API key management tabs (the latter is in Chapter 9).

Create a model group

1. Choose Add model group.
2. Set the group badge, fill in name (unique in the tenant) and description. Put intended tasks and cost tier in the description so members can see them when they choose.
3. Choose Add models and tick models to join. Added models can be dragged to change display order; click Remove for ones you do not need.
4. After save, continue to configure availability scope (Chapter 7). A new group is granted to nobody, so members cannot see it.

Group detail has tabs such as Basic information, Visibility & quotas, Call activity, Traffic rules, and Operation Audit. The left of Basic information is the Publish switch and group-level quota; the right is the member model table. Group target-specific prices are set per upstream under member models, because billing is settled on the upstream that actually handled the request.

Naming and split suggestions

• Name and split by purpose, for example General chat, Code assistant, Low-cost batch. Do not use a model SKU as the group name.
• When you need different available people, cost caps, or data-handling requirements, split into two groups rather than distinguishing by convention inside one group.
• For callers that need to handle images, put at least one multimodal model in the group.

6.7 Group order and metering priority

When to use: one person can use the same model through more than one group, and you need to decide which group usage is counted in.

The order of the group list is metering priority: among groups the caller can use that contain the requested model, usage is counted in the highest-ordered group that still has quota this month. When every group's quota is exhausted, the call is denied with a quota error. The Metering group field on the call detail shows which group this call was actually counted in.

Procedure

1. In the Model Groups list, drag a group row to the target position.
2. After you release, the reorder impact is shown first: how many people are affected and how metering destinations change.
3. Confirm to save the new order. If another administrator added, deleted, or reordered groups in the meantime, save is refused; refresh and reorder again.

Note: group order only affects which group usage is counted in. It does not affect which upstream a call goes to, or whether the call is allowed.

6.8 Publish and disable a group

The Publish switch on the left of the Basic information tab is the group's master switch:

• On: members in scope can see and call models in the group.
• Off: the group is taken down; calls through that group are denied. Authorization, quotas, and API keys are kept and resume after you turn it on again.

Turning it off first opens a confirm box listing the consequences of disable: how many API keys are bound to the group, traffic rules whose conditions reference the group, and functions in Global settings that are affected. Disable and restore are both written to operation audit.

Note: while disabled, the error the caller sees may look like credential invalid. When diagnosing a sudden "cannot call", first confirm the Publish switch, then check keys. Disabling a provider (Chapter 4) affects all of that provider's upstreams, a wider blast radius than disabling one group.

6.9 Prompt templates and versions

When to use: you want to add one system prompt (identity statement, output-format requirement, safety constraint) to a class of model calls, and you want that text to have versions, be switchable, and be roll-backable like code.

Prompts are maintained in the tenant-level prompt library, with versions and two labels, production and latest. They are referenced by the Prompt action of a traffic rule, by label, with an injection position. Prompts do not hang on a model or a model group: when they should apply only to a model or a group, narrow the range in the traffic rule with a Model or Model group condition.

Create a prompt

Procedure

1. Open Admin Console › Prompts and choose New prompt. The top of the dialog states that this prompt is visible to all workspaces in the tenant and can be referenced by traffic rules.
2. Fill in Name, Group, Content, Commit message. Write variables in the content as {{name}} or [name].
3. Choose Save. The first version is automatically labeled production and latest.

What you will see: after save you enter the detail page. Tabs are Overview, Referenced by traffic rules, Versions, Metrics. Upper right is New version, Send to agent, Delete. The header marks the source as Tenant. Overview shows the body as-is; the Variables area below lists recognized variables.

The gateway can render only this set of system variables: {{tenant}} (current tenant), {{principal.user}} (caller member), {{principal.agent}} (caller agent), {{principal.credential}} (credential of this call), {{route}} (matched route), {{request_id}} (platform call ID), {{client_request_id}} (the caller's X-Request-Id; empty when not sent or invalid).

Note: user input is not filled into template variables. Variables in the body outside this set (for example {{产品名}}) are not replaced with values from the caller. Saving a rule that references that prompt will error.

Reference a prompt in a traffic rule

Procedure

1. Open Admin Console › Traffic › Traffic rules, create or edit a rule, and narrow the range with conditions, for example Model group = General chat and Call type = Model call.
2. Add the disposition action Prompt, pick the prompt, a label (for example production), and the injection position (prepend or append).
3. Save and confirm the rule (full traffic-rule operations are in Chapter 15).

What you will see: for a call that hits the rule, in Operations overview › Requests, Request sent upstream, the first item is the injected system message, variables already rendered to actual values, with the user message after it. That message is not in Client request.

Manage versions, switch, and roll back

Procedure

1. On the prompt detail choose New version, change the content, fill in a commit message. Tick Also label as production version if it should take effect immediately.
2. After save, view all versions and each version's commit message and labels on the Versions tab.
3. Switch: select a version and label it as the production version.
4. Roll back: select an old version, choose Create a new version from this content, then label the new version as production. That both restores the old content and keeps a record of who rolled back and when.

What you will see: after you add a version, the version count goes from 1 to 2. The newest created version has latest; the one labeled production has production.

Note:

• When the production label is referenced by a rule, it resolves to a fixed version. After you move the label to a new version, rules that reference it need to be republished to take effect, in about ten-odd seconds.
• Roll back is create a new version from the old content, not moving a pointer back, so version history stays complete and version numbers do not go backwards.
• Injection has only prepend and append, and both append; they do not overwrite a system message the caller already sent. User input always sits between the two.
• Whether a prompt has an effect depends on whether the model follows system instructions. To intercept prompt injection, combine with Content detection · Conversation content model judge (Chapter 14); do not rely on the prompt alone.

References

The Referenced by traffic rules tab lists traffic rules that reference it, with name, priority, and status:

Before you change or delete a prompt, confirm the blast radius here. An empty list means no rule currently references it.

6.10 Delete a group and reference protection

Procedure: at the end of a Model Groups list row, … › Delete. The platform first checks references, then shows a confirm box.

What you will see

• When nothing blocks delete, the confirm box only has an irreversible warning and Cancel / Delete.
• When references remain, Cannot delete this model group yet pops up. The group lists Global settings functions that are using this group, plus protected records such as published prices and billing records. After you handle them, choose I have handled this, check again.
• When traffic-rule Model group conditions still reference it, the confirm box lists those rules with Go change.

After a group is deleted, its authorization and quotas lapse, and API keys bound to that group can no longer call. The confirm box does not list availability-scope and API-key detail; check those yourself before delete. In production, first turn off Publish, watch for a while, confirm nobody is using it, then delete. All deletes are written to operation audit (Chapter 22).

6.11 Operation audit

When to use: look back at who recently changed a model or a model group, and when it was disabled.

The Operation Audit tab of model and model-group detail records changes on that object. Each record has operator, time, source, and action type. Each on and off of the group Publish switch is recorded as one row, so "who turned the group off and when" can be found here directly.

Note: this tab only covers the current object. For cross-object queries (for example "which groups did this person change today"), go to Admin Console › Audit & compliance › Operation Audit and search by operator, time, and keyword (Chapter 22).

Capability notes

---

Previous: 5. Model pricing and categories · Next: 7. Configure model-group availability scope and quotas