12. Troubleshooting and submitting support

Back to overview · Full table of contents · Download full PDF

This chapter is the order of handling when a problem appears: first confirm the three most common causes yourself, then collect troubleshooting information in one pass, then match against install-stage, diagnostic-code, and post-install access-and-call scenarios. If it is still unsolved, prepare materials per section 12.6 and submit support. The readers are the customer system administrator and the implementation engineer. Every collection command in this chapter is read-only and can be run without worry.

12.1 Confirm three things yourself first

Applies to: a caller reports "the call does not work", "it was rejected", or "it suddenly stopped working", but platform services themselves are running normally.

Most of those reports are not failures. They are the platform rejecting per configuration. The reject conclusion is in two places; you do not need logs:

• The reason field in the error body the caller received (OpenAI and Anthropic error bodies also carry request_id, matching response header x-request-id);
• The call record found in console Gateway requests by x-request-id.

Procedure

1. Ask the caller for this call's x-request-id and the reason in the body.
2. Find this call in console Gateway requests by x-request-id and check caller, model, and result.
3. Use the table below to decide which class it is, and check configuration under Where to confirm.

What you will see

• For these three classes, platform service status is healthy. The same member succeeds with a request that is in scope, still has quota, and does not hit a rule.
• Handling is changing configuration (add to scope, adjust quota, change a rule) or having the caller change the request. No service restart is needed.

Note: all three classes of reject leave a call record. If Gateway requests cannot find this call by x-request-id at all, the request did not reach the gateway (wrong address, network down, or a front device intercepted it). Troubleshoot per section 12.5.

12.2 Collect information in one pass

Applies to: a problem section 12.1 cannot explain, or any problem that will be submitted to support. Collect first, then act — after services are restarted, some on-site information is gone.

Procedure

1. Run the four commands below on the install host. All four are read-only and do not change any service state.

# Install host · root · read-only (collect troubleshooting information; output goes to the staging directory)
sudo zenova status --json > /root/zenova-install/status.json
sudo zenova doctor --all --save /root/zenova-install/doctor
sudo zenova logs control-plane --since '30 min ago' > /root/zenova-install/control-plane.log
sudo cp /opt/zenova/state/install.log /root/zenova-install/

2. When the problem happened more than 30 minutes ago, change --since to a time that covers the moment it happened, for example '2 hours ago'. The --since value is passed through to journalctl.
3. When the problem involves connectivity to an external address (model provider, notification channel, object storage), add one targeted doctor:

# Install host · root · read-only (layered DNS, TCP, TLS, HTTP check of a named address)
sudo zenova doctor --target <target-address>

What you will see

• status.json, a doctor/ directory, control-plane.log, and install.log appear under /root/zenova-install/.
• When the zenova doctor --target report notes "direct connect; may not match production egress", the result is only this host's direct connection to that address. It does not represent connectivity of the gateway's actual egress (for example through an egress proxy).

Note: zenova status lists only services that start with zenova-. View postgres, clickhouse, caddy, secret-service, and spicedb with systemctl status <name>. For a problem on a remote gateway node, collect on the node host with sudo zenova status, sudo zenova logs node-edge, and sudo zenova logs node-caddy.

Note: the materials above may contain internal addresses, account names, and request-content fragments. Redact them per section 12.6 before sending them out. Do not collect or submit /opt/zenova/.env, /etc/zenova/secret-service/, or /opt/zenova/.env.runtime/root-password-initial. Those three hold the master key, secret-service material, and the initial password.

12.3 Install-stage failure table

Applies to: zenova install (including --check-only preflight) stops before or after confirm-install, and the screen prints a prompt.

Before confirm-install the installer writes no files and creates no services. A stop before confirm: after fixing the cause, rerun the same command. This host has nothing to clean.

Procedure

1. Look up the exact on-screen text in the table below.
2. Follow the Remedy column, then rerun the original command.
3. If it is not in the table, record the exact on-screen text and the diagnostic code, and check section 12.4.

A few more cases that appear only after confirm-install:

What you will see

• After the fix, a rerun continues from where it last stopped: a stop before confirm answers from the start; a stop after confirm reuses existing install progress.

Note: passing --check-only preflight does not mean the real install will pass. Preflight does not include the business-ingress port check, the existing-dependency probe, post-unpack execution-rights check, or the notification test. Problems in the second table above can appear after preflight has passed.

Note: do not delete progress files under /opt/zenova/state/ or delete directories by hand to "start over". To install from scratch, use wipe-and-reinstall in Chapter 11.

12.4 How to read diagnostic codes

Applies to: output from the installer, zenova doctor, post-install acceptance, upgrade, wipe-and-reinstall, or backup/restore carries a code such as [HOST_DISK_LOW].

There are 69 diagnostic codes, in ten groups by prefix. Codes are stable: the same problem keeps the same code across versions and Chinese/English UI. You can report the code to support as-is; you do not need to paraphrase the on-screen text. The full list and remedies are on the product documentation site's "Install and operations diagnostic codes" page.

Procedure

1. Look at the prefix first to decide which class of problem it is.
2. Then look at parameters after the code (such as path, have_gib, need_gib); they give the concrete object and numbers.
3. Follow the on-screen remedy or the table below. If it is still unclear, submit the code and the full output together to support.

Examples of common codes:

Note: zenova doctor exits 2 when there is a blocking item, 0 when there is none. Inspection scripts can use that. Warning items do not affect the exit code, but should be recorded in the handover notes.

12.5 Access and call problems after install

The console will not open

Procedure

1. First open the console on the intranet using the Direct access on this host URL printed at the end of install. It is built from the admin ingress's actual port and has no front forward. If it opens, this host is healthy; the problem is the load balancer, reverse proxy, NAT, or network policy. Go to step 4.
2. If Direct access on this host also will not open, inspect service status on the install host:

# Install host · root · read-only (inspect each service's status)
sudo zenova status
systemctl status caddy postgres clickhouse secret-service spicedb

3. Restart a service that is not running and inspect logs:

# Install host · root · write (restart a named service)
sudo zenova restart <service>
sudo zenova logs <component> --since '10 min ago'

4. This host is healthy but external access does not work: check the host firewall, cloud security group, and DNS. When a front TLS or proxy is used, confirm the local port the front device forwards to matches what was filled in at install.

What you will see

• After restart, service status is running again and the console opens.
• While the control plane is unavailable, gateway nodes keep serving with the last configuration. Existing business calls are not affected.

Note: zenova restart accepts only services that start with zenova-. For a few seconds while the control plane switches slot, the internal ingress returning 503 is expected; the ingress proxy holds new requests for up to 5 seconds. A readiness check on the central gateway-node data plane (3080) returning 503 with reason no_tenant_routes means there is not yet a tenant or model routing. It recovers after a tenant is created and models are configured. It is not a failure.

The call returns an error

On failure, all three gateway protocol ingresses (OpenAI, Anthropic, Gemini) return a unified error envelope. The body carries a stable reason. HTTP status is decided by reason and is the same across the three protocols. Response header x-should-retry states whether a retry is meaningful.

Under the same HTTP status, different reason values are handled completely differently. Looking only at the status code goes the wrong way: the same 429 may be a member quota exhausted (adjust the quota) or an upstream provider rate limit (retry later); the same 503 may be model routing not configured (change configuration) or the upstream temporarily unavailable (retry later). Always judge by reason.

What you will see

• When Who handles it is the caller, the caller succeeds after changing the request. When it is the administrator, a retry will not change the result; configuration must be adjusted. When it is upstream, a retry may succeed.
• When a streaming request fails after output has started, the gateway sends an in-stream error frame of the same shape as the chosen protocol's error body, then ends the stream normally. Content and usage already output are billed and shown as usual. reason is still read from the table above.

Note: the table lists only the most common reasons. The full list (HTTP status, error type, and whether retryable for each reason) is on the product documentation site's Error reference page. For a reason not in the table, submit the exact reason text together with x-request-id to support.

Note: when a model-provider test or zenova doctor --target reports a connectivity failure, the failure is one of seven classes: dnsFailed (name resolution), proxyConnectFailed (egress-proxy connect), connectFailed (target TCP connect), tlsFailed (TLS handshake and certificate), timeout, ssrfBlocked (outbound safety policy), invalidEndpointUrl (illegal address format). Find the owner of the matching layer by class; do not start from the first layer every time.

12.6 What to prepare when submitting support

Applies to: after handling per sections 12.1–12.5 the problem is still there, and you need support from the delivering party or the vendor.

Procedure

1. Collect the four materials per section 12.2.
2. Organize the problem information in the table below. Provide what you can; write "none" when you do not have it. Do not leave a blank for support to guess.

3. Redact all materials (see below).
4. Organize the materials in this order and pack them for submission:

What you will see

• When the materials are complete, support can locate by request ID and diagnostic code without going back and forth.

Note: the following must be redacted or must not be submitted:

Note: do not send the initial password or any credential over instant messaging, email, or a ticket. Before a screenshot or screen recording, confirm the frame has no password, full key, or recovery code.

Capability notes

Previous: 11. Wipe-and-reinstall and changing product