8. Required post-install work and day-to-day operations

Back to overview · Full table of contents · Download full PDF

This chapter: a finished install is not a finished handover. The first three sections are three things that must be done on install day — copy key material off the host, take the first platform backup and move it off-host, and clean temporary files. The last four sections are the commands used in day-to-day operations, how to read logs, how to change deployment configuration, and emergency recovery of the root password and the admin-network CIDR. The readers are the implementation engineer and the customer system administrator who takes over operations. Every command in this chapter runs as root on the install host and does not depend on whether the console is available.

8.1 Off-host backup of key material

Applies to: must be done on install day; redo later when the master key is rotated or secret-service material changes.

Prerequisite: install is complete; a controlled storage location isolated from the install host has been agreed with the customer (a different host, different storage, preferably a different site), and a keeper named.

A platform backup (section 8.2) includes /opt/zenova/.env (which holds the master key) but does not include secret-service material /etc/zenova/secret-service/. The secret-service keys/ directory encrypts every credential the platform stores (model-provider keys, object-storage keys, and similar). Without it, even with a platform backup, you cannot restore a usable platform onto a new host.

Procedure

1. Pack the key material and compute the digest.

# Install host · root · write (creates the key-material archive under /root)
umask 077
tar -C / -czf /root/zenova-keys-$(hostname)-$(date +%Y%m%d).tar.gz \
    etc/zenova/secret-service opt/zenova/.env
sha256sum /root/zenova-keys-*.tar.gz

2. Copy the archive to the storage location isolated from the install host, using the customer's controlled method.
3. Recompute SHA-256 at the storage location and compare it character by character with the output of step 1.
4. Fill in the register below and file it with the handover materials (the register itself contains no key content).
5. After the copy succeeds and the digest matches, delete the archive on this host (see section 8.3).

What you will see

• /root contains zenova-keys-<hostname>-<date>.tar.gz, readable only by root (guaranteed by umask 077).
• sha256sum prints one line: a 64-character hex digest and the file name.

Note: High risk. This archive contains the master key and every encryption key of the secret service. Leakage is equivalent to leaking every credential the platform stores; loss means the platform cannot be restored from backup. Handle it as the customer's highest-classification data: do not send it over instant messaging, email, a ticket, or a cloud-drive share link; do not store it in the same place as the platform-backup file (section 8.2); when the keeper changes, leave a trace on the register.

Note: with a custom install root (--install-dir), replace opt/zenova/.env in the command with <install-root>/.env (without the leading /).

8.2 First platform backup and off-host copy

Applies to: immediately after install and Chapter 7 acceptance; also before every upgrade and every important configuration change.

Prerequisite: enough free space on the install host; see the notes in this section.

Procedure

1. Back up immediately, list the backup ID, then export it as a single archive.

# Install host · root · write (creates the local backup and the export archive)
sudo zenova backup
sudo zenova backup list
sudo zenova backup export <backup-id> --output /root/zenova-install/first-install.zbackup

2. Compute the export-file digest, copy it to a controlled location in a different failure domain from the install host (a different host, different storage device, preferably a different site), and confirm the digest matches at the destination.

# Install host · root · read-only
ls -l /root/zenova-install/first-install.zbackup
sha256sum /root/zenova-install/first-install.zbackup

3. Record in the handover materials: backup ID, export file name, digest, off-host location, keeper, date.

What you will see

• zenova backup list lists local backups, newest first. The backup ID looks like zb-20261020T020000Z-1a2b3c4d (zb- + UTC time + 8-character random string).
• Local backups are stored under /opt/zenova/data/backups/<backup-id>/.
• The export file mode is 0600.

Backup scope:

Note: the export file is not compressed, not additionally encrypted, and contains the master key. Store it at the highest classification, separate from the key-material archive in section 8.1. Do not put them on the same host or in the same bucket.

Note: space needed is about "local PG data directory + local object-data directory + 1 GiB". The local backup and the export file each take a copy. Before export, confirm both /opt/zenova and /root/zenova-install have headroom.

Note: local backups sit under /opt/zenova/data, on the same volume as platform data. Leaving only a local backup does not count as a backup — a host or disk failure loses them together, and wipe-and-reinstall (Chapter 11) deletes them as well. Always copy off-host.

Note: restore requires the same version, product, deployment form, and platform. A backup from an older version cannot be restored onto a newer install. After an upgrade, take a new backup and copy it off-host.

8.3 Clean temporary files

Applies to: after sections 8.1 and 8.2; do a final pass when handover acceptance finishes.

Prerequisite: the key-material archive and the platform-backup export file have both been copied off-host, and the digest matches at the destination.

Procedure

1. Handle each class of file per the table.

2. Delete the secrets file and the archives that have been copied off-host.

# Install host · root · delete (secrets file and the copied-off key-material archive and backup export)
rm -f /root/zenova-install/secrets.toml
rm -f /root/zenova-keys-*.tar.gz
rm -f /root/zenova-install/first-install.zbackup
ls -la /root/zenova-install/ /root/

What you will see: ls no longer shows secrets.toml, zenova-keys-*.tar.gz, or first-install.zbackup; the installer, offline package, and answers.toml are still there.

Note: delete only the files listed above. Do not delete /opt/zenova/state/answers.toml (install record, used by upgrade and reconfigure) or any file under /opt/zenova/.env.runtime/. The initial-password file root-password-initial becomes invalid after the root password is changed and does not need to be deleted by hand.

Note: do not delete the key-material archive before the off-host copy and digest check finish. If the copied copy is then found corrupt, pack again per section 8.1.

8.4 Day-to-day operations command cheat sheet

Applies to: daily inspection, incident handling, change windows. All commands run as root on the install host and do not depend on the console.

Procedure

Pick the command by "when to use it".

What you will see: each command prints its result directly in the terminal. zenova doctor exits 0 when there is no blocking item, 2 when there is a blocking item.

Note: zenova start/stop/restart accept only service names that start with zenova-, for example sudo zenova restart zenova-edge. postgres, clickhouse, caddy, secret-service, and spicedb are managed with systemctl.

Note: before restore (zenova restore), take another backup of the current environment. Restore conditions: same version, product, deployment form, and platform; no upgrade in progress; control-plane group services already stopped. After restore and before starting the control-plane group, restart spicedb, secret-service, and rustfs first (omit rustfs when bundled object storage is not used).

Note: never clean or "reinstall" the platform with hand-written rm -rf, docker rm, systemctl disable, or similar. When a wipe is needed, use only the wipe-and-reinstall flow in Chapter 11.

8.5 How to read logs

Applies to: zenova status shows a service abnormal, install or upgrade failed, or you need to give logs to support.

Procedure

1. View logs by component. zenova logs resolves the systemd unit currently corresponding to that component (the control plane locates the slot currently taking traffic) and then calls journalctl.

# Install host · root · read-only
sudo zenova logs control-plane --since '30 min ago'
sudo zenova logs edge --follow

2. For services not in the component list, view by unit name with journalctl.

# Install host · root · read-only
journalctl -u [email protected] --since '30 min ago' --no-pager
journalctl -u spicedb.service --since '30 min ago' --no-pager

3. Process logs for install, resume, and reconfigure are in install.log.

# Install host · root · read-only
tail -n 200 /opt/zenova/state/install.log

4. Logs of a remote gateway node are viewed on the node host.

# Node host · root · read-only
sudo zenova logs node-edge --since '30 min ago'
sudo zenova logs node-caddy --since '30 min ago'

5. When sending to support, collect everything once with the commands below.

# Install host · root · write (creates files only under /root/zenova-install)
sudo zenova status --json > /root/zenova-install/status.json
sudo zenova doctor --all --save /root/zenova-install/doctor
sudo zenova logs control-plane --since '30 min ago' > /root/zenova-install/control-plane.log
sudo cp /opt/zenova/state/install.log /root/zenova-install/

Component-to-unit mapping (Linux; zenova logs has 8 components):

Other service units (view with journalctl -u <unit>): zenova-worker@<slot>, zenova-updater, spicedb, rustfs, and capability components such as qdrant, docling, zenova-runtime-store@<slot>. Which units actually exist follows the capabilities installed on this host.

What you will see

• zenova logs output is the content of journalctl -u <unit>. Without --follow it prints once and exits; with --follow it keeps following until Ctrl+C.
• A wrong component name reports "unknown log component …" and lists every available component.

Note: service logs on Linux are stored by journald. --since is passed through to journalctl --since and accepts formats journalctl accepts, such as '30 min ago' or '2026-10-20 09:00'. There is no directory such as /opt/zenova/<service>/logs/; do not look for logs on that path.

Note: logs may contain request content and internal addresses. Review them against the customer's data-management rules before sending them to support. Do not send .env, key material, or the initial-password file.

8.6 Change deployment configuration

Applies to: access-URL change, object-storage parameter adjustment, operations-notification channel change, resource-capacity adjustment, ingress-port change, and similar post-install configuration changes.

Prerequisite: no upgrade is in progress; the previous change has finished taking effect. Back up first per section 8.2 before the change.

Procedure

1. Prefer the console. A platform administrator opens System maintenance › Deployment configuration, chooses Edit on the target item, fills in the new value, and saves. The page has seven groups:

2. When the console is unavailable or the change must be scripted, use the command line. Ordinary items take the value on the command line:

# Install host · root · write (change the access URL; related services reload or restart)
sudo zenova config set PUBLIC_URL <new-access-url>
echo "exit code: $?"

3. Sensitive items must not be written on the command line. Enter them at the prompt (not echoed), or read them from standard input with --value-stdin:

# Install host · root · write (change the object-storage secret access key; enter at the prompt, not echoed)
sudo zenova config set LP_OBJECT_STORE_ACCESS_KEY_SECRET

4. After the change, check service status with sudo zenova status and make one real business call per section 7.3.

The 18 keys zenova config set can change:

Key names follow the command's actual prompt. A key not in the list reports "<key> is not a changeable deployment-configuration item" and lists every changeable key.

What you will see

• Console: "saved; applying the new value", then the number of services restarted. Sensitive items show only Set or Not set; the previous value is not echoed.
• Command line: first "saved; applying the new value…", then possibly "restarted: …" and "install record updated: …", last line "the change has taken effect".
• An empty value clears that item.

Note: the object-storage mode cannot be changed online after install. The console reports "online change is not supported: back up, reinstall with the new choice, then restore". When a change is needed, schedule a maintenance window and follow backup → reinstall → restore.

Note: ingress ports and bind NICs can be changed only under Ingress listen on the console; they cannot be changed with zenova config set. After changing the business ingress, external callers and remote gateway nodes must use the new port. After changing the admin ingress, the current page URL is invalid; use the new URL the page gives. The installer opens firewall ports only on first install. After a port change, operations adjust the firewall, load balancer, and the public access URL under Gateway → Entry & certificates.

Note: changing Resource capacity restarts the control plane. The console is briefly unavailable; gateway requests are not affected.

Note: do not edit files under /opt/zenova/.env.runtime/ or /opt/zenova/etc/ by hand. They are generated by the installer and configuration compilation. Hand edits are overwritten on the next change, upgrade, or reconfigure, or they prevent services from starting. /opt/zenova/.env contains the master key; do not edit it by hand either.

Note: credentials of existing-service databases and object storage are rotated by the customer administrator. The Deployment configuration page does not provide credential rotation for those dependencies.

8.7 Recover the root password and two-factor authentication; emergency admin-network access

Applies to: root forgot the password, or both the two-factor device and the recovery codes are lost; or the Admin-plane access CIDRs whitelist is wrong, so the console cannot be reached from the customer network.

Procedure

A. View the initial password before the first password change

1. Before the initial password has been changed, you can view it again:

# Install host · root · read-only
sudo zenova root-password show

B. Reset root

1. When root forgot the password, or both two-factor authentication and the recovery codes are lost:

# Install host · root · write (reset the root password, clear two-factor authentication, and invalidate sessions)
sudo zenova root-password set

2. Sign in with the new password printed in the terminal and change the password as the page requires. If Require two-step verification is on at platform scope, the page then asks you to bind two-factor authentication again (authenticator app or passkey); store the new recovery codes. If it is off, bind again by following steps 5–6 in Section 6.1. Password change and binding each have a 10-minute limit; after timeout, sign in again.

C. Emergency admin-network access

1. View the current whitelist:

# Install host · root · read-only
sudo zenova admin-network show

2. Append a CIDR allowed to reach the admin console:

# Install host · root · write (append an allowed admin-plane CIDR)
sudo zenova admin-network allow --cidr <cidr>

3. Or turn the whitelist off:

# Install host · root · write (disable the admin-plane CIDR whitelist)
sudo zenova admin-network disable

4. After access is restored, sign in as root, open Platform accounts & sign-in › Sign-in methods, and reconfigure the correct CIDRs under the Allow only these CIDRs to reach the admin console switch.

What you will see

• root-password show: prints the initial password before the first password change; after the change the initial password is invalid.
• root-password set: prints a newly generated random password; root's existing two-factor authentication and recovery codes are cleared, every root session is invalidated, and the action is written to the audit log.
• admin-network: the change takes effect on every instance in about 10 seconds. Access to the console from this host's loopback is always allowed.

Note: the new password printed by root-password set is like the initial password: do not send it over instant messaging, email, or a ticket, and do not record or screenshot it for anyone off the host. Before running it, confirm the terminal is not being recorded or shared.

Note: with an http:// access URL the sign-in page does not offer passkeys; two-factor authentication can use only an authenticator app.

Note: these two command groups recover only platform root. When a tenant administrator loses two-factor authentication, a platform administrator opens that tenant's detail under Tenants and Reset two-factor authentication on the Admins tab.

Note: before turning on Allow only these CIDRs to reach the admin console, add both the implementing party's and the customer's operations CIDRs. admin-network disable is for emergency only; after recovery, turn it back on and check the CIDRs.

Capability notes

Previous: 7. Installation acceptance · Next: 9. Add a remote gateway node