10. Backup, restore, and upgrade

Back to overview · Full table of contents · Download full PDF

This chapter: what a platform backup covers, how to create and copy it off-host, how to restore from a backup, what to prepare before an upgrade, how to upgrade from the console or the command line, and what to do when a problem appears after upgrade. Restore changes currently running data. Read every Note in the matching section before you run it. The readers are the customer system administrator and the implementation engineer.

10.1 Backup scope

zenova backup and the console Backup & Restore page use the same backup. It backs up only what the installer deployed and manages.

Backups are stored under /opt/zenova/data/backups/<backup-id>/. Space estimate:

Space needed ≈ local PG data directory + local object-data directory + 1 GiB

Procedure

1. Estimate space for one backup.

# Install host · root · read-only (estimate space for a backup)
du -sh /opt/zenova/data/*
df -h /opt/zenova/data

2. From the du output, take the sizes of the local PostgreSQL and local object-storage data directories (directory names follow the actual output), add 1 GiB, and compare with free space from df.

Note: for the two classes that are not included, agree in writing with the customer before handover who is responsible and how often they back up, and agree that restore will bring them to the same point in time as the platform backup. Restoring only the platform backup while existing PG or S3 is not restored in sync produces inconsistent data.

Note: secret-service material is not in the platform backup. When restoring onto another host, without the off-host backup of the original host's /etc/zenova/secret-service/, every credential the platform stored cannot be decrypted.

10.2 Create, export, and copy off-host

Procedure

1. Create a backup and record the backup ID printed on screen (looks like zb-20261020T020000Z-1a2b3c4d).

# Install host · root · write (create a backup)
sudo zenova backup

2. List local backups and confirm that ID.

# Install host · root · read-only
sudo zenova backup list

3. Export as a .zbackup archive.

# Install host · root · write (export the archive)
sudo zenova backup export <backup-id> --output /root/zenova-install/<backup-id>.zbackup
sha256sum /root/zenova-install/<backup-id>.zbackup

4. Copy the archive and digest to a controlled location in a different failure domain from the install host (a different host, different site, or offline media). Register storage location, keeper, digest, and date. After the copy succeeds, delete the on-host export file.
5. Delete local backups that are no longer needed, by ID.

# Install host · root · delete (delete one local backup; it cannot be recovered)
sudo zenova backup delete <backup-id>

Command line versus console (console is the Backup & Restore tab under Maintenance on the platform admin console):

What you will see

• After a command-line backup finishes, it prints the backup ID.
• After Create backup on the console, it reports "backup has started; it will appear in the list when finished." The job goes through Prepare, Install state, Platform database, Analytics database, Object storage, Verify.
• Backup list shows Created at, Version, Initiator, Size, Included content. Components not included show as (not included) with the reason.
• The export file mode is 0600.

Note: the export file is not compressed, not additionally encrypted, and contains the master key. Anyone who has it can unpack the credentials the platform stored. Keep it at the highest classification. Do not send it over instant messaging, email, or a ticket.

Note: a backup that exists only on the install host does not count as a backup. When the host disk fails, or wipe-and-reinstall runs (which deletes every local backup), the local backups are gone with it.

Note: backups are triggered by hand from the page or the command line. The platform does not provide scheduled automatic backup or expiry cleanup. Periodic backup, off-site copy, and cleanup are arranged by the customer in their operations process. When a backup, restore, or upgrade is already in progress, a new backup is blocked.

10.3 Restore

Applies to: data was deleted or corrupted by mistake and the platform must return to the state of a given backup; or verifying that a backup works on a rehearsal host.

Prerequisite: all of the following must hold, or restore is refused.

Note: restore overwrites current data with the backup. Every write after the backup (configuration changes, call records, usage, audit, and similar) is permanently lost and cannot be undone. Before restore, take another backup of the current environment, export it, and copy it off-host, as a way back if restore fails or the wrong backup is chosen.

Note: existing PostgreSQL and existing S3 are not in restore scope. When they are in use, the corresponding administrators restore their data to the same point in time as the platform backup using their backup mechanisms; otherwise platform data is inconsistent.

Procedure

1. First take another backup of the current environment, export it, and copy it off-host (see section 10.2).

# Install host · root · write (protective backup before restore)
sudo zenova backup
sudo zenova backup list
sudo zenova backup export <current-backup-id> --output /root/zenova-install/<current-backup-id>.zbackup

2. Stop the control-plane group services. Use zenova status to see service names; skip components that are not enabled.

# Install host · root · write (stop the control-plane group; console and admin APIs unavailable)
sudo zenova status
sudo zenova stop zenova-worker
sudo zenova stop zenova-runtime-store
sudo zenova stop zenova-control-plane

3. Run restore. The argument may be a backup ID or a .zbackup archive (import first, then restore). When prompted, type the confirmation word RESTORE.

# Install host · root · write (overwrite current data with the backup; cannot be undone)
sudo zenova restore <backup-id-or-.zbackup-file>

4. After restore finishes and before starting the control-plane group, restart the secret service, authorization service, and bundled object storage so they load the restored data. Omit rustfs when bundled object storage is not used.

# Install host · root · write (restart dependency services; load restored data)
systemctl restart spicedb secret-service rustfs
systemctl status spicedb secret-service rustfs --no-pager

5. Start the control-plane group.

# Install host · root · write (start the control-plane group)
sudo zenova start zenova-control-plane
sudo zenova start zenova-runtime-store
sudo zenova start zenova-worker
sudo zenova status

6. Check: sign in to the console; confirm gateway nodes are online; run acceptance once under Deployment configuration › Latest acceptance; complete one real call per section 7.3, and confirm that model, routing, permission, audit, and alert configuration match the backup.
7. Record the point in time that can be restored to, restore duration, missing content, and external-dependency data that must be rebuilt.

What you will see

• zenova restore first requires RESTORE; any other input does not run. Completion output follows the actual output.
• When restore is started from the console (backup row Restore → Restore from backup → type RESTORE → Start restore), the page shows Restoring, then Stop console services, restore of each component, and Start console services, and finishes with "Restore completed."
• Restore is refused while the control plane is still running, or while an upgrade or backup job is in progress.

Note: during restore the old data is kept until the new data is fully in place. Once restore has finished, overwritten data can be recovered only from the protective backup in step 1.

Note: when restoring onto another host, first complete an install of the same version, product, and form, put the off-host backup of the original host's /etc/zenova/secret-service/ back in place, then run the steps above. Before overwriting a production environment, complete one restore rehearsal on an isolated rehearsal host.

10.4 Prepare before upgrade

Procedure

1. Read the target version's release notes. Confirm what changed, known issues, and whether a database change is involved.
2. Confirm you can upgrade directly from the current version to the target version. When the current version is earlier than the target version's "lowest version that can upgrade directly", upgrade to an intermediate version first, then to the target version.
3. Confirm every capability currently enabled. The target-version offline package must include every currently enabled capability. Missing any enabled component fails the pre-upgrade check.

# Install host · root · read-only (view current version and services)
sudo zenova status

You can also check version and each capability's state under Current version on console System maintenance › System Updates.

4. For the current product, platform, and every enabled capability, download the target-version installer and offline package from the download site, check SHA-256 per Chapter 2, and copy them onto removable media to bring on site. The platform does not fetch upgrade packages online. Upgrade packages are always uploaded on site on the System Updates page or run from the command line.
5. When there are remote gateway nodes, confirm the target-version offline package contains node packages for those nodes' OS. When it does, the control-plane upgrade upgrades online nodes one by one before the console is switched (see section 10.5). When it does not (for example the node and control-plane platforms differ), the pre-upgrade Gateway nodes check fails; first upgrade those nodes by hand per Chapter 9 Upgrade a node. Nodes already at the target version are skipped. A node is drained during upgrade. Drain does not change DNS or the load balancer: when the ingress pool has only one node, the public ingress is unavailable while it upgrades.
6. Complete one backup, export it, copy it to a different failure domain, and record the backup ID (see section 10.2).
7. Agree a maintenance window with the customer and notify console users.

Note: the pre-upgrade check decides whether this run is a fast switch or a downtime upgrade, and states the impact on the System Updates page. On a fast switch the console switch takes a few seconds; operations are slightly slow or briefly unavailable, and gateway calls are not affected. When a downtime upgrade is required (a large-table change exceeds the switch-window budget, a pending analytics-database downtime migration, and similar), the upgrader stops the old services to finish the migration. During that time both the console and gateway calls on the central node are unavailable. Remote gateway nodes that were already upgraded to the target version keep serving with the configuration already issued (a downtime migration does not start if a node is offline or not yet at the target version). The "about 600 seconds" shown in preflight is a fixed hint; actual duration follows data volume. Downtime upgrade supports only single-host deployments. Schedule the maintenance window against the impact the check gives.

Note: the platform does not provide a rollback command. After an upgrade finishes, returning to the old version can only restore the pre-upgrade backup (see section 10.6). That backup cannot be skipped.

Note: when the current version has Operations observability enabled, before upgrading to a version that no longer provides that capability you must disable Operations observability on the current version's System maintenance › System Updates. Otherwise the pre-upgrade check refuses the upgrade: Deployment configuration fails, and expanding View technical details (command line zenova upgrade prints a "Details:" line) shows "capability 'Operations observability' has been retired in the new version; disable it on the current version's System Updates page first, then upgrade". Capability packages complete also reports that the upgrade package is missing an enabled capability. They are the same cause; disable it as prompted. You do not need another upgrade package. After the upgrade, you may delete the prometheus, alertmanager, grafana, loki, and tempo directories under <install-directory>/data/ by hand to reclaim disk.

10.5 Upgrade

Prerequisite: every preparation in section 10.4 is done, and no other upgrade is in progress.

Upgrade from the console

Procedure

1. As root, open System maintenance › System Updates.
2. Upload the .zpkg prepared in section 10.4. If the upload is interrupted, you can Resume upload.
3. After upload finishes, the system runs the pre-upgrade check automatically. Check each result. When an item fails, follow the prompt, then click Recheck.

Check names and count on the console follow the actual UI and may also include infrastructure version, database-change duration, version order, and similar. When PostgreSQL or ClickHouse deployed by the installer does not meet the version requirement, run sudo zenova infra upgrade <infrastructure-package> in a maintenance window, then recheck. When using an existing database, the database administrator upgrades it.

4. After every item passes, click Upgrade and confirm with the root password in the dialog.
5. Watch progress until the upgrade finishes. Do not shut the server down.
6. After the upgrade: under Upgrade history, check version, result, duration, and operator; click Run acceptance under Deployment configuration › Latest acceptance; complete one real call per section 7.3.

What you will see

• The upgrade goes through nine stages. Stages that do not apply are skipped:

• When there are remote gateway nodes, the Gateway nodes stage upgrades online nodes one by one before the console is switched: each is drained, the package is swapped, it passes ingress self-check, then it returns to the ingress pool. Progress looks like "gateway node gw-sh swapping package". If any node upgrade fails, the console is not switched. The result is "not successful; restored to the previous version", listing the failed nodes and reasons. Failed nodes automatically switch back to the old version and keep serving; nodes already upgraded stay on the new version.
• The check result states this run's impact: on a fast switch the console switch takes a few seconds and gateway calls are not affected; on a downtime upgrade the console and gateway calls on the central node are unavailable during the upgrade, and remote gateway nodes keep serving (see section 10.4).
• The result is one of Completed, Completed, some items need attention, Not successful; restored to the previous version, or Needs manual handling (follow the actual UI).

Upgrade from the command line

Procedure

1. Unpack the target-version installer in the install-host staging directory and run the upgrade with the new-version installer.

# Install host · root · write (upgrade to the target version)
cd /root/zenova-install
tar xzf <target-version-installer>.tar.gz
sudo ./zenova upgrade <target-version-offline-package>.zpkg
echo $?

2. Judge the result by exit code; after it finishes, check the same way as console step 6.

What you will see

• The command completes upload, pre-upgrade check, and switch in order, and keeps showing progress. It is the same flow and the same upgrade record as a console upgrade.
• Exit codes:

Note: you must run zenova upgrade with the new-version installer. The already-installed old-version program may not recognize a new-version upgrade package.

Note: only one upgrade can run at a time. Console-started and command-line-started upgrades can see each other. Do not start another from one place while one is running in the other.

10.6 Problems after upgrade

Applies to: the upgrade failed, or after it finished the new version has a problem that cannot be accepted.

The platform does not provide a rollback command. Handling depends on the upgrade result:

Note: restoring the pre-upgrade backup loses every write after that backup. Restore requires this host's version to match the backup's version. First install this host onto the original version per Chapter 11, then restore per section 10.3. That is why section 10.4 requires a backup copied off-host before upgrade.

Procedure

1. Inspect the current version and the last upgrade's result.

# Install host · root · read-only
sudo zenova status
sudo zenova upgrade status

2. Choose handling from the table above. After it is handled, complete one real call per section 7.3 and run acceptance once under Deployment configuration › Latest acceptance.

Note: when there are remote gateway nodes, after handling, check each node's status and version in the node list; follow the actual output.

Capability notes

Previous: 9. Add a remote gateway node · Next: 11. Wipe-and-reinstall and changing product