Upgrade and add components

This page shows how to move a self-hosted Netzilo Server to a newer release, one component at a time, and how to add components that a server installed earlier lacks: the AI Assistant's worker, the clientless-access gateway and the published-applications reverse proxy.

An upgrade replaces containers. Your data stays in Docker volumes and in the files of the compose folder, which an upgrade does not touch. The component you replace stops for the few seconds Docker needs to swap it; tunnels between devices keep running.

How it works

Restart management and zitadel in a quiet period. The others can be updated any time. Update management and dashboard together, management first: there is no version check between them, and a mismatch shows as odd dashboard behaviour.

Two rules keep an upgrade safe:

• The identity provider only moves forward. Once zitadel has started on a newer image, never put the older image back: its database changes cannot be undone, and the old image breaks every sign-in. Roll back other components if needed and leave zitadel where it is.
• Management migrates its database when it starts. If that fails, the log shows auto migrate: and the container restarts in a loop. Roll back the image and restore the database dump together, see Roll back.

Devices do not need to be upgraded with the server. See Install Netzilo to update clients.

Before you start

1. Check that the server is healthy. An upgrade does not fix a broken server; it hides the cause.

   if [ -f /opt/netzilo/run/docker-compose.yml ]; then C=/opt/netzilo/run; else C=/opt/netzilo; fi
   cd "$C" && sudo docker compose ps
   

   CopyCopied!

   Every service must be Up, and postgres, redis and support-worker healthy.

2. Back up. Run the backup from Backup and restore. It must end with BACKUP OK. With an external database, also take a snapshot at your database provider.

3. Record what is running. This list is what you roll back to:

   STAMP=$(date -u +%Y%m%dT%H%MZ)
   for s in management signal dashboard zitadel gateway support-worker caddy coturn redis db; do
     id=$(sudo docker compose images -q "$s" 2>/dev/null)
     [ -n "$id" ] && printf "%-15s %s\n" "$s" "$(sudo docker image inspect --format '{{index .RepoDigests 0}}' "$id")"
   done | sudo tee "/opt/netzilo/pre-upgrade-$STAMP.txt"
   

   CopyCopied!

4. Check disk space with df -h /var/lib/docker. New images are downloaded next to the old ones. With less than a few gigabytes free, remove unused images first (sudo docker image prune -f).

Then follow the section for how the server was installed: the one-line installer, or an AWS or Azure Marketplace image.

Upgrade a server installed with the one-line installer

Servers installed with the one-line installer keep their files in /opt/netzilo. Newer releases of the installer upgrade an existing server in place. Older releases erase it. The download address is the same for both, so check the script every time:

curl -fsSL https://pkg.netzilo.com/download/install-netzilo.sh > /tmp/nz-installer.sh
grep -q -- '--upgrade' /tmp/nz-installer.sh && echo "upgrade mode: available" || echo "upgrade mode: NOT available - do not run it"
grep -q 'install <component>' /tmp/nz-installer.sh && echo "component commands: available" || echo "component commands: NOT available"

If the upgrade mode is not available, upgrade by hand as described in Upgrade one component by hand.

When it is available, these commands work:

<component> is one of zitadel, management, dashboard, signal, support-worker, gateway, reverse-proxy.

curl -fsSL https://pkg.netzilo.com/download/install-netzilo.sh | sudo bash -s -- --upgrade
curl -fsSL https://pkg.netzilo.com/download/install-netzilo.sh | sudo bash -s -- update gateway
curl -fsSL https://pkg.netzilo.com/download/install-netzilo.sh | sudo bash -s -- install reverse-proxy

Every command asks for one confirmation; --yes skips it. Never pass --yes together with --reinstall on a server you want to keep. Set NETZILO_IMAGE_TAG to move to a specific release instead of the latest one. The commands refuse servers in the marketplace layout (/opt/netzilo/run); upgrade those as described in Upgrade an AWS or Azure Marketplace image.

What the full upgrade does

1. Backup. Copies every configuration file, the certificates, CREDENTIALS and the list of running images into /opt/netzilo/backups/upgrade-<time>/, and takes a full database dump, postgres-all.sql.gz. If the dump fails, nothing changes. With an external database it stops unless you set NETZILO_UPGRADE_DB_BACKED_UP=1 to confirm you took a snapshot.
2. Configuration. Adds only what newer releases need and the server lacks, for example the AI Assistant's worker and, on a server with HTTPS, the clientless-access gateway. Existing values are never changed.
3. Images. Pulls the latest Netzilo images and recreates them one at a time, the identity provider first, waiting for each to answer before the next. A service that does not come back stops the upgrade and prints how to roll back.
4. Report. Lists the images before and after, the backup folder and the rollback lines.

The full upgrade never moves Caddy, coturn, PostgreSQL or Redis, and never adds published applications, which need a domain and DNS you choose (install reverse-proxy).

When the upgrade adds the gateway to a Let's Encrypt server, it moves Caddy's certificates into a named volume and restarts Caddy once: sign-in is down for a few seconds. Afterwards, open port 8443/tcp and switch clientless access on, see Clientless-access gateway.

Backup folders

update and install write /opt/netzilo/backups/<command>-<component>-<time>/ before they change anything, with a database dump for zitadel and management, the two components that change the database. These folders roll back one command. They are not a backup of the server and do not replace Backup and restore.

Each folder holds every secret of the server. Once the server works, copy the folder somewhere safe or delete it:

sudo ls -1 /opt/netzilo/backups/
sudo rm -rf /opt/netzilo/backups/<folder>

Upgrade an AWS or Azure Marketplace image

On marketplace images every image is pinned to a digest, including the database, cache, proxy and relay. docker compose pull changes nothing until you change a reference. Update one Netzilo component at a time:

1. Do the steps in Before you start.
2. In /opt/netzilo/run/docker-compose.yml, change the component's image from ghcr.io/netzilo/<image>@sha256:... to the release tag Netzilo names, or to latest.
3. Pull and recreate that one service, then check it, as in Upgrade one component by hand.
4. Pin it again to the digest now running, as in Pin what you checked.

Leave the references of caddy, coturn, postgres and redis as they are.

Upgrade one component by hand

Use this on marketplace images, and on one-line installs whose installer has no upgrade mode. Update in this order, and check each component before the next: zitadel, management, dashboard, gateway, signal, support-worker, then reverse-proxy if you run it.

cd "$C"
S=management
sudo docker compose pull "$S"
sudo docker compose up -d --no-deps "$S"
sudo docker compose logs -f --since 2m "$S"

--no-deps keeps Compose from touching anything else. Never run a bare docker compose pull followed by docker compose up -d: on a one-line install that also moves Caddy, coturn and Redis to whatever their newest version is.

Pin what you checked

Pin a component you checked to the digest now running, so that a later pull cannot move it by accident. Image names: net-management, net-signal, net-dashboard, net-gateway, net-support-worker, net-reverse-proxy, zitadel-build.

cd "$C"
S=management; IMG=ghcr.io/netzilo/net-management
NEW=$(sudo docker image inspect --format '{{index .RepoDigests 0}}' "$(sudo docker compose images -q $S)")
BAK="docker-compose.yml.$(date -u +%Y%m%dT%H%MZ).bak"; sudo cp docker-compose.yml "$BAK"
sudo sed -E "s#^([[:space:]]*image:[[:space:]]*)${IMG//./\\.}[@:].*#\1$NEW#" "$BAK" | sudo tee docker-compose.yml >/dev/null
grep -n "$IMG" docker-compose.yml

On a one-line install, also pin caddy, coturn/coturn and redis the same way before your first upgrade, so that they move only when you decide. Never change the postgres:16 tag to a newer major version: that needs a dump, a new database and a restore.

Roll back

After a one-line installer command. The command printed its rollback lines and the backup folder. When the database is fine (any failed update or install, or an upgrade after which something misbehaves but sign-in works), put the files back. This also puts the old images back, because docker-compose.yml names them:

cd /opt/netzilo
B=/opt/netzilo/backups/<the folder the command printed>
sudo cp -a "$B"/{docker-compose.yml,management.json,zitadel.env,dashboard.env,Caddyfile} .
sudo docker compose up -d --remove-orphans && sudo docker compose restart caddy

After a manual upgrade. Put back the digest you recorded before the upgrade and recreate only that service. The old image is still on the host, so this needs no network:

cd "$C"
S=management; IMG=ghcr.io/netzilo/net-management
OLD=$(grep "^$S " /opt/netzilo/pre-upgrade-<STAMP>.txt | awk '{print $2}')
BAK="docker-compose.yml.$(date -u +%Y%m%dT%H%MZ).bak"; sudo cp docker-compose.yml "$BAK"
sudo sed -E "s#^([[:space:]]*image:[[:space:]]*)${IMG//./\\.}[@:].*#\1$OLD#" "$BAK" | sudo tee docker-compose.yml >/dev/null
sudo docker compose up -d --no-deps "$S"

Management keeps restarting with auto migrate:. The new version changed the database and the old one cannot read it. Put the old image back and restore the database from before the upgrade, never only one of the two. Use the restore in Backup and restore with the backup you took before the upgrade. Every change since then is lost.

The identity provider. If zitadel started cleanly on the new image, leave it there. Restore its database only when it failed to start and sign-in is already down.

When the rollback window has passed, remove old images with sudo docker image prune -f.

Add a component to an older server

Servers installed before a component existed run without it. Never re-run the installer to add one.

Before adding the gateway or the reverse proxy, management and the dashboard must be from a release that has the feature. Upgrade them first.

Troubleshooting

docker compose pull says the image is up to date. On a one-line install, nothing newer is published: that is a correct result. On a marketplace image, the reference is a digest, which never changes: edit it first.

A one-line installer command refuses to run. It found the marketplace layout (/opt/netzilo/run). Upgrade by hand.

manifest unknown for a ghcr.io/netzilo image. The tag in your compose file does not exist. Check it with docker manifest inspect <image>. The images are public and need no login.

After an upgrade, the dashboard behaves oddly. Management and the dashboard are from different releases. Update the one that is behind.

Related Documentation

• Backup and restore - back up before every upgrade
• Server reference - containers, images and files
• Troubleshooting - when a container does not come back