Backup and restore

A self-hosted Netzilo Server does not back itself up. This page gives you a backup that either captures everything a restore needs or fails loudly, a daily schedule, a way to prove that a backup restores, and the restore procedures for the same server, a new server and an external database.

What a restore needs

The folders that the one-line installer's update, install and --upgrade commands leave in /opt/netzilo/backups/ roll back one command. They are not a backup of the server: they have no management data, no checksums and no off-server copy.

Step 1: Install the backup script

The script below works on every installation: one-line install or marketplace image, bundled or external database. It writes one folder per run under /var/backups/netzilo, checks every dump it takes, writes checksums and a MANIFEST, and prints BACKUP OK only when everything a restore needs was captured. Anything missing makes it exit with an error and rename the folder to <time>.FAILED.

Install it once and run it by hand:

sudo tee /usr/local/sbin/netzilo-backup.sh >/dev/null <<'EOF'
#!/usr/bin/env bash
# Netzilo Server backup. Fails closed: non-zero exit, a .FAILED directory, and no
# "complete=true" line unless every required artifact was captured and verified.
set -Eeuo pipefail
umask 077

ROOT=${NETZILO_BACKUP_ROOT:-/var/backups/netzilo}
STATE=/opt/netzilo/netzilo-state.env
if [ -f /opt/netzilo/run/docker-compose.yml ]; then C=/opt/netzilo/run; else C=/opt/netzilo; fi
TS=$(date -u +%Y%m%dT%H%M%SZ)
B="$ROOT/$TS"
install -d -m 700 "$ROOT" "$B"

fail() { echo "BACKUP FAILED: $*" >&2; exit 1; }
trap 'rc=$?; if [ "$rc" -ne 0 ]; then echo "exit $rc" > "$B/FAILED"; mv "$B" "$B.FAILED" 2>/dev/null || true; echo "BACKUP FAILED: $B.FAILED" >&2; fi' EXIT
st() { sed -n "s/^$1=//p" "$STATE" 2>/dev/null | head -n 1 || true; }

cd "$C"

# 1. Required configuration must exist and be non-empty.
REQUIRED="docker-compose.yml Caddyfile management.json zitadel.env dashboard.env turnserver.conf"
if grep -q 'auto_https off' Caddyfile; then REQUIRED="$REQUIRED certs/fullchain.pem certs/privkey.pem"; fi
for f in $REQUIRED; do [ -s "$C/$f" ] || fail "required file missing or empty: $C/$f"; done
MISSING_OPTIONAL=""
for f in "$C/.env" "$C/machinekey" /opt/netzilo/CREDENTIALS "$STATE" /opt/netzilo/metering.env /opt/netzilo/.provisioned; do
  [ -e "$f" ] || MISSING_OPTIONAL="$MISSING_OPTIONAL $f"
done

# 2. Database client: bundled container or external server.
if grep -q 'container_name: postgres' docker-compose.yml; then
  MODE=container
  PGU=$(sed -n 's/^[[:space:]]*- POSTGRES_USER=//p' docker-compose.yml | head -n 1); PGU=${PGU:-postgres}
  pgc() { docker exec -i -e PGUSER="$PGU" postgres "$@"; }
else
  MODE=external
  [ -s "$STATE" ] || fail "external database mode needs $STATE"
  PGIMG=$(st IMAGE_POSTGRES); PGIMG=${PGIMG:-postgres:16}
  export PGPASSWORD; PGPASSWORD=$(st DB_ADMIN_PASSWORD)
  [ -n "$PGPASSWORD" ] || fail "DB_ADMIN_PASSWORD not found in $STATE"
  pgc() { docker run --rm -i -e PGPASSWORD -e PGHOST="$(st DB_HOST)" -e PGPORT="$(st DB_PORT)" \
            -e PGUSER="$(st DB_ADMIN_USER)" -e PGSSLMODE="$(st DB_SSLMODE)" "$PGIMG" "$@"; }
fi
DB_VERSION=$(pgc psql -X -d postgres -tAc 'SHOW server_version')
PGDUMP_VERSION=$(pgc pg_dump --version)

# 3. Database dumps, each proven readable.
for db in netzilo zitadel; do
  pgc pg_dump -Fc -d "$db" > "$B/$db.dump"
  [ -s "$B/$db.dump" ] || fail "$db.dump is empty"
  pgc pg_restore --list < "$B/$db.dump" > /dev/null || fail "$db.dump is not a readable archive"
done
ZROLE=$(sed -n 's/^ZITADEL_DATABASE_POSTGRES_USER_USERNAME=//p' zitadel.env | head -n 1)
if [ "$MODE" = container ]; then
  pgc pg_dumpall --globals-only > "$B/globals.sql"
  grep -q "CREATE ROLE $ZROLE;" "$B/globals.sql" || fail "globals.sql does not contain role $ZROLE"
elif ! pgc pg_dumpall --globals-only --no-role-passwords > "$B/globals.sql" 2>/dev/null; then
  rm -f "$B/globals.sql"; MISSING_OPTIONAL="$MISSING_OPTIONAL globals.sql(external)"
fi

# 4. Management data volume, resolved from the container's mount.
MV=$(docker inspect -f '{{range .Mounts}}{{if eq .Destination "/var/lib/netzilo"}}{{.Name}}{{end}}{{end}}' management)
[ -n "$MV" ] || fail "cannot find the management data volume"
MP=$(docker volume inspect -f '{{.Mountpoint}}' "$MV")
tar -C "$MP" -czf "$B/management-data.tgz" .

# 5. Configuration and state: all of /opt/netzilo, then prove the required members are in it.
tar -C / -czf "$B/config.tgz" opt/netzilo
for f in $REQUIRED; do
  tar -tzf "$B/config.tgz" "${C#/}/$f" > /dev/null 2>&1 || fail "config.tgz lacks ${C#/}/$f"
done

# 6. What was running: service, image reference, registry digest, local image id.
SERVICES=$(docker compose config --services)
for s in $SERVICES; do
  cid=$(docker compose ps -a -q "$s")
  if [ -z "$cid" ]; then echo "$s not-created - -"; continue; fi
  iid=$(docker inspect -f '{{.Image}}' "$cid")
  echo "$s $(docker inspect -f '{{.Config.Image}}' "$cid") $(docker image inspect -f '{{if .RepoDigests}}{{index .RepoDigests 0}}{{else}}local-only{{end}}' "$iid") $iid"
done > "$B/images.txt"

# 7. Checksums, manifest, and the completeness flag written last.
cd "$B"
FILES="netzilo.dump zitadel.dump management-data.tgz config.tgz images.txt"
if [ -f globals.sql ]; then FILES="$FILES globals.sql"; fi
sha256sum $FILES > SHA256SUMS
{
  echo "format=netzilo-backup-1"
  echo "created_utc=$TS"
  echo "host=$(hostname)"
  echo "compose_dir=$C"
  echo "domain=$(st NETBIRD_DOMAIN)"
  if grep -q 'auto_https off' "$C/Caddyfile"; then echo "tls_mode=provided-or-selfsigned"; else echo "tls_mode=letsencrypt"; fi
  echo "db_mode=$MODE"
  echo "db_engine=postgresql"
  echo "db_server_version=$DB_VERSION"
  echo "pg_dump=$PGDUMP_VERSION"
  echo "management_volume=$MV"
  echo "optional_missing=${MISSING_OPTIONAL:- none}"
} > MANIFEST
sha256sum -c --quiet SHA256SUMS
echo "complete=true" >> MANIFEST
chmod 600 "$B"/*
echo "BACKUP OK: $B"
EOF
sudo chmod 700 /usr/local/sbin/netzilo-backup.sh
sudo /usr/local/sbin/netzilo-backup.sh      # first run by hand; it must end with "BACKUP OK"

A good backup folder holds netzilo.dump, zitadel.dump, globals.sql (bundled database), management-data.tgz, config.tgz, images.txt, SHA256SUMS and a MANIFEST whose last line is complete=true. MANIFEST and images.txt hold no secrets. Everything else does: the folder holds every credential of the server. The script creates it readable by root only; keep it that way.

The two database dumps are taken a few seconds apart. For a strictly matching pair, for example before a major change, stop management and zitadel while the backup runs and start them afterwards. Sign-in and the dashboard are down meanwhile; tunnels are not affected.

External database. The same script dumps both databases over the network with the details in netzilo-state.env. Also take a snapshot at your database provider: that is the primary copy. Managed databases usually refuse to export role passwords, so globals.sql is optional in this mode.

Step 2: Schedule it and copy it off the server

Run the backup daily at 02:30 and keep 14 days. Old backups are deleted only after a successful run, so a series of failures never deletes the last good backup:

sudo tee /etc/cron.d/netzilo-backup >/dev/null <<'EOF'
30 2 * * * root /usr/local/sbin/netzilo-backup.sh >>/var/log/netzilo-backup.log 2>&1 && find /var/backups/netzilo -mindepth 1 -maxdepth 1 -mtime +14 -exec rm -rf {} +
EOF

A failing cron job is silent. Have your monitoring alert when the last line of /var/log/netzilo-backup.log is not BACKUP OK.

Copy every complete backup folder off the server, to storage that is encrypted and restricted like the server itself, and check the copy there with sha256sum -c SHA256SUMS. A backup on the server's own disk does not survive the loss of the server.

Check the latest backup before a risky change

Run this right before an upgrade, a restore, a re-installation or a decommission. It passes only for a complete backup with valid checksums, less than an hour old:

L=$(sudo find /var/backups/netzilo -mindepth 1 -maxdepth 1 -type d ! -name '*.FAILED' | sort | tail -n 1)
if [ -n "$L" ] && [ "$(sudo tail -n 1 "$L/MANIFEST")" = complete=true ] \
   && sudo sh -c 'cd "$1" && sha256sum -c --quiet SHA256SUMS' _ "$L" \
   && [ -n "$(sudo find "$L" -maxdepth 0 -mmin -60)" ]; then
  echo "BACKUP CHECK PASSED: $L"
else
  echo "BACKUP CHECK FAILED: run /usr/local/sbin/netzilo-backup.sh and fix what it reports"
fi

If it fails, do not start the change.

Prove that a backup restores

There are three levels of proof. Only the last shows that you can recover the server.

1. Readable. The script checks every dump and every checksum on each run. That proves the files are intact, not that they restore.

2. The data restores. Restore both dumps into a throwaway database with no network, which is deleted afterwards. Replace <TS> with the backup's folder name in /var/backups/netzilo:

   sudo B=/var/backups/netzilo/<TS> bash -s <<'EOF'
   set -Eeuo pipefail
   cd "$B"
   [ "$(tail -n 1 MANIFEST)" = complete=true ] || { echo "REFUSING: not a complete backup"; exit 1; }
   sha256sum -c --quiet SHA256SUMS
   IMG=$(awk '$1=="db"{print $2}' images.txt); IMG=${IMG:-postgres:16}
   N=netzilo-restore-test
   docker rm -f "$N" >/dev/null 2>&1 || true
   trap 'docker rm -f "$N" >/dev/null 2>&1 || true' EXIT
   docker run -d --name "$N" --network none -e POSTGRES_HOST_AUTH_METHOD=trust "$IMG" >/dev/null
   for i in $(seq 1 60); do docker logs "$N" 2>&1 | grep -q 'init process complete' && break; sleep 2; done
   for i in $(seq 1 30); do docker exec "$N" pg_isready -U postgres -q && break; sleep 2; done
   docker exec "$N" pg_isready -U postgres -q
   if [ -f globals.sql ]; then
     grep -vx 'CREATE ROLE postgres;' globals.sql | docker exec -i "$N" psql -X -q -v ON_ERROR_STOP=1 -U postgres -d postgres
   fi
   docker exec -i "$N" pg_restore -U postgres --create --exit-on-error -d postgres < netzilo.dump
   docker exec -i "$N" pg_restore -U postgres --create --exit-on-error -d postgres < zitadel.dump
   for t in accounts users peers; do
     echo "$t: $(docker exec "$N" psql -X -U postgres -d netzilo -tAc "SELECT count(*) FROM $t")"
   done
   echo "DATA RESTORE TEST PASSED"
   EOF
   

   CopyCopied!

   Compare the counts with the live server, for example sudo docker exec postgres psql -U postgres -d netzilo -tAc "SELECT count(*) FROM peers". They should match as of the time of the backup.

3. The server restores (rehearsal). On a throwaway host of the same OS and size, with no public DNS name and a firewall that admits only your address on 22 and 443, restore the backup as in Restore on a new server. Before you start the stack there:

   • in its docker-compose.yml, point every extra_hosts entry for the domain at 172.20.0.1, so the copy talks to itself and never to production;
   • create a short-lived self-signed certificate for the domain, switch the copy to your own certificate and trust it on that host, because Let's Encrypt cannot issue for a domain that points elsewhere;
   • on a marketplace image, run sudo systemctl disable --now netzilo-metering.timer so the copy never reports usage.

   Point your own machine's hosts file at the rehearsal host and sign in. The rehearsal passes when you can sign in as an admin, the zitadel log has no unable to decrypt key, the device, user, group and policy counts match, and a setup key from before the backup is listed in the dashboard. Then destroy the host and its disks, and remove the hosts file entry: the host held every secret of the production server.

Record the date, the backup used, the result and how long the restore took.

Restore on the same server

Use this after data corruption or a failed upgrade. It replaces both databases with the backup's and keeps the current ones under a prerestore_ name, so a failed restore can be undone. Every change since the backup is lost.

It needs free disk space for a second copy of both databases (df -h /var/lib/docker). After a failed upgrade, first put the old image references back in docker-compose.yml (see Upgrade and add components): the restore starts the stack on whatever images the compose file names. Replace <TS> with the backup's folder name in /var/backups/netzilo.

sudo B=/var/backups/netzilo/<TS> bash -s <<'EOF'
set -Eeuo pipefail
if [ -f /opt/netzilo/run/docker-compose.yml ]; then C=/opt/netzilo/run; else C=/opt/netzilo; fi
cd "$B"
[ "$(tail -n 1 MANIFEST)" = complete=true ] || { echo "REFUSING: $B is not a complete backup"; exit 1; }
sha256sum -c --quiet SHA256SUMS || { echo "REFUSING: checksum mismatch in $B"; exit 1; }
grep -qx 'db_mode=container' MANIFEST || { echo "REFUSING: external database; restore it at your database provider"; exit 1; }
cd "$C"
PGU=$(sed -n 's/^[[:space:]]*- POSTGRES_USER=//p' docker-compose.yml | head -n 1); PGU=${PGU:-postgres}
OLD=prerestore_$(date -u +%Y%m%d_%H%M%S)   # digits only: Postgres lowercases unquoted names
trap 'rc=$?; [ "$rc" -eq 0 ] || echo "RESTORE FAILED (exit $rc). Stack left STOPPED. Do not start it; follow the recovery steps. Suffix: $OLD" >&2' EXIT
sql() { docker exec -i postgres psql -X -q -v ON_ERROR_STOP=1 -U "$PGU" -d postgres "$@"; }

docker compose stop $(docker compose config --services | grep -vxE 'db|redis')
docker compose up -d db redis
for i in $(seq 1 60); do docker exec postgres pg_isready -U "$PGU" -q && break; sleep 2; done
docker exec postgres pg_isready -U "$PGU" -q

sql -c "ALTER DATABASE netzilo RENAME TO netzilo_$OLD"
sql -c "ALTER DATABASE zitadel RENAME TO zitadel_$OLD"
docker exec -i postgres pg_restore -U "$PGU" --create --exit-on-error -d postgres < "$B/netzilo.dump"
docker exec -i postgres pg_restore -U "$PGU" --create --exit-on-error -d postgres < "$B/zitadel.dump"
docker compose exec -T redis redis-cli FLUSHALL        # the cache describes the replaced data
echo "RESTORED. Previous databases kept as netzilo_$OLD and zitadel_$OLD"
docker compose up -d
EOF

Then check the server (see Install on your own server) and sign in. When the restored state is confirmed, drop the kept copies to free the space:

sudo docker exec postgres psql -U postgres -c 'DROP DATABASE netzilo_<suffix>' -c 'DROP DATABASE zitadel_<suffix>'

If the script reports RESTORE FAILED. The stack is stopped and the original databases are intact under the _<suffix> names it printed. Put back the ones that were renamed, then start the stack:

if [ -f /opt/netzilo/run/docker-compose.yml ]; then C=/opt/netzilo/run; else C=/opt/netzilo; fi
cd "$C"
sudo docker exec postgres psql -X -v ON_ERROR_STOP=1 -U postgres -d postgres \
  -c 'DROP DATABASE IF EXISTS netzilo' -c 'ALTER DATABASE netzilo_<suffix> RENAME TO netzilo'
sudo docker exec postgres psql -X -v ON_ERROR_STOP=1 -U postgres -d postgres \
  -c 'DROP DATABASE IF EXISTS zitadel' -c 'ALTER DATABASE zitadel_<suffix> RENAME TO zitadel'
sudo docker compose up -d

Run only the lines for the databases that were renamed. List them with sudo docker exec postgres psql -U postgres -lqt | cut -d'|' -f1.

A dump from a one-line installer command. The postgres-all.sql.gz in /opt/netzilo/backups/ is plain SQL, not the format above. The command that took it printed how to roll back; see Upgrade and add components.

Restore on a new server

Use this when the server is lost. Two things cannot change:

• The domain. It is stored in the identity provider's database and in the configuration files. Point the domain's A record at the new server.
• The master key and the data encryption key. They are restored from config.tgz.

1. Build the new host with the same OS and the same or a larger size, install Docker (docker-ce and docker-compose-plugin), and open the firewall ports listed in Server reference. If the new host is a fresh marketplace instance, its first boot already installed an empty server: remove it with cd /opt/netzilo/run && sudo docker compose down --volumes. It holds no data yet.

2. Copy the backup folder to the host and check it. Stop if either check fails:

   cd <backup folder> && [ "$(tail -n 1 MANIFEST)" = complete=true ] && sha256sum -c SHA256SUMS
   

   CopyCopied!

3. Restore the configuration with sudo tar -C / -xzf config.tgz. This restores /opt/netzilo with all configuration and secrets. Then set C to the compose folder (/opt/netzilo, or /opt/netzilo/run on a marketplace image).

4. Adjust $C/docker-compose.yml for the new host:

   • Change every extra_hosts entry for the domain from the old server's address to the new server's public address (or to 172.20.0.1 when that address is private).
   • Pin every image to the version recorded in the backup, so the data is opened by the same versions that wrote it. Upgrade afterwards.

   cd "$C"
   sudo cp docker-compose.yml "docker-compose.yml.$(date -u +%Y%m%dT%H%M%SZ).bak"
   while read -r s ref dig iid; do
     case "$dig" in *@sha256:*) ;; *) continue ;; esac
     sudo awk -v r="$ref" -v d="$dig" '$1=="image:" && $2==r { $0 = substr($0, 1, index($0, r) - 1) d } { print }' \
       docker-compose.yml | sudo tee docker-compose.yml.new >/dev/null
     sudo mv docker-compose.yml.new docker-compose.yml
   done < <backup>/images.txt
   grep -n 'image:' docker-compose.yml
   

   CopyCopied!

   On a server without internet access, docker load the same images first.

5. Restore the data:

   sudo B=<backup> bash -s <<'EOF'
   set -Eeuo pipefail
   if [ -f /opt/netzilo/run/docker-compose.yml ]; then C=/opt/netzilo/run; else C=/opt/netzilo; fi
   cd "$C"
   trap 'rc=$?; [ "$rc" -eq 0 ] || echo "RESTORE FAILED (exit $rc). Stack left STOPPED. Fix the cause, remove the volumes of this new install and run this step again." >&2' EXIT
   PGU=$(sed -n 's/^[[:space:]]*- POSTGRES_USER=//p' docker-compose.yml | head -n 1); PGU=${PGU:-postgres}
   sql() { docker exec -i postgres psql -X -q -v ON_ERROR_STOP=1 -U "$PGU" "$@"; }
   
   docker compose up -d db
   for i in $(seq 1 60); do docker exec postgres pg_isready -U "$PGU" -q && break; sleep 2; done
   docker exec postgres pg_isready -U "$PGU" -q
   # A new cluster holds only an empty "netzilo" database; refuse anything else.
   [ "$(sql -d netzilo -tAc "SELECT count(*) FROM information_schema.tables WHERE table_schema='public'")" = 0 ] \
     || { echo "REFUSING: the netzilo database on this host is not empty"; exit 1; }
   [ -z "$(sql -d postgres -tAc "SELECT 1 FROM pg_database WHERE datname='zitadel'")" ] \
     || { echo "REFUSING: a zitadel database already exists on this host"; exit 1; }
   grep -vx "CREATE ROLE $PGU;" "$B/globals.sql" | sql -d postgres
   sql -d postgres -c 'DROP DATABASE netzilo'
   docker exec -i postgres pg_restore -U "$PGU" --create --exit-on-error -d postgres < "$B/netzilo.dump"
   docker exec -i postgres pg_restore -U "$PGU" --create --exit-on-error -d postgres < "$B/zitadel.dump"
   
   docker compose up --no-start --no-deps management
   MV=$(docker inspect -f '{{range .Mounts}}{{if eq .Destination "/var/lib/netzilo"}}{{.Name}}{{end}}{{end}}' management)
   [ -n "$MV" ] || { echo "cannot find the management data volume"; exit 1; }
   tar -C "$(docker volume inspect -f '{{.Mountpoint}}' "$MV")" -xzf "$B/management-data.tgz"
   echo "DATA RESTORED"
   docker compose up -d
   EOF
   

   CopyCopied!

6. Check the server and sign in. Let's Encrypt issues a certificate on the first HTTPS request once DNS points at the new host.

Devices reconnect by themselves: the domain and the server's identity are the same, and their keys live on the devices. Never run the installer to "recreate" a server: it creates a new, empty one.

Restore an external database

Prefer your database provider's point-in-time restore of the netzilo and zitadel databases: it restores roles and permissions with them. To restore from the dumps instead, into an empty database server:

1. Create the identity provider's login role first, with the name and password from zitadel.env (ZITADEL_DATABASE_POSTGRES_USER_USERNAME and ZITADEL_DATABASE_POSTGRES_USER_PASSWORD).
2. Run pg_restore --create --exit-on-error -d postgres for each dump, with the connection details from netzilo-state.env. Check that each one succeeds before the next.
3. Restore config.tgz and the management data volume as in Restore on a new server, and start the stack only after every step succeeded.

Change the server's domain

A server cannot be moved to another domain in place. The domain is part of the identity provider's database, the configuration files and every device's settings. To change it:

1. Back up the old server and keep it running until the new one works.
2. Install a new server on the new domain, see Install on your own server.
3. Recreate users (or connect the same identity provider), groups, policies, routes and DNS settings. The REST API can export them from the old server and import them into the new one.
4. Enrol the devices again: netzilo down, then netzilo up --management-url https://<new domain>, or a new setup key for servers.

Troubleshooting

The backup ends with BACKUP FAILED: required file missing or empty. A configuration file the restore needs is missing in the compose folder. Find out why before you change anything else; the backup cannot be completed without it.

The backup ends with cannot find the management data volume. The management container is not running. Start the stack and run the backup again.

The restore says REFUSING: the netzilo database on this host is not empty. The new host already has data, usually from a marketplace first boot. Remove that empty install (step 1) and start again.

After a restore, zitadel logs unable to decrypt key. zitadel.env does not belong to the restored database. Restore config.tgz from the same backup as the dumps.

After a restore, devices show peer is not registered. The backup is older than those devices' enrolment. Enrol them again.

Related Documentation

• Upgrade and add components - back up before every upgrade
• Server reference - files, volumes and secrets
• Troubleshooting - diagnose a server that does not start