Database encryption

Netzilo Server can encrypt the sensitive columns of its management database with a key held in a key store you control: HashiCorp Vault, AWS KMS, Azure Key Vault, or a key in the configuration file. A copy of the database, such as a dump, a backup, a replica or a disk snapshot, then holds ciphertext for those columns, and reading them needs the key store as well.

Database encryption is off unless you turn it on. Nothing turns it on by itself, and an upgrade never does. Netzilo Cloud runs with it on.

This page shows what it protects, how to set up each key store, how to turn it on for a server installed with the one-line installer or from the AWS or Azure Marketplace image, and how to operate and troubleshoot it. Every command runs on the server, as root or with sudo, from the compose folder ($C, see Server reference).

What it protects

It protects against anyone who gets the management database without the key store: a lost or stolen dump or backup, a database replica, a disk or volume snapshot, a database administrator who is not a Netzilo administrator.

It does not protect against someone who controls the running server: the management container, or the host with management.json and the key store access the server has. The server needs these values to work and reads them. The dashboard and the REST API show them as before, masked where they were masked.

What is encrypted:

What stays readable in the database, by design: object names and ids, peers and their addresses, groups, policies, routes, timestamps, the event type of each activity, and the AI facts used for counting (agent, rule, model, server, tool, source, surface). Searching events keeps working: keyword search uses keyed tokens of the words, so a dump shows tokens, not words, and filtering by user works.

The identity provider's database (zitadel) is not part of this; it is encrypted with its own master key. DataStoreEncryptionKey in management.json is a separate setting and is not used by this feature.

How it works

• Two keys. A random 256-bit data key encrypts the columns. The key store's key wraps the data key, and only the wrapped form is stored, in the table encryption_keys of the same database. The key store's key never leaves the key store.
• The key store is used at start, not per request. Management unwraps the data key once when it starts and keeps it in memory. A key store outage does not affect a running server; it stops the next start. Management retries an unreachable key store for about 30 seconds, then exits, and the container restarts and tries again. A definite refusal, such as permission denied or a disabled or deleted key, is not retried.
• Two value formats. Most columns use AES-256-GCM with a fresh nonce per value; long text is compressed first, so the event and chat tables usually get smaller. The few columns the server looks values up by (setup key, token hash, event initiator email) use deterministic AES-SIV: equal values give equal ciphertext, which a dump reveals, and nothing else. Every value is bound to its table and column, so a value copied into another column does not decrypt.
• Turning it on encrypts existing data. At the first start with encryption on, management encrypts the small tables (setup keys, tokens, credentials, peers, profiles) in seconds before it serves, then the large ones (events, AI messages, AI Assistant history) in the background while it serves, at roughly a thousand rows a second. Both forms are read until the pass finishes. Progress is kept, so a restart continues where it stopped.
• Redis. Cache entries that carry stored objects are encrypted with the same key. There is nothing to do on the Redis side.
• Cost. Request latency and CPU with encryption on stay within the normal variation of the same server without it.

Choose a key store

local is the weakest choice: the key sits in management.json. It protects a database dump only as long as that dump is never stored together with management.json. The backup in Backup and restore copies both into one folder, so with local, keep the key out of that backup and store it separately (see Backups and restore).

Before you start

1. Back up. Turning encryption on rewrites rows; a backup taken before is your way back. Run the backup check and do not start if it fails.

2. Check that the running management has the feature:

   cd "$C" && sudo docker compose exec -T management /go/bin/netzilo-mgmt encryption --help
   

   CopyCopied!

   A list with status, decrypt, rewrap and keygen means yes. unknown command "encryption" means you upgrade management first, see Servers installed earlier.

3. Database. The bundled PostgreSQL, an external PostgreSQL, SQLite and the Redis-cached store are supported. The legacy JSON file store is not.

4. Agree on two rules with whoever runs the key store:

   • the key store must be reachable from the server at every start;
   • losing the key store's key (deleted, purged, or an old version disabled) makes the encrypted columns unreadable for good, in the database and in every backup. Turn on the key store's own deletion protection; each section below says how.

5. Every management instance that uses the database gets the same EncryptionConfig. Several instances exist only in custom deployments.

Set up the key store

Do these steps from a workstation or from the server, with the key store's own tools. Each section ends with an EncryptionConfig block that you add to management.json in Turn it on.

HashiCorp Vault

export VAULT_ADDR=https://vault.example.com:8200
vault secrets enable transit                               # once per Vault
vault write -f transit/keys/netzilo-management             # the key; aes256-gcm96 by default

vault policy write netzilo-management - <<'EOF'
path "transit/encrypt/netzilo-management" { capabilities = ["update"] }
path "transit/decrypt/netzilo-management" { capabilities = ["update"] }
EOF

Then choose how management authenticates:

Management renews a token every 15 minutes, and signs in again with AppRole or Kubernetes when a token expires.

"EncryptionConfig": {
  "Mode": "on",
  "Provider": "vault",
  "Vault": {
    "Address": "https://vault.example.com:8200",
    "Token": "<token>",
    "Mount": "transit",
    "KeyName": "netzilo-management"
  }
}

Deletion protection. Keep the key's deletion_allowed false, which is the default, and never raise its min_decryption_version past the version that wrapped the stored key. Back up Vault itself: losing Vault's data loses the key.

AWS KMS

Create a symmetric key and an alias:

aws kms create-key --description "Netzilo Server database encryption" \
  --key-usage ENCRYPT_DECRYPT --key-spec SYMMETRIC_DEFAULT --query KeyMetadata.Arn --output text
aws kms create-alias --alias-name alias/netzilo-management --target-key-id <key-arn>
aws kms enable-key-rotation --key-id <key-arn>              # optional; transparent to management

Allow the server's identity to use the key. On the AWS Marketplace image that is the instance role the stack created. Add an inline policy to it:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": ["kms:Encrypt", "kms:Decrypt"],
    "Resource": "<key-arn>",
    "Condition": { "StringEquals": { "kms:EncryptionContext:netzilo": "netzilo-management-dek" } }
  }]
}

The condition is optional. Management always sends that encryption context, so the condition limits the role to this use of the key.

The container and the instance role. Management runs in a container, one network hop further from the instance metadata service than the host. With IMDSv2 required, as on the AWS image, the hop limit must be at least 2 or the container gets no credentials:

aws ec2 modify-instance-metadata-options --instance-id <instance-id> \
  --http-tokens required --http-endpoint enabled --http-put-response-hop-limit 2

"EncryptionConfig": {
  "Mode": "on",
  "Provider": "awskms",
  "AWSKMS": { "Region": "us-east-1", "KeyID": "alias/netzilo-management" }
}

Outside AWS, put AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY of an IAM user limited to the policy above in the management service's environment (step 4 of Turn it on).

Deletion protection. Never schedule the key for deletion. KMS waits 7 to 30 days before it deletes a key, and aws kms cancel-key-deletion undoes it in that time. A disabled key stops every start of management until it is enabled again.

Azure Key Vault

Create a vault with role-based access and purge protection, and an RSA key that may only wrap and unwrap:

az keyvault create -g <resource-group> -n <vault-name> -l <region> --sku premium \
  --enable-rbac-authorization true --enable-purge-protection true --retention-days 90
az keyvault key create --vault-name <vault-name> -n netzilo-management \
  --kty RSA-HSM --size 3072 --ops wrapKey unwrapKey          # --kty RSA on the standard SKU
az lock create -n netzilo-key-no-delete --lock-type CanNotDelete \
  --resource "$(az keyvault show -g <resource-group> -n <vault-name> --query id -o tsv)"

Creating the key needs data-plane access to the vault from where az runs: the Key Vault Crypto Officer role on the vault, and network access. Create the key before you restrict the vault's network.

Allow the server's identity to use the key, and nothing else:

KEY_SCOPE="$(az keyvault show -g <resource-group> -n <vault-name> --query id -o tsv)/keys/netzilo-management"
az role assignment create --assignee-object-id <principal-id> --assignee-principal-type ServicePrincipal \
  --role "Key Vault Crypto Service Encryption User" --scope "$KEY_SCOPE"

The role grants get, wrap and unwrap on that key. A new role assignment can take a few minutes to apply; a start in that time fails with a 403 and the container retries.

Network. Management reaches the vault at https://<vault-name>.vault.azure.net. If you close the vault's public access, give the server a path: a private endpoint in the server's virtual network, with the privatelink.vaultcore.azure.net private DNS zone linked to it, or a service endpoint on the server's subnet with that subnet allowed in the vault's firewall.

"EncryptionConfig": {
  "Mode": "on",
  "Provider": "azurekeyvault",
  "AzureKeyVault": { "KeyURL": "https://<vault-name>.vault.azure.net/keys/netzilo-management" }
}

Use the URL without a version. Management then wraps with the newest key version and, after a rotation, wraps the stored key again with the new version at its next start. A URL that ends in a version pins that version.

Management signs in with Azure's default credential chain: environment variables, workload identity, then managed identity. Outside Azure, put AZURE_TENANT_ID, AZURE_CLIENT_ID and AZURE_CLIENT_SECRET (or AZURE_CLIENT_CERTIFICATE_PATH) of the service principal in the management service's environment (step 4 of Turn it on).

Deletion protection. Purge protection and the delete lock above. Never disable or delete an older key version while a stored key or a backup may still be wrapped with it, see Rotate the key store's key.

Local key

Create a key:

cd "$C" && sudo docker compose exec -T management /go/bin/netzilo-mgmt encryption keygen

"EncryptionConfig": {
  "Mode": "on",
  "Provider": "local",
  "Local": { "Key": "<output of keygen>" }
}

Keep a copy of the key somewhere other than the server and its backups. Without it the encrypted columns cannot be read.

Turn it on

Compose installs: one-line installer, AWS and Azure images

All three use the same compose layout; only the compose folder differs.

1. Do the steps in Before you start.

2. Find the compose folder and keep a copy of the current file:

   if [ -f /opt/netzilo/run/docker-compose.yml ]; then C=/opt/netzilo/run; else C=/opt/netzilo; fi
   cd "$C" && sudo cp -a management.json "management.json.pre-encryption"
   

   CopyCopied!

3. Add your block from Set up the key store as a top-level key of management.json, keeping everything else. For example, for Vault:

   sudo python3 - <<'PY'
   import json
   path = "management.json"
   cfg = json.load(open(path))
   cfg["EncryptionConfig"] = {
       "Mode": "on",
       "Provider": "vault",
       "Vault": {"Address": "https://vault.example.com:8200", "Token": "<token>",
                 "Mount": "transit", "KeyName": "netzilo-management"},
   }
   with open(path, "r+") as f:            # same file, same owner and mode
       f.seek(0); f.write(json.dumps(cfg, indent=2) + "\n"); f.truncate()
   PY
   sudo python3 -c 'import json; print(json.load(open("management.json"))["EncryptionConfig"]["Provider"])'
   

   CopyCopied!

4. Only if the key store needs something outside management.json, such as cloud credentials outside their cloud or a CA file for Vault, add it to the management service in docker-compose.yml:

     management:
       environment:
         # … the existing lines stay …
         - AZURE_TENANT_ID=<tenant-id>
         - AZURE_CLIENT_ID=<app-id>
         - AZURE_CLIENT_SECRET=<secret>
       volumes:
         # … the existing lines stay …
         - ./vault-ca.pem:/etc/netzilo/vault-ca.pem:ro    # then "CACert": "/etc/netzilo/vault-ca.pem"
   

   CopyCopied!

   sudo docker compose config -q must print nothing.

5. Recreate management. The dashboard and the API are unavailable for the seconds this takes; connected devices keep their tunnels.

   sudo docker compose up -d --force-recreate management
   sudo docker compose logs -f management | grep -m 3 -i "database encryption"
   

   CopyCopied!

   Expected, in this order:

   database encryption: created data encryption key <id>, wrapped by <provider>
   database encryption on: key <id> wrapped by <provider> (<key reference>), 0 retired key(s) readable
   database encryption: encrypting setup_keys done, <n> rows in <time>
   

   CopyCopied!

   One encrypting <table> done line follows per table; events and ai_messages come last and can take minutes. A start that fails logs failed creating Store: … database encryption: … and the container restarts; see Troubleshooting. To back out, put management.json.pre-encryption back and recreate management again. Nothing was encrypted if no key was created.

6. Check it, then check that management answers as described in Upgrade one component by hand.

Servers installed earlier

Upgrade first, then turn it on. Encryption needs a management release that has it. Upgrade management the usual way, with encryption still off: update management (or the full upgrade) on a one-line install, or a new management image reference on an AWS or Azure image, see Upgrade and add components. Confirm with the check in step 2 of Before you start, then follow the steps above.

Upgrades keep it on. The one-line installer's upgrade and update commands read management.json and write it back with every key, so the EncryptionConfig block stays. The marketplace procedure changes only image references. After every upgrade, before you call it done:

cd "$C"
sudo python3 -c 'import json; print(json.load(open("management.json")).get("EncryptionConfig", {}).get("Mode"))'   # on
sudo docker compose config | grep -E "AZURE_|AWS_|NETZILO_ENCRYPTION_" | sed 's/=.*/=…/'   # the variables you added, if any
sudo docker compose logs --since 10m management | grep "database encryption on:"

Custom deployments and Kubernetes

• Configuration. Mount management.json with the block from a Secret, or set the environment variables on the container. The file and the variables combine; a variable wins.
• Identity without stored secrets. On AKS, workload identity: a user-assigned managed identity with a federated credential for system:serviceaccount:<namespace>:<service-account> and the role from Azure Key Vault on the key; the service account annotated azure.workload.identity/client-id: <client-id>; the pod labelled azure.workload.identity/use: "true" and run under that service account. On EKS, IRSA: the service account annotated with an IAM role that has the policy from AWS KMS. With Vault, KubernetesRole.
• Private networking. If the key store is reachable only privately, check that the cluster's DNS resolves it to the private address and that no NetworkPolicy blocks the traffic.
• Readiness. Give the management container a readiness probe on its listening port; a TCP check is enough. Management opens the port only after the key is unwrapped, so a pod that cannot reach the key store never becomes ready and a rolling update keeps the old pod serving. Wait for the rollout in your pipeline (helm upgrade --wait, kubectl rollout status) so a failure is visible.
• Rolling updates. Roll out the release that has encryption first, with encryption still off, then turn it on in a second rollout. An instance still running with encryption off stops writing secrets as soon as another turns it on, and logs another instance turned encryption on, until it is replaced.
• Shutdown. Give the pod a termination grace period of at least 30 seconds.

Environment variables

Every field can come from the environment instead of the file; a variable that is set wins over the file.

Check it

cd "$C" && sudo docker compose exec -T management \
  /go/bin/netzilo-mgmt encryption status --count --log-file console

configured mode: on
stored key: 3f9c… wrapped by vault (https://vault.example.com:8200/v1/transit/keys/netzilo-management), created 2026-01-15 10:00:02
PASS                         DONE  ROWS    UPDATED
enabled                      true  0       2026-01-15 10:00:02
enc:setup_keys               true  12      2026-01-15 10:00:03
…
running instances with encryption on: 1
TABLE                    PLAINTEXT ROWS  ENCRYPTED ROWS
setup_keys               0               12
…

--count scans every table and takes a while on a large event table. Leave it out for a quick look.

Day-to-day operations

Backups and restore

• Backups need no change. A dump holds the ciphertext and the wrapped data key. The backup in Backup and restore stays the same.
• A restore needs the key store. Restore the whole database and the same management.json with its EncryptionConfig, and make sure the restored server reaches the key store with an identity that may use the key. A new VM or cluster has a new identity: grant it as in Set up the key store before the first start. The key store's key, or the key version that wrapped the data key when the backup was taken, must still exist and be enabled.
• Restore the whole database, not single tables. The data key travels with the data in encryption_keys. A database whose encrypted rows came back without that table makes management refuse to start, instead of making a new key.
• A copy of production for testing reads the encrypted columns only with access to the same key store key.
• Local key. The backup copies management.json with the dump, which puts the key next to the data. Either set the key with NETZILO_ENCRYPTION_LOCAL_KEY from outside the compose folder, or store those backups as carefully as the key.

Rotate the key store's key

Keep older versions usable (Vault: do not raise min_decryption_version past them; Azure: do not disable them) as long as you may restore a backup taken before the rotation.

Rotating the key store's key does not re-encrypt the data. To replace the data key itself, turn encryption off and on again; that creates a new data key.

Move to another key store or key: rewrap

Changing Provider, or the Vault, KMS or Key Vault key, in the configuration alone makes management refuse to start with key provider mismatch. Move the wrapped data key instead; the data is not touched:

cd "$C"
sudo cp -a management.json management.old.json
# edit management.json: the new EncryptionConfig; keep management.old.json as it was
sudo docker compose stop management
sudo docker compose run --rm --no-deps -T \
  -v "$PWD/management.old.json:/etc/netzilo/management.old.json:ro" \
  --entrypoint /go/bin/netzilo-mgmt management \
  encryption rewrap --old-config /etc/netzilo/management.old.json --log-file console
# expect: rewrapped the key with <new provider>
sudo docker compose up -d management

Both key stores must be reachable from the server while the command runs. Keep the old key until the backups taken before the move have expired.

Turn it off: decrypt

Setting Mode to off alone makes management refuse to start while the database holds encrypted values. Decrypt in place first:

cd "$C"
sudo docker compose stop management
sudo docker compose run --rm --no-deps -T --entrypoint /go/bin/netzilo-mgmt management \
  encryption decrypt --log-file console
# then set "Mode": "off" in management.json (or remove the block), and:
sudo docker compose up -d management

• The command needs the key store, so the provider settings must still be there. It waits up to two minutes for the stopped instance's last heartbeat to age out.
• While it runs, and if it is interrupted, no instance starts with encryption on. Run it again to finish, or encryption decrypt --cancel to abandon it and keep encryption on.
• --force decrypts even though an instance reported recently (that instance stops using the key within 30 seconds), and retires the key even if some values did not decrypt.
• At the end the key is retired, not deleted: it stays in encryption_keys, read-only. Turning encryption on again later creates a new data key.

What never to do

• Delete or disable the key store's key, or an older version of it that is still in use.
• Delete or edit rows of encryption_keys or encryption_progress.
• Change the provider or the key in the configuration without rewrap.
• Set Mode to off without decrypt.
• Restore single tables into an encrypted database, or copy rows between databases with different keys.

Troubleshooting

A start failure appears in sudo docker compose logs management as failed creating Store: <data directory>: database encryption: <reason>. A configuration error stops management earlier, names the field, and has no failed creating Store line.

When you contact Netzilo support, include the database encryption lines of the management log and the output of encryption status. Never send management.json, tokens, keys or a database dump.

Security notes

• Audit the key store. Each management start unwraps the data key once. Turn on the key store's audit log (a Vault audit device, AWS CloudTrail for KMS Decrypt, Key Vault diagnostic logs) and alert on unwraps that do not match a management start.
• Least privilege. Management needs only encrypt and decrypt (Vault, KMS) or wrap and unwrap (Key Vault) on one key. Nobody else needs those rights for daily operation.
• management.json is a secret, more so with a Vault token or a local key in it. Prefer identities without stored secrets: instance roles, managed identities, workload identity.
• Copies made earlier stay readable. Turning encryption on protects copies made from then on. Dumps and backups made before stay in plain text; expire them as your retention policy allows, and rotate any credential that may have been in one that was exposed.

Related Documentation

• Backup and restore - back up before you turn it on, and restore with the key store
• Upgrade and add components - upgrade management before you turn it on
• Server reference - the compose folder, management.json and the secrets a server holds
• Hosting on AWS - the AWS Marketplace image and its instance role
• Hosting on Azure - the Azure Marketplace image and its managed identity