Troubleshooting

This page helps you diagnose a self-hosted Netzilo Server: a dashboard that does not load, a container that keeps restarting, sign-in problems, devices that cannot connect, relay problems, a full disk and a failed upgrade. Each entry gives the symptom, the cause and the fix.

For problems on a device, see Troubleshooting client issues.

Start with a first check

Run this on the server. It finds the compose folder, shows the state of every container and the last log lines, and tests the server from the outside:

if [ -f /opt/netzilo/run/docker-compose.yml ]; then C=/opt/netzilo/run; else C=/opt/netzilo; fi
cd "$C"
sudo docker compose ps
sudo docker compose logs --tail=100 management | tail -n 60
sudo docker compose logs --tail=100 zitadel | grep -iE "error|fatal|warn" | tail -n 30
sudo docker compose logs --tail=60 caddy
df -h /; free -h; uptime
D=$(grep -oE '^[^*].*:443' Caddyfile | head -1 | sed 's/:443//'); echo "domain=$D"
curl -sS -o /dev/null -w "dashboard %{http_code}\n" https://$D/
curl -sS -o /dev/null -w "oidc %{http_code}\n" https://$D/.well-known/openid-configuration
curl -sS -o /dev/null -w "api %{http_code}\n" https://$D/api/users

On a healthy server every container is Up, postgres, redis and support-worker are healthy, the dashboard and the sign-in discovery answer 200, and the API answers 401.

Nothing loads, or a certificate warning

The browser cannot connect at all. Ports 80 and 443 are blocked, Caddy is down, or DNS is wrong. Check sudo docker compose ps caddy, the cloud security group, and that getent hosts <domain> prints the server's public IP.

The browser shows a certificate warning, and the certificate's issuer is Caddy Local Authority. Caddy could not get a Let's Encrypt certificate: DNS does not point at the server, port 80 is blocked, the server has no outbound HTTPS, or Let's Encrypt's rate limit was hit. Check sudo docker compose logs caddy | grep -iE "acme|challenge|obtain|rate". Fix DNS or the ports; Caddy retries by itself. When rate-limited (too many certificates already issued), wait, or use your own certificate.

Caddy's log shows repeated failures for *.<your domain>. Expected and harmless on a Let's Encrypt server. The certificate for your domain is issued separately.

502 Bad Gateway right after an install or restart. The containers are still starting. Wait a minute; if it stays, see A container keeps restarting.

The sign-in page says "Instance not found. Make sure you got the domain right." You opened the server by another name: an alias, a CNAME or its IP address. Use the exact domain the server was installed with.

Your own certificate expired. Renewing it is your job. See Server reference.

A container keeps restarting

docker compose ps shows Restarting, Exit or unhealthy. Read the container's log with sudo docker compose logs --since 30m <service>.

The log line TrustedPeers are configured to default value from management is informational and can be ignored.

Cannot sign in to the dashboard

The admin password is not accepted on the first sign-in. The credentials are in /opt/netzilo/CREDENTIALS on the server; the username is the admin e-mail. On AWS they are also in the SSM parameter /netzilo/<stack name>/admin-credentials and in the CloudFormation outputs.

Sign-in loops back to the sign-in button. You opened the server by another name than its domain. Use the exact domain.

The dashboard says "Oops, something went wrong" after sign-in. The token could not be validated. Check that curl https://<domain>/.well-known/openid-configuration shows "issuer":"https://<domain>". The exact reason is logged by management only at debug level, see Server reference.

The dashboard spins forever after sign-in. The dashboard cannot reach the API. curl -sS https://<domain>/api/users -H "Authorization: Bearer x" must answer 401, not a 5xx. A 5xx is logged by management as got a handler error:.

Password-reset or invitation e-mails never arrive. No SMTP server is configured. See Identity provider and SSO.

A self-hosted dashboard shows plan limits or "Upgrade Plan". The server is not running as Enterprise. On a marketplace image the switch happens after the first sign-in; on an installer server, contact support@netzilo.com.

For other sign-in messages, such as User is locked, see Identity provider and SSO.

Devices cannot connect to the server

Run netzilo status -d on an affected device. It tells you which part to look at.

Devices connect but cannot see each other: check the policies. Changes reach devices within about 30 seconds; netzilo refresh applies them at once.

If Integrations > Networking has a TURN/STUN integration switched on, devices use those servers instead of the server's own relay. Check there first.

Relay (STUN and TURN) problems

The relay runs in host network mode. Devices receive stun:<domain>:3478, turn:<domain>:3478 and turns:<domain>:5349?transport=tcp, with the username self and a password that must match the user=self:... line in turnserver.conf.

Check the listeners and the credentials on the server. This compares the two copies of the password without printing it:

cd "$C"
sudo ss -lunp | grep -E ':3478|:5349'
A=$(sudo jq -r '.TURNConfig.Turns[0].Password' management.json | tr -d '\n' | shasum -a 256 | cut -c1-12)
B=$(sudo sed -n 's/^user=self://p' turnserver.conf | tr -d '\n' | shasum -a 256 | cut -c1-12)
[ "$A" = "$B" ] && echo "TURN credentials consistent" || echo "MISMATCH: management.json vs turnserver.conf"

A mismatch happens only after someone edited one of the files. Make turnserver.conf match management.json and run sudo docker compose restart coturn.

Test the relay on the server. The relay image includes test tools:

cd "$C"
PW=$(sudo jq -r '.TURNConfig.Turns[0].Password' management.json)
sudo docker compose exec coturn turnutils_stunclient -p 3478 127.0.0.1
sudo docker compose exec coturn turnutils_uclient -v -y -n 3 -u self -w "$PW" -p 3478 127.0.0.1
unset PW

A good run reports the allocation and the packets sent and received. 401 means the password was refused. This proves the relay works on the server, not that it is reachable from outside.

Test the relay from a device's network. Open the WebRTC Trickle ICE test page in a browser at that site, remove the default servers, add turn:<domain>:3478 with the username self and the password from management.json, and gather candidates:

• srflx and relay candidates both appear: STUN and TURN work from that site.
• srflx is missing: UDP 3478 is blocked between that site and the server.
• srflx is present, relay is missing: the relay refused the allocation. Check the credentials, and the coturn log for 401.
• Both appear, but relayed devices still pass no traffic: inbound 49152-65535/udp is blocked at the server, or the server sits behind 1:1 NAT (every cloud VM) without the public address configured. Add external-ip=<public IP>/<private IP> to turnserver.conf and run sudo docker compose restart coturn.

Also check the host notes for Hetzner and Oracle Cloud in Install on your own server.

Browsers with the extension cannot reach internal sites

Start with the test in Clientless-access gateway. It tells server-side problems from browser-side ones in one step. On older AWS and Azure deployments, port 8443 is often not open yet.

Slow dashboard or API

Management logs too many clients already. The database has run out of connections. In docker-compose.yml, lower the management variables NETBIRD_DB_MAX_OPEN_CONNS to 60 and NETBIRD_DB_MAX_IDLE_CONNS to 30, and recreate management. Or move to 4 vCPU and 16 GB.

Sign-ins are slow or fail during a login storm. Raise ZITADEL_DATABASE_POSTGRES_MAXOPENCONNS in zitadel.env from 20 to 40 and recreate zitadel. Keep the total below the database's 100 connections.

Disk full

df -h /
sudo du -sh /var/lib/docker/containers /var/lib/docker/volumes/*netzilo_* /var/lib/docker/overlay2 2>/dev/null
sudo docker system df

Safe ways to free space: remove unused images (sudo docker image prune -f), empty the container logs (sudo truncate -s 0 /var/lib/docker/containers/*/*-json.log) and then limit them as in Install on your own server, and delete old backups under /var/backups/netzilo. Never run docker volume prune on this server: the database is a volume. Afterwards, run sudo docker compose restart postgres redis management zitadel.

After a reboot or cloud maintenance

All containers start again by themselves. If they do not:

sudo systemctl status docker
cd "$C" && sudo docker compose up -d && sudo docker compose ps

On AWS, an instance without an Elastic IP gets a new public IP: update the domain's A record. On Azure the public IP is static.

An upgrade went wrong

If you used a one-line installer command, it printed the rollback and left a backup folder in /opt/netzilo/backups/. Otherwise, roll back the one component that fails to the image you recorded before the upgrade. If no record exists, sudo docker image ls still shows the previous image on the host. See Upgrade and add components. Do not put an older identity provider image back once zitadel started on the new one.

Marketplace metering

The log is /var/log/netzilo-metering.log. Run it at once with sudo systemctl start netzilo-metering.service.

Do not turn metering off on a pay-as-you-go listing: it is your contractual billing.

Contact support

Write to support@netzilo.com with the output of the first check, the time the problem started, and netzilo status -d from one affected device. Never send zitadel.env, management.json, CREDENTIALS, database dumps or tokens. Review logs taken at debug level line by line before you send them: they can contain tokens and e-mail addresses.

Related Documentation

• Server reference - containers, files, ports and logs
• Upgrade and add components - roll back a failed upgrade
• Backup and restore - restore after data loss
• Troubleshooting client issues - problems on devices