Cluster

​

How it works

• Change feed. Each server keeps zones, forwarding, blocklists and its other answer data in memory. Whenever one of them changes it, the server also writes a row to the database’s change feed, and the others load that part again. On Postgres they are woken at once with LISTEN/NOTIFY; on MySQL they poll four times a second (Postgres also polls every 2 seconds, in case its LISTEN connection drops). A node that was away for more than a day loads everything.
• Leader. Jobs that write shared state run on one node at a time, the holder of a lease in the database. The leader renews it every 5 seconds; when it stops (crash, lost network), another node takes over after 15 to 20 seconds. A node that shuts down cleanly hands the lease over at once. Every new leader raises the lease’s generation, so a node that was cut off and comes back finds it has lost the lease.
• Nodes. Each server has a stable node id (made on first start and kept in the data folder as node-id, or set with DNZ_NODE_ID) and reports its address, version and the last change it loaded every 5 seconds.

​

Set it up

CREATE USER dnz WITH PASSWORD 'change-me';
CREATE DATABASE dnz OWNER dnz;

host  dnz  dnz  10.0.0.0/24  scram-sha-256

docker run -d --name dnz --restart unless-stopped \
  -p 53:53/udp -p 53:53/tcp -p 8080:8080 \
  -v dnz-data:/data \
  -e DNZ_INSTANCE_NAME=ns1 \
  -e DNZ_NODE_ID=ns1 \
  -e DNZ_CLUSTER_ADDRESS=http://10.0.0.11:8080 \
  -e DNZ_DATABASE_URL='postgres://dnz:change-me@db.internal:5432/dnz' \
  terrificci/dnz:latest

sudo tee -a /etc/dnz/dnz.env >/dev/null <<'EOF'

# Cluster node (docs/cluster.md)
DNZ_INSTANCE_NAME=ns1
DNZ_NODE_ID=ns1
DNZ_CLUSTER_ADDRESS=http://10.0.0.11:8080
DNZ_DNS_ADDR=0.0.0.0:53
DNZ_DATABASE_URL=postgres://dnz:change-me@db.internal:5432/dnz
EOF
sudo systemctl restart dnz

docker run -d --name dnz --restart unless-stopped \
  -p 53:53/udp -p 53:53/tcp -p 8080:8080 \
  -v dnz-data:/data \
  -e DNZ_INSTANCE_NAME=ns2 \
  -e DNZ_NODE_ID=ns2 \
  -e DNZ_CLUSTER_ADDRESS=http://10.0.0.12:8080 \
  -e DNZ_DATABASE_URL='postgres://dnz:change-me@db.internal:5432/dnz' \
  terrificci/dnz:latest

dnz cluster --server http://10.0.0.11:8080
dig @10.0.0.11 www.example.com +short
dig @10.0.0.12 www.example.com +short

Postgres cluster · change feed at 42 · leader ns1 (lease generation 1)

ID   NAME           STATE  ADDRESS                DNS           VERSION  HEARTBEAT  CHANGES
ns1  ns1 * (this)   up     http://10.0.0.11:8080  10.0.0.11:53  1.2.0    2s ago     42/42
ns2  ns2            up     http://10.0.0.12:8080  10.0.0.12:53  1.2.0    1s ago     42/42

​

Clients: NS records or anycast

example.com.      3600  IN  NS  ns1.example.com.
example.com.      3600  IN  NS  ns2.example.com.
ns1.example.com.  3600  IN  A   203.0.113.11
ns2.example.com.  3600  IN  A   203.0.113.12

sudo ip addr add 198.51.100.53/32 dev lo
# in /etc/dnz/dnz.env (or -e with Docker and --network host):
# DNZ_DNS_ADDR=198.51.100.53:53

protocol static dnz_anycast {
  ipv4;
  route 198.51.100.53/32 via "lo";
}

protocol bgp upstream {
  local as 65053;
  neighbor 10.0.0.1 as 65000;
  ipv4 { import none; export where proto = "dnz_anycast"; };
}

#!/bin/sh
# /usr/local/bin/dnz-anycast-check
if dig @198.51.100.53 www.example.com +time=1 +tries=2 +short >/dev/null 2>&1; then
  birdc enable dnz_anycast >/dev/null
else
  birdc disable dnz_anycast >/dev/null
fi

​

Watching it

• Settings → Cluster (superadmin): each node’s address, DNS address, version, last heartbeat, whether it leads, and the change it has loaded against the feed’s last one. A node is down after 30 seconds without a heartbeat and behind when it hasn’t loaded a change that is 30 seconds old. Nodes gone for good can be removed from the list.
• dnz cluster shows the same in the terminal.
• Alerts: the leader sends one when a node goes down or falls behind, and again when it is back, to the notification providers subscribed to cluster_node.
• The topology map shows the other nodes next to the DNZ box, colored by state.
• GET /api/cluster returns it as JSON; GET /api/health on each node is the liveness check for load balancers and Docker.

​

Upgrading

​

Good to know

• Query logs and statistics are in the shared database: the Overview and the logs show the traffic of all nodes together.
• GeoIP: each node keeps its own GeoIP database in its data folder; automatic updates download it on every node.
• dnz-proxy: each dnz-proxy node streams its configuration from the DNZ node it connects to. ACME HTTP-01 challenges are handed out by the leader, which runs the issuance: point dnz-proxy nodes at the leader’s address, or use DNS-01 through DNZ’s own zones, which works from any node.
• NOTIFY from a primary may arrive at any node; the leader transfers the zone within seconds.
• Removing a node: stop it, then remove it under Settings → Cluster (or dnz cluster forget). Nodes not seen for 7 days are removed by themselves.
• Containers without a data volume get a new node id on every start unless DNZ_NODE_ID is set.

​

Troubleshooting