Skip to main content
dnz-proxy is DNZ’s own reverse proxy, built on Cloudflare’s Pingora. It is the second proxy backend next to Traefik. It serves the same routers, services and middlewares you set up for a proxy in DNZ (the same objects a Traefik would get), and adds what DNZ can do because it controls the proxy:
  • Many servers per proxy. Any number of servers (“nodes”) join a proxy. DNZ pushes the configuration to all of them the moment it changes, and DNS answers the proxy’s names with the addresses of the nodes that are up.
  • Certificates from DNZ, issued over ACME with DNS-01 through DNZ’s own zones or a Cloudflare token per domain, or HTTP-01 answered by every node. See Certificates.
  • Analytics and request logs for every proxied hostname.
  • Caching, a firewall, and an “under attack” challenge as middlewares.
  • TCP and UDP routing.
Proxying is off by default. Turn it on with Settings → Proxy → dnz-proxy, or DNZ_PROXY_BACKEND=dnz. See Turning proxying on.

Nodes

A node is a server running the dnz-proxy binary. It ships with DNZ, in the same Docker image as dnz. A node needs only DNZ’s HTTP address. It never has to be on the same machine.

Running a node

Settings → Proxy → Set up dnz-proxy shows these commands filled in, for systemd, Docker and a plain command line.

Joining

A node joins with the connect key, in one of three ways:
  • Added by address. On the proxy’s Nodes tab, Add node with the address it calls DNZ from. A node calling from there joins at once.
  • Approved. Any other node shows up as New node waiting on the Proxy page. Approve it into a proxy (check its public addresses), or ignore it. At most 20 wait at once.
  • Automatically. A proxy with Let nodes join without approval (its Settings tab) takes every node that names it.
Once joined, a node gets a token of its own, kept in its state directory. The connect key isn’t needed any more, and replacing the connect key doesn’t affect joined nodes. Removing a node revokes its token; with the connect key still set, it asks to join again.

Configuration and failures

Each node keeps a stream open to DNZ (GET /api/proxy/dnz/stream). When anything the proxy serves changes, DNZ rebuilds its configuration within a second and sends it to every node, which swaps it in without dropping connections. A node stores the last configuration on disk and keeps serving it while DNZ is away or restarting. It reconnects by itself.
If DNZ is behind a reverse proxy, it must not buffer responses, or the stream stalls. DNZ sends X-Accel-Buffering: no. See HTTPS.
Nodes report every 5 seconds: the configuration they serve, their listeners, the health of each upstream, cache use, warnings (a rule or middleware they couldn’t use), and their traffic.

DNS and health

With dnz-proxy, a proxied name is answered with the public addresses of the proxy’s healthy nodes and a short TTL, so a node that goes down leaves the answers quickly. The TTL is TTL of proxied answers on the proxy’s Settings tab, 60 seconds by default. A node is healthy while it reports (within 15 seconds) and, with Check the nodes’ ports from DNZ on, while DNZ can connect to its HTTPS (or HTTP) port.
  • If no node is healthy, DNS answers with all of them rather than with nothing.
  • A proxy without nodes answers with its own addresses.
Drain takes a node out of the answers for maintenance. It keeps serving visitors who still reach it.

Managing nodes from the CLI

These need a superadmin. See Proxy commands.

Running one on DNZ’s server

Settings → Proxy → Run dnz-proxy on this server starts a node as a child of DNZ, joined to the chosen proxy as local. DNZ uses the dnz-proxy next to its own binary, or on the PATH, or the one DNZ_PROXY_BIN names. It restarts the node if it exits and shows its last log lines.
Ports 80 and 443 need root or CAP_NET_BIND_SERVICE. The systemd unit and the Docker image allow it.

Entry points

A proxy’s nodes listen on its entry points, on the proxy’s Entry points tab. Each has a name, an address (:443 listens on every address, IPv6 and IPv4), and a protocol: http, https, tcp or udp. A proxy starts with web (:80, HTTP) and websecure (:443, HTTPS). Routers name entry points as they do with Traefik. A router with TLS answers on HTTPS entry points, one without on HTTP ones. Per entry point:
  • PROXY protocol from: addresses or ranges of load balancers in front of the nodes that send a PROXY protocol header (v1 or v2). The visitor’s address comes from it.
  • Trusted forwarders: peers whose X-Forwarded-* headers are kept. From anyone else they are replaced.
  • Timeouts for idle connections and for reading a request.

Routing

Routers, services and middlewares work as described in Routers, services and middlewares. dnz-proxy reads them as Traefik v3 does:
  • Rules: Host, HostRegexp, Path, PathPrefix, PathRegexp, Method, Header, HeaderRegexp, Query, QueryRegexp, ClientIP with &&, ||, ! and parentheses. Priority is by the rule’s length unless set.
  • Services: load balancing (wrr, p2c, hrw, leasttime), sticky cookies, active and passive health checks, weighted, mirroring (without request bodies) and failover services. HTTP/1.1, HTTP/2 and h2c upstreams, WebSocket and gRPC.
  • Headers to the upstream: X-Forwarded-For, -Proto, -Host, -Port, -Server and X-Real-Ip.
  • Certificates: HTTPS entry points pick the certificate by the name the visitor asks for (SNI), in this order: DNZ’s issued and uploaded certificates, the TLS certificates of the proxy, the default certificate, then a self-signed one.

Client certificates

Client certificates follow clientAuth in the router’s TLS options: clientAuthType, and caFiles as paths on the node or PEM text. CAs alone mean RequireAndVerifyClientCert, as in Traefik.
  • The handshake asks for a certificate when the name the visitor asks for belongs to such a router, and checks that the visitor holds its key.
  • Whether one is required and signed by the CAs is checked once the visit is routed. A visit that fails gets a 403 (Traefik ends the handshake instead), or a 421 when it came over a connection made for another name (HTTP/2 reuses connections), so the browser retries on one of its own.
  • A name served by routers with different TLS options gets the default options.
  • Routers without a Host rule get the default options’ client certificate setting.
  • TCP routers that end TLS don’t ask for client certificates.

Middlewares it runs

redirectScheme, redirectRegex, addPrefix, stripPrefix(Regex), replacePath(Regex), encodedCharacters, chain, headers (custom, CORS, security), basicAuth, digestAuth, forwardAuth, ipAllowList, passTLSClientCert, rateLimit, inFlightReq, compress, contentType, grpcWeb, errors, retry, circuitBreaker, buffering.
Plugins are not supported. A router that uses one is turned off, with a warning on the node.
Answers 400 to paths with an encoded /, \, ;, %, ? or # that isn’t allowed. The query isn’t checked.
Writes X-Forwarded-Tls-Client-Cert (each certificate as base64 DER, joined with ,) and X-Forwarded-Tls-Client-Cert-Info (URL-escaped Subject="…";Issuer="…";…) as Traefik does. Copies of these headers sent by the visitor are dropped.
Turns application/grpc-web and application/grpc-web-text (base64) requests into gRPC, and the answer back, with the trailers written into the body. CORS preflights from allowOrigins are answered. The service needs an HTTP/2 server (h2c:// or https://).
Counts per node in memory, or with redis in Redis, shared by every node that uses it. One endpoint is a single Redis, several are a cluster. User, password, database, timeouts and TLS (CA, client certificate and key, insecureSkipVerify) work as in Traefik.Buckets are keyed by the middleware’s name and the client, so proxies sharing a Redis share the counts of same-named middlewares. While Redis can’t be reached, each node counts on its own and logs it.

TCP and UDP

  • TCP routers on tcp entry points route by HostSNI (TLS passthrough, or TLS ended on the node with its certificates), HostSNI(`*`) for everything, ClientIP and ALPN.
  • TCP services balance by weight, send a PROXY protocol header if asked, and may speak TLS to the server.
  • TCP middlewares: ipAllowList and inFlightConn.
  • UDP routers on udp entry points send each client’s datagrams to one server of their service. Sessions end after 3 seconds without traffic.

dnz-proxy’s middlewares

These appear under dnz-proxy in Add middleware while dnz-proxy is the backend. Traefik never gets them.

Cache (dnzCache)

Caches answers on each node.
  • Answers are stored for as long as Cache-Control (s-maxage, max-age) or Expires allows, or for the default TTL when the origin says nothing. With Respect the origin off, the default TTL always applies.
  • Never stored: no-store, private, answers that set cookies or vary on anything but Accept-Encoding, and statuses outside the list.
  • Visits with Authorization or one of the bypass cookies go to the origin. Prefixes in the style of wordpress_logged_in* work.
The cache key is the host, path and query (unless Query in key is off), chosen headers and cookies, and the accepted encoding.
  • Stale while revalidate serves an expired answer while one visit fetches it again.
  • Stale if error serves it while the origin fails.
Answers carry X-Cache: HIT | MISS | STALE | BYPASS and Age. Each node caches in memory (256 MiB by default) and, with a disk cache directory and size, on disk too. Both are on the proxy’s Settings tab. Purge cache on the proxy clears everything, a hostname, a URL or a path prefix on every node:
Proxy → open the proxy → Purge cache.

Firewall (dnzFirewall)

An ordered list of rules; the first that matches decides. A rule has conditions, all or any of which must hold, on the client’s address or range, country, host, path, method, a header, a query parameter or the user agent. Each condition uses is, is not, in range, contains, starts with or matches (a regular expression). Countries need a GeoIP database. Set GeoIP database URL on the proxy’s Settings tab to an .mmdb (or .mmdb.gz) file, such as MaxMind GeoLite2 Country or DB-IP’s IP to Country Lite. Each node downloads it.

Under attack (dnzChallenge)

Browsers get a short page that solves a small proof of work. A cookie, signed for the visitor’s network and valid for Valid for, then lets them through.
  • Bots can be let through by user agent and by address.
  • Requests that aren’t a browser page (APIs, forms) get a 403 until a page has been visited.
  • Difficulty is in bits: each one doubles the work. 16 takes well under a second.

Analytics

Proxy → Analytics shows, per proxy and hostname, over the last hour to 30 days:
  • requests by status class, bandwidth, and latency (p50, p95, p99)
  • cache hit ratio, blocked and challenged visits
  • top hosts, paths, clients, countries, user agents and statuses
  • a request log with filters and a live view
Counters are kept per minute for two days and per hour for 90 days. Each node sends up to Logged per second, per node log lines; errors, blocks and challenges get ten times as many. Logs are kept for 7 days by default.

Troubleshooting

Check that it can reach DNZ at --dnz and has the current connect key. A node that calls from an address no node was added for waits under New node waiting on the Proxy page. At most 20 wait at once. With a self-signed certificate on DNZ, add --insecure.
The configuration stream goes through GET /api/proxy/dnz/stream. A reverse proxy in front of DNZ that buffers responses holds it back. Turn buffering off for DNZ.
A node leaves the answers once it stops reporting for 15 seconds, or once DNZ can’t connect to its port when Check the nodes’ ports from DNZ is on. Resolvers may keep the old answer for the TTL of proxied answers. If no node is healthy, DNS answers with all of them.
It uses something the node couldn’t run, such as a plugin middleware. The node’s report lists the warning.
DNZ looks for dnz-proxy next to its own binary, then on the PATH; set DNZ_PROXY_BIN to point at it. Ports 80 and 443 need root or CAP_NET_BIND_SERVICE. DNZ shows the node’s last log lines.

Next steps

Certificates

Let DNZ issue and renew certificates for the nodes.

dnz-proxy API

Nodes, entry points, analytics and purges over HTTP.