RogerBW's Blog: HAProxy, Anubis, LigHTTPD

I've been setting up Anubis. This turns out to be not trivial, especially if you only know the most basic bits of Docker. So here's what I did, based on the documentation, and a lot of advice, and a lot of trial and error. (This is mostly for Future Me, but I hope it may be useful to other people too.)

Note that there is a hatively packaged version which is probably less trouble, but it was buried so far down in the docs that I didn't find it.

This is all going in front of an existing web server; as it happens it's running LigHTTPD, and itself routes queries between static front-end pages and more fiddly stuff dealt with by another server (originally that was Apache with mod_perl). The basic approach is to stick HAProxy out on the front to do SSL termination: if it gets and can verify an Anubis cookie, it routes the query directly to the LigHTTPD service, and if not it sends it to Anubis instead. (The normal approach sends everything to Anubis and it then redirects to the back-end service or a verification page. That still happens for cases in which Anubis is configured to pass without a challenge.)

You will need haproxy, docker.io, docker-cli, docker-compose (assuming a Debian system). I'm using Debian-supplied packages for everything.

Some censored items in the config below: $SECRET$ is a 512-bit shared secret in hex string form, generated by openssl rand -hex 64. $HOST1$ and "$HOST2" are the names of the underlying services; I'm assuming a single back-end web server that distinguishes based on host name, and of course if you only have one such service it all gets rather easier, but I'm putting in two to make it clear how that works. Because of the existing setup of the system where I was first deploying this, Anubis will run on port localhost:8078 and the actual web service on port localhost:8079, but obviously you cah change those ports ad lib; just change them everywhere.

/etc/haproxy/haproxy.cfg has this:

frontend FE-application
  mode http
  bind :80
  # ssl offloading on port 443 using a certificate from /etc/haproxy/ssl/ directory
  bind :443 ssl crt /etc/haproxy/ssl/ alpn h2,http/1.1 ssl-min-ver TLSv1.2 no-tls-tickets
  timeout client 50000ms

  # set X-Real-IP header required for Anubis
  http-request set-header X-Real-IP "%[src]"

  # redirect HTTP to HTTPS
  http-request redirect scheme https code 301 unless { ssl_fc }
  # add HSTS header
  http-response set-header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload"

  # route to Anubis backend by default
  acl acl_anubis_required hdr(host) -i "$HOST1$"
  acl acl_anubis_required hdr(host) -i "$HOST2$"
  acl acl_anubis_ignore path /excluded/path

(if you don't have any excluded paths, remove this and references to
acl_anubis_ignore below.)

  # default_backend BE-anubis-application
# get payload of the JWT such as algorithm, expire time, restrictions
  http-request set-var(txn.anubis_jwt_alg) req.cook(techaro.lol-anubis-auth),jwt_header_query('$.alg') if acl_anubis_required !acl_anubis_ignore
  http-request set-var(txn.anubis_jwt_exp) cook(techaro.lol-anubis-auth),jwt_payload_query('$.exp','int') if acl_anubis_required !acl_anubis_ignore
  http-request set-var(txn.anubis_jwt_res) cook(techaro.lol-anubis-auth),jwt_payload_query('$.restriction') if acl_anubis_required !acl_anubis_ignore
  http-request set-var(txn.srcip) req.fhdr(X-Real-IP) if acl_anubis_required !acl_anubis_ignore
  http-request set-var(txn.now) date() if acl_anubis_required !acl_anubis_ignore
  use_backend BE-anubis if acl_anubis_required !acl_anubis_ignore !{ req.cook(techaro.lol-anubis-auth) -m found }

  # use Anubis if JWT has wrong algorithm, is expired, restrictions don't match or isn't signed with the correct key
  use_backend BE-anubis if acl_anubis_required !acl_anubis_ignore !{ var(txn.anubis_jwt_alg) -m str HS512 }
  use_backend BE-anubis if acl_anubis_required !acl_anubis_ignore { var(txn.anubis_jwt_exp),sub(txn.now) -m int lt 0 }
  use_backend BE-anubis if acl_anubis_required !acl_anubis_ignore !{ var(txn.srcip),digest(sha256),hex,lower,strcmp(txn.anubis_jwt_res) eq 0 }
  use_backend BE-anubis if acl_anubis_required !acl_anubis_ignore !{ cook(techaro.lol-anubis-auth),jwt_verify(txn.anubis_jwt_alg,"$SECRET$") -m int 1 }

  # custom routing in HAProxy
  use_backend BE-service if { hdr(host) -i "$HOST1$" }
  use_backend BE-service if { hdr(host) -i "$HOST2$" }

backend BE-anubis
  mode http
  server anubis 127.0.0.1:8078 maxconn 32
  timeout connect 5000ms
  timeout server 50000ms

backend BE-service
  mode http
  server service localhost:8079 maxconn 32
  option forwardfor
  http-request set-header X-Real-IP %[src]
  timeout connect 5000ms
  timeout server 50000ms

Into /etc/haproxy/ssl/ go the SSL cert and key files that the LigHTTPD server was using. (If using Let's Encrypt, concatenate fullchain.pem and privkey.pem.)

Now /etc/anubis/docker-compose.yml:

---
services:
  anubis:
    image: ghcr.io/techarohq/anubis:latest
    container_name: anubis
    restart: unless-stopped
    environment:
      BIND: ":8078"
      DIFFICULTY: "4"
      METRICS_BIND: ":9090"
      COOKIE_DYNAMIC_DOMAIN: "true"
      TARGET: "http://0.0.0.0"
      HS512_SECRET: "$SECRET$"
      REDIRECT_DOMAINS: "$HOST1$, $HOST2$"
    ports:
      - 127.0.0.1:8078:8078

Start this service with docker compose up -d from the directory where that file is. (Ew.)

And /etc/lighttpd/lighttpd.conf needs to include:

server.port                 = 8079
server.bind                 = "localhost"

while (in Debian) a file in /etc/lighttpd/conf-enabled/ needs some customisation for logging:

server.modules += ( "mod_accesslog" )

accesslog.filename = "/var/log/lighttpd/access.log"
accesslog.format = "%{x-real-ip}i %V %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\""

By default the log format would start with "%h"; this modifies the logging so that it stores the real external IP address, forwarded by HAProxy in a custom header.

At this point you can shut down the old lighttpd, bring up the new one, and bring up haproxy, and it should all work.

RogerBW's Blog

Add A Comment