10 min readTechnology Guides

How to Compile NGINX with QUIC + HTTP/3 and Brotli on Debian 11

Step by step instructions using Cloudflare’s Quiche patch and Google’s Brotli module

Introduction

Introduction to QUIC + HTTP/3

QUIC is a lower-latency transport-layer protocol, initially developed by Google, that uses multiplexed UDP streams.# The Internet Engineering Task Force (IETF) formalized QUIC as RFC 9000 in May 2021. HTTP/3 is the application-layer protocol that maps HTTP over QUIC, which the IETF formalized as RFC 9114 in June 2022. HTTP/3’s improvements over HTTP/2 include faster TLS handshakes, zero round-trip time connection resumption (0-RTT), and TLS 1.3 encryption by default.# Chrome, Edge, Firefox, and Opera support HTTP/3 by default, and Safari offers support as an experimental feature.#, #, # Synthetic benchmarks aside, HTTP/3 appears to have similar performance to HTTP/2.#

Comparison of HTTP 1.1, HTTP/2, and HTTP/3 Protocol Stacks
Comparison of HTTP 1.1, HTTP/2, and HTTP/3 Protocol Stacks | Figure by Sedrubal (Wikimedia)

Introduction to Brotli

Brotli is gzip’s successor for HTTP compression. It is a lossless data compression algorithm developed by Google.# Although Brotli uses more CPU resources to compress data than gzip, and marginally more resources to decompress data than gzip,# the resulting smaller transmission payloads often result in faster resource loading.# All major browsers accept Brotli encoding.#

The Problem with NGINX

NGINX is the most popular web server application.# However, its developers have been slow to add support for QUIC + HTTP/3 and Brotli into the mainline distribution. They are developing QUIC + HTTP/3 support in a separate branch, and they have restricted Brotli support to paying NGINX Plus customers. Of the top five most popular web server applications, Cloudflare Server (#3), LiteSpeed (#4), and Microsoft IIS (#5) each support QUIC + HTTP/3;#, #, #, # and NGINX is the only application of the five that doesn’t support Brotli.#, #, #, #

One Solution

One way to add QUIC + HTTP/3 and Brotli support to NGINX is to build NGINX from source, including Cloudflare’s unofficial Quiche patch (which is reportedly more mature than NGINX’s QUIC branch#) and Google’s Brotli module.

Cloudflare and Google’s respective GitHub pages include brief instructions for how to compile NGINX with support for QUIC + HTTP/3 and Brotli. This tutorial’s aim is to provide end-to-end instructions, starting with a minimal Debian 11 (Bullseye) installation and ending with a fully-operational NGINX installation. We’ll compile and install build dependencies; pull the source code for NGINX, Quiche, and Brotli; patch, compile, and install NGINX; configure NGINX; test that NGINX serves QUIC + HTTP/3 and Brotli; and then configure systemd to manage NGINX.

Compiling and Installing NGINX

Prerequisites

I wrote this tutorial based on a minimal installation of Debian 11 (Bullseye). If you’re using a different distribution of Linux, you may need to alter some of the package management commands, pre-compiled dependencies, and build configurations shown below.

You’ll need a system with an active internet connection to download dependencies and source code. This tutorial assumes you’ll be running commands as a non-root user with sudo privileges. (If you’ll be running commands as root, omit sudo as applicable.) Lastly, if you already have NGINX installed on your system, you’ll need to remove it before proceeding.

Installing Pre-Compiled Dependencies

Let’s begin by installing pre-compiled binaries from Debian’s repository. We’ll need git and wget to download source code. Compiling CMake and NGINX will require GCC, which is available through build-essential. Compiling CMake will require the OpenSSL developer library, which is available as libssl-dev. (Normally building NGINX would also require the OpenSSL developer library, but in this case, NGINX will use the BoringSSL library as provided through Quiche.) Lastly, building the NGINX HTTP rewrite module will require the PCRE developer library, which is available as libpcre++-dev, and the gzip module will require the zlib developer library, which is available as zlib1g-dev.

sudo apt update
sudo apt install build-essential git libpcre++-dev libssl-dev wget zlib1g-dev -y

Building Rust from Source

Now let’s build our remaining dependencies. Quiche builds using Rust. The Debian repository has an old version of Rust that breaks building NGINX,# so we’ll need to pull and compile Rust from source.

First, let’s make a working directory inside /tmp for easy cleanup:

mkdir /tmp/rustscript
cd /tmp/rustscript

The installation script for Rust includes a prompt mid-installation. To automate a default installation, let’s run the following:

wget https://sh.rustup.rs -O rust.sh
chmod +x rust.sh
./rust.sh -y
source "$HOME/.cargo/env"

Building CMake from Source

We also need to build CMake, which BoringSSL (the SSL library included with Quiche) uses to build itself.

First, let’s make a working directory inside /tmp for easy cleanup:

mkdir /tmp/cmakebuild
cd /tmp/cmakebuild

Now let’s pull the latest stable branch, unpack, build, and install CMake:

wget https://github.com/Kitware/CMake/releases/download/v3.22.5/cmake-3.22.5.tar.gz
tar xf cmake-3.22.5.tar.gz
cd /tmp/cmakebuild/cmake-3.22.5
./bootstrap
make
sudo make install

Preparing the NGINX Source Code

Now that we’ve prepared our build environment, let’s pull and modify the NGINX source code.

First, let’s make a working directory inside /tmp for easy cleanup:

mkdir /tmp/nginxbuild
cd /tmp/nginxbuild

Next let’s pull and unpack NGINX v1.16.1, which Cloudflare’s unofficial Quiche patch is built against.#

wget https://nginx.org/download/nginx-1.16.1.tar.gz
tar xf nginx-1.16.1.tar.gz

Now let’s patch NGINX with Cloudflare’s unofficial Quiche patch:

git clone --recursive https://github.com/cloudflare/quiche
cd /tmp/nginxbuild/nginx-1.16.1
patch -p01 < /tmp/nginxbuild/quiche/nginx/nginx-1.16.patch

Finally, let’s add Google’s Brotli module:

cd /tmp/nginxbuild
git clone --recursive https://github.com/google/ngx_brotli

Building and Installing NGINX

Now that we’ve prepared the source code for NGINX, let’s set its build configurations. (Refer to NGINX’s configure documentation for an explanation of options.)

The configuration options shown below in sections 1–4 mimic those of Debian’s distribution of NGINX, which may be shown by running nginx -V on a system with a Debian-packaged NGINX installation. Section 5 includes settings for Quiche. Section 6 adds Google’s Brotli module to NGINX. Section 7 contains my own configurations, which add the versions of Quiche and Brotli to the output of nginx -v and install the NGINX binary to /usr/local/sbin/nginx, which is in the privileged system $PATH.

cd /tmp/nginxbuild/nginx-1.16.1
./configure \
  \
  --with-cc-opt='-g -O2 -fstack-protector-strong -Wformat -Werror=format-security -fPIC -Wdate-time -D_FORTIFY_SOURCE=2' \
  --with-ld-opt='-Wl,-z,relro -Wl,-z,now -fPIC' \
  \
  --prefix=/usr/share/nginx \
  --conf-path=/etc/nginx/nginx.conf \
  --error-log-path=/var/log/nginx/error.log \
  --http-log-path=/var/log/nginx/access.log \
  --lock-path=/var/lock/nginx.lock \
  --modules-path=/usr/lib/nginx/modules \
  --pid-path=/run/nginx.pid \
  \
  --http-client-body-temp-path=/var/lib/nginx/body \
  --http-fastcgi-temp-path=/var/lib/nginx/fastcgi \
  --http-proxy-temp-path=/var/lib/nginx/proxy \
  --http-scgi-temp-path=/var/lib/nginx/scgi \
  --http-uwsgi-temp-path=/var/lib/nginx/uwsgi \
  \
  --with-compat \
  --with-debug \
  --with-http_addition_module \
  --with-http_auth_request_module \
  --with-http_dav_module \
  --with-http_gunzip_module \
  --with-http_gzip_static_module \
  --with-http_realip_module \
  --with-http_slice_module \
  --with-http_ssl_module \
  --with-http_stub_status_module \
  --with-http_sub_module \
  --with-http_v2_module \
  --with-pcre-jit \
  --with-threads \
  \
  --with-http_v3_module \
  --with-openssl=/tmp/nginxbuild/quiche/quiche/deps/boringssl \
  --with-quiche=/tmp/nginxbuild/quiche \
  \
  --add-module=/tmp/nginxbuild/ngx_brotli \
  \
  --build="brotli_$(git --git-dir=/tmp/nginxbuild/ngx_brotli/.git describe --tags) quiche_$(git --git-dir=/tmp/nginxbuild/quiche/.git describe --tags)" \
  --sbin-path=/usr/local/sbin/nginx

Now that we’ve set the build configuration, let’s build and install NGINX:

make
sudo make install

Managing Compile Bloat

Evaluating Filesystem Size Before and After Compiling NGINX

Compiling NGINX with QUIC + HTTP/3 and Brotli creates 2.8 GiB of intermediate files, which is likely undesirable, especially in a container environment. Below I’ll trace where the intermediate files are created and then show how to recover file system space.

For simplicity, I’ll display file sizes in terminal outputs using the -h flag, which displays, for instance, 31G instead of 31861548 1K-blocks. The -h flag displays sizes in base 2, and it always rounds up to the nearest figure shown. In my own descriptions, I’ll calculate values using kibibytes, convert them to the largest applicable KiB/MiB/GiB unit, and then round them to the nearest decimal so to prevent rounding errors.

I benchmarked filesystem size by installing Debian 11.4 on a single ext4 partition without any optional package bundles. I updated the repository cache, installed openssh-server and sudo, and configured a non-root user with SSH access and sudo permissions. That placed the filesystem size at 1.3 GiB:

admin@debian:~$ df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        31G  1.3G   28G   5% /

Here’s how file space was distributed:

admin@debian:~$ sudo du -sh /* 2>&- | grep -E 'K|M|G' | sort -h
4.0K    /mnt
4.0K    /opt
4.0K    /srv
8.0K    /media
16K     /lost+found
32K     /tmp
36K     /root
48K     /home
572K    /run
2.8M    /etc
79M     /boot
257M    /var
946M    /usr

After compiling and installing NGINX, the filesystem size became 4 GiB:

admin@debian:~$ df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        31G  4.1G   25G  15% /

Here’s how file space was distributed after compiling and installing NGINX:

admin@debian:~$ sudo du -sh /* 2>&- | grep -E 'K|M|G' | sort -h
4.0K    /mnt
4.0K    /opt
4.0K    /srv
8.0K    /media
16K     /lost+found
36K     /root
572K    /run
3.6M    /etc
79M     /boot
260M    /var
1.2G    /tmp
1.3G    /home
1.4G    /usr

Even though compiling and installing NGINX took an additional 2.8 GiB of file space, only 36 MiB was for NGINX itself:

admin@debian:/$ sudo find / -name nginx -exec du -sh {} \; | grep -v "/tmp" | sort -h
4.0K    /var/log/nginx
16K     /usr/share/nginx
72K     /etc/nginx
36M     /usr/local/sbin/nginx

Here’s the distribution of file space for those 2.8 GiB of intermediate files, which I calculated using the above measures as [post-compile file space distribution] - [NGINX installation files] - [pre-compile file space distribution].

764 KiB     /etc
3.4 MiB     /var
453 MiB     /usr
1.1 GiB     /tmp
1.2 GiB     /home

Cleaning Compile Bloat

We can easily recover the intermediary file space consumed inside /home and /tmp. /etc, /usr, and /var contain a mixture of build intermediaries and built NGINX dependencies, so they won’t be as easy to recover file system space from.

First, let’s clean the home directory by removing directories that Rust created, removing the Cargo environment variable from .bashrc and .profile, and removing the HSTS file:

cd ~
rm -rf .cargo .rustup
sed -i '$d' .bashrc
sed -i '$d' .profile
rm .wget-hsts

Next, let’s clean the /tmp directory by removing the directories we created earlier:

cd /tmp
sudo rm -rf cmakebuild nginxbuild rustscript

There isn’t a simple way to revert /etc, /usr, and /var to their original states. The intermediary files created in these directories include a combination of build intermediaries and newly-built NGINX dependencies. (See this changes report made with pkgdiff for a list of every file that changed.) If we wished to keep intermediary files to a minimum, we could package our custom NGINX installation and then install it and the necessary dependencies on a target system. For this tutorial, though, removing the pre-compiled dependencies we originally installed is satisfactory. Doing so frees an additional 101 MiB of space on our file system:

sudo apt purge --auto-remove build-essential git libpcre++-dev libssl-dev wget zlib1g-dev -y

Let’s check our progress:

admin@debian:~$ df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        31G  1.7G   28G   6% /

All together, we’ve brought the filesystem size back down to 1.6 GiB (remember that -h always rounds up), or just 356 MiB above the filesystem size before we compiled and installed NGINX.

Serving Hello World with NGINX

Now that we’ve compiled NGINX with support for QUIC + HTTP/3 and Brotli and recovered file system space, let’s test a basic NGINX configuration to make sure everything’s working.

Creating a Needed Directory

NGINX doesn’t create the parent directory for its temporary directories. Let’s create it so NGINX can run:

sudo mkdir /var/lib/nginx/

Creating an SSL Certificate

We’ll need an SSL certificate for serving HTTP/3. Let’s create a self-signed certificate for testing:

sudo openssl ecparam -out /etc/nginx/selfsign.key -name prime256v1 -genkey
sudo openssl req -new -key /etc/nginx/selfsign.key -x509 -sha256 -days 7200 -out /etc/nginx/selfsign.pem

Running a Basic NGINX Configuration

Now let’s replace the stock NGINX configuration with a short configuration file.

First, let’s clear the stock NGINX configuration…

sudo bash -c "cat /dev/null > /etc/nginx/nginx.conf"

… and paste the below configuration into /etc/nginx/nginx.conf with our text editor of choice running as sudo:

events {
    worker_connections 1024;
}

http {

    # Brotli
    brotli on;
    
    server {
        
        # QUIC requires 'reuseport' if listening on multiple workers
        # declare 'reuseport' on first port use only
        listen 443 ssl http2 reuseport;
        # QUIC
        listen 443 quic;
        
        ssl_certificate selfsign.pem;
        ssl_certificate_key selfsign.key;
        # QUIC requires TLSv1.3
        ssl_protocols TLSv1.2 TLSv1.3;
        
        return 200 '<!DOCTYPE html><title>Hello World</title>Hello World';
        default_type text/html;
        # HTTP/3
        add_header alt-svc 'h3=":443"; ma=86400';
        
    }

}

Then run NGINX:

sudo nginx

Verifying QUIC + HTTP/3

There are three methods we can use to verify that NGINX is serving over QUIC + HTTP/3:

  1. We can compile a custom curl binary with support for QUIC + HTTP/3 on the host system and then run:
    curl -kI --http3 https://localhost
  2. We can use an online tool like LiteSpeed’s HTTP/3 Check. (This method requires a publicly reachable hostname mapped to our server.)
  3. We can use the developer tools inside a browser that supports QUIC + HTTP/3 to verify the connection protocol.

Verifying Brotli

There are three methods we can use to verify that NGINX is serving Brotli encoded responses:

  1. We can run curl from the host system:
    curl -kIH 'Accept-Encoding: br' https://localhost
  2. We can use an online tool like KeyCDN Tools Brotli Test. (This method requires a publicly reachable hostname mapped to our server.)
  3. We can use the developer tools inside a browser that supports Brotli to verify the content encoding.

Managing NGINX with systemd

We can use systemd to manage NGINX on our system. Since we’re not installing NGINX from a package, we’ll need to create the systemd file ourselves.

First, let’s kill the NGINX process we started earlier:

sudo pkill nginx

Creating a Custom systemd File

Next, let’s create our systemd file. Our systemd file will be a copy of the systemd file Debian packages with NGINX with two changes:

  1. We’ll change the location of the systemd file from /usr/lib/systemd/system/nginx.service to /etc/systemd/system/nginx.service to indicate that we’ve built a custom NGINX installation rather than installed it from a package.#
  2. We’ll change the binary location to match that of our installation.

Let’s paste the below configuration into /etc/systemd/system/nginx.service with our text editor of choice running as sudo:

# Stop dance for nginx
# =======================
#
# ExecStop sends SIGSTOP (graceful stop) to the nginx process.
# If, after 5s (--retry QUIT/5) nginx is still running, systemd takes control
# and sends SIGTERM (fast shutdown) to the main process.
# After another 5s (TimeoutStopSec=5), and if nginx is alive, systemd sends
# SIGKILL to all the remaining processes in the process group (KillMode=mixed).
#
# nginx signals reference doc:
# http://nginx.org/en/docs/control.html
#
[Unit]
Description=A high performance web server and a reverse proxy server
Documentation=man:nginx(8)
After=network.target

[Service]
Type=forking
PIDFile=/run/nginx.pid
ExecStartPre=/usr/local/sbin/nginx -t -q -g 'daemon on; master_process on;'
ExecStart=/usr/local/sbin/nginx -g 'daemon on; master_process on;'
ExecReload=/usr/local/sbin/nginx -g 'daemon on; master_process on;' -s reload
ExecStop=-/sbin/start-stop-daemon --quiet --stop --retry QUIT/5 --pidfile /run/nginx.pid
TimeoutStopSec=5
KillMode=mixed

[Install]
WantedBy=multi-user.target

Running NGINX with systemd

Now that we have our systemd file, we need to enable it so NGINX continues running across system restarts:

sudo systemctl enable nginx

Finally, let’s start NGINX and verify that it’s running:

sudo systemctl start nginx
sudo systemctl status nginx

And there we have it! We now have NGINX running, serving HTTP/3 content over QUIC, and encoding content with Brotli.