Scripts I Use to Automate NGINX

I self-host my own Ghost blog. Ghost is a Node.js/Express application, and I use NGINX for SSL termination, firewall, upstream caching, and re-writing responses. Below are scripts I use to compile NGINX with extended features, to automate updating NGINX’s configuration after I make changes on my local machine, and to maintain NGINX’s SSL certificates during runtime:

#1 Compile NGINX

I built NGINX with support for Brotli and QUIC + HTTP/3 to enable improved data compression and faster SSL handshakes on subsequent connections (0-RTT). Since NGINX doesn’t include support for either in its mainline distribution, I compiled it from source and included Cloudflare’s unofficial Quiche patch and Google’s Brotli module. I wrote the script below to help me maintain consistency between rebuilds.

See my blog post How to Compile NGINX with QUIC + HTTP/3 and Brotli on Debian 11 for a full explanation of the below script.

Note #1 - The latest Rust installer (after time of writing the script) does not add cargo to sudo’s environment $PATH and breaks compiling NGINX. The solution is to edit visudo after building Rust.

Note #2 - I've temporarily disabled HTTP/3 on my blog because Quiche bug #640 causes POST requests with a body payload to fail. My anticipated resolution is to recompile NGINX with QUIC + HTTP/3 support using NGINX’s QUIC development branch and the QuicTLS OpenSSL fork.

#!/usr/bin/env bash

# Install Pre-Compiled Dependencies
sudo apt update
sudo apt install build-essential git libpcre++-dev libssl-dev wget zlib1g-dev -y

# Build Rust from Source
mkdir /tmp/rustscript
cd /tmp/rustscript
wget https://sh.rustup.rs -O rust.sh
chmod +x rust.sh
./rust.sh -y
source "$HOME/.cargo/env"

# Build CMake from Source
mkdir /tmp/cmakebuild
cd /tmp/cmakebuild
wget https://github.com/Kitware/CMake/releases/download/v3.22.5/cmake-3.22.5.tar.gz
tar xf cmake-3.22.5.tar.gz
cd /tmp/cmakebuild/cmake-3.22.5
./bootstrap
make
sudo make install

# Pull NGINX Source Code
mkdir /tmp/nginxbuild
cd /tmp/nginxbuild
wget https://nginx.org/download/nginx-1.16.1.tar.gz
tar xf nginx-1.16.1.tar.gz

# Patch NGINX with Cloudflare's Quiche patch
git clone --recursive https://github.com/cloudflare/quiche
cd /tmp/nginxbuild/nginx-1.16.1
patch -p01 < /tmp/nginxbuild/quiche/nginx/nginx-1.16.patch

# Add Google's Brotli module
cd /tmp/nginxbuild
git clone --recursive https://github.com/google/ngx_brotli

# Build and Install NGINX
cd /tmp/nginxbuild/nginx-1.16.1
./configure \
  \
  --with-cc-opt='-g -O2 -fstack-protector-strong -Wformat -Werror=format-security -fPIC -Wdate-time -D_FORTIFY_SOURCE=2' \
  --with-ld-opt='-Wl,-z,relro -Wl,-z,now -fPIC' \
  \
  --prefix=/usr/share/nginx \
  --conf-path=/etc/nginx/nginx.conf \
  --error-log-path=/var/log/nginx/error.log \
  --http-log-path=/var/log/nginx/access.log \
  --lock-path=/var/lock/nginx.lock \
  --modules-path=/usr/lib/nginx/modules \
  --pid-path=/run/nginx.pid \
  \
  --http-client-body-temp-path=/var/lib/nginx/body \
  --http-fastcgi-temp-path=/var/lib/nginx/fastcgi \
  --http-proxy-temp-path=/var/lib/nginx/proxy \
  --http-scgi-temp-path=/var/lib/nginx/scgi \
  --http-uwsgi-temp-path=/var/lib/nginx/uwsgi \
  \
  --with-compat \
  --with-debug \
  --with-http_addition_module \
  --with-http_auth_request_module \
  --with-http_dav_module \
  --with-http_gunzip_module \
  --with-http_gzip_static_module \
  --with-http_realip_module \
  --with-http_slice_module \
  --with-http_ssl_module \
  --with-http_stub_status_module \
  --with-http_sub_module \
  --with-http_v2_module \
  --with-pcre-jit \
  --with-threads \
  \
  --with-http_v3_module \
  --with-openssl=/tmp/nginxbuild/quiche/quiche/deps/boringssl \
  --with-quiche=/tmp/nginxbuild/quiche \
  \
  --add-module=/tmp/nginxbuild/ngx_brotli \
  \
  --build="brotli_$(git --git-dir=/tmp/nginxbuild/ngx_brotli/.git describe --tags) quiche_$(git --git-dir=/tmp/nginxbuild/quiche/.git describe --tags)" \
  --sbin-path=/usr/local/sbin/nginx
make
sudo make install

# Clean Compile Bloat
cd
rm -rf .cargo .rustup
sed -i '$d' .bashrc
sed -i '$d' .profile
rm .wget-hsts
cd /tmp
sudo rm -rf cmakebuild nginxbuild rustscript
sudo apt purge --auto-remove build-essential git libpcre++-dev libssl-dev wget zlib1g-dev -y

# Create NGINX Temp Directory
sudo mkdir /var/lib/nginx/

#2 Update NGINX Configuration

I make changes to the NGINX configuration for this blog frequently on my local machine. To streamline uploading the configuration to my server, I upload the configuration as a zip package using SSH. Then I use this script to unpack the files, validate the new configuration, clear NGINX’s upstream cache, and either reload NGINX or restart NGINX and clear its log files.

#!/usr/bin/env bash

# 1) Test that x.zip exists. Else, abort script
if [ ! -f "/home/foo/x.zip" ]; then
  echo "The zip file is missing. Aborting script."
  exit 1
fi

# 2) Replace configuration files with zip contents
cd /etc/nginx
sudo rm -rf _misc global nginx.conf sites snippets
sudo unzip -q /home/foo/x.zip -d /etc/nginx/
sudo rm /home/foo/x.zip

# 3) Test if new NGINX configuration is okay. Else, abort script
TESTRESULT=$(sudo nginx -t 2>&1 >/dev/null)
if [[ ! $TESTRESULT =~ "nginx: configuration file /etc/nginx/nginx.conf test is successful" ]]; then
  echo "Aborting script due NGINX configuration error..."
  echo $TESTRESULT
  exit 1
fi
echo "NGINX configuration test is successful"

# 4) Ask whether to reload or restart NGINX
echo "Do you wish to..."
echo " 1) Reload NGINX"
echo " 2) Restart NGINX and clear its logs"

# 5) Do: a) clear cache, b-1) reload NGINX, b-2) clear logs + restart NGINX
while true; do
  read opt
  case $opt in
    1 ) sudo find /var/lib/nginx -mindepth 2 -maxdepth 2 -wholename "*cache*/*" -exec rm -rf {} \; && sudo systemctl reload nginx && echo "Cleared cache and reloaded NGINX"; exit;;
    2 ) sudo find /var/lib/nginx -mindepth 2 -maxdepth 2 -wholename "*cache*/*" -exec rm -rf {} \; && sudo rm -rf /var/log/nginx/* && sudo systemctl restart nginx && echo "Cleared cache, cleared logs, and restarted NGINX"; exit;;
  esac
done

#3 Automate SSL Management

I use a combination of scripts to (A) automate certificate generation and renewal on my server, (B) manage OCSP stapling, and (C) help maintain forward secrecy.

A) Issue & Renew SSL Certificates

I use Let’s Encrypt with Certbot and the Cloudflare DNS verification plugin to generate SSL certificates for my blog every 60 days. Certbot generates its own cron tasks and renewal scripts based on the initial certbot command issued to generate the certificate.

Since my session ticket key rotation script (part C below) reloads NGINX twice daily, I skip using Certbot’s post-deploy hook to reload NGINX.

issue-alexmathews.sh (for initial certificate generation, called manually)

#!/usr/bin/env bash

certbot certonly \
--cert-name alexmathews \
--force-renewal \
--key-type ecdsa \
--elliptic-curve secp256r1 \
--dns-cloudflare \
--dns-cloudflare-credentials /etc/letsencrypt/cloudflare-token.txt \
--dns-cloudflare-propagation-seconds 30 \
-d alexmathews.blog,static.alexmathews.blog

NGINX configuration

ssl_certificate /etc/letsencrypt/live/alexmathews/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/alexmathews/privkey.pem;

B) Staple OCSP Responses

NGINX doesn’t pre-load OCSP responses for stapling, which causes the first incoming SSL connection to NGINX to fail after restarting NGINX. To circumvent this behavior, I manually query and store the OCSP response for my SSL certificates every three days. (Let’s Encrypt’s OCSP response is valid for seven.)

My session ticket key rotation script (part C below) takes care of reloading NGINX after the OCSP response updates.

cron task (root)

7 17 */3 * * /etc/nginx/run/fetch-ocsp.sh

fetch-ocsp.sh

#!/usr/bin/env bash

# delete the current stored OCSP response
rm /etc/nginx/run/alexmathews.ocsp.der

# read the OCSP responder information from the SSL certificate
# then query the OCSP responder and save the response
openssl ocsp \
-issuer /etc/letsencrypt/live/alexmathews/chain.pem \
-cert /etc/letsencrypt/live/alexmathews/cert.pem \
-url $( openssl x509 -noout -ocsp_uri -in /etc/letsencrypt/live/alexmathews/cert.pem ) \
-noverify \
-no_nonce \
-respout /etc/nginx/run/alexmathews.ocsp.der

# set secure permissions
chmod 600 /etc/nginx/run/alexmathews.ocsp.der

NGINX configuration

ssl_stapling on;
ssl_stapling_file /etc/nginx/run/alexmathews.ocsp.der;

C) Maintain Forward Secrecy

Each of the SSL ciphers I use on my blog support forward secrecy. I also enable session tickets to support zero round-trip time resumption of SSL connections (0-RTT).

Security-conscious NGINX configurations typically disable SSL session tickets. By default, NGINX generates a session ticket key on restart, which persists indefinitely and becomes a vulnerability for maintaining forward secrecy.

To circumvent creating a vulnerability in forward secrecy, I rotate my session ticket keys every twelve hours. If I rotated only a single key, open connections would break each time I rotated my keys. Instead, I rotate my active key into a prior key and load both in NGINX to maintain SSL connections through key rotation.

Since my forward secrecy requirements have the shortest cycle time, reloading NGINX with this script is sufficient to update NGINX when parts A and B (above) change as well.

cron task (root)

37 */12 * * * /etc/nginx/run/rotate-ssl-session-ticket-key.sh

rotate-ssl-session-ticket-key.sh

#!/usr/bin/env bash

# rotate the current session ticket key to prior status
mv /etc/nginx/run/current-ssl-session-ticket.key /etc/nginx/run/prior-ssl-session-ticket.key

# generate a new session ticket key and set secure permissions
openssl rand 80 > /etc/nginx/run/current-ssl-session-ticket.key
chmod 600 /etc/nginx/run/current-ssl-session-ticket.key

# reload NGINX if the configuration checks as valid
nginx -t 2>/dev/null && systemctl reload nginx

NGINX configuration

ssl_session_ticket_key run/current-ssl-session-ticket.key;
ssl_session_ticket_key run/prior-ssl-session-ticket.key;