Update (03-04-2024)
Added some versioning to my dockerfile to better compare versions. I also cleaned up the build from yesterday and finally decided to add a ENV for setting the boringssl version also. Since their build broke my dockerfile we better update it manually. I also updated nginx to 1.25.4 and NJS to 0.8.3 and verified the build
Remco Loup.
Introduction
In here you will find a working Dockerfile I created with the below defined compatibility needs for myself. This is posted here for my own archive and to share of course. I added the required support for the following.
- HTTP/2 HPACK
- HTTP/3
- Boringssl
- Brotli support
This version of Nginx supports HTTP/3 quic and Brotli compression as a main benefit with some other fixes. Not having Brotli support these days is just a step back in the delorean.
More info about how HPACK accelerates your HTTP/2. And here you can read more details about BoringSSL
Brotli compared to GZIP
As mentioned this is a Nginx version which supports Brotli. Brotli is very useful as it is a better compression algorithm then GZIP. If you are still not convinced watch the video below.
Nginx base Docker file
This is the base of my dockerfile which takes care of all the things mentioned above. You can of course edit this or use different versions of alpine or Nginx regarding this wont break building the other modules.
I use this image instead of the official Nginx to load all my web projects. See this as a base image to define all your running webserver projects in. Only you get "free" Brotli compression and HTTP/3 support with it so you can make your sites faster and more up to date.
##################################################
# Nginx with Quiche (HTTP/3) and Brotli
# VERSION 1.04 / 03-04-2024
##################################################
# This build will compile Nginx with Brotli and
# HTTP/3 support as the main benefit. We also clean
# this build to remove all build dependencies we
# dont need to run this image after compile.
# it is based on Alpine and therefore a lean and
# mean image to run.
##################################################
FROM alpine:latest as builder
# Set the version of Nginx and NJS to build with and the path to the BoringSSL directory we will use to build Nginx
ENV NGINX_VERSION 1.25.4
ENV NJS_VERSION 0.8.3
ENV BORINGSSL_VERSION 68c6fd8943ffba4e5054ff3a9befa8882b6b226a
ENV BORINGSSL="/tmp/boring-nginx"
# Build-time metadata as defined at https://label-schema.org
ARG BUILD_DATE
ARG VCS_REF
# Install dependencies for building Nginx
RUN addgroup -S nginx \
&& adduser -D -S -h /var/cache/nginx -s /sbin/nologin -G nginx nginx \
&& apk update \
&& apk upgrade \
&& apk add --no-cache ca-certificates openssl \
&& update-ca-certificates \
&& apk add --no-cache --virtual .build-deps \
gcc \
libc-dev \
make \
pcre-dev \
zlib-dev \
linux-headers \
gnupg \
libxslt-dev \
gd-dev \
geoip-dev \
perl-dev \
&& apk add --no-cache --virtual .brotli-build-deps \
autoconf \
libtool \
automake \
git \
g++ \
cmake \
go \
perl \
rust \
cargo \
patch \
libxml2-dev \
byacc \
flex \
libstdc++ \
libmaxminddb-dev \
lmdb-dev \
file \
openrc
# Install BoringSSL and build it from source to get the crypto libraries
RUN mkdir $BORINGSSL \
&& cd $BORINGSSL \
&& git clone https://boringssl.googlesource.com/boringssl \
&& cd boringssl \
&& git checkout -q ${BORINGSSL_VERSION} \
&& mkdir build \
&& cd $BORINGSSL/boringssl/build \
&& cmake -DBUILD_SHARED_LIBS=1 .. \
&& make
# Make an .openssl directory for nginx and then symlink BoringSSL's include directory tree
RUN mkdir -p "$BORINGSSL/boringssl/.openssl/lib" \
&& cd "$BORINGSSL/boringssl/.openssl" \
&& ln -s ../include include \
# Copy the BoringSSL crypto libraries to .openssl/lib so nginx can use them for the Nginx compile
&& cd "$BORINGSSL/boringssl" \
&& cp "build/crypto/libcrypto.so" ".openssl/lib" \
&& cp "build/ssl/libssl.so" ".openssl/lib"
# Copy the BoringSSL libraries to /usr/lib so nginx can find them when starting Nginx
RUN cp "$BORINGSSL/boringssl/.openssl/lib/libssl.so" /usr/lib/
RUN cp "$BORINGSSL/boringssl/.openssl/lib/libcrypto.so" /usr/lib/
# Clone the Nginx Brotli and NJS repos to get the latest versions
RUN mkdir /usr/src \
&& cd /usr/src \
&& git clone --depth=1 --recursive --shallow-submodules https://github.com/google/ngx_brotli \
&& git clone --branch $NJS_VERSION --depth=1 --recursive --shallow-submodules https://github.com/nginx/njs
# Build Nginx with Brotli and HTTP/3 support and then clean up the build dependencies we don't need
RUN cd /usr/src \
&& wget -qO nginx.tar.gz https://nginx.org/download/nginx-$NGINX_VERSION.tar.gz \
&& wget -qO nginx.tar.gz.asc https://nginx.org/download/nginx-$NGINX_VERSION.tar.gz.asc \
&& rm -rf "$GNUPGHOME" nginx.tar.gz.asc \
&& tar -zxC /usr/src -f nginx.tar.gz \
&& rm nginx.tar.gz \
&& cd /usr/src/nginx-$NGINX_VERSION \
&& mkdir /root/.cargo \
&& echo $'[net]\ngit-fetch-with-cli = true' > /root/.cargo/config.toml \
&& ./configure --prefix=/etc/nginx \
--sbin-path=/usr/sbin/nginx \
--modules-path=/usr/lib/nginx/modules \
--conf-path=/etc/nginx/nginx.conf \
--error-log-path=/var/log/nginx/error.log \
--http-log-path=/var/log/nginx/access.log \
--pid-path=/var/run/nginx.pid \
--lock-path=/var/run/nginx.lock \
--http-client-body-temp-path=/var/cache/nginx/client_temp \
--http-proxy-temp-path=/var/cache/nginx/proxy_temp \
--http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp \
--http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp \
--http-scgi-temp-path=/var/cache/nginx/scgi_temp \
--user=nginx \
--group=nginx \
--with-pcre-jit \
--with-http_ssl_module \
--with-http_realip_module \
--with-http_addition_module \
--with-http_sub_module \
--with-http_dav_module \
--with-http_flv_module \
--with-http_mp4_module \
--with-http_gunzip_module \
--with-http_gzip_static_module \
--with-http_random_index_module \
--with-http_secure_link_module \
--with-http_stub_status_module \
--with-http_auth_request_module \
--with-http_xslt_module=dynamic \
--with-http_image_filter_module=dynamic \
--with-http_geoip_module=dynamic \
--with-http_perl_module=dynamic \
--with-threads \
--with-stream \
--with-stream_ssl_module \
--with-stream_ssl_preread_module \
--with-stream_realip_module \
--with-stream_geoip_module=dynamic \
--with-http_slice_module \
--with-mail \
--with-mail_ssl_module \
--with-compat \
--with-file-aio \
--with-http_v2_module \
--with-http_v3_module \
--add-module=/usr/src/ngx_brotli \
--add-module=/usr/src/njs/nginx \
--with-cc-opt=-Wno-error \
--with-select_module \
--with-poll_module \
# --build is used to set the version of the image and the versions of the modules we are using
--build="docker-nginx-http3-$VCS_REF-$BUILD_DATE ngx_brotli-$(git --git-dir=/usr/src/ngx_brotli/.git rev-parse --short HEAD) njs-$(git --git-dir=/usr/src/njs/.git rev-parse --short HEAD)" \
# with-cc-opt is necessary to find the BoringSSL headers we copied to /usr/include
--with-cc-opt="-I$BORINGSSL/boringssl/include" \
# with-ld-opt is necessary to find the BoringSSL libraries we copied to /usr/lib
--with-ld-opt="-L$BORINGSSL/boringssl/build/ssl \
-L$BORINGSSL/boringssl/build/crypto" \
# Fix "Error 127" manually during build
&& touch "$BORINGSSL/boringssl/.openssl/include/openssl/ssl.h" \
&& make -j$(getconf _NPROCESSORS_ONLN) \
&& make -j$(getconf _NPROCESSORS_ONLN) install \
&& rm -rf /etc/nginx/html/ \
&& mkdir /etc/nginx/conf.d/ \
&& mkdir -p /usr/share/nginx/html/ \
&& install -m644 html/index.html /usr/share/nginx/html/ \
&& install -m644 html/50x.html /usr/share/nginx/html/ \
&& ln -s /usr/lib/nginx/modules /etc/nginx/modules \
&& strip /usr/sbin/nginx* \
&& strip /usr/lib/nginx/modules/*.so \
&& rm -rf /etc/nginx/*.default /etc/nginx/*.so \
&& rm -rf /usr/src \
\
# Bring in gettext so we can get `envsubst`, then throw
# the rest away. To do this, we need to install `gettext`
# then move `envsubst` out of the way so `gettext` can
# be deleted completely, then move `envsubst` back.
# read more about envsubst here: https://nickjanetakis.com/blog/using-envsubst-to-merge-environment-variables-into-config-files
&& apk add --no-cache --virtual .gettext "gettext>=0.21-r2" \
&& mv /usr/bin/envsubst /tmp/ \
\
# Determine which dynamic modules to include in the final image and clean up build dependencies
&& runDeps="$( \
scanelf --needed --nobanner /usr/sbin/nginx /usr/lib/nginx/modules/*.so /tmp/envsubst \
| awk '{ gsub(/,/, "\nso:", $2); print "so:" $2 }' \
| sort -u \
| xargs -r apk info --installed \
| sort -u \
)" \
&& apk add --no-cache --virtual .nginx-rundeps $runDeps \
# Clean up our build by removing build dependencies.
&& apk del .brotli-build-deps \
&& apk del .build-deps \
&& apk del .gettext \
# added some extra necesarry modules not fetched by scanelf.
&& apk add --no-cache --virtual .nginx-rundeps g++ libstdc++ libxml2 pcre brotli-libs \
&& rm -rf /root/.cargo \
&& rm -rf /var/cache/apk/* \
&& mv /tmp/envsubst /usr/local/bin/ \
# Create a self-signed certificate for localhost
&& mkdir -p /etc/ssl/private \
&& openssl req -x509 -newkey rsa:4096 -nodes -keyout /etc/ssl/private/localhost.key -out /etc/ssl/localhost.pem -days 365 -sha256 -subj '/CN=localhost' \
# forward request and error logs to docker log collector to get output from `docker logs`
&& ln -sf /dev/stdout /var/log/nginx/access.log \
&& ln -sf /dev/stderr /var/log/nginx/error.log
# expose ports on the docker virtual network
# you still need to use -p or -P to open/forward these ports on host
EXPOSE 80 443
STOPSIGNAL SIGTERM
CMD ["nginx", "-g", "daemon off;"]
Nginx.conf sample
Here you will find a sample nginx.conf which give you an idea of how i run my nginx environment.
The most important difference in this file compared to the normal nginx.conf should be the brotli compression settings.
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log notice;
pid /var/run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;
events {
worker_connections 1024;
}
http {
##
# Basic Settings
##
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
proxy_connect_timeout 300s;
proxy_send_timeout 300s;
proxy_read_timeout 300s;
fastcgi_send_timeout 300s;
fastcgi_read_timeout 300s;
include /etc/nginx/mime.types;
default_type application/octet-stream;
##
# SSL Settings
##
ssl_protocols TLSv1.2 TLSv1.3;
ssl_prefer_server_ciphers on;
##
# Logging Settings
##
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
##
# Gzip Settings
##
gzip on;
gzip_vary on;
gzip_proxied any;
gzip_comp_level 6;
gzip_buffers 16 8k;
gzip_http_version 1.1;
gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;
##
# Brotli Settings (modify the brotli_comp_level to match your optimal cpu load vs compression)
##
brotli on;
brotli_static on;
brotli_comp_level 11;
brotli_types
text/plain
text/css
text/xml
text/javascript
text/x-component
application/xml
application/xml+rss
application/javascript
application/json
application/atom+xml
application/vnd.ms-fontobject
application/x-font-ttf
application/x-font-opentype
application/x-font-truetype
application/x-web-app-manifest+json
application/xhtml+xml
application/octet-stream
font/opentype
font/truetype
font/eot
font/otf
font/woff
font/woff2
image/svg+xml
image/x-icon
image/vnd.microsoft.icon
image/bmp;
##
# Virtual Host Configs
##
include /etc/nginx/conf.d/*.conf;
}
Site.conf sample
You need to watch out that the higher you set the brotli compression the more cpu intensive it becomes. TTFB (time to first byte) can become an issue when the compression ratio is set to high with to slow kubernetes/ high loads or everything in between. so you need to test on real world scenario's and adjust accordingly. This is not only a Brotli issue but Gzip also has the same disadvantages when setting compression levels to high while the decrease in file size isn't worth it.
You can use this file to take all the HTTP/3 settings defined here below. This will optimize and enable HTTP/3 support for your site.
server {
# HTTP/3 and Quic Listen
listen 443 quic reuseport;
http3 on;
# HTTP/3 Add Alt-Svc header negotiation.
add_header alt-svc 'h2=":443"; ma=86400, h3-29=":443"; ma=86400, h3=":443"; ma=86400' always;
# HTTP/2 Listen
listen 443 ssl http2;
http2 on;
# Certificate locations for HTTPS.
ssl_certificate /etc/ssl/certs/wildcard-cert-23.crt;
ssl_certificate_key /etc/ssl/private/wildcard-cert-23.key;
# Enable SSL settings for performance.
ssl_session_cache shared:SSL:50m;
ssl_session_timeout 1d;
ssl_session_tickets off;
# Enable SSL settings for security.
ssl_prefer_server_ciphers on;
ssl_protocols TLSv1.3;
ssl_ciphers HIGH:!aNULL:!MD5;
# Base settings (root and hostname).
root /var/www/html;
index index.php index.html index.htm;
server_name kubernetes.servername.nl;
# HTTP/2 settings
http2_push_preload on;
# /End of HTTP/2.
# HTTP/3 settings
# Enable TLSv1.3's 0-RTT. Use $ssl_early_data when reverse proxying to
# prevent replay attacks.
# @see: http://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_early_data
ssl_early_data on;
quic_retry on;
quic_gso on;
quic_host_key /etc/ssl/private/wildcard-cert-23.key;
# /End of HTTP/3.
location / {
try_files $uri $uri/ /index.php?$query_string;
}
location ~ \.php$ {
try_files $uri =404;
# Bunch of PHP and FPM settings
fastcgi_split_path_info ^(.+\.php)(/.+)$;
fastcgi_pass your-php-fpm-instance:9000;
fastcgi_index index.php;
include fastcgi_params;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
fastcgi_param PATH_INFO $fastcgi_path_info
# Add a bunch of other headers based on your desired settings.
add_header X-protocol $server_protocol always;
add_header X-Cache $upstream_cache_status always;
add_header X-Frame-Options SAMEORIGIN always;
add_header X-Content-Type-Options nosniff always;
add_header X-XSS-Protection "1; mode=block";
add_header Strict-Transport-Security "max-age=31536000; includeSubdomains; preload" always;
add_header Referrer-Policy strict-origin-when-cross-origin;
add_header Pragma public;
add_header Cache-Control "public";
}
}
Multiple H3 Connections from a single Nginx
Here i will explain how you can make multiple server blocks within Nginx work with HTTP/3. This has some caveats since http3 uses UDP with the setting reuseport which makes it unable to assign port 443 multiple times when using HTTP/3
add_header alt-svc 'h2=":443"; ma=86400, h3-29=":443"; ma=86400, h3=":443"; ma=86400' always;This part of the header informs that our HTTP/3 connection accepts port 443. When you have a single nginx container setup and you have more server blocks configured you will get issues regarding the reuse for port 443.
With Http3 you can only define a single port once within a server block. So if you have more server blocks you have the option to let your website communicate over different ports then 443 using this alt-svc header which tells the browser it accepts HTTP/3 and on which port.
add_header alt-svc 'h2=":443"; ma=86400, h3-29=":443"; ma=86400, h3=":443"; ma=86400' always;
add_header alt-svc 'h2=":443"; ma=86400, h3-29=":444"; ma=86400, h3=":444"; ma=86400' always;Check out the differences in the h3 ports here. This way you can make multiple server blocks use h3 by specifying unique ports.
All you need to do is make your instance listen to the right port specified in your alt-svc header. If there is a successful connection possible the browser will try to switch to h3. Below a sample how you can listen to port 443 and 444 with HTTP3
server {
server_name http3-port-443
# HTTP/3 and Quic Listen
listen 443 quic reuseport;
http3 on;
# HTTP/3 Add Alt-Svc header negotiation.
add_header alt-svc 'h2=":443"; ma=86400, h3-29=":443"; ma=86400, h3=":443"; ma=86400' always;
# HTTP/2 Listen
listen 443 ssl http2;
http2 on;
# HTTP/3 settings
# Enable TLSv1.3's 0-RTT. Use $ssl_early_data when reverse proxying to
# prevent replay attacks.
# @see: http://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_early_data
ssl_early_data on;
quic_retry on;
quic_gso on;
quic_host_key /etc/ssl/private/wildcard-cert-23.key;
# /End of HTTP/3.
location / {
try_files $uri $uri/ /index.php?$query_string;
}
}server {
server_name http3-port-444
# HTTP/3 and Quic Listen
listen 444 quic reuseport;
http3 on;
# HTTP/3 Add Alt-Svc header negotiation.
add_header alt-svc 'h2=":443"; ma=86400, h3-29=":444"; ma=86400, h3=":444"; ma=86400' always;
# HTTP/2 Listen
listen 443 ssl http2;
http2 on;
# HTTP/3 settings
# Enable TLSv1.3's 0-RTT. Use $ssl_early_data when reverse proxying to
# prevent replay attacks.
# @see: http://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_early_data
ssl_early_data on;
quic_retry on;
quic_gso on;
quic_host_key /etc/ssl/private/wildcard-cert-23.key;
# /End of HTTP/3.
location / {
try_files $uri $uri/ /index.php?$query_string;
}
}It is that easy to make your Reverse proxy or multiple server blocks establish a H3 connection. You do have to make the right port forwards within your firewall/network also.
When using docker you configure this as shown below. When specifying UDP you have to specify every port seperately in docker. Below is my sample which includes UDP ports 443 up to 449.
services:
############################################################
# proxy webservice to connect to multiple nginx instances
############################################################
reverse-proxy:
image: custom-build-nginx:latest
restart: unless-stopped
ports:
- '80:80'
- '443:443'
- '443:443/udp'
- '444:444/udp'
- '445:445/udp'
- '446:446/udp'
- '447:447/udp'
- '448:448/udp'
- '449:449/udp'
volumes:Update (02-04-2024)
I only recently found out that an update to BoringSSL done around 28 february broke my build of BoringSSL. below the updated dockerfile that compiles again. I had to build libssl as a shared library for it to function again. I will cleanup this image a bit more but it is at least working again.
Update (18-03-2024)
Added a section to explain how you can use multiple H3 connections when you have a Nginx that has more than 1 server block. This also holds the key when you use Nginx as a Reverse Proxy and want to make use of H3.
Update (04-09-2023)
I found out that this build didnt compile anymore due to a change in the brotli source code? It might had something to do with updates from github i didnt version in this Dockerfile. I completely reworked this file to work with the updated Nginx 1.25.2 which has "built in" HTTP/3 support. I also made sure we have native Brotli support again. This should keep functioning from now on.
Update (30-08-2023)
I removed the modsec support there was before. This was giving random Errors and since i wont use it i removed it. It might return later.
