In a world with DHCP address, being able to access your home PC or server remotely is a Pita. So, if you’re are “blessed"with an ISP that assigns a dynamic public IP address, you face a significant challenge: your IP changes periodically, breaking your ability to connect to your home device using a static domain.

This is where this free Dynamic DNS (DDNS) updater script for TransIP comes to the rescue. By seamlessly updating your DNS records whenever your external IP changes, this solution ensures your domain (e.g., vpn.example.com) always points to the correct IP. Here’s why this script is a must for anyone leveraging TransIP’s DNS services.

Key Features of the Script

1. Automated DNS Updates:
   • Automatically checks your current external IP and compares it to the DNS A-record associated with your domain.
   • Creates or updates the A-record to match your latest external IP.
2. Dynamic DNS for Dynamic IPs:
   • Perfect for environments where your public IP changes frequently, such as home internet connections with dynamic DHCP assignments.
3. Secure Authentication:
   • Leverages TransIP’s API with a private key for secure communication.
   • Generates short-lived tokens for added security (valid for only 10 minutes per session).
4. Customizable Settings:
   • Easily configure domain, subdomain, TTL (time-to-live), and logging options.
   • Adjust the script to match your specific requirements.
5. Logging and Error Handling:
   • Comprehensive logging ensures you can track every step of the update process.
   • Built-in error handling for smoother execution and debugging.

How It Works

1. Initial Setup:
   • Set up a TransIP account and generate a private key for API access at https://www.transip.nl/cp/account/api/
   • Configure the script with your domain name, subdomain, and private key path.
2. Fetching External IP:
   • The script retrieves your current external IP using a public API like ipinfo.io.
3. Token Generation:
   • Generates a short-lived access token using TransIP’s authentication endpoint.
4. DNS Record Validation:
   • Checks if the current DNS A-record matches your external IP.
   • If it differs, the script updates the record; if it’s missing, it creates a new one.
5. DNS Update:
   • Uses TransIP’s secure API to update or create the DNS A-record with your current assigned external IP.
6. Logging and Notifications:
   • Logs each step and provides clear messages in case of issues.

Prerequisites

Before using this script, ensure you have:

1. A TransIP account with API access enabled.
2. A registered domain in your TransIP account.
3. A static private key for secure API authentication.
4. Dependencies installed on your system:
   • curl, openssl, and jq.

sudo apt update
sudo apt install -y curl openssl jq

Why Use This Solution?

Unlike payed third-party DDNS providers, this script gives you free control over your DNS management, ensuring privacy and avoiding vendor lock-in. It’s tailored for you.

#!/bin/bash

# Configuration settings
LOGIN_NAME="your-login-name"  # Fill in your TransIP login name
PRIVATE_KEY_PATH="/path/transip_private_key.pem"  # Path to your private key file
AUTH_URL="https://api.transip.nl/v6/auth"
TOKEN_FILE="/var/tmp/transip_token.json"  # File to store token and expiration date
DOMAIN="example.com"  # Your main or test domain
SUBDOMAIN="vpn"  # The subdomain you want to update (e.g., test.example.com)
RECORD_TYPE="A"
TTL=60  # Time-to-live for DNS records
LOG_FILE="/var/log/transip_dyndns.log"

# Function to update a log file
log_message() {
    echo "$(date '+%Y-%m-%d %H:%M:%S') - $1" > $LOG_FILE
}

# Function to log an error and stop the script
handle_error() {
    log_message "ERROR: $1"
    echo "ERROR: $1" >&2
    exit 1
}

# Function to obtain a new access token with a short validity period (10 minutes)
generate_new_token() {
    log_message "Generating a new token with a validity period of 10 minutes."
    NONCE=$(head /dev/urandom | tr -dc A-Za-z0-9 | head -c 16)
    TIMESTAMP=$(date +%s)
    LABEL="Set DNS to external DHCP $TIMESTAMP"
    REQUEST_BODY=$(printf '{"login":"%s","nonce":"%s","read_only":false,"expiration_time":"10 minutes","label":"%s","global_key":true}' "$LOGIN_NAME" "$NONCE" "$LABEL")
    log_message "Request Body: $REQUEST_BODY"
    SIGNATURE=$(echo -n "$REQUEST_BODY" | openssl dgst -sha512 -sign "$PRIVATE_KEY_PATH" | base64 | tr -d '\n')
    log_message "Generated signature: $SIGNATURE"
    RESPONSE=$(curl -s -X POST "$AUTH_URL" \
        -H "Signature: $SIGNATURE" \
        -H "Content-Type: application/json" \
        -d "$REQUEST_BODY") || handle_error "Failed to connect to TransIP API"
    log_message "API-response received: $RESPONSE"
    ACCESS_TOKEN=$(echo "$RESPONSE" | jq -r '.token // empty')
    if [ -z "$ACCESS_TOKEN" ]; then
        handle_error "The received access token is empty or invalid. Check the API-response: $RESPONSE"
    fi
    log_message "Received token: $ACCESS_TOKEN"
}

# Function to get the current DNS A-record from TransIP
get_dns_record() {
    # Get API response and store status code
    RESPONSE=$(curl -s -X GET \
        -H "Authorization: Bearer $ACCESS_TOKEN" \
        "https://api.transip.nl/v6/domains/$DOMAIN/dns")

    # Check if response contains JSON and no errors
    if echo "$RESPONSE" | jq empty 2>/dev/null; then
        RECORD_CONTENT=$(echo $RESPONSE | jq -r '.dnsEntries[] | select(.name=="'$SUBDOMAIN'" and .type=="'$RECORD_TYPE'") | .content')
    else
        handle_error "API response does not contain valid JSON or contains an error: $RESPONSE"
    fi

    # Check if we found a valid record
    if [ -z "$RECORD_CONTENT" ]; then
        log_message "No valid DNS record found for $SUBDOMAIN.$DOMAIN. Record may need to be created."
        return 1
    fi

    echo "$RECORD_CONTENT"
    return 0
}

# Function to update or create DNS record
update_or_create_dns_record() {
    # Check if record exists
    CURRENT_DNS_IP=$(get_dns_record)

    if [ -z "$CURRENT_DNS_IP" ]; then
        log_message "No DNS A-record found for $SUBDOMAIN.$DOMAIN. Creating record."
        # Create request body for new record (TTL set to 60 seconds)
        DNS_ENTRY=$(jq -n \
        --arg name "$SUBDOMAIN" \
        --arg expire "60" \
        --arg type "$RECORD_TYPE" \
        --arg content "$CURRENT_IP" \
        '{"dnsEntry": {"name": $name, "expire": ($expire | tonumber), "type": $type, "content": $content}}')

        log_message "DNS entry (create): $DNS_ENTRY"

        # Execute POST request to create new DNS entry with better status code handling
        RESPONSE=$(curl -s -w "%{http_code}" -X POST \
            -H "Authorization: Bearer $ACCESS_TOKEN" \
            -H "Content-Type: application/json" \
            -d "$DNS_ENTRY" \
            "https://api.transip.nl/v6/domains/$DOMAIN/dns")

        HTTP_STATUS_UPDATE=$(echo "$RESPONSE" | tail -n1)
        RESPONSE_BODY=$(echo "$RESPONSE" | head -n -1)

        log_message "API response for creating DNS entry: $RESPONSE_BODY"
        log_message "HTTP status code received for creation: $HTTP_STATUS_UPDATE"

        if [ "$HTTP_STATUS_UPDATE" -ne 201 ]; then
            handle_error "API call for creating DNS entry failed with status code $HTTP_STATUS_UPDATE."
        fi

        log_message "New DNS A-record successfully created."
    else
        log_message "Updating DNS A-record for $SUBDOMAIN.$DOMAIN to IP $CURRENT_IP"
        # Create request body for updating the record
        DNS_ENTRY=$(jq -n \
        --arg name "$SUBDOMAIN" \
        --arg expire "$TTL" \
        --arg type "$RECORD_TYPE" \
        --arg content "$CURRENT_IP" \
        '{"dnsEntry": {"name": $name, "expire": ($expire | tonumber), "type": $type, "content": $content}}')

        log_message "DNS entry (update): $DNS_ENTRY"

        # Execute PATCH request to update the existing DNS record
        RESPONSE=$(curl -s -w "%{http_code}" -X PATCH \
            -H "Authorization: Bearer $ACCESS_TOKEN" \
            -H "Content-Type: application/json" \
            -d "$DNS_ENTRY" \
            "https://api.transip.nl/v6/domains/$DOMAIN/dns")

        HTTP_STATUS_UPDATE=$(echo "$RESPONSE" | tail -n1)
        RESPONSE_UPDATE_BODY=$(echo "$RESPONSE" | head -n -1)

        log_message "API update response received: $RESPONSE_UPDATE_BODY"
        log_message "HTTP status code received for update: $HTTP_STATUS_UPDATE"

        # Accept both 200 and 204 as successful updates
        if [ "$HTTP_STATUS_UPDATE" -ne 200 ] && [ "$HTTP_STATUS_UPDATE" -ne 204 ]; then
            handle_error "API call for DNS update failed with status code $HTTP_STATUS_UPDATE."
        else
            log_message "DNS A-record successfully updated to $CURRENT_IP"
        fi
    fi
}

# Get the current external IP
CURRENT_IP=$(curl -s http://ipinfo.io/ip) || handle_error "Failed to get external IP"

# Log the start
log_message "---- Start new run ----"
log_message "Current external IP: $CURRENT_IP"

# Always generate a short lived new token.
generate_new_token


# Try to update or create the DNS record
update_or_create_dns_record

log_message "---- End of run ----"

Managing SSL certificates for domains hosted on TransIP can be cumbersome without automation. TransIP luckily offers a robust API for DNS management, but to integrate it seamlessly with Certbot for dns-01 validation, a custom solution is necessary. Since I couldn’t find a ready-made script for this purpose, I created one that securely handles authentication, DNS updates, and certificate issuance.

You can go to the full API documentation here:

TransIP API

This guide explains the full setup using Docker, including how to generate the API key pair in TransIP’s web interface and use it for automated certificate management.

How TransIP authentication works

TransIP uses an RSA key pair for API authentication. Here’s how to set it up:

1. Generate the Key Pair in TransIP:
   • Log in to your TransIP control panel.
   • Go to API Settings under your account and enable API access.
   • Generate a new API key pair.
   • Download/Save the private key (transip_private_key.pem) securely and create a secret from it to use in your Docker setup.

docker secret create transip_private_key transip_private_key.pem

1. Keep the Private Key Secure:
   • Store the private key somewhere safe after creating a secret from it.
   • This private key is needed to sign requests to TransIP’s API.
2. How It Works:
   • The script uses the private key to sign an authentication request.
   • TransIP’s API responds with a short-lived token (10 minutes by default), which is then used to manage the DNS records.
   • It will create a new token for convenience on every run. This made it more resilient then checking for an active token and using that which made the whole script less resilient and cumbersome.

What this setup is going to do

1. Automates SSL Certificates:
   • Automates issuance and renewal of wildcard and domain-specific certificates.
2. Handles DNS Challenges:
   • Manages dns-01 challenges by creating and deleting DNS TXT records via the TransIP API.
3. Seamless Integration:
   • Encapsulates everything in a Docker container for portability and simplicity.
   • You can remove or add domains in this script on the -d rule part. I put the examples in for a better explanation.

We will pipe a add or clean command to the bash script to take the necessary actions as defined in the script. So we use the –manual-auth-hook and the manual-cleanup-hook here. This makes sure that a fail in the add section still cleans up after itself for example.

You can also add a –force to this script. but your will run out of attempts to Letsencrypt quickly since they only allow you to do a few calls every 24 hours. But this will make sure it refreshes the certificates always when you need to test or verify something.

version: "3.8"

services:
  certbot:
    image: certbot/certbot:latest
    container_name: certbot-transip
    secrets:
      - transip_private_key  # Reference the secret you created earlier
    # adjust these volumes to point to the right .pem and script. right now this dockerfile expects the files to be in the same folder as the docker-certbot-transip.yml
    volumes:
      - "./certbot-transip.sh:/usr/local/bin/certbot-transip.sh:ro"
      - "./letsencrypt:/etc/letsencrypt"
    environment:
    #These Variables are going to be piped in our bash script.
      LOGIN_NAME: "your-transip-login"        # Replace with your TransIP login name
      PRIVATE_KEY_PATH: "/etc/transip_private_key.pem"
      DOMAIN: "example.com"                  # Replace with your domain
      TTL: "60"                              # Optional: Time-to-live for DNS challenge
      CERTBOT_EMAIL: "you@example.com"       # Replace with your valid email as requested by letsencrypt.
    entrypoint:
      - sh
      - -c
      - |
        apk update && apk add --no-cache bash curl jq bind-tools && \
        chmod +x /usr/local/bin/certbot-transip.sh && \
        certbot certonly --manual --preferred-challenges dns \
          --manual-auth-hook "/usr/local/bin/certbot-transip.sh add" \
          --manual-cleanup-hook "/usr/local/bin/certbot-transip.sh clean" \
          --cert-name "$DOMAIN" \
          -d "*.$DOMAIN" \
          -d "example2.$DOMAIN" \
          -d "example3.$DOMAIN" \
          --expand \
          --non-interactive --agree-tos \
          --email "$CERTBOT_EMAIL"

secrets:
  transip_private_key:
    external: true  # Secret must already exist in Docker

The Bash script

This is the “advanced” part that will do its magic and make everything work with TransIP. I tried to make it a bit more resilient then standard also. So this should hopefully help in creating a resilient and good working environment.

You can also use this TransIP script to run it locally creating certificates or in a Kubectl or Helm chart.

#!/bin/bash

# Configuration settings
LOGIN_NAME=${LOGIN_NAME}
TTL=${TTL:-60}  # Default value 60 if TTL is not set
PRIVATE_KEY_PATH=${PRIVATE_KEY_PATH}
DOMAIN=${DOMAIN}

AUTH_URL="https://api.transip.nl/v6/auth"
LOG_FILE="/var/log/transip_dyndns.log"

# Function to update a log file
#log_message() {
#    echo "$(date '+%Y-%m-%d %H:%M:%S') - $1" >> $LOG_FILE
#}

# Function to log messages to stdout
log_message() {
    echo "$(date '+%Y-%m-%d %H:%M:%S') - $1"
}

# Function to log an error and stop the script
handle_error() {
    log_message "ERROR: $1"
    echo "ERROR: $1" >&2
    exit 1
}

# Function to obtain a new access token with a short validity period (10 minutes)
generate_new_token() {
    log_message "Generating a new token with a validity period of 10 minutes."

    NONCE=$(head /dev/urandom | tr -dc A-Za-z0-9 | head -c 16)
    TIMESTAMP=$(date +%s)
    LABEL="Certbot DNS update $TIMESTAMP"

    REQUEST_BODY=$(printf '{"login":"%s","nonce":"%s","read_only":false,"expiration_time":"10 minutes","label":"%s","global_key":true}' "$LOGIN_NAME" "$NONCE" "$LABEL")

    log_message "Request Body: $REQUEST_BODY"

    SIGNATURE=$(echo -n "$REQUEST_BODY" | openssl dgst -sha512 -sign "$PRIVATE_KEY_PATH" | base64 | tr -d '\n')

    log_message "Generated signature: $SIGNATURE"

    RESPONSE=$(curl -s -X POST "$AUTH_URL" \
        -H "Signature: $SIGNATURE" \
        -H "Content-Type: application/json" \
        -d "$REQUEST_BODY") || handle_error "Failed to connect to TransIP API"

    log_message "API-response received: $RESPONSE"

    ACCESS_TOKEN=$(echo "$RESPONSE" | jq -r '.token // empty')
    if [ -z "$ACCESS_TOKEN" ]; then
        handle_error "The received access token is empty or invalid. Check the API-response: $RESPONSE"
    fi

    log_message "Received token: $ACCESS_TOKEN"
}

# Function to add or update a TXT record
add_or_update_txt_record() {
    local full_domain="$1"
    local value="$2"

    # Extract the subdomain part from the full domain
    local name="_acme-challenge.${full_domain%%.*}"

    if [ -z "$name" ] || [ -z "$value" ]; then
        log_message "ERROR: Name or value for TXT record is missing."
        exit 1
    fi

    # Check if the DNS record already exists
    RESPONSE=$(curl -s -X GET \
        -H "Authorization: Bearer $ACCESS_TOKEN" \
        "https://api.transip.nl/v6/domains/$DOMAIN/dns")

    # Search for all existing records with the same name and type TXT
    MATCHING_RECORDS=$(echo "$RESPONSE" | jq -r --arg name "$name" '[.dnsEntries[] | select(.name==$name and .type=="TXT")]')

    # Check if there are multiple or no matching records
    MATCHING_COUNT=$(echo "$MATCHING_RECORDS" | jq 'length')

    # Ensure MATCHING_COUNT is always an integer
    if ! [[ "$MATCHING_COUNT" =~ ^[0-9]+$ ]]; then
        MATCHING_COUNT=0
    fi

    if [ "$MATCHING_COUNT" -eq 0 ]; then
        log_message "No existing TXT record found for $name. Creating a new record."
    elif [ "$MATCHING_COUNT" -ge 1 ]; then
        log_message "Multiple or one existing TXT record found for $name. Removing all existing records."

        # Remove all existing records using the full details of each record
        echo "$MATCHING_RECORDS" | jq -c '.[]' | while read -r record; do
            existing_name=$(echo "$record" | jq -r '.name')
            existing_expire=$(echo "$record" | jq -r '.expire')
            existing_content=$(echo "$record" | jq -r '.content')

            # Log the details of the record to be removed
            log_message "Remove record: name=$existing_name, expire=$existing_expire, content=$existing_content"

            # Make the DELETE call using the full DNS-entry details
            DELETE_ENTRY=$(jq -n \
            --arg name "$existing_name" \
            --arg expire "$existing_expire" \
            --arg type "TXT" \
            --arg content "$existing_content" \
            '{"dnsEntry": {"name": $name, "expire": ($expire | tonumber), "type": $type, "content": $content}}')

            DELETE_RESPONSE=$(curl -s -X DELETE \
                -H "Authorization: Bearer $ACCESS_TOKEN" \
                -H "Content-Type: application/json" \
                -d "$DELETE_ENTRY" \
                "https://api.transip.nl/v6/domains/$DOMAIN/dns")

            # Log the response of the DELETE call
            log_message "Response after removing $existing_name: $DELETE_RESPONSE"

            # Check if the removal was successful
            if [[ "$DELETE_RESPONSE" == *"error"* ]]; then
                log_message "ERROR: The record $existing_name could not be removed."
            else
                log_message "Record $existing_name successfully removed."
            fi
        done
    fi

    # Create a new record
    DNS_ENTRY=$(jq -n \
    --arg name "$name" \
    --arg expire "$TTL" \
    --arg type "TXT" \
    --arg content "$value" \
    '{"dnsEntry": {"name": $name, "expire": ($expire | tonumber), "type": $type, "content": $content}}')

    RESPONSE=$(curl -s -X POST \
        -H "Authorization: Bearer $ACCESS_TOKEN" \
        -H "Content-Type: application/json" \
        -d "$DNS_ENTRY" \
        "https://api.transip.nl/v6/domains/$DOMAIN/dns")

    log_message "API-response for TXT record creation: $RESPONSE"

    if [[ "$RESPONSE" == *"error"* ]]; then
        handle_error "The TXT record could not be created. Check the API-response: $RESPONSE"
    else
        log_message "TXT record successfully added for $name with value $value"
    fi
}

# Function to update an existing TXT record
update_txt_record() {
    local name="$1"
    local value="$2"

    DNS_ENTRY=$(jq -n \
    --arg name "$name" \
    --arg expire "$TTL" \
    --arg type "TXT" \
    --arg content "$value" \
    '{"dnsEntry": {"name": $name, "expire": ($expire | tonumber), "type": $type, "content": $content}}')

    RESPONSE=$(curl -s -X PATCH \
        -H "Authorization: Bearer $ACCESS_TOKEN" \
        -H "Content-Type: application/json" \
        -d "$DNS_ENTRY" \
        "https://api.transip.nl/v6/domains/$DOMAIN/dns")

    log_message "API-response for TXT record update: $RESPONSE"

    if [[ "$RESPONSE" == *"error"* ]]; then
        handle_error "The TXT record could not be updated. Check the API-response: $RESPONSE"
    else
        log_message "TXT record successfully updated for $name with value $value"
    fi
}

# Function to remove a TXT record for `dns-01` validation
remove_txt_record() {
    local full_domain="$1"

    # Extract the subdomain part from the full domain
    local name="_acme-challenge.${full_domain%%.*}"

    # Check if the DNS record exists
    RESPONSE=$(curl -s -X GET \
        -H "Authorization: Bearer $ACCESS_TOKEN" \
        "https://api.transip.nl/v6/domains/$DOMAIN/dns")

    # Search for all records with the same name and type TXT
    MATCHING_RECORDS=$(echo "$RESPONSE" | jq -r --arg name "$name" '[.dnsEntries[] | select(.name==$name and .type=="TXT")]')

    # Check if there are existing records that need to be removed
    MATCHING_COUNT=$(echo "$MATCHING_RECORDS" | jq 'length')

    if [ "$MATCHING_COUNT" -eq 0 ]; then
        log_message "No existing TXT record found for $name to remove."
    else
        log_message "Removing all existing TXT records for $name."

        # Loop through all found records and remove them
        echo "$MATCHING_RECORDS" | jq -c '.[]' | while read -r record; do
            existing_name=$(echo "$record" | jq -r '.name')
            existing_expire=$(echo "$record" | jq -r '.expire')
            existing_content=$(echo "$record" | jq -r '.content')

            # Make the DELETE call using the full details
            DELETE_ENTRY=$(jq -n \
            --arg name "$existing_name" \
            --arg expire "$existing_expire" \
            --arg type "TXT" \
            --arg content "$existing_content" \
            '{"dnsEntry": {"name": $name, "expire": ($expire | tonumber), "type": $type, "content": $content}}')

            DELETE_RESPONSE=$(curl -s -X DELETE \
                -H "Authorization: Bearer $ACCESS_TOKEN" \
                -H "Content-Type: application/json" \
                -d "$DELETE_ENTRY" \
                "https://api.transip.nl/v6/domains/$DOMAIN/dns")

            # Check if the removal was successful
            if [[ "$DELETE_RESPONSE" == *"error"* ]]; then
                log_message "ERROR: The TXT record $existing_name could not be removed."
            else
                log_message "TXT record $existing_name successfully removed."
            fi
        done
    fi
}

# Always generate a new token for each run
generate_new_token

# Certbot passes these values when adding and removing the record
if [ "$1" == "add" ]; then
    add_or_update_txt_record "$CERTBOT_DOMAIN" "$CERTBOT_VALIDATION"
    sleep 25  # Wait for DNS update to propagate
elif [ "$1" == "clean" ]; then
    remove_txt_record "$CERTBOT_DOMAIN"
fi

Steps to run

1. Download the Private Key: Save the private key file (transip_private_key.pem) from TransIP’s API settings somewhere on your device you are going to run this from.
2. Set Up Docker Compose: Place the docker-compose.yml and certbot-transip.sh files in the same directory. (or edit the script)
3. Run the Container: Start the container (cd to the path where your yml is)

docker-compose up --build

Monitor the Log output: The log output of the docker container will show the progress of DNS challenges and certificate issuance.

Thats all there is to it!

I will also post a version with a working helm chart soon. This was a little bigger culprit to make sure it all works correctly and updates all certificates through different namespaces where i needed the certificates to be available on different domains.

The API: https://api.protonvpn.ch/vpn/logicals has been formally shut down according to Proton Support. The only thing it returns now is:

{"Code":0,"Error":"Invalid access token","Details":{}}

So this post is sadly obsolete now. Somebody started a petition to get this feature back.

The deprecated guide which no longer functions as of 06/2025 For historical reasons.

When ProtonVPN discontinued its command-line interface (CLI) tool, users of headless systems like Raspberry Pi were left without an easy method to manage VPN connections. Without a graphical user interface, manually switching VPN servers has become a cumbersome task. Also selecting the fastest VPN in a pool of servers has become impossible.

To solve this problem, I developed a Python script that automatically selects the fastest ProtonVPN server based on latency and load, then updates your WireGuard configuration for you. This guide will walk you through setting up WireGuard for ProtonVPN, installing the script, and automating your VPN connection to ensure your Raspberry Pi is always connected to the best server.

Why do i need this Protonss script?

This Python script automates the process of connecting your Raspberry Pi to the fastest ProtonVPN server available, using WireGuard as the VPN protocol. It checks the latency and load of multiple VPN servers from ProtonVPN’s API, selects the one with the best performance, updates your WireGuard configuration with the new server information, and reconnects your VPN only if a significantly faster server is found (nobody actually likes reconnects). This script also logs the entire process for easier troubleshooting.

1. No more manual VPN switching: Without ProtonVPN’s CLI, switching to the fastest server would require manual work. This script automates the process by regularly checking the available servers and updating the connection based on real-time performance.
2. Optimizes connection speed: The script ensures you’re always connected to the VPN server with the lowest latency and load. This is especially useful for maintaining high-speed connections in countries with fast or low-traffic ProtonVPN servers.
3. Reduces downtime: By automating the reconnection process, the script helps minimize potential downtimes caused by poor server performance or slow connections.
4. Tailored for Raspberry Pi: If you’re running a headless Raspberry Pi (without a monitor), this script is perfect for automating VPN tasks without requiring constant oversight.
5. Security & Privacy: By ensuring your Raspberry Pi always connects to a VPN, it helps protect your data and online activity, especially if you’re using the Pi as a torrenting or networking device.

In short, this script automates finding the optimal ProtonVPN server, providing faster speeds, more reliability, and convenience for headless systems like a Raspberry Pi.

Let’s dive in!

First, you need to install WireGuard, which is a fast, lightweight VPN protocol which we will use to connect to our ProtonVPN server.

# First, update your package list and install WireGuard:
sudo apt update
sudo apt install wireguard curl resolvconf

Generate a WireGuard configuration from ProtonVPN

• Log in to your ProtonVPN account on their website.
• Go to the “Downloads” section in your dashboard.
• Generate a new configuration file under “WireGuard Configurations.”
• If you’re using torrents, enable NAT-PMP.

Save this configuration file on your Raspberry Pi:

# Save the configuration file to your Raspberry Pi:
# Place it in the WireGuard directory and rename it for simplicity:
sudo nano /etc/wireguard/protonvpn.conf

Once you’ve placed the configuration file in the correct directory, verify the connection.

# Verify connection to ProtonVPN:
sudo wg-quick up protonvpn

# This should give an ip address different to the one frome your isp
curl ifconfig.me 

# Verify connection to ProtonVPN:
sudo wg-quick down protonvpn

curl ifconfig.me 

If these two ip addresses from ifconfig.me are different your are connected to a VPN. even if you dont know your ip address since its a DHCP address or other.

Installing our Python environment for Protonss.

When developing and running Python scripts on your Raspberry Pi, it’s best to use a Python virtual environment. A virtual environment allows you to install and manage Python packages in an isolated setting, ensuring that your system’s default Python configuration remains untouched.

This way, any changes made to your environment will only affect the project you’re working on—such as installing specific packages like requests—without causing conflicts with other Python projects or system-level dependencies.

Here’s how you can set up and use a virtual environment for our script:

1. Install virtual environment: First, install the python3-venv package, which allows you to create and manage virtual environments.

sudo apt install -y python3-venv python3-full

2. Create a virtual environment: Once installed, you can create a virtual environment. This will generate an isolated environment in your specified directory (in this case, ~/protonss).

python3 -m venv ~/protonss

3. Activate the virtual environment: After creating the environment, you’ll need to activate it. This tells your system to use the Python and packages from the virtual environment, instead of the system-wide Python installation.

source ~/protonss/bin/activate

• Now that the environment is active, any Python package you install (such as requests) will be installed into this isolated environment.

pip install requests

Using this virtual environment in the future: Each time you want to run Python scripts that depend on this virtual environment, you’ll need to activate it again using the source ~/venv/bin/activate command. Once activated, any Python commands or packages you use will run in this environment.

Installing our ProtonSS script to automate a connection to the fastest VPN.

We will create a configuration file to hold all the variables that the Python script will use. For example, you can define the path to your WireGuard configuration, log file, and the number of pings to test latency.

cd ~/protonss

Sample config.ini:

nano config.ini

[Settings]
# Path to your WireGuard configuration file
wg_config_path = /etc/wireguard/protonvpn.conf

# Path to your log file
log_file = /var/log/wireguard_updater.log

# Number of pings to measure latency
ping_count = 3

# Maximum score difference before reconnection 0 - 100
# Average score of servers is between 10-40 it seems. so setting this to high results in no switches from servers
score_threshold = 5.0

# Comma-separated list of preferred countries
preferred_countries = US

# Comma-separated list of preferred cities
# If specified, servers must be in these cities (unless they're in a country without any specified city)
# Preferred cities to connect to. If specified, servers must be in these cities
# (unless they're in a country without any specified city).
# Example: preferred_cities = Phoenix, Los Angeles, Seattle
# This allows you to target specific cities within your preferred countries.
# If left empty, any city within the preferred countries will be considered.
# Note: City names are case sensitive and must match ProtonVPN's naming exactly.
# Multiple cities should be comma-separated.
preferred_cities =Phoenix

# Weight for latency in the server score
# latency is subjective since this is probably measured through the vpn and only counts as server latency.
# Checking the latency without a vpn isnt recommended since the proton servers wont reply back without vpn.
# Value cant be higher then 1. 0.3 translates to 30% of the score_treshold.
latency_weight = 0.3

# Weight for server load in the server score
# Load is the best factor 0.7 translates to 70% of the weight of the score_threshold. this cant be higher then 1.
load_weight = 0.7

# Enable or disable pinging servers (True or False)
# Since ping might not be reliable enough for you to keep on switching servers this can be disabled here.
ping_enabled = True

# Specify the interface to use for ping tests. By setting this option you can ping to the fastest vpn server
# this will ensure the lowest latency VPN from your location.
ping_interface = eth0

[Features]
# Choose which type of servers you want to use
# Options: p2p, plus, secure_core, tor
# Default is p2p if not specified
feature_type = p2p

# Feature explanation:
# Each server has exactly ONE feature number:
# 1, 17 = Secure Core servers
# 2 = TOR servers
# 0, 8, 16, 24 = Plus-only servers
# 4, 12, 20, 28 = P2P servers (these are also Plus servers)

You can customize the variables as needed. For example, leaving preferred_countries empty will make the script check a LOT! of servers, which will make the script run a very very long time!

Create the Python script

Now, we are going to create the Python script that will automate your VPN connection. We first need to prepare a few steps

The first line of the script should be a shebang, which ensures the script always runs in the correct Python environment:

source ~/protonss/bin/activate
which python

This will provide you with the pad you need to put on the first rule of this script

/home/pi/venv/bin/python

Modify the script by adding a shebang: Add the following line at the very top of your Python script. Replace /home/{user}/venv/bin/python with the actual path you got when you ran the which python command earlier. Make sure to keep the #! at the beginning of the line—this tells the system which interpreter to use to run the script. we need the absolute path so ~/ wont work.

nano protonvpn_auto_connect.py

#!/home/{user}/protonss/bin/python

import requests
import subprocess
import os
import sys
import configparser
import logging
import re
from datetime import datetime
import math
import shutil

# Get the directory of the current script
script_dir = os.path.dirname(os.path.abspath(__file__))

# Load the config file from the same directory as the script
config = configparser.ConfigParser()
config.read(os.path.join(script_dir, 'config.ini'))

WG_CONFIG_PATH = config.get('Settings', 'wg_config_path')
LOG_FILE = config.get('Settings', 'log_file')
PING_COUNT = config.getint('Settings', 'ping_count')
SCORE_THRESHOLD = config.getfloat('Settings', 'score_threshold', fallback=5.0)
PREFERRED_COUNTRIES = config.get('Settings', 'preferred_countries', fallback='').split(',')
PREFERRED_CITIES = config.get('Settings', 'preferred_cities', fallback='').split(',')
PING_INTERFACE = config.get('Settings', 'ping_interface', fallback=None)

# Weighting factors for latency and load from the config
LATENCY_WEIGHT = config.getfloat('Settings', 'latency_weight', fallback=0.3)
LOAD_WEIGHT = config.getfloat('Settings', 'load_weight', fallback=0.7)

# Ping option from config
PING_ENABLED = config.getboolean('Settings', 'ping_enabled', fallback=True)

# Clean up country codes (remove spaces and convert to uppercase)
PREFERRED_COUNTRIES = [country.strip().upper() for country in PREFERRED_COUNTRIES if country.strip()]

# Clean up city names (remove spaces)
PREFERRED_CITIES = [city.strip() for city in PREFERRED_CITIES if city.strip()]

# Set up logging
logging.basicConfig(
    filename=LOG_FILE,
    level=logging.INFO,
    format='%(asctime)s [%(levelname)s] %(message)s',
    datefmt='%Y-%m-%d %H:%M:%S'
)

logging.info('--- ProtonVPN Auto Connect Script Start ---')

# Feature definitions
FEATURE_SECURE_CORE = [1, 17]  # Secure Core servers
FEATURE_TOR = [2]         # TOR servers
FEATURE_PLUS = [0, 8, 16, 24]  # Plus-only servers
FEATURE_P2P = [4, 12, 20, 28]    # P2P servers (these are also Plus servers)
# check https://api.protonmail.ch/vpn/logicals for more info. 
# be aware that the secure core and tor server are very limited so check if one is available for your country as stated in the config.ini file.
# FREE servers are not included in the api response. so they are already left out.

def get_feature_description(features):
    # If no feature type is specified in config.ini, default to P2P
    # Features is a number representing server capabilities
    if not features:  # If features is 0 or None
        return "P2P"  # Default to P2P
    elif features in FEATURE_SECURE_CORE:
        return "Secure Core"
    elif features in FEATURE_TOR:
        return "TOR" 
    elif features in FEATURE_P2P:
        return "P2P"
    elif features in FEATURE_PLUS:
        return "Plus"
    else:
        return "P2P"  # Default to P2P for any unrecognized feature

def get_wanted_features():
    """Get the wanted feature type based on config priorities"""
    feature_type = config.get('Features', 'feature_type', fallback='p2p').lower()
    
    if feature_type == 'secure_core':
        return FEATURE_SECURE_CORE
    elif feature_type == 'tor':
        return FEATURE_TOR
    elif feature_type == 'plus':
        return FEATURE_PLUS
    else:  # Default to P2P
        return FEATURE_P2P

# Function to fetch the list of servers from the ProtonVPN API
def get_servers(full_list=False):
    try:
        url = 'https://api.protonmail.ch/vpn/logicals'
        response = requests.get(url)
        response.raise_for_status()
        data = response.json()
        logging.info("API response data received.")
        return data['LogicalServers']
    except requests.RequestException as e:
        logging.error(f'Error fetching server list: {e}')
        sys.exit(1)
    except ValueError as e:
        logging.error(f'Error parsing JSON: {e}')
        sys.exit(1)

# Function to filter P2P servers and servers in preferred countries
def filter_servers(servers):
    filtered_servers = []
    
    # Get feature preferences from config (nu maar één type)
    wanted_features = get_wanted_features()
    
    logging.info(f"Filtering servers with preferences - Countries: {PREFERRED_COUNTRIES or 'Any'}, "
                f"Cities: {PREFERRED_CITIES or 'Any'}, Features: {[get_feature_description(f) for f in wanted_features]}")
    
    for server in servers:
        if not isinstance(server, dict):
            logging.error(f"Unexpected type for server: {type(server)} - Value: {server}")
            continue
            
        features = server.get('Features', 0)
        country = server['ExitCountry']
        city = server.get('City', '')

        # Check if server features match our wanted features
        if features in wanted_features:
            # Check country and city preferences
            if country in PREFERRED_COUNTRIES:
                if not PREFERRED_CITIES or city in PREFERRED_CITIES:
                    filtered_servers.append(server)
                    logging.info(f"Server matched - Country: {country}, City: {city}, "
                              f"Features: {get_feature_description(features)}")
            
    if not filtered_servers:
        logging.warning(f"No servers found matching filters. Countries: {PREFERRED_COUNTRIES}, "
                      f"Cities: {PREFERRED_CITIES}, Features: {[get_feature_description(f) for f in wanted_features]}")
    else:
        logging.info(f"Found {len(filtered_servers)} matching servers")
        
    return filtered_servers

# Function to measure latency by pinging the server (optional)
def get_latency(hostname):
    if not PING_ENABLED:
        logging.info(f"Ping disabled, skipping latency check for {hostname}.")
        return 0  # Return 0 latency if ping is disabled

    latencies = []
    for i in range(PING_COUNT):
        try:
            # Add the interface option if it's specified
            ping_command = ['ping', '-c', '1', '-W', '1', hostname]
            if PING_INTERFACE:
                ping_command.insert(1, '-I')
                ping_command.insert(2, PING_INTERFACE)

            result = subprocess.run(ping_command, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
            if result.returncode == 0:
                output = result.stdout.decode()
                latency_line = [line for line in output.split('\n') if 'time=' in line][0]
                latency = float(latency_line.split('time=')[1].split(' ms')[0])
                latencies.append(latency)
            else:
                latencies.append(float('inf'))
        except Exception as e:
            latencies.append(float('inf'))
            logging.error(f'Ping error for {hostname}: {e}')

    # Return the lowest (best) latency value
    best_latency = min(latencies)
    
    # Handle inf latency
    if math.isinf(best_latency):
        logging.warning(f"Latency for {hostname} is inf. Assigning high latency value.")
        best_latency = 1000  # Assign a high latency value

    return best_latency

# Function to select the best server based on a weighted combination of latency and load
def select_best_server(servers):
    best_server = None
    best_score = float('inf')
    
    for server in servers:
        server_load = server.get('Load')
        
        # Handle inf load
        if server_load is None or math.isinf(server_load):
            logging.warning(f"Server {server['Name']} has inf load. Assigning high load value.")
            server_load = 100  # Assign a high load value if it's inf or None
        else:
            server_load = int(server_load)
        
        for srv in server['Servers']:
            exit_ip = srv.get('ExitIP')
            hostname = exit_ip

            # Get latency, only if ping is enabled
            latency = get_latency(hostname)

            # Combine latency and load using the configured weights
            score = (latency * LATENCY_WEIGHT) + (server_load * LOAD_WEIGHT)
            
            logging.info(f"Server {server['Name']} - Latency: {latency:.2f} ms, Load: {server_load}%, Score: {score:.2f}")
            
            if score < best_score:
                best_score = score
                best_server = server
                best_server_instance = srv

    return best_server, best_server_instance, best_score

# Function to read the current server IP from the WireGuard configuration
def get_current_server_ip():
    try:
        with open(WG_CONFIG_PATH, 'r') as f:
            config_text = f.read()
        # Extract the current Endpoint IP
        match = re.search(r'Endpoint\s*=\s*([\d\.]+):\d+', config_text)
        if match:
            logging.info(f"Current server IP found in WireGuard configuration: {match.group(1)}")
            return match.group(1)
        else:
            logging.warning('No current Endpoint IP found in WireGuard configuration.')
            return None
    except Exception as e:
        logging.error(f'Error reading the WireGuard configuration: {e}')
        return None

# Function to find the current server's performance in the live server data
def get_current_server_performance(servers, current_ip):
    if current_ip is None:
        logging.warning('Current server IP is None, skipping performance check.')
        return None, None, float('inf')
    
    for server in servers:
        for srv in server['Servers']:
            if srv['ExitIP'] == current_ip:
                server_load = server.get('Load')
                if server_load is None or math.isinf(server_load):
                    logging.warning(f"Server {server['Name']} (IP: {current_ip}) has inf load. Assigning high load value.")
                    server_load = 100  # Assign a high load value
                else:
                    server_load = int(server_load)

                latency = get_latency(current_ip) if PING_ENABLED else 0
                score = (latency * LATENCY_WEIGHT) + (server_load * LOAD_WEIGHT)

                logging.info(f"Current server {server['Name']} - Latency: {latency:.2f} ms, Load: {server_load}%, Score: {score:.2f}")
                return server, srv, score
    logging.warning(f"Current server with IP {current_ip} not found in live data.")
    return None, None, float('inf')

# Function to find the current server in the full ProtonVPN list if not found in filtered data
def find_current_server_full_list(current_ip):
    servers = get_servers(full_list=True)
    return get_current_server_performance(servers, current_ip)

# Function to update the WireGuard configuration file
def update_wg_config(server, server_instance, best_score):
    try:
        endpoint_ip = server_instance['EntryIP']
        endpoint_port = 51820  # Default WireGuard port
        endpoint = f"{endpoint_ip}:{endpoint_port}"
        public_key = server_instance['X25519PublicKey']
        server_name = server['Name']

        # Read the current WireGuard configuration
        with open(WG_CONFIG_PATH, 'r') as f:
            config_text = f.read()

        # Strip comments that start with # under [Peer] before PublicKey
        config_text = re.sub(r'#.*\n', '', config_text, flags=re.M)

        # Update the Endpoint in the configuration
        config_text = re.sub(r'Endpoint\s*=.*', f'Endpoint = {endpoint}', config_text)

        # Update the PublicKey in the configuration
        config_text = re.sub(r'PublicKey\s*=.*', f'PublicKey = {public_key}', config_text)

        # Add the new comment indicating the chosen server and score
        config_text = re.sub(r'\[Peer\]\n', f'[Peer]\n# {server_name} - Auto-generated by ProtonVPN Auto Connect - Score: {best_score:.2f}\n', config_text)

        # Write the updated configuration back to the file
        with open(WG_CONFIG_PATH, 'w') as f:
            f.write(config_text)
        os.chmod(WG_CONFIG_PATH, 0o600)

        logging.info(f"WireGuard configuration updated with server: {server_name} ({server['ExitCountry']}), Endpoint: {endpoint}, PublicKey: {public_key}, Score: {best_score:.2f}")
    except Exception as e:
        logging.error(f'Error updating the WireGuard configuration: {e}')
        sys.exit(1)

# Function to restart the WireGuard interface
def is_resolvconf_available():
    """Check if resolvconf is available on the system."""
    return shutil.which("resolvconf") is not None

def is_wg_quick_available():
    """Check if wg-quick is available on the system."""
    return shutil.which("wg-quick") is not None

def restart_wg():
    interface_name = os.path.basename(WG_CONFIG_PATH).split('.')[0]
    
    # Check if wg-quick is available before proceeding
    if not is_wg_quick_available():
        logging.error("wg-quick not found. Ensure WireGuard is installed correctly.")
        sys.exit(1)

    try:
        # Check if resolvconf is available before proceeding
        if not is_resolvconf_available():
            logging.warning("resolvconf not found, DNS settings may not be updated correctly.")

        # Bring down the interface
        subprocess.run(['wg-quick', 'down', interface_name], stdout=subprocess.PIPE, stderr=subprocess.PIPE)

        # Bring up the interface
        subprocess.run(['wg-quick', 'up', interface_name], check=True)
        logging.info(f'WireGuard interface {interface_name} started.')

    except subprocess.CalledProcessError as e:
        logging.error(f'Error starting the WireGuard interface: {e}')
        sys.exit(1)

# Main function
def main():
    logging.info('--- Starting server selection process ---')
    
    # Fetch the list of servers
    servers = get_servers()
    if not isinstance(servers, list):
        logging.error(f'Unexpected type for servers: {type(servers)}')
        sys.exit(1)

    # Filter the servers
    filtered_servers = filter_servers(servers)
    if not filtered_servers:
        logging.error('No suitable servers available after filtering.')
        sys.exit(1)

    # Get the current server's IP address from the WireGuard configuration
    current_ip = get_current_server_ip()

    # Get the current server's performance if it exists
    current_server, current_server_instance, current_score = get_current_server_performance(filtered_servers, current_ip)

    # If the current server is not found, check the full list of servers
    if current_server is None:
        logging.info(f"Current server not found in filtered list, checking full server list...")
        current_server, current_server_instance, current_score = find_current_server_full_list(current_ip)

    # Select the best server based on weighted latency and load
    best_server, best_server_instance, best_score = select_best_server(filtered_servers)
    if not best_server:
        logging.error('No suitable server found.')
        sys.exit(1)

    if current_server:
        logging.info(f"Current server {current_server['Name']} (IP: {current_ip}) - Score: {current_score:.2f}")
    
    logging.info(f"Selected best server {best_server['Name']} - Score: {best_score:.2f}")
    
    if current_server and current_score <= best_score + SCORE_THRESHOLD:
        logging.info(f"Current server {current_server['Name']} is within threshold. No switch necessary.")
    else:
        logging.info(f"Switching to server {best_server['Name']} due to better performance.")
        update_wg_config(best_server, best_server_instance, best_score)
        restart_wg()

    logging.info('--- ProtonVPN Auto Connect Script End ---')

if __name__ == '__main__':
    main()

Testing the script

chmod +x protonvpn_auto_connect.py

Once the script is saved, give it execution permissions and run it

sudo ./protonvpn_auto_connect.py

cat /var/log/wireguard_updater.log
tail -f /var/log/wireguard_updater.log

you can tail or cat the log file to check its output in the logs. Tail is preferred since you can keep watching it while running your script. Just open a second ssh with the tail to keep reviewing it.

2024-09-25 22:19:10 [INFO] --- ProtonVPN Auto Connect Script Start ---
2024-09-25 22:19:10 [INFO] --- Starting server selection process ---
2024-09-25 22:19:11 [INFO] API response data received.
2024-09-25 22:19:11 [INFO] Current server IP found in WireGuard configuration: 79.135.105.20
2024-09-25 22:19:11 [WARNING] Current server with IP 79.135.105.20 not found in filtered data.
2024-09-25 22:19:11 [INFO] Current server not found in filtered list, checking full server list...
2024-09-25 22:19:12 [INFO] API response data received.
2024-09-25 22:19:13 [INFO] Current server BH#1 - Latency: 26.90 ms, Load: 10%, Score: 15.07
2024-09-25 22:19:14 [INFO] Server AO#1 - Latency: 27.10 ms, Load: 7%, Score: 13.03
2024-09-25 22:19:15 [INFO] Server AO#2 - Latency: 26.60 ms, Load: 14%, Score: 17.78
2024-09-25 22:19:16 [INFO] Server AO#3 - Latency: 27.10 ms, Load: 9%, Score: 14.43
2024-09-25 22:19:17 [INFO] Server AO#4 - Latency: 26.60 ms, Load: 16%, Score: 19.18
2024-09-25 22:19:17 [INFO] Current server BH#1 (IP: 79.135.105.20) - Score: 15.07
2024-09-25 22:19:17 [INFO] Selected best server AO#1 - Score: 13.03
2024-09-25 22:19:17 [INFO] Current server BH#1 is within threshold. No switch necessary.
2024-09-25 22:19:17 [INFO] --- ProtonVPN Auto Connect Script End ---

sudo cat /etc/wireguard/protonvpn.conf

You can also verify that everything works by checkin the protonvpn.conf file or by checking your external ip address.

[Peer]
# BH#3 - Auto-generated by ProtonVPN Auto Connect - Score: 7.74

you will see something like this above. This comment will update when it connects to another VPN and the ipaddress inside the peer also changes with it off course.

curl ifconfig.me #this wil give the external ip in the cli.

As an additional step you can also check your external ip. this should be something different then the ip address provided by your provider.

Running this script every hour.

sudo crontab -e

0 * * * * cd /home/{user}/protonss && /home/{user}/protonss/protonvpn_auto_connect.py >> /home/{user}/protonss/protonvpn_cron.log 2>&1

Here we defined it to run every hour. but can you schedule it how you want.

sudo crontab -l

Save the crontab and verify the cron is installed.

Optional: Block specific traffic if your VPN fails

If you want to ensure no traffic leaves your device if the VPN connection fails, you can use iptables to block traffic when the WireGuard interface is down.

sudo apt-get install iptables

Verify that you have installed iptables or install it with the above command.

ifconfig -a

protonvpn: flags=209<UP,POINTOPOINT,RUNNING,NOARP>  mtu 1420
        inet 10.2.1.3  netmask 255.255.255.255  destination 10.2.1.3
        unspec 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  txqueuelen 1000  (UNSPEC)
        RX packets 612  bytes 101768 (99.3 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 765  bytes 101144 (98.7 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

First check the name of your network interface. If you named your file protonvpn.conf it should be named protonvpn.

sudo iptables -A OUTPUT -p tcp --dport 51413 ! -o protonvpn -j REJECT --reject-with icmp-port-unreachable
sudo iptables -A OUTPUT -p udp --dport 51413 ! -o protonvpn -j REJECT --reject-with icmp-port-unreachable

you can add any rule to your iptables. The 2 rules below make sure this port 51413 is rejected when the network interface called protonvpn isn’t available. There is a rule for UDP and TCP traffic in place.

sudo iptables -L OUTPUT -v -n

After adding all your required rules you can verify them with this command .

sudo apt-get install iptables-persistent

And make these rules persistent so you dont have to add them on every reboot.

sudo netfilter-persistent save

Conclusion

With this Python script, your Raspberry Pi will always be connected to the fastest ProtonVPN server available, ensuring you have the best possible VPN performance without manual intervention. Whether you’re using your Pi for networking, torrenting, or general online security, this automation ensures optimal performance and reliable privacy protection.

Update - 20-12-2024

Added Preferred_cities as an option to shorten the US country select. That included over 2000 servers to check otherwise. The performance check can at least get a bit shorter this way :-).

And i found out what each Feature section for every server in the API includes. I will explain this below. I adjusted my code accordingly to work with this correctly. Be aware that the config you made earlier should reflect the type of server you are connecting to. If your Configuration isnt configured to connect to a Secure Core dont select a Secure Core in this script also. Reflect to the created ProtonVPN Configuration.

Feature explanation:

Each server has exactly ONE feature number:

• 1, 17 = Secure Core servers
• 2 = TOR servers
• 0, 8, 16, 24 = Plus-only servers
• 4, 12, 20, 28 = P2P servers (these are also Plus servers it seems)

        {
            "Name": "US-AZ#105",
            "EntryCountry": "US",
            "ExitCountry": "US",
            "Domain": "node-us-260.protonvpn.net",
            "Tier": 2,
            "Features": 12,

After some more investigation regarding questions about the Free servers from ProtonVPN, it appears that ProtonVPN has removed public API access to their free server endpoints. The /vpn/servers endpoint that previously should have provided information about free servers now returns a 404 error:

{    "Code": 404,    "Error": "Path not found",    "Details": {}}

Currently, only the Plus server endpoint is accessible via the public

API:url = ‘https://api.protonmail.ch/vpn/logicals’

This is likely a deliberate change by ProtonVPN. So sadly this script cant support the Free servers.

Introduction

This post aims to shed light on database performance within Azure, focusing on a direct comparison between MySQL, MariaDB, and PostgreSQL. Despite abundant literature on the subject, practical tests that measure real-life performance are scarce.

Many discussions revolve around the differences between these databases rather than their actual performance. When I started this research, my priority wasn’t the differences but finding a performant database that could serve as a viable Azure-based alternative for our MariaDB Monolithic setup. Initially, we thought switching from MariaDB to MySQL was a straightforward replacement with almost no performance impact. We were wrong there!

“There are many opinions about performance, But we needed facts”

We encountered a massive regression in performance in every query. Performance is crucial to us, and it was perplexing to see MySQL lag behind MariaDB so much. Upon discovering this, we set up a MariaDB VM ourselves in Azure, which proved to be performant again but introduced a lot of additional maintenance and management challenges, such as ensuring uptime, hot standby, and backups.

This led me to explore PostgreSQL, hoping for better performance than what MySQL offered. With this introduction, let’s dive into the tests we conducted, including a complex query and a set of small queries, to test the performance differences between the three databases and their implementations within Azure. We also included Docker tests to be able to determine whether the performance discrepancies are database-related or if Azure is the contributing factor.

Testing our complex Query

To assess the databases, we ran a complex query that aggregated data from multiple tables, representing a realistic workload for our system. For confidentiality reasons, we’ve altered table and column names in this description. This query aimed to evaluate how each database handles data aggregation and complex joins.

WITH AggregatedSkateData AS (
SELECT
    e.id AS eventID,
    e.theme_cid AS themeCID,
    s.id AS skateID,
    p.firstName || ' ' || p.lastName AS participantName,
    l.label AS locationLabel,
    s.priceOffered,
    s.inspectionCompleted,
    s.registrationDate,
    s.eventStartDate,
    org.name AS organizerName,
    'https://skateeventimages.net/' || im.path AS organizerImageURL,
    s.registrationCode,
    COUNT(DISTINCT CASE WHEN pa_last.status_cid = 0 OR pa_last.id IS NULL THEN sp.id END) AS partsPending,
    COUNT(DISTINCT CASE WHEN pa_last.status_cid = 1 THEN sp.id END) AS partsRequested,
    COUNT(DISTINCT CASE WHEN pa_last.status_cid = 2 THEN sp.id END) AS partsConfirmed,
    COUNT(DISTINCT CASE WHEN pa_last.status_cid = 8 THEN sp.id END) AS partsDenied,
    MAX(CASE WHEN ti_unscanned.id IS NOT NULL THEN 1 ELSE 0 END) AS hasUnscannedTickets,
    s.isPublic
FROM
    skateEvents s
    INNER JOIN events e ON s.eventID = e.id
    INNER JOIN participants p ON p.id = e.participantID
    LEFT JOIN organizers org ON org.id = e.organizerID
    LEFT JOIN images im ON im.id = org.profileImageID
    LEFT JOIN locations l ON l.id = s.locationID
    LEFT JOIN skateParts sp ON sp.eventID = s.id
    LEFT JOIN partApprovals pa ON pa.id = sp.partApprovalID
    LEFT JOIN ticketsUnscanned ti_unscanned ON ti_unscanned.id = sp.ticketID AND ti_unscanned.scannedAt IS NULL
    LEFT JOIN partApprovalActions pa_last ON pa_last.id = (
        SELECT pa2.id
        FROM partApprovalActions pa2
        WHERE pa2.partID = pa.id
        ORDER BY pa2.id DESC
        LIMIT 1
    )
WHERE
    s.registrationDate IS NOT NULL
    AND e.cancelledAt IS NULL
    AND e.deletedAt IS NULL
    AND s.isArchived = false
    AND s.deletedAt IS NULL
GROUP BY
    e.id, e.theme_cid, s.id, p.id, p.firstName, p.lastName, org.id, im.id, l.id, l.label, s.priceOffered, s.inspectionCompleted, s.registrationDate, s.eventStartDate, org.name, im.path, s.registrationCode, s.isPublic
)
SELECT * FROM AggregatedSkateData
WHERE
    TO_CHAR(s.registrationDate, 'YYYY-MM') = '2024-03'
    OR partsDenied + partsRequested + partsPending > 0
ORDER BY s.registrationDate DESC

Azure, complicated query

We conducted these tests within the same Azure zone for containers and databases to avoid interzone latency. Below are the specifications and results for transparency. It’s worth noting that for MariaDB and MySQL, the performance specifications didn’t markedly affect outcomes, provided we avoided the first (burstable) tier. PostgreSQL was the exception, showing noticeable improvements in large query runtimes.

Using HeidiSQL via the Azure VPN, we determined our best results out of five runs.

MySQL Flexible, Zone 2 (Standard_E4ds_v5, 450 iops)
Found rows: 130
Duration for 1 query: 13,875 sec

MariaDB VM, Zone 2 (Standard B4s v2 (4 vcpus, 16 GiB memory))
Found rows: 130
Duration for 1 query: 10,625 sec
23.4% better then MySQL

Postgres Flexible, Zone 3 (Standard_E4ds_v50 p10, 500 iops)
Found rows: 130
Duration for 1 query: 3,953 sec
71.5% better then MySQL

Local Docker, complicated query

Local hardware we tested on

12th Gen Intel(R) Core(TM) i9-12900H, 2900 Mhz, 14 Core(s), 20 Logical Processor(s)
Installed Physical Memory (RAM) 32,0 GB

MySQL 8.3 Docker
Found rows: 130
Duration for 1 query: 12,437 sec

MariaDB 11.4 Docker
Found rows: 130
Duration for 1 query: 9,484 sec

Postgres 16.2 Docker
Found rows: 130
Duration for 1 query: 3,969 sec

This concludes our complicated query tests. It is awesome to see Postgres perform so well here! We used to run in MariaDB before we started our move to Azure. It was very sad to see such bad performance in MySQL the moment we started testing with this new setup.

Testing our set small Query’s

The next test is when we fire more then 20.000 transactions per second against each database which will build us a feed in JSON that takes a certain amount of time to complete. we stress our database better with a lot of query’s which will give us a average runtime for every database.
There is also some PHP logic here since this is a PHP script we run to generate the feed. But it runs on the same hardware/docker image all the time. so the difference in runtime comes from the database.
For this we moved our containers to the same zone as our database is running to also take away potential issues with zone latency.

We are running this script within our Laravel framework. We are running our PHP against Swoole for the best possible performance within PHP 8.

Azure, small query’s

In this test we also introduce container performance aspects .
We run our containers and database again in the same Zone in Azure (Western Europe). When using Azure we run with a lot less performance than we have available in our docker environment.

For this we can only use these results to compare the differences within docker and Azure itself, not between them. Comparing the two together wont mean anything due to this hardware difference. We only want to observe the difference in runtime against the databases. and Docker is a baseline to check if the differences match up with the differences we see in Azure.

Our Node Size in Azure is a standard_b2s, 2 vCPUs with 4 GiB of RAM

API, Zone 2 -> MariaDB VM, Zone 2
24.59s runtime (spot nodes)

API, Zone 2 -> MySQL Flexible, Zone 2
27.87s runtime (spot nodes)

API, Zone 3 -> Postgres Flexible, Zone 3
24.92s runtime (spot nodes)

Local Docker, small query’s

For the small query tests within local Docker we utilize a system powered by a 12th Gen Intel(R) Core(TM) i9-12900H CPU, running at 2900 MHz with 14 cores and 20 logical processors, complemented by 32 GB of installed RAM.

These Docker images are configured to be memory-bound, meaning their performance was primarily dependent on the available memory rather than being limited by the CPU. The CPU settings were configured to allow unrestricted performance, ensuring that Docker can utilize as much CPU resource as needed.

Before each test run against a Database, I made sure to restart the Docker container to refresh its connection. After restarting, I allowed a 30-second pause to ensure that the container had fully stabilized and was running idle. This pause is crucial for ensuring more consistent test conditions. Once the container was ready, I initiated the feed generator script to begin building the feed, assessing how effectively our different database setups could handle small queries under these controlled conditions.

API -> MariaDB 11.4 Docker
22.88s runtime

API -> MySQL 8.3 Docker
23.74s runtime

API -> Postgres 16.2 Docker
20.77s runtime

MariaDB query cache

In our past use of MariaDB within our monolithic setup, we made significant use of the query cache functionality. This feature played a crucial role in delivering performance for repeated queries. This efficiency persists until the cache requires rebuilding, typically triggered by updates or inserts into the cached query’s tables. Lets do a second run on our complex query with Query cache enabled.

Local MariaDB Docker
Found rows: 130
Duration for 1 query: 0,000 sec

However, the performance benefits of the query cache aren’t universally consistent. For instance, when executing an average of 20,000 transactions per second with small queries, the runtime dropped to some extent. This variance highlights that the query cache doesn’t always produce the same level of optimization, particularly under high-load conditions. but more about that topic later.

API -> Local MariaDB Docker (query cache enabled)
17.30s runtime

API, Zone 2 -> MariaDB VM, Zone 2 (query cache enabled)
22.72s runtime (spot nodes)

The query cache in MariaDB works by storing the result set of SELECT queries, along with the query text. When an identical query is executed again (with the same text and under the same database user privileges), MariaDB can bypass the query execution phase, pulling the result directly from the cache. This feature can drastically reduce query execution times for frequently run queries that do not alter data.

It’s important to approach the use of MariaDB’s query cache with caution. Despite its benefits, the query cache was deprecated and disabled by default in newer versions of MariaDB 10.1.7 (and MySQL as of version 5.7.20) due to concerns over its impact on performance in environments with high levels of concurrent transactions. Nonetheless, for specific scenarios, particularly those with a predominant read workload, the query cache can still significantly enhance performance. It is completely removed in MySQL 8 and up.

Observation on Azure’s Flexible Servers

These tests highlight that the performance of databases on Azure’s flexible servers has improved significantly, aligning closely with the performance seen in local Docker environments. This marks a decent improvement from the previous Azure database single servers, which typically exhibited lesser performance due to some form of overhead introduced by how it was it was running in the Azure cloud.

Azure Latency anomaly

As an added bonus, we conducted some additional tests to measure interzone latency when connecting from Zone 1 or 2 to a PostgreSQL database in Zone 3 within Azure.

API, Zone 2 -> Azure PostgreSQL Flexible, Zone 3
The runtime was 44.23 seconds.

Interestingly, the runtime doubled again, a phenomenon observed exclusively when communications occur between Zones 1-2 and Zone 3, or vice versa. This pattern suggests that any container requiring a database connection experiences significant latency under these conditions, whereas latency is nearly negligible when the interaction is between Zone 1 and Zone 2.

For those who might have missed it, I’ve previously discussed a latency anomaly encountered in Zone 3. The conventional wisdom suggests maintaining the database connection within the same zone to minimize latency. However, our tests show that the latency impact is minimal when operating between Zones 1 and 2, but it becomes significantly pronounced when crossing from Zones 1 or 2 to Zone 3.

This testing further confirms that the latency issue is not exclusive to MySQL Flexible services but also affects PostgreSQL Flexible services within Azure at least, indicating a broader pattern across Azure’s database offerings.

Latency Anomaly: An attempt to untangle Azure Performance issues in Western EuropeUnveiling a significant Azure performance issue: Zone 3’s unexpected latency anomaly in the Western Europe region challenges cloud deployment strategies All Subjects MatterRemco Loup.

Conclusion

From a performance related evaluation, it’s clear that choosing the right database is crucial and should match your app’s specific needs for speed and efficiency. The results show MySQL tends to be slower. MariaDB works well and has a handy query cache feature, but Azure has no support for MariaDB anymore. This makes PostgreSQL the most viable choice for performance-focused users in Azure, as it excels in handling both complex and numerous simple queries effectively as shown in the tests above.

This analysis hopefully helps in selecting the right database technology and underlines how important careful planning is for boosting your app performance. For those using Azure and looking for top tier performance, PostgreSQL is the go-to option in my test results.

There are other reasons to choose your database on. But i only wanted to focus on the performance aspect in this post. This was our main concern since we are in the proces of moving from a Monolithic server to Azure. There are more things to consider when switching databases. The advantage here is that all three databases can be integrated quite easily if you use either of them.

Postgres is the best compatible here with its strict routines. The complicated query above is written for Postgres and runs in MariaDB without a single change. For MySQL the TO_CHAR in the last line only needs to be replaced with DATE_FORMAT.

Introduction

As we embark on a transition from a traditional VM setup hosted at TransIP to the scalable cloud infrastructure of Microsoft Azure, our journey is as much about discovery as it is about performance. Guided by the expertise of Pionative, we’re not just moving our environment; we are attempting to reinvent it in Azure, ensuring that every step we take is carefully tested and analyzed. Our goal? To ensure optimal performance in this new architecture and to uncover any potential bottlenecks or hidden issues that might lurk within our new cloud architecture as early as possible.

Azure regions and Zones

Azure’s infrastructure spans the globe, organized into regions for localized services, and further divided into zones for high availability. Regions are specific geographical areas hosting Azure services, while zones are physical locations within these regions, each with independent utilities to isolate from failures.

The Western Europe Region

The Western Europe region in Azure is structured into three distinct availability zones. These zones provide an additional layer of redundancy and fault tolerance by physically separating data centers within the region. This setup is designed to ensure that even in the event of a complete zone failure, the services can continue to operate unaffected in the other zones, offering robust disaster recovery solutions for businesses operating in Europe. Where Zone 1,2 and 3 are exactly located geographically is unknown to me.

Performance anomaly in the Western Europe Region

Our testing regime led us to a anomaly that has become the focal point of our Azure exploration for this post: a distinct latency issue within Zone 3 of the Western Europe region. This discovery wasn’t immediate; it emerged from days of performance testing and a keen observation of patterns that deviated from our expectations. This all because we just started to learn the way’s of Azure. And it’s difference in performance as opposed to our current setup.

We first started to exclude our own pods, code, performance settings within the node pool. Database performance settings and such.

Methodology and findings

Our final approach was quite thorough to verify this result. We deployed two node pools across each of Microsoft Azure’s three zones within the Western Europe region, aiming to rule out any node-specific variances. All these pools had the exact same performance setup.

Through extensive testing, including a battery of database calls and running our cron jobs integral to our application’s functionality, we noticed a consistent and surprising pattern. Irrespective of the zone’s reported capacity or congestion levels.

Interactions with the database were significantly slower when either the database or the connecting pods were situated in Zone 3.

Performance results

Below is our situation running a Cronjob that takes us locally within Docker around 32 seconds to run against our current setup, which isn’t comparable to MS Azure. But it did provide us a performance Matrix and goal. The script (cronjob) we ran fires around 32,000 queries to build a feed from all the products on our website. Yes, that’s a lot, but that is not the matter in this case. We wanted to uncover the performance issue. We know this script can be optimized to perform better.

MySQL Database in Zone 2
Zone 1 – pool 1: 33 seconds
Zone 1 – pool 2: 32 seconds

Zone 2 –  pool 1:  32 seconds
Zone 2 –  pool 2:  32 seconds

Zone 3 –  pool 1: 115 seconds
Zone 3 –  pool 2: 117 seconds

Then, to put the icing on the cake, we switched the setup with a database running in Zone 3. And see below the expected results we hoped to see to conclude this search.

MySQL Database on Zone 3
Zone 1: 120 seconds
Zone 2: 122 seconds
Zone 3: 33 seconds

We started to see the same consistency every time, even when we ran less but larger database calls.

MySQL Database on zone 2
16 DB queries single threaded.

Zone 1: Total time for queries from log: 9.57 ms
Zone 2: Total time for queries from log: 9 ms
Zone 3: Total time for queries from log: 49.96 ms

Website performance in Azure

When we run our Database in Zone 2 while having the API pod in Zone 1, the initial load time for our website decreases by at least 400 milliseconds for the first page load, compared to the configuration where the API pod is in Zone 3.

This means that the setup with the database in Zone 2 and the API pod in Zone 1 results in a much faster initial response time for the website compared to having the API pod located in Zone 3 and the Database in Zone 2.

There is a margin of error here, but we can conclude from these results that when you want to run your API demographically separated in two zones, there is no way you can consistently perform when using Zone 3. Zones 1 and 2 are the way to go.

We must be honest that this performance anomaly only happens when you have a pod with a database connection. Running our website (which has no database connection) in Zone 3 measured almost no impact here when the API pod ran in Zone 1 and the database resided in Zone 2.

Diving deeper in Zone 3’s latency issues.

This performance discrepancy in Zone 3 was astonishing to us. According to Microsoft’s best practices, the physical proximity of application servers to their respective databases within the same region should minimize latency. Our findings, however, painted a different picture, suggesting that within-region zone selection could profoundly impact application performance.

Our investigation was quite comprehensive, eliminating potential variables such as our own codebase, PHP environment configurations, MySQL performance, and much more. The consistent latency issues, particularly affecting PHP’s single-threaded cron jobs, finally pointed us towards a more systemic issue within Zone 3’s network architecture or deployment somewhere.

Collaboration and learning

Our journey, backed by the collaboration with Pionative, underscores the importance of partnership in navigating the cloud’s complexity. As we dove deeper into this latency anomaly, our shared efforts aim not only to optimize our deployment but also to contribute our findings back to a broader Azure community.

This latency impact is too large to ignore in our production environment.

Implications and next steps

This anomaly in Zone 3 opens up discussions on optimal zone selection and deployment strategies within Azure’s Western Europe region. Is this the reason that Zone 3 is the least populated zone? Online there isn’t much to find about this it looks like. So i wrote this post since this cost us literally days to pinpoint this performance difference.

This will impact our production environment. We are going to run our DB in Zone 1 or 2. and the API must reside in either of these zones. Zone 3 can be used for pods that talk with the API.

This latency anomaly in Zone 3 was more then a challenge for us. We took it as a opportunity to learn, adapt, and optimize our knowledge. As we continue to collaborate, test, and refine our approach, our journey into Azure is shaping up to be a very interesting technical exploration which is broadening our toolset for the future.

This post is a follow up on GIT deployment strategies within Kubernetes environments. Make sure you read this post first:

CI/CD pipeline with Docker and Git branching strategyExplore advanced CI/CD with Docker & Git. Master deploying Docker artifacts via Git branching. Dive into strategies for DEV, STAGING, LIVE environments. Perfect your pipeline now! All Subjects MatterRemco Loup.

In a CI/CD pipeline centered around Docker artifacts with the focus on a singular branch, implementing an efficient hotfix strategy requires a small deviation to ensure rapid deployment of critical fixes. Let’s work out our approach to hotfixing in this Docker-centric environment, given the explained branching architecture in the above post.

Identifying the need for a Hotfix

Hotfixes are a critical part of maintaining the stability and reliability of any software in production. They are typically required under specific circumstances that demand immediate attention. Understanding when to implement a hotfix is key to managing your CI/CD pipeline effectively, especially when using Docker artifacts and streamlined branching strategies like focusing primarily on a Single branch. Here are our scenarios where a hotfix can be necessary:

1. Security Vulnerabilities: One of the most urgent triggers for a hotfix is the discovery of a security vulnerability within your application. Whether it’s an exposed data risk, a potential for unauthorized access, or any flaw that could be exploited maliciously, quick action is required to patch the vulnerability and protect your users and data.
2. Critical Bugs Impacting User Experience: If a significant bug slips through your testing processes and reaches production, it can severely impact the user experience. This could be anything from a broken core functionality that prevents users from using your application as intended to severe performance issues leading to timeouts or crashes. If the issue significantly affects your user base, a hotfix is warranted.
3. Compliance Issues: Certain applications may have legal or regulatory requirements. If a compliance issue is identified that could affect your legal standing or user privacy, addressing it swiftly through a hotfix becomes a priority to avoid penalties or breaches of trust.
4. Data Loss Risks: Scenarios where there’s a risk of data loss or corruption for users require immediate attention. Whether it’s a flaw in how data is saved, updated, or retrieved, a hotfix should be deployed to mitigate any potential loss or corruption of user data.
5. Payment or Transaction Failures: For applications dealing with payments or any form of transactions, issues that prevent users from completing transactions or that might lead to financial discrepancies are critical. Such problems not only affect user trust but can also have financial repercussions for both users and the service provider, necessitating a hotfix.

Deploying a Hotfix

When a critical issue is identified within the production environment, the immediate goal is to develop, test, and deploy a hotfix as quickly as possible. Given that our primary branch for development is MAIN and that we generate Docker artifacts for deployment, we need a process that allows for rapid intervention without disrupting the ongoing work.

1. Creating a Hotfix Branch: Instead of branching off from your Production branch (as LIVE is not represented anymore in the Git repository in this strategy), the hotfix branch is created from the commit currently deployed in production. This ensures that the Hotfix is based on the exact state of the version running on the production environment. The specific commit can be identified through tags or Docker image labels that correlate with the production deployment.
2. Developing and Testing the Hotfix: The critical fix is developed within this hotfix branch. The urgency of the situation necessitates a focused approach, targeting the problem with minimal changes to expedite the testing and deployment process. Despite the rush, adherence to the Definition of Done (DoD) is crucial to ensure the stability and reliability of the hotfix. This can mean a condensed version of the DoD, prioritizing critical tests and checks that ensure the fix will not introduce new issues.
3. Creating a Docker Artifact for the Hotfix: Once the hotfix meets the agreed DoD criteria, an automated process builds a Docker image from the hotfix branch. This Docker artifact is then tagged appropriately, indicating its purpose and urgency.
4. Deploying the Hotfix: The newly created Docker artifact is deployed directly into the production environment, bypassing the usual DEV to STAGING to PRODUCTION environment strategy. This deployment is only justified by the critical nature of the fix and the testing it has undergone.
5. Merging Back and Maintaining Consistency: After the hotfix is successfully running in production, it’s essential to merge the changes back into the MAIN branch and any other relevant branches or tags. Only this step ensures that the fix is incorporated into the ongoing development work and future releases, preventing regression!!

Ensuring Rapid Deployment and Minimizing Disruption

• Automated Processes: Automation plays a key role in this strategy, from the creation of the Docker artifact to its deployment. Automated tests, builds, and deployments ensure that the hotfix process is as fast and efficient as possible.
• Communication and Monitoring: Rapid deployment of hotfixes requires clear communication among team members and close monitoring of the production environment post-deployment to ensure the fix is effective and does not introduce new issues.
• Documentation and Review: Documenting the hotfix process and conducting post-mortem reviews help in refining the hotfix strategy, ensuring that the team can respond even more effectively to future issues.

Conclusion

In a CI/CD pipeline using Docker and primarily operating through Git Flow, hotfixing requires a streamlined approach that leverages automation and focuses on rapid, targeted fixes. By creating hotfix branches based on the production-deployed commit, developing and testing fixes according to an well defined DoD, and deploying Docker artifacts directly into production, teams can address critical issues swiftly while ensuring the ongoing stability and integrity of their application.

Let’s dive into managing a CI/CD pipeline using Docker with a focus on Git branching strategy that involves development (DEV), staging (STAGING), and production (LIVE) branches. Our goal is to clarify the role each branch plays in the process, particularly how Docker artifacts are created and managed within this framework.

Docker deployment using Git Flow with multiple branches

The core of our branching strategy involved three branches:

1. DEV: This is where all development work happens. Every new feature, bug fix, or enhancement starts in a feature branch, branched off from DEV. Once the work on a feature branch is complete, reviewed and meets the Definition of Done (DoD), it’s merged back into DEV.
2. STAGING: This branch mirrors a pre-production environment. It’s where integrated features from DEV get tested as a whole and accepted by stakeholders to ensure they work together seamlessly.
3. MASTER: The branch that reflects the current state of our live environment. Only thoroughly tested and stakeholder-approved changes from STAGING make it to Production.

Git Branching and Branching StrategyIn this post, we will go over why branching is required, the difference between development, staging… DEV CommunityPreetham

This was our variation of a Git Flow strategy, we used until now. When you start working with Artifacts this all changes back to a normal Git Flow structure. We still have environments running (Dev, Staging and Prod). The real difference is that code commits get done once instead of getting merged into a branch running on Staging or in Production.

Instead we now have one Branch we called it Main. This is where a runnable artifact gets created after a successful code merge using Docker images. Everything deployed into our Staging or Production environment is a specific version of an earlier created Artifact.

The biggest change is that STAGING and PROD aren’t branches anymore where you merge code into. They have instead became environments that run a specific version of a Docker image. This all has its own challenges, especially when you need to Hotfix. Hotfixing has it’s own post which you can read after this post. Lets break down all of this without introducing hotfixing step by step first. It is comprehensive enough already.

What is an Artifact?

A Docker artifact, often referred to interchangeably with a Docker image, is a lightweight, standalone, executable package that includes everything needed to run a piece of software, including the code, a runtime, libraries, environment variables, and config files. In simple terms, it’s like a snapshot of your application and its environment, bundled together, ready to be deployed and run anywhere Docker is installed, ensuring consistency across different environments.

The Importance of DoD When Merging code

A comprehensive and clear Definition of Done (DoD) plays a pivotal role in this new process, even more when merging changes into a single branch that triggers a Docker image which should be runnable and deployable:

• Quality Assurance: The DoD ensures that all changes, including new features, enhancements, and hotfixes merged back into MAIN, meet a consistent quality standard. This includes code quality checks, successful completion of automated tests, and adherence to security best practices.
• Preventing Regression: By applying the DoD to all merges into MAIN, the team can prevent regressions and ensure that new changes do not negatively affect existing functionality. This is crucial for maintaining the integrity and stability of the codebase.
• Streamlining Deployments: A clear and comprehensive DoD facilitates smoother deployments by ensuring that only fully vetted and ready-to-deploy changes are merged into MAIN. This reduces the risk of wasting resources on building Docker images that are bugged or broken, and therefore contributes to more efficient and reliable artifacts.

Creating and Managing Artifacts

The first thing for this proces to work is when a forked branch is merged back into MAIN, this triggers an automated process in our CI/CD pipeline. Here’s what happens:

1. Automated Testing: First, automated tests run to ensure the new changes meet our quality standards. This is crucial for maintaining a stable branch.
2. Docker Artifact Creation: Upon successful running al tests, a Docker image (our Artifact) is created from the MAIN branch. This is because the MAIN branch is the single branch that represents the latest state of every DONE development, This is where we generate our artifacts from that can become “shippable” products.
3. Why using MAIN for Docker Artifacts?: Creating artifacts at this stage allows us to catch any integration issue. Since every feature merged into MAIN is going to be considered ready for a release, generating an artifact here ensures that we always have a tested and potentially deployable version of our product at hand.

Transferring from Branches to Environments

In traditional Git workflows, branches like STAGING and PROD are used to represent different stages of the development process, from testing to production. However, in a Docker-centric CI/CD pipeline, the focus shifts from using these branches for code integration to using them as references for deployment environments.

Deploying to STAGING

• Creating Docker Artifacts: Once development is complete and merged into the MAIN branch, a Docker artifact is automatically created. This artifact is a snapshot of your application at that point in time, encapsulating all the changes that were merged.
• Deployment, Not Merging: The STAGING environment’s role is to test this Docker artifact in conditions that closely mimic the production environment. Instead of merging code from MAIN to a STAGING branch, you deploy the Docker artifact to a STAGING environment. This means the code itself isn’t moving between branches; rather, the compiled and containerized version of your application is being tested.
• Purpose of STAGING: The STAGING environment is essential for comprehensive testing, including performance and security assessments. It’s a final check to ensure everything works as expected in a production-like setting without directly affecting your actual users.

Going to Production

• Stakeholder Approval: Once your Docker artifact has been rigorously tested in STAGING and approved by stakeholders, it’s ready to move to production. This step is crucial for ensuring that all features, fixes, and updates meet the necessary standards and expectations.
• Deployment to Production: The transition from STAGING to PROD involves deploying the tested Docker artifact to your production environment. This deployment process ensures consistency, as the exact artifact tested in STAGING is what gets deployed to PROD, eliminating any discrepancies between tested and deployed code.

Key takeaways

• No More Branch Merging for Deployment: In this streamlined workflow, STAGING and PROD branches are no longer used for merging code. Instead, they reference specific deployment environments where Docker artifacts are deployed for testing and production.
• Focus on Artifacts: The emphasis is on creating, testing, and deploying Docker artifacts. These artifacts represent stable versions of your application ready for STAGING and, eventually, PROD deployment.
• Consistency and Reliability: By deploying the same Docker artifact from STAGING to PROD, you ensure that what has been tested is exactly what gets released, increasing the reliability and predictability of your deployments.

This approach simplifies the deployment process, reduces the risk of errors, and ensures that every release is thoroughly tested in an environment that mirrors production as closely as possible.

Are multiple Branches still needed?

Given this new workflow, the STAGING and PROD branches in Git branching serve no active purpose anymore and therefore they have became obsolete. They helped track which versions of the code have been deployed to each environment which we have solved by running a specific version of an Artifact.

Conclusion

Incorporating Docker into our Git branching strategy streamlines the development-to-deployment process by ensuring that every merge into MAIN triggers a Docker image with the potential to go to production. The key to this strategy is the creation of Docker artifacts early in the development cycle, allowing for continuous integration and delivery while maintaining high standards of quality and stability.

By understanding the changes to Git branches and the process of creating and deploying Docker artifacts, we can rely on the complexities of CI/CD with confidence, ensuring that all new features and fixes are delivered to users as efficiently and reliably as possible.

Hotfixing

We havent spoken about hotfixing yet. As i said this deserves its own Post. You can read all about Hotfixing and its caveats here.

Hotfixing with Docker Artifacts in a Single Branch StrategyMaster hotfixes in Kubernetes with Docker. Learn to rapidly deploy critical fixes in a CI/CD pipeline focusing on streamlined branching. Perfect for tackling urgent issues without disrupting flow. Dive in now. All Subjects MatterRemco Loup.

Some background to understand our issue better

We are working on a new website with a great emphasis on images. We already have www.333travel.nl which of course being a travel product already heavily relies on images. Our new website has even higher res images loaded because there is a great emphasis on the luxury feel. This site is aimed towards the luxury traveller www.travelplatinum.nl. The project that served our images static.333travel.nl is already a few years running with no reported issues for the 333travel website.

The issue

The issue at hand got reported from our home workers. They noted that when working from home a lot of the images displayed were sometimes broken on TravelPlatinum (and a refresh mostly fixed them). They also replied that it was all working fine when working from within the office. So that made us start a quest in finding the culprit.

We noticed that in the browsers we use (Edge and Chrome) there were reports from the failed images in the console which reported. ERR_HTTP2_PROTOCOL_ERROR. With that message in the pocket we started our search for the issue searching through the logs on the server. We are running Nginx as our webserver.

We quickly found a trail that appeared for every image that we thought didn’t load correctly. Since we weren’t able to simulate the issue ourselves yet.

0.0.0.0 - - [06/Nov/2023:15:45:02 +0100] “GET /web-images/10/2023/653775730fc66/Jazan—Al-qahar-mountains-1200.webp HTTP/2.0” 206 183064 “https://www.travelplatinum.nl/” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36 Edg/119.0.0.0”

Pay attention to the 206 call here. This means partial content loaded.

https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/206

This was odd since our webserver didn’t support partial loading of images. We never send headers that replied with something like a range which are required to be able to do partial content.

Our attempt to fix the issue

So that was the first issue we dove in. I found out that Nginx by default sends a header that replies Content-Type: bytes. I don’t know why I would want our image server to respond to a 206 request with an ok message. So we disabled this with the following setting in our server block.

server {
server_name static.333travel.nl;
max_ranges 0;
}

We replied back to the persons experiencing this issue to report to us if it happened again. Which sadly was quickly the case. The error message did change in our logging. Edge and chrome kept spitting out the same useless ERR_HTTP2_PROTOCOL_ERROR in the console tab. In the timeline we also saw these images stopping to load at a certain point.

0.0.0.0 - - [08/Nov/2023:14:22:27 +0100] “GET /web-images/06/2020/5ed7ad48d3d79/mexico-yucatan-chichen-itza-ik-kil-cenote.jpeg HTTP/2.0” 200 0 “https://vue-boss.333office.nl/” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36 Edg/119.0.0.0”

We now get a header with a length of 0 bytes and a response with http 200 (ok). This is getting real strange here. The header when inspected in the browser returns the right amount of bytes in the response header. So why does the log of Nginx state it replies with a size of 0. This is where things got weird and we didn’t really know what to do next. We first had to be able to reproduce the issue ourself so we could really start to troubleshoot.

I tried to simulate a fault with ab (apache benchmark). I started conservative with 500 requests and 50 concurrent requests. The issue never triggered for me. i then moved in small steps upwards to end on 10.000 requests with 700 concurrent requests without ever triggering the reported issue. The server load also appears to handle everything fine. It was an image of around 500kb i requested there.

The Facts

1. Setting a slow profile in the network tab when inspecting didn’t simulate the issue for us.
2. The error was still only reported working from home from 2 users.
3. When in the office they haven’t had a single issue.
4. Loading the site 333travel.nl didn’t report the issue at all. Every image worked there also for the 2 persons working from home.
5. Using a 5g connection testing from all various places we could think off didn’t simulate the issue once for me.
6. Using ab (apache benchmark) doing 10.000 requests with 700 concurrent requests never triggered the issue at hand.

There was still one difference and that was the domain serving the images. 333travel got replied with a Cross origin header of same_site which of course is true since the domain is the same. So I added a header to the static.

server {
server_name static.333travel.nl;
max_ranges 0;
add_header ‘Access-Control-Allow-Origin’ ‘*’;
}

This way every source wanting to use images from our static.333travel.nl gets the same reply. This didn’t solve anything about the issue reported.

We still experienced issues to simulate the issue ourself. We were pretty sure that this only got triggered on a somehow bad connection visiting TravelPlatinum. Visiting 333travel which served the images from the same static server never showed the issue we are experiencing on TravelPlatinum.

Simulating a bad WiFi connection

We badly wanted to simulate a bad WiFi connection. I moved myself to a spot where there was bad WiFi outside of our office and I started to finally see the issue for myself. Finally something to start our testing on. While it was still odd we needed a somehow bad/slow WiFi to simulate this. You would say setting your profile to slow 3g in the browsers developer tools would behave the same.

Now we finally were able to start testing from our development environment. We started to suspect the project itself doing something weird with maybe something javascript related closing the connection or something. So we started to create a stripped down website from TravelPlatinum to test without javascript.

The no image loading issue was still there..

We decided to take in the exact same html from 333travel.nl to test with that to create a same environment. Since 333travel still was going strong every refresh.

The no image loading issue was still there..

We badly wanted to be able simulate this even better since our bad WiFi spot wasn’t at a desk or something, we were sitting on a bunch of stairs debugging the issue from there. I decided to create a 2^nd WiFi hotspot in our office with a throttled connection of 3mbit and see where that takes us. This was it! We could now simulate every refresh a lot of broken images on TravelPlatinum! We are getting somewhere at least. Bear in mind that the 333travel website still was loading every image with no issues.

We moved away from the nginx logs at this moment in time since we uncovered that going in that direction nothing was going to be solved at this moment. Nginx still replied an ok on a header with 0 bytes. If that’s ok for nginx with the right headers in place, that message is not going to help us solve the issue at this moment.

Testing with Firefox

The moment I created this WiFi hotspot we were just about ready to test images with just loading the html image tag and nothing more loaded in the project. This still broke our images. I then suggested to fire up Firefox just to see if that behaved differently since I suspected it to be a browser related thing on how it responds on what is happening. Or at least it could output maybe a different error. Something we could focus our search for the solution on.

We saw that Firefox was indeed loading the images line by line on the 3mbit connection. But it suddenly stopped loading leaving broken images in place which were about 50% loaded or not at all. This was finally something of a change to see! Still Edge/Chrome displayed no image at all and gave me a useless error message. I now finally started to realize that the static was actually sending the images and the browser received them but the connection got closed to soon somehow.

The solution

And that was when I finally found the issue at hand. A header i already set ages ago on a global level in Nginx in a performance tuning session.

send_timeout 2;

What send timeout in my opinion does is keeping the connection open for X seconds after not receiving data. The default is 60 seconds and I adjusted it to 2 seconds so we would not have open connections for a whole 60 seconds doing nothing.

Why would that be the issue since I clearly saw Firefox visually loading the image line by line? There is constant data being send then right? Why would this setting then be related to closing the connection.

Sets a timeout for transmitting a response to the client. The timeout is set only between two successive write operations, not for the transmission of the whole response. If the client does not receive anything within this time, the connection is closed.

Well it appears this does get triggered on slower connections. This is getting interpreted as a stalled request by Nginx. And it closes the whole connection to in our case the static.333travel.nl. So the partially loaded image and every image after that not being loaded wont get loaded again since the connection to static is closed.

Why simulating a slow connection in the developer tools wont trigger this issue is still something I don’t know for sure. I have some theories, a bad WiFi also introduces more latency and lost packets getting resend. But I don’t know the real reason. I dont know how these browsers simulate a slow connection.

Setting this send_timeout to 8 seconds as below solved our issue completely.

server {
server_name static.333travel.nl;
max_ranges 0;
add_header ‘Access-Control-Allow-Origin’ ‘*’;
send_timeout 8;
}

Conclusion

We have learned a lot, and we solved the issue. The odd thing remains that all this time it only triggered on our TravelPlatinum project. We were obviously balancing on the limit with the send_timeout setting since the image load is greater on TravelPlatinum.

Another important thing to note, setting a slow connection in your developer tools is not a real simulation from a slow connection. You can simulate all you want but you need real life cases to really measure if everything is working fine.

Notepad++ regexp

For example checking your php-fpm access/slow log file. Use mark in notepad++ to mark all lines that are slower then x/y.

10000ms en trager 

^.*\s(\d{5,}\.\d{3}ms).*

5000ms en trager 

^.*\s(5\d{3}\.\d{2,}ms|\d{5,}\.\d{2,}ms).*

DNS tools to automatically check the transfer of your DNS

sudo apt update
sudo apt install dnsutil

nano check_nameservers.sh

#!/bin/bash

DOMAIN="jouwdomein.com"
EXPECTED_NS1="ns1.cloudflare.com."
EXPECTED_NS2="ns2.cloudflare.com."

# Get the current nameservers
CURRENT_NS=$(dig NS $DOMAIN +short)

# Check if the current nameservers match the expected ones
if [[ $CURRENT_NS == *"$EXPECTED_NS1"* && $CURRENT_NS == *"$EXPECTED_NS2"* ]]; then
    echo "Nameservers are correctly set to Cloudflare."
else
    echo "Nameservers have not yet switched to Cloudflare."
fi

chmod +x check_nameservers.sh

watch -n 1 ./check_nameservers.sh

Update (13-11-2024)

Somehow i lost the creating of a new image to copy all the builder items from. It changed nothing about the way the image worked but the image size is much smaller now. It is under 100mb from more then 2gb. Somewhere in my previous debugging we lost this step. So the lastest update can be found below again.

Latency Anomaly: An attempt to untangle Azure Performance issues in Western EuropeUnveiling a significant Azure performance issue: Zone 3’s unexpected latency anomaly in the Western Europe region challenges cloud deployment strategies All Subjects MatterRemco Loup.

Introduction

In here you will find a working Dockerfile I created with the below defined compatibility needs for myself. This is posted here for my own archive and to share of course. I added the required support for the following.

• HTTP/2 HPACK
• HTTP/3
• Boringssl
• Brotli support

This version of Nginx supports HTTP/3 quic and Brotli compression as a main benefit with some other fixes. Not having Brotli support these days is just a step back in the delorean.

More info about how HPACK accelerates your HTTP/2. And here you can read more details about BoringSSL

Brotli compared to GZIP

As mentioned this is a Nginx version which supports Brotli. Brotli is very useful as it is a better compression algorithm then GZIP. If you are still not convinced watch the video below.

Nginx base Docker file

This is the base of my dockerfile which takes care of all the things mentioned above. You can of course edit this or use different versions of alpine or Nginx regarding this wont break building the other modules. I also updated Nginx, NJS and BoringSSL to the latest versions.

I use this image instead of the official Nginx to load all my web projects. See this as a base image to define all your running webserver projects in. Only you get “free” Brotli compression and HTTP/3 support with it so you can make your sites faster and more up to date.

##################################################
# Nginx with Quiche (HTTP/3) and Brotli
# VERSION 1.05 / 13-11-2024
##################################################
# This build compiles Nginx with Brotli and
# HTTP/3 support. It uses a multi-stage build to
# minimize the final image size by copying only
# the necessary artifacts from the builder image.
# Based on Alpine for a lean and efficient image.
##################################################

# Builder Stage
FROM alpine:latest AS builder

# Set the versions of Nginx, NJS, and BoringSSL
ENV NGINX_VERSION=1.27.2
ENV NJS_VERSION=0.8.7
ENV BORINGSSL_VERSION=c59bf8bf189dcbde868e04efcd53b705ed155231
ENV BORINGSSL="/tmp/boring-nginx"

# Build-time metadata
ARG BUILD_DATE
ARG VCS_REF

# Install dependencies for building Nginx
RUN addgroup -S nginx \
  && adduser -D -S -h /var/cache/nginx -s /sbin/nologin -G nginx nginx \
  && apk update \
  && apk upgrade \
  && apk add --no-cache ca-certificates openssl \
  && update-ca-certificates \
  && apk add --no-cache --virtual .build-deps \
      gcc libc-dev make pcre-dev zlib-dev linux-headers gnupg \
      libxslt-dev gd-dev geoip-dev perl-dev \
  && apk add --no-cache --virtual .brotli-build-deps \
      autoconf libtool automake git g++ cmake go perl rust cargo patch \
      libxml2-dev byacc flex libstdc++ libmaxminddb-dev lmdb-dev file openrc

# Install and build BoringSSL
RUN mkdir $BORINGSSL \
  && cd $BORINGSSL \
  && git clone https://boringssl.googlesource.com/boringssl \
  && cd boringssl \
  && git checkout -q ${BORINGSSL_VERSION} \
  && mkdir build \
  && cd build \
  && cmake -DBUILD_SHARED_LIBS=1 .. \
  && make

# Prepare BoringSSL for Nginx compilation
RUN mkdir -p "$BORINGSSL/boringssl/.openssl/lib" \
  && cd "$BORINGSSL/boringssl/.openssl" \
  && ln -s ../include include \
  && cd "$BORINGSSL/boringssl" \
  && cp "build/crypto/libcrypto.so" ".openssl/lib" \
  && cp "build/ssl/libssl.so" ".openssl/lib" \
  && cp ".openssl/lib/libssl.so" /usr/lib/ \
  && cp ".openssl/lib/libcrypto.so" /usr/lib/

# Clone Nginx modules
RUN mkdir /usr/src \
  && cd /usr/src \
  && git clone --depth=1 --recursive --shallow-submodules https://github.com/google/ngx_brotli \
  && git clone --branch $NJS_VERSION --depth=1 --recursive --shallow-submodules https://github.com/nginx/njs

# Build Nginx with modules
RUN cd /usr/src \
  && wget -qO nginx.tar.gz https://nginx.org/download/nginx-$NGINX_VERSION.tar.gz \
  && tar -zxC /usr/src -f nginx.tar.gz \
  && rm nginx.tar.gz \
  && cd /usr/src/nginx-$NGINX_VERSION \
  && mkdir /root/.cargo \
  && echo $'[net]\ngit-fetch-with-cli = true' > /root/.cargo/config.toml \
  && ./configure --prefix=/etc/nginx \
      --sbin-path=/usr/sbin/nginx \
      --modules-path=/usr/lib/nginx/modules \
      --conf-path=/etc/nginx/nginx.conf \
      --error-log-path=/var/log/nginx/error.log \
      --http-log-path=/var/log/nginx/access.log \
      --pid-path=/var/run/nginx.pid \
      --lock-path=/var/run/nginx.lock \
      --http-client-body-temp-path=/var/cache/nginx/client_temp \
      --http-proxy-temp-path=/var/cache/nginx/proxy_temp \
      --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp \
      --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp \
      --http-scgi-temp-path=/var/cache/nginx/scgi_temp \
      --user=nginx \
      --group=nginx \
      --with-pcre-jit \
      --with-http_ssl_module \
      --with-http_realip_module \
      --with-http_addition_module \
      --with-http_sub_module \
      --with-http_dav_module \
      --with-http_flv_module \
      --with-http_mp4_module \
      --with-http_gunzip_module \
      --with-http_gzip_static_module \
      --with-http_random_index_module \
      --with-http_secure_link_module \
      --with-http_stub_status_module \
      --with-http_auth_request_module \
      --with-http_xslt_module=dynamic \
      --with-http_image_filter_module=dynamic \
      --with-http_geoip_module=dynamic \
      --with-http_perl_module=dynamic \
      --with-threads \
      --with-stream \
      --with-stream_ssl_module \
      --with-stream_ssl_preread_module \
      --with-stream_realip_module \
      --with-stream_geoip_module=dynamic \
      --with-http_slice_module \
      --with-mail \
      --with-mail_ssl_module \
      --with-compat \
      --with-file-aio \
      --with-http_v2_module \
      --with-http_v3_module \
      --add-module=/usr/src/ngx_brotli \
      --add-module=/usr/src/njs/nginx \
      --with-cc-opt="-I$BORINGSSL/boringssl/include -Wno-error" \
      --with-ld-opt="-L$BORINGSSL/boringssl/build/ssl -L$BORINGSSL/boringssl/build/crypto" \
      --with-select_module \
      --with-poll_module \
      --build="docker-nginx-http3-$VCS_REF-$BUILD_DATE ngx_brotli-$(git --git-dir=/usr/src/ngx_brotli/.git rev-parse --short HEAD) njs-$(git --git-dir=/usr/src/njs/.git rev-parse --short HEAD)" \
  && touch "$BORINGSSL/boringssl/.openssl/include/openssl/ssl.h" \
  && make -j$(getconf _NPROCESSORS_ONLN) \
  && make install \
  && rm -rf /etc/nginx/html/ \
  && mkdir /etc/nginx/conf.d/ \
  && mkdir -p /usr/share/nginx/html/ \
  && install -m644 html/index.html /usr/share/nginx/html/ \
  && install -m644 html/50x.html /usr/share/nginx/html/ \
  && ln -s /usr/lib/nginx/modules /etc/nginx/modules \
  && strip /usr/sbin/nginx* \
  && strip /usr/lib/nginx/modules/*.so \
  && rm -rf /etc/nginx/*.default /etc/nginx/*.so \
  && rm -rf /usr/src \
  && apk add --no-cache --virtual .gettext gettext \
  && mv /usr/bin/envsubst /tmp/ \
  \
  # Determine runtime dependencies
  && runDeps="$( \
      scanelf --needed --nobanner /usr/sbin/nginx /usr/lib/nginx/modules/*.so /tmp/envsubst \
      | awk '{ gsub(/,/, "\nso:", $2); print "so:" $2 }' \
      | sort -u \
      | xargs -r apk info --installed \
      | sort -u \
    )" \
  && echo "$runDeps" > /nginx-run-deps.txt \
  \
  # Cleanup
  && apk del .brotli-build-deps .build-deps .gettext \
  && rm -rf /root/.cargo \
  && rm -rf /var/cache/apk/* \
  && mv /tmp/envsubst /usr/local/bin/ \
  && mkdir -p /etc/ssl/private \
  && openssl req -x509 -newkey rsa:4096 -nodes -keyout /etc/ssl/private/localhost.key -out /etc/ssl/localhost.pem -days 365 -sha256 -subj '/CN=localhost' \
  && ln -sf /dev/stdout /var/log/nginx/access.log \
  && ln -sf /dev/stderr /var/log/nginx/error.log

# Final Stage
FROM alpine:latest

# Copy Nginx and related files from the builder stage
COPY --from=builder /usr/sbin/nginx /usr/sbin/nginx
COPY --from=builder /etc/nginx /etc/nginx
COPY --from=builder /usr/lib/nginx /usr/lib/nginx
COPY --from=builder /usr/share/nginx /usr/share/nginx
COPY --from=builder /usr/local/bin/envsubst /usr/local/bin/envsubst
COPY --from=builder /var/log/nginx /var/log/nginx
COPY --from=builder /var/cache/nginx /var/cache/nginx
COPY --from=builder /var/run /var/run
COPY --from=builder /etc/ssl /etc/ssl
COPY --from=builder /usr/lib/libssl.so* /usr/lib/
COPY --from=builder /usr/lib/libcrypto.so* /usr/lib/
COPY --from=builder /nginx-run-deps.txt /nginx-run-deps.txt

# Install runtime dependencies
RUN apk add --no-cache ca-certificates openssl libstdc++ libgcc \
  && apk add --no-cache $(cat /nginx-run-deps.txt) \
  && rm /nginx-run-deps.txt \
  && update-ca-certificates \
  \
  # Forward logs to Docker
  && ln -sf /dev/stdout /var/log/nginx/access.log \
  && ln -sf /dev/stderr /var/log/nginx/error.log

# Create nginx user and group
RUN addgroup -S nginx \
  && adduser -D -S -h /var/cache/nginx -s /sbin/nologin -G nginx nginx

# Expose ports
EXPOSE 80 443

STOPSIGNAL SIGTERM

CMD ["nginx", "-g", "daemon off;"]

Nginx.conf sample

Here you will find a sample nginx.conf which give you an idea of how i run my nginx environment.

The most important difference in this file compared to the normal nginx.conf should be the brotli compression settings.

user nginx;
worker_processes auto;

error_log  /var/log/nginx/error.log notice;
pid        /var/run/nginx.pid;

include /etc/nginx/modules-enabled/*.conf;

events {
	worker_connections 1024;
}

http {
	##
	# Basic Settings
	##
	sendfile on;
	tcp_nopush on;
	tcp_nodelay on;
	keepalive_timeout 65;
	types_hash_max_size 2048;

	proxy_connect_timeout  300s;
	proxy_send_timeout  300s;
	proxy_read_timeout  300s;
	fastcgi_send_timeout 300s;
	fastcgi_read_timeout 300s;

	include /etc/nginx/mime.types;
	default_type application/octet-stream;

	##
	# SSL Settings
	##
	ssl_protocols TLSv1.2 TLSv1.3;
	ssl_prefer_server_ciphers on;

	##
	# Logging Settings
	##
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

	##
	# Gzip Settings
	##
	gzip on;
	gzip_vary on;
	gzip_proxied any;
	gzip_comp_level 6;
	gzip_buffers 16 8k;
	gzip_http_version 1.1;
	gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;

	##
	# Brotli Settings (modify the brotli_comp_level to match your optimal cpu load vs compression)
	##
	brotli on;
	brotli_static on;
	brotli_comp_level 11;
	brotli_types
	text/plain
	text/css
	text/xml
	text/javascript
	text/x-component
	application/xml
	application/xml+rss
	application/javascript
	application/json
	application/atom+xml
	application/vnd.ms-fontobject
	application/x-font-ttf
	application/x-font-opentype
	application/x-font-truetype
	application/x-web-app-manifest+json
	application/xhtml+xml
	application/octet-stream
	font/opentype
	font/truetype
	font/eot
	font/otf
	font/woff
	font/woff2
	image/svg+xml
	image/x-icon
	image/vnd.microsoft.icon
	image/bmp;

	##
	# Virtual Host Configs
	##
	include /etc/nginx/conf.d/*.conf;
}

Site.conf sample

You need to watch out that the higher you set the brotli compression the more cpu intensive it becomes. TTFB (time to first byte) can become an issue when the compression ratio is set to high with to slow kubernetes/ high loads or everything in between. so you need to test on real world scenario’s and adjust accordingly. This is not only a Brotli issue but Gzip also has the same disadvantages when setting compression levels to high while the decrease in file size isn’t worth it.

You can use this file to take all the HTTP/3 settings defined here below. This will optimize and enable HTTP/3 support for your site.

server {
       	# HTTP/3 and Quic Listen
      	listen 443 quic reuseport;
        http3 on;

        # HTTP/3 Add Alt-Svc header negotiation.
        add_header alt-svc 'h2=":443"; ma=86400, h3-29=":443"; ma=86400, h3=":443"; ma=86400' always;
        
        # HTTP/2 Listen
        listen 443 ssl http2;
        http2 on;

        # Certificate locations for HTTPS.
	    ssl_certificate /etc/ssl/certs/wildcard-cert-23.crt;
        ssl_certificate_key /etc/ssl/private/wildcard-cert-23.key;

        # Enable SSL settings for performance.
        ssl_session_cache shared:SSL:50m;
        ssl_session_timeout 1d;
        ssl_session_tickets off;
        # Enable SSL settings for security.
        ssl_prefer_server_ciphers on;
        ssl_protocols TLSv1.3;
        ssl_ciphers HIGH:!aNULL:!MD5;

        # Base settings (root and hostname).
        root /var/www/html;
        index index.php index.html index.htm;
        server_name kubernetes.servername.nl;

        # HTTP/2 settings
        http2_push_preload on;
        # /End of HTTP/2.

        # HTTP/3 settings
        # Enable TLSv1.3's 0-RTT. Use $ssl_early_data when reverse proxying to
        # prevent replay attacks.
        # @see: http://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_early_data
        ssl_early_data on;
        quic_retry on;
        quic_gso on;
        quic_host_key /etc/ssl/private/wildcard-cert-23.key;
        # /End of HTTP/3.

        location / {
                try_files $uri $uri/ /index.php?$query_string;
	    }

        location ~ \.php$ {
                try_files $uri =404;
                # Bunch of PHP and FPM settings
                fastcgi_split_path_info ^(.+\.php)(/.+)$;
                fastcgi_pass your-php-fpm-instance:9000;
                fastcgi_index index.php;
                include fastcgi_params;
                fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
                fastcgi_param PATH_INFO $fastcgi_path_info
                
                # Add a bunch of other headers based on your desired settings.
                add_header X-protocol $server_protocol always;   
                add_header X-Cache $upstream_cache_status always;
                add_header X-Frame-Options SAMEORIGIN always;
                add_header X-Content-Type-Options nosniff always;
                add_header X-XSS-Protection "1; mode=block";
                add_header Strict-Transport-Security "max-age=31536000; includeSubdomains; preload" always;
                add_header Referrer-Policy strict-origin-when-cross-origin;
                add_header Pragma public;
                add_header Cache-Control "public";
        }
}

Multiple H3 Connections from a single Nginx

Here i will explain how you can make multiple server blocks within Nginx work with HTTP/3. This has some caveats since http3 uses UDP with the setting reuseport which makes it unable to assign port 443 multiple times when using HTTP/3

add_header alt-svc 'h2=":443"; ma=86400, h3-29=":443"; ma=86400, h3=":443"; ma=86400' always;

This part of the header informs that our HTTP/3 connection accepts port 443. When you have a single nginx container setup and you have more server blocks configured you will get issues regarding the reuse for port 443.

With Http3 you can only define a single port once within a server block. So if you have more server blocks you have the option to let your website communicate over different ports then 443 using this alt-svc header which tells the browser it accepts HTTP/3 and on which port.

add_header alt-svc 'h2=":443"; ma=86400, h3-29=":443"; ma=86400, h3=":443"; ma=86400' always;

add_header alt-svc 'h2=":443"; ma=86400, h3-29=":444"; ma=86400, h3=":444"; ma=86400' always;

Check out the differences in the h3 ports here. This way you can make multiple server blocks use h3 by specifying unique ports.

All you need to do is make your instance listen to the right port specified in your alt-svc header. If there is a successful connection possible the browser will try to switch to h3. Below a sample how you can listen to port 443 and 444 with HTTP3

server {

        server_name http3-port-443

       	# HTTP/3 and Quic Listen
      	listen 443 quic reuseport;
        http3 on;

        # HTTP/3 Add Alt-Svc header negotiation.
        add_header alt-svc 'h2=":443"; ma=86400, h3-29=":443"; ma=86400, h3=":443"; ma=86400' always;
        
        # HTTP/2 Listen
        listen 443 ssl http2;
        http2 on;
        
        # HTTP/3 settings
        # Enable TLSv1.3's 0-RTT. Use $ssl_early_data when reverse proxying to
        # prevent replay attacks.
        # @see: http://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_early_data
        ssl_early_data on;
        quic_retry on;
        quic_gso on;
        quic_host_key /etc/ssl/private/wildcard-cert-23.key;
        # /End of HTTP/3.

        location / {
                try_files $uri $uri/ /index.php?$query_string;
	}
}

server {

        server_name http3-port-444

       	# HTTP/3 and Quic Listen
      	listen 444 quic reuseport;
        http3 on;

        # HTTP/3 Add Alt-Svc header negotiation.
        add_header alt-svc 'h2=":443"; ma=86400, h3-29=":444"; ma=86400, h3=":444"; ma=86400' always;
        
        # HTTP/2 Listen
        listen 443 ssl http2;
        http2 on;
        
        # HTTP/3 settings
        # Enable TLSv1.3's 0-RTT. Use $ssl_early_data when reverse proxying to
        # prevent replay attacks.
        # @see: http://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_early_data
        ssl_early_data on;
        quic_retry on;
        quic_gso on;
        quic_host_key /etc/ssl/private/wildcard-cert-23.key;
        # /End of HTTP/3.

        location / {
                try_files $uri $uri/ /index.php?$query_string;
	}
}

It is that easy to make your Reverse proxy or multiple server blocks establish a H3 connection. You do have to make the right port forwards within your firewall/network also.

When using docker you configure this as shown below. When specifying UDP you have to specify every port seperately in docker. Below is my sample which includes UDP ports 443 up to 449.

services:
  ############################################################
  # proxy webservice to connect to multiple nginx instances
  ############################################################
  reverse-proxy:
    image: custom-build-nginx:latest
    restart: unless-stopped
    ports:
      - '80:80'
      - '443:443'
      - '443:443/udp'
      - '444:444/udp'
      - '445:445/udp'
      - '446:446/udp'
      - '447:447/udp'
      - '448:448/udp'
      - '449:449/udp'
    volumes:

Update (03-04-2024)

Added some versioning to my dockerfile to better compare versions. I also cleaned up the build from yesterday and finally decided to add a ENV for setting the boringssl version also. Since their build broke my dockerfile we better update it manually. I also updated nginx to 1.25.4 and NJS to 0.8.3 and verified the build

Update (02-04-2024)

I only recently found out that an update to BoringSSL done around 28 february broke my build of BoringSSL. below the updated dockerfile that compiles again. I had to build libssl as a shared library for it to function again. I will cleanup this image a bit more but it is at least working again.

Update (18-03-2024)

Added a section to explain how you can use multiple H3 connections when you have a Nginx that has more than 1 server block. This also holds the key when you use Nginx as a Reverse Proxy and want to make use of H3.

Update (04-09-2023)

I found out that this build didnt compile anymore due to a change in the brotli source code? It might had something to do with updates from github i didnt version in this Dockerfile. I completely reworked this file to work with the updated Nginx 1.25.2 which has “built in” HTTP/3 support. I also made sure we have native Brotli support again. This should keep functioning from now on.

Update (30-08-2023)

I removed the modsec support there was before. This was giving random Errors and since i wont use it i removed it. It might return later.

Introduction

In this post i will explain my particular setup of how i use my windows terminal along with some other tips. So if you want to impress someone while working in your favorite coffee place by making them feel like you work for Nasa. You are in the right place! All jokes aside, working with something day in day out it at least may look more appealing if possible, right?

Lets start with a list below of everything i have installed after which i will explain what they all do.

ZSH and the p10k theme

GitHub - ohmyzsh/ohmyzsh: 🙃 A delightful community-driven (with 2,100+ contributors) framework for managing your zsh configuration. Includes 300+ optional plugins (rails, git, macOS, hub, docker, homebrew, node, php, python, etc), 140+ themes to spice up your morning, and an auto-update tool so that makes it easy to keep up with the latest updates from the community.🙃 A delightful community-driven (with 2,100+ contributors) framework for managing your zsh configuration. Includes 300+ optional plugins (rails, git, macOS, hub, docker, homebrew, node, php, pyth… GitHubohmyzsh

GitHub - romkatv/powerlevel10k: A Zsh themeA Zsh theme. Contribute to romkatv/powerlevel10k development by creating an account on GitHub. GitHubromkatv

Quick start guide | SpaceVimA quick start guide which will tell you how to install and configure SpaceVim, also provides a list of resources for learning SpaceVim.SpaceVimShidong Wang

I have installed the above items for the look and feel of my Shell. I use Windows Terminal and this makes my terminal look good and distinguishes the right information with colors added to the shell.

ohmyzshis the framework for managing my ZSH configuration. The theme i installed is p10k. This is a highly customizable theme.

SpaceVimis one the of the best visual editors for editing everything you need in Linux. Go check it out by using the link.

Here a screenshot of how my personal shell looks like. I will also share the Json config below of my windows terminal. The font i use is the default nerd font which comes with p10k. This is the Meslo Gorgeous monospace Nerd font.

The Colors i use are called Atom One Dark. And are a subtle pastel colored font

In the json file below you can find some nice benefits in making your experience even more customized.

Kali linux

I have Kali linux installed working in a seamless overlay. This means you can run Kali visually native within your Windows. See the screenshot below how neat this looks! Follow the link below for how to get this working with Kali.

Win-KeX | Kali Linux DocumentationWindows Subsystem for Linux 2 & Win-KeX Kali Linux

"commandline": "wsl -d kali-linux kex --sl --wtstart -s",

External SSH in Windows Terminal

You can also easily login to (external) servers requiring tokens with Windows Terminal.

"commandline": "ssh -i \"~/.ssh/id_rsa\" username@ip-address",

Look at how to configure this using the link below.

Windows Terminal config

Here is the json for the Windows Terminal with the last part the color theme i am using. and the settings for the look and feel i use. Also present are examples for running kali seamless and an SSH session with tokens.

    "profiles": 
    {
        "defaults": 
        {
            "antialiasingMode": "cleartype",
            "backgroundImage": null,
            "backgroundImageAlignment": "bottom",
            "backgroundImageOpacity": 0.8,
            "colorScheme": "Atom One Dark",
            "cursorShape": "vintage",
            "experimental.retroTerminalEffect": false,
            "font": 
            {
                "face": "MesloLGS NF"
            },
            "opacity": 75,
            "useAcrylic": true
        },
        "list": 
        [
            {
                "commandline": "wsl -d kali-linux kex --sl --wtstart -s",
                "guid": "{55ca431a-3a87-5fb3-83cd-11ececc031d2}",
                "hidden": false,
                "icon": "C:\\Users\\icons\\kali-menu.png",
                "name": "Kali Seamless overlay",
                "startingDirectory": "%USERPROFILE%"
            },
            {
                "commandline": "ssh -i \"~/.ssh/id_rsa\" username@ip-address",
                "guid": "{0caa0dad-35be-5f56-a8ff-afceaeaa6103}",
                "hidden": false,
                "name": "External test server"
            }
        ]
    },
    "schemes": 
    [
        {
            "background": "#282C34",
            "black": "#000000",
            "blue": "#61AFEF",
            "brightBlack": "#5C6370",
            "brightBlue": "#61AFEF",
            "brightCyan": "#56B6C2",
            "brightGreen": "#98C379",
            "brightPurple": "#C678DD",
            "brightRed": "#E06C75",
            "brightWhite": "#FFFFFF",
            "brightYellow": "#D19A66",
            "cursorColor": "#FFFFFF",
            "cyan": "#56B6C2",
            "foreground": "#CCCCCC",
            "green": "#98C379",
            "name": "Atom One Dark",
            "purple": "#C678DD",
            "red": "#E06C75",
            "selectionBackground": "#FFFFFF",
            "white": "#ABB2BF",
            "yellow": "#D19A66"
        }
    ],
    "themes": [],
    "useAcrylicInTabRow": true

Addins which make your shell look and function even better

Below you will find some more links which are going to make your shell experience even better. It is a collection of tools i use daily. You can already look at the bookmarks and get an idea of what it does. I would recommend to read through every link and decide if this is also going to make your life easier.

I personally like Exa the most. This is a replacement for the ’ls’ command which creates a lot more visibility using colors and icons for folders and such. Also htop is something i already use for years as the goto process manager.

GitHub - BurntSushi/ripgrep: ripgrep recursively searches directories for a regex pattern while respecting your gitignoreripgrep recursively searches directories for a regex pattern while respecting your gitignore - GitHub - BurntSushi/ripgrep: ripgrep recursively searches directories for a regex pattern while respec… GitHubBurntSushi

GitHub - nvbn/thefuck: Magnificent app which corrects your previous console command.Magnificent app which corrects your previous console command. - GitHub - nvbn/thefuck: Magnificent app which corrects your previous console command. GitHubnvbn

GitHub - rupa/z: z - jump aroundz - jump around. Contribute to rupa/z development by creating an account on GitHub. GitHubrupa

GitHub - ogham/exa: A modern replacement for ‘ls’.A modern replacement for ‘ls’. Contribute to ogham/exa development by creating an account on GitHub. GitHubogham

htop - an interactive process vieweran interactive process viewer