In Part 1, I wrote about why we built an AI travel interface from scratch, what we learned from our first failed attempt in 2019, and why a plugin can never get you to a fully integrated conversational experience. This part is the how. The architecture, the search pipeline, the memory problem that took four attempts to solve, and the UI/UX work that makes 25 seconds feel like 5.

The architecture

Since this is a tech blog and not a marketing page, let me show you what’s actually running. This isn’t a weekend project. This is production infrastructure.

Frontend (React + TypeScript + Zustand)
    │
    │  WebSocket (real-time, bidirectional)
    │
FastAPI Backend
    ├── Query Orchestrator
    │     ├── LLM Client (multi-provider with automatic fallback)
    │     ├── Tool Registry (9 specialized tools)
    │     └── Observer System (async post-turn analysis)
    ├── RAG Pipeline
    │     ├── Vector Database (3 content collections)
    │     ├── Embedding Engine
    │     └── Reranker (dual provider, switchable)
    ├── Conversation Memory (dual storage)
    │     ├── SQL (message history)
    │     └── Vector DB (semantic search over past messages)
    ├── Security Layer
    │     ├── Input Guard (prompt injection defense)
    │     ├── Output Guard (credential leak prevention)
    │     └── Rate Limiting + IP Management
    ├── MCP Server (Model Context Protocol)
    │     └── External AI assistant access to our data
    └── Lead Management
          ├── Proposal Generation
          └── Specialist Handoff

The frontend is a React application with real-time WebSocket communication. No polling. No request-response cycles where you wait for a full answer. Every token streams in real-time, buffered at 50ms intervals to prevent UI flicker. Product results push to the interface the moment they’re found, before the LLM even starts composing its response.

The backend is Python/FastAPI with a modular tool system. The LLM doesn’t just generate text. It has access to 9 specialized tools and decides autonomously which ones to use. Need to search trips? It calls search_products. Need destination information? It calls search_knowledge. Need to recall what was discussed earlier? It queries conversation memory semantically.

The LLM isn’t generating answers. It’s orchestrating tools and composing a response from real data.

The LLM layer runs with a multi-provider fallback system. If the primary provider goes down, the system automatically switches to the secondary. No downtime. No errors. The customer never notices.

And there’s already an MCP server running, the protocol I described in my Web 4.0 post. External AI assistants can already query our travel data through it. It’s early days, but the infrastructure for AI-to-business communication is live. When Lisa’s AI assistant wants to search 333travel’s products, the endpoint already exists.

The RAG pipeline: where the real work happens

RAG, Retrieval-Augmented Generation, is the backbone of the entire system. It’s what separates “GPT with a knowledge base” from “an AI that actually knows your product.” But RAG done poorly is almost worse than no RAG at all.

We maintain three separate vector collections:

When a customer asks something, the query gets embedded and thrown against the relevant collection. But here’s the critical part: we don’t just take the top vector matches and call it a day.

Vector similarity alone is mediocre. It gets you in the right ballpark, but it doesn’t get you the best results. So we do a broad fetch, significantly more results than we need, and then run them through a reranker. The reranker evaluates each query-document pair for actual semantic relevance, not just embedding proximity.

The difference is enormous. It’s the difference between “here are trips that contain some of the words you used” and “here are trips that actually match what you’re looking for.”

But it goes further than that. Before the query even hits the vector database, it runs through a fuzzy normalization layer. Every country and location name in our system is cached at startup. When a customer types “Bali” we know they mean Indonesia. When they write “Tailand” we know what they meant. Aliases, spelling variations, language differences, all resolved before the search begins. Without this, you get empty results for queries that should have matched. And on top of that, the search combines semantic similarity with metadata filters. Country, duration, product type, these aren’t just fields in a database. They’re active filters that work together with the vector search to narrow results before the reranker even starts.

After reranking, the results split into two paths. The full set of reranked results pushes to the frontend immediately, but blurred. The customer sees product cards appearing while Joy is still thinking. It signals progress without overwhelming. Meanwhile, the LLM gets a stripped-down version of those results, only the fields it needs to reason about. This projection saves roughly a third of the tokens, which means faster responses and lower costs. Then Joy composes her answer and discusses the 4 or 5 trips that genuinely fit the question. Only those discussed products become visible to the customer. The rest disappear. What the customer sees is a curated, considered selection. Not a list of 10 search results. A recommendation of 5 that actually make sense.

Broad fetch → Rerank → Project. Three steps that make the difference between a search result and a recommendation.

The memory problem

This is the part that took the most iterations to get right.

Conversational AI has a dirty secret: it doesn’t remember anything. Every LLM call is stateless. The model has no idea what was said before unless you explicitly feed it the context. And there’s a limit to how much context you can feed.

For a simple Q&A chatbot, this doesn’t matter. Someone asks a question, gets an answer, done.

But for a travel assistant? Memory is everything.

A customer says: “I want to go to Thailand for two weeks.” Three turns later: “Actually, make it three weeks. And I want to include Laos.” Five turns later: “What was that trip you showed me earlier with the cooking class?”

Without memory, every turn is a blank slate. The system doesn’t know about Thailand. Doesn’t know about the three-week change. Doesn’t know about Laos. Definitely doesn’t know about the cooking class trip from seven messages ago.

Without memory, every message is a first date. Your AI has no idea what happened before.

This wasn’t solved in one attempt. Not in two. Not even in three.

The first version was simple: feed the last 10 question-answer pairs into the LLM context. It worked for short conversations. But our conversations aren’t short. Customers explore, compare, come back to earlier ideas, change their mind. Ten turns isn’t enough when someone has been chatting for twenty minutes about three different countries.

So we tried generating summaries in the background. The idea was sound: condense the conversation into key points so the LLM always has the full picture without eating up the entire context window. The reality was different. The LLM started hallucinating details into the summaries. Mixing up countries. Attributing preferences the customer never mentioned. The summary was supposed to be the source of truth, and it was making things up.

Then we designed a three-stage memory system: short-term, ephemeral, and long-term. The ephemeral layer was supposed to track which topics belonged to which destination, which preferences applied to which part of the trip. On paper, elegant. In practice, a nightmare. It required the AI to understand complex relational context, like “when they said budget was flexible, they meant for Thailand, not for Vietnam.” It couldn’t. Too many edge cases. Too fragile.

What we landed on was simpler and more robust. Every message gets embedded as a vector in a dedicated memory collection in the same database we use for product search. When the customer says “that trip with the cooking class,” the system runs a semantic search across all past messages in that session. It finds the relevant message even if the exact words don’t match. Maybe Joy called it a “culinary experience” three turns ago. Vector search doesn’t care about word matching. It matches meaning. Combined with the last 10 exchanges in the prompt for immediate context, Joy doesn’t need to make that tool call for every question. She has the recent conversation right there, and the full history searchable when she needs it.

Four attempts. Each one taught us something about what AI can and can’t do with context. The final version works not because it’s the smartest design, but because it’s the most honest about the limitations.

AI memory isn’t one problem. It’s two: recalling what was said, and understanding what was meant. We needed two different systems to solve them.

UI/UX: the invisible work

Here’s where most AI projects die, and they don’t even realize it.

You can have the most sophisticated RAG pipeline in the world. The smartest retrieval. The best reranking. But if the interface feels clunky, slow, or confusing, none of it matters. The customer doesn’t care about your architecture. They care about how it feels.

If your AI is smart but your interface is slow, you built a genius locked in a closet.

We went deep on this. And “deep” means dozens of iterations on things most people would consider details. But details are the product.

Streaming that feels natural. LLM responses stream token by token over WebSocket. But raw streaming creates flickering text that’s jittery and unpleasant. We buffer at 50ms intervals. Fast enough to feel real-time, slow enough to prevent constant re-renders. The difference is subtle but massive in terms of perceived quality.

The speed tradeoff. Let’s be honest about something: a product search with a full answer takes 20 to 30 seconds. That’s long. We know it’s long. But we made a deliberate choice: quality over speed. The broad fetch, the reranking, the semantic matching, every step in the pipeline exists because it makes the results better. Cut any of them and you get faster responses with worse recommendations. We’d rather have Joy work for 25 seconds and deliver 5 trips that genuinely fit than return 10 mediocre matches in 8 seconds.

So the question became: how do we make 25 seconds not feel like 25 seconds? That’s where the UX work came in. Joy shows what she’s doing. Searching products. Looking up destination information. Checking conversation history. Every tool call is visible as it happens. Product cards push in blurred while she’s still composing her answer. Once the answer is ready, only the trips Joy actually discusses become visible. The rest fade away. The customer never stares at a blank screen. They see the process unfold. And it turns out that watching something work feels completely different from waiting for something to finish.

Part of the latency is also a language problem. Dutch is poorly represented in most embedding and language models. The semantic understanding that works beautifully in English degrades when your customers and your product data are in Dutch. It means more processing, more careful matching, more effort to get the same level of quality.

What is GPT-NL? GPT-NL is a Dutch initiative to build a large language model specifically trained on and optimized for the Dutch language.

GPT-NL: Een verantwoord taalmodel voor NederlandIn Nederland wordt gewerkt aan de ontwikkeling van een eigen open taalmodel genaamd GPT-NL. Wij gebruiken alleen data om het model te trainen als we die rechtmatig kunnen verkrijgen en bieden zo een verantwoord alternatief op bestaande LLMs. Dit model is essentieel voor het versterken en behouden van onze digitale soevereiniteit. GPT-NL

Where current models like GPT and Claude treat Dutch as a non existent language, GPT-NL aims to make Dutch a first-class citizen in AI. For applications like ours, where both the customer and the product data are Dutch, a model that truly understands the language could significantly improve semantic search quality and reduce the processing overhead we currently need to compensate for that gap. We’re watching this project closely.

And then there’s model selection. The quality of the conversation depends heavily on which LLM you use. We run on Sonnet 4.6 as our primary, and the difference with smaller or cheaper models is not subtle. It’s the difference between a conversation that feels natural and one that feels scripted. Haiku, the faster and cheaper option, feels like a downgrade the moment you switch. GPT serves as our automatic fallback, but it’s a step back in conversational quality. These aren’t interchangeable commodities. The model you choose defines the experience your customer gets.

Instant page loads. The entire conversation persists in local storage. When you refresh the page or come back later, your conversation is already there. Instantly. No loading spinner. No “reconnecting…” message. The sync with the server happens silently in the background.

Markdown normalization. LLMs have opinions about formatting. Claude loves bold headers with emoji. GPT loves numbered lists. We normalize everything server-side. Bold-with-emoji gets converted to clean headers. Horizontal rules get stripped. Font weights get tuned down. The result looks like it was designed, not generated. The customer should never feel like they’re reading AI output.

Session continuity. Your session survives across page refreshes, browser closes, and reconnections. Come back a day later, your conversation is still there. Your saved trips are still there. And if a session does expire, you get a clean overlay explaining what happened, not a broken page or a cryptic error.

The lead flow. When a customer is ready to talk to a specialist, the system generates a travel proposal pre-filled with what it already knows from the conversation. Destinations discussed, trip preferences, products explored. Minimal friction. Maximum context. The human specialist picks up exactly where Joy left off. That’s not a handoff. That’s the whole point. Joy exists to make that human conversation better, not to avoid it.

Every one of these details took multiple iterations. Every one of them is invisible to the customer. That’s the point.

Security: an honest note

I’ll keep this section brief and honest. Security isn’t something you finish. It’s something you work on constantly.

We have a dual guard system. An input guard before the LLM that catches prompt injection attempts, and an output guard after the LLM that catches credential leaks or system prompt fragments in responses. Both are deterministic and add virtually zero latency.

The system prompt is hardened with anti-jailbreak instructions. There’s rate limiting, bot protection, and IP management for persistent abuse.

Is it bulletproof? No. Nothing is. But it’s built with defense in depth in mind, and we iterate on it as new attack patterns emerge. If you’re building any AI interface that talks to the public and you haven’t thought about input/output guards, you have a problem waiting to happen.

The bigger picture: an extra channel, not a replacement

Let me be very clear about what Joy is and what she isn’t.

Joy is not a replacement for our website. She’s not a replacement for phone or email. She’s an additional channel. A new way for customers to explore our products, alongside everything that already exists. She will never be a replacement for human contact. We value that the most in our company. Joy is a means to human contact. She helps customers figure out what they want, so that when they talk to one of our specialists, that conversation starts at a completely different level.

Joy isn’t the destination. She’s the road that leads to a real conversation with a real person.

The MCP server is already live, which means AI assistants can already query our data programmatically. When the Web 4.0 vision materializes, when Lisa’s AI assistant autonomously searches for her perfect Thailand trip, the endpoint already exists. Not because we scrambled to build it. Because we’ve been exploring this infrastructure for over a year.

Joy isn’t the end product. She’s the beginning of a channel strategy that includes both human-to-AI and AI-to-AI interactions. And she’s a test, a very serious, very thorough test, of whether conversational AI can actually add value for travel customers today.

The results so far have been better than I expected. Not flawless, nothing is, but genuinely impressive in how natural the conversations feel and how relevant the recommendations are. When you see a customer describe a vague idea and Joy comes back with trips that actually fit, you realize this isn’t a gimmick. It works. And it works because of everything underneath it.

So, what are we trying to solve again?

I opened with this question because it’s the one that guided everything we built. Not “how do we use AI?” Not “what’s our competitors doing?” Not “how do we look innovative?”

Just: what problem are we solving, and does this solution actually work?

We built Joy to find out. With real architecture. Real data. Real customers. And the willingness to say “it’s not ready yet” if that turned out to be the case.

If you’re thinking about adding AI to your product, start there. Not with the tool. Not with the vendor pitch. Not with the pressure to keep up.

Start with the question. Build from the answer.

Your customers deserve better than a FAQ with a face. They deserve a real conversation. And eventually, a real person. Make sure your AI leads them there, not away from it.

If you haven’t read Part 1, that’s where the story starts: why we built this, what we learned from failing at it in 2019, and why you can’t get here with a plugin.

Can a conversation be a travel channel? Part 1: We built one to find out.We built a full AI travel interface from scratch, not to say we do AI, but to test whether a conversation can actually be a meaningful channel for personalized travel. This is what it took, why a plugin will never get you there, and why most AI chatbots are just expensive search bars. All Subjects MatterRemco Loup.

We built an AI-powered travel interface from scratch. Not a chatbot. Not a widget. Not a GPT-wrapper with a smiley face bolted onto a website. A full conversational interface with its own vector database, its own retrieval pipeline, its own memory system, and real-time streaming over WebSockets. Not to replace anything. To test whether conversational AI is actually ready to be a meaningful channel for travel. Here’s what we found, what it took, and why we think most of what’s out there right now is noise.

This is a two-part story. This first part covers the why: what problem we’re solving, what we learned from failing at this before, and why a plugin will never get you here. Part 2 goes deeper on the architecture: the RAG pipeline, the memory system, the UI/UX decisions, and the technical tradeoffs we made.

What are you actually trying to solve?

That’s the question I keep coming back to. Every time I see a company announce their new “AI-powered” feature. Every time someone asks me what we’re doing with AI. Every time a vendor pitches yet another plugin.

It’s the most important question in tech right now. And almost nobody is asking it.

Instead, the entire industry is working backwards: starting with “we need to do something with AI” and then looking for a place to put it. Companies hear “AI” and they hear “innovation” and they think they’re falling behind if they don’t do something right now.

The pressure to “do something with AI” has become louder than the question of what are you trying to achieve.

And so what happens? Companies reach for the most visible, most marketable application they can find: a chatbot. Slap it on the website. Give it a name and a friendly face. Connect it to a knowledge base. Done. “We do AI now.”

Here’s the problem: the integration is always limited. Some connect to a product database. Some pull live availability. But as a plugin, there’s a ceiling to how deep that integration goes. It’s always working with whatever data it gets access to, in whatever format the API provides, with whatever context the vendor decided was enough. The result is an interaction that looks conversational on the surface but is fundamentally constrained in what it can actually do.

And because it’s a plugin, it can never be fully seamless. It will always be a separate layer on top of your website, not part of it. A different design language. A different interaction model. A different experience. The customer feels it, even if they can’t articulate it. It’s the difference between a room that was designed as a whole and a room where someone placed a kiosk in the corner.

A smarter gatekeeper is still a gatekeeper. Your customers wanted a door.

We’ve been down this road ourselves. Back in 2019, we built the first version of Joy on Flow.ai. An intent-based chatbot that could understand natural language, extract travel preferences from a single sentence, and generate leads for our specialists. It worked. Technically. But customer engagement was too low to justify keeping it alive. We retired Joy in early 2022.

The lesson wasn’t that the technology was bad. The lesson was that people have learned to distrust chatbots. And rightfully so. For years, companies have used chatbots as a barrier to contact, even when they claim otherwise. Customers know this. They’ve experienced it. After three wrong answers, you don’t think “let me try again.” You think “how do I get to a real person?” Some people don’t even bother engaging. They see a chat bubble with a face and their immediate reaction is to close it. Not because they’re impatient, but because experience taught them it’s a waste of time.

That’s not a technology problem. That’s a trust problem. And it’s one you can’t solve by making the same format smarter.

On top of that, the interface of a chatbot plugin is simply limiting. A small window in the corner of your screen. A few lines of text. Maybe a button or two. You can’t show rich product cards in that. You can’t display a map of a trip. You can’t let someone browse, compare, save trips for later. The format dictates the experience, and the format is a box.

That experience is exactly why we didn’t build another chatbot. Not a smarter one. Not a better one. Something fundamentally different. A full conversational interface, built from scratch, designed to inspire. Where someone can explore destinations, discover trips they didn’t know existed, and when they’re ready, transition into contact with a specialist after a conversation that actually felt good. Not after three failed attempts to get past a gatekeeper.

The real question was never “how do we add AI to our website?” The real question is: where does AI genuinely create value that wasn’t possible before? That’s what we set out to test.

Why I built what I built

I didn’t build this because I think websites are dead or because I wanted to say we “do something with AI.” I’ve written before about how much I despise that mentality.

I built this because of a question I’ve been testing for over a year now: is conversational AI truly ready to be a real channel?

In my Web 4.0 post, I wrote about the autonomous web, a world where AI assistants don’t just retrieve information but act as independent digital consumers on behalf of their humans. Where Lisa tells her AI assistant “find me a two-week trip to Thailand” and the AI contacts travel companies through MCP-enabled interfaces, negotiates, refines, and presents options. Without Lisa ever opening a browser.

That might sound like removing humans from the equation. It’s not. The technology enables full autonomy. But that doesn’t mean we want to remove the human. It means the customer gets to choose. And our default will always be: lead to a person. More on that later.

That future is coming. Maybe not tomorrow, but it’s coming. And if you’re a travel company that takes itself seriously, you’d better be ready for it.

So I built Joy, an AI travel assistant for 333travel, not as a product launch, not as a marketing play, but as a serious test. Can we build a conversational interface that actually adds value? That understands our products? That doesn’t hallucinate as much? That creates a genuinely useful experience as an additional channel next to our website, phone, and email?

I didn’t build this to replace our website. I built this to answer a question: can a conversation be a meaningful channel for personalized travel?

What this actually is

Let me be specific about what Joy does:

Joy sits on top of our entire product database. Every roundtrip, every hotel, every excursion, every destination, all embedded as vectors in a dedicated database. When a customer says “I want something adventurous in Southeast Asia, around two weeks, not too touristy,” Joy doesn’t keyword-match against a PDF. She searches semantically across hundreds of products, reranks results based on actual relevance, and responds with options that genuinely fit the intent.

And she remembers the conversation. Not just the last message, the whole thread. She knows what trips were discussed, what was saved, what was dismissed. When you say “that trip you mentioned earlier with the cooking class,” she understands what you mean even if she originally called it a “culinary experience.”

But here’s the key difference with any chatbot out there: she doesn’t just know travel in general. She knows our travel. Our specific products. Our specific itineraries. Our destinations that our team personally visited. That’s not something you can easily bolt on with a plugin, no matter how smart the LLM behind it is.

And the moment you want human contact? Joy hands the full context to a real travel specialist who knows exactly what you’ve been exploring. No “please describe your inquiry again.” No cold transfer. Full continuity.

Joy doesn’t replace conversations with specialists. She makes them better.

What she doesn’t do (yet): she doesn’t handle customer service queries, airline information, or post-booking support. This is purely a product discovery and inspiration channel. We’ll get there. But we’re not going to pretend she does things she doesn’t. And honestly, when someone reaches out to customer service, there’s already a problem. That’s the worst possible moment to put a machine between you and your customer. That’s when you show up. Personally. So we are still debating about what we want her to do there. The nuance is always in the details.

Why you can’t get here with a plugin

You cannot achieve what we built by installing a plugin. Not with ChatGPT. Not with any off-the-shelf “AI widget.” Not with any SaaS tool that promises to “make your website smarter in minutes.”

And it’s not because the AI models aren’t good enough. The models are fine. The problem is everything around them.

The entire system depends on data that we own, structure, and control. Every trip in our database has been visited by our own team. Every itinerary is our own creation. Every piece of destination knowledge lives in our own systems. This isn’t data we license from a third party or pull from a brochure catalog.

That means we can embed it. We can structure it exactly how we need it for semantic search. We can update it the moment something changes. We control the full pipeline from raw product data to vector embedding to retrieval to response.

You can’t build a smart interface on top of piles of dumb data.

If your product data lives in someone else’s system, if you’re reselling trips from a catalog, if your “database” is a collection of PDFs from tour operators, you’re at the mercy of whatever structure someone else decided on. You can’t build intelligence on top of data you don’t understand or control.

And many companies aren’t just starting from zero, they’re starting from a deficit. Their existing systems are already inadequate. The booking engine is outdated. The product data is scattered across spreadsheets and PDFs. The customer journey has gaps everywhere. And instead of fixing those fundamentals, they think it is smart to layer an AI chatbot on top of it. As if intelligence on top of dysfunction creates something functional. It doesn’t. It just makes the dysfunction harder to diagnose.

You’re not solving a problem. You’re decorating one.

Building what we built required three things:

Owning our data. Every trip is ours. Visited by our team. Structured in our database. Embedded in our vector store. You can’t build semantic search over data you don’t structure yourself.

Owning our systems. Our own backend, frontend, deployment pipeline. When we needed a dual reranker, we built it. When we needed semantic conversation memory, we built it. When we needed an MCP server for Web 4.0 readiness, we built it. No vendor approval needed. No feature requests. No roadmap dependencies.

Owning the entire chain. From the moment a customer opens the chat to the moment they’re talking to a specialist, every step is ours. The AI knows our products because we made the products. The specialist knows the conversation because our system feeds it to them.

The AI is the last 10% of the work. The first 90% is building a company that can support it.

In Part 2, I’ll show you how it is actually running under the hood. The architecture, the search pipeline, the memory system that took many attempts to get right, and why 25 seconds of waiting can feel like 5 when you design it right.

Can a conversation be a travel channel? Part 2: The technical architectureThe technical deep-dive into how we built Joy: a RAG pipeline with vector search and reranking, a memory system that took four iterations, and the UI/UX decisions that make a 25-second response feel instant. All Subjects MatterRemco Loup.

I built a “production” app using Vibecoding. 97,089 lines generated. Only 25,056 made it to production. Four complete refactors. The AI didn’t fail because of bad prompts. It failed because AI forgets, loses context, and creates duplicate logic no matter how well you guide it. Vibecoding isn’t about letting AI build for you. It’s about knowing your system well enough to filter the noise. Without that knowledge, you’re only shipping guesses.

What Vibecoding actually means

People think vibecoding means letting AI build your app while you watch. Reality check kicking in here, this is not at all the case if you need an actual working product with the expected result every time.

Vibecoding still means you take control. You prompt with precision. You experiment. You guide. You filter. It’s still hands-on work, just with different leverage.

“A fool with a tool is still a fool”

And if you don’t know what you’re doing? The result is the same as it always was: a mess. Vibecoding without the right knowledge is just plain wrong. And the main problem with AI, is that it keeps producing code after code after code. It just does what you ask. it isn’t critical about that. So if you don’t know what you are doing you are contradicting yourself all the time with programmed functionality that never works together within your app.

Ai is king in fallbacks. I would have had 5 different fallbacks for every api call that fails. Shouldn’t we just make the right way work instead of making fallbacks. AI is also lazy, it rather suppresses notices then fixing them. especially if it won’t work on the first try.

I built a production app with AI (but not because of AI)

I used Windsurf to build an entire application. No human developers. No code written by me directly. we use Sonnet 4.5 as our knowledge.

The result after a lot of frustration: a full production game with backend logic, login flows, async data writes, mail queues, admin security, monitoring, and cartesian algorithms that update based on user interaction.

The AI wrote all of it. I haven’t touched a single line of code. So even the simples css change. I let AI do it.

But here’s what nobody tells often: I had to completely refactor the entire project four times. And not because I prompted poorly. I used KISS principles, DRY standards, PSR guidelines. We wrote decisions and created implementation documents on forehand. We used guidelines for the languages that we used.

It didn’t matter in the end.

AI forgets. It loses context. It starts creating duplicate functions without realizing it already wrote that logic three files ago. The more complex your project gets, the more it drifts. And the more it isn’t done right in the first attempt AI thinks it needs to maintain backwards compatibility, complicating your code even more. No matter how you prompt it the first time or with project rules.

Four complete refactors later, I learned: AI doesn’t forget context. It never had it ;).

The only reason the final version works is because I knew exactly what I was building. I had the system in my head. I could spot when AI was repeating itself. I knew when to stop it, when to redirect it, when to start over. I checked every file for duplicate fallback data. duplicates and so on. I also used the very usefull npm package knip

https://www.npmjs.com/package/knip

You can’t vibecode something into production without understanding your product. Period. And you need to accept that bug fixing is relying on your AI to fix it without breaking anything else, Good luck with that…

The data tells the rest of our story

During our Vibecoding, Windsurf produced:

• 97,089 lines of code generated
• 2,925 prompts sent
• 5,693 credits used
• 15,734 tool calls (including 7,000 code edits)

What actually made it into production:

• 25,056 lines of code

Over 70,000 lines got scrapped, replaced or refactored.

Now, you might think this means you just need to “guide AI better” or “prompt more precisely.” That’s not the point I wanted to make.

The point is: you can’t prevent AI spew right now. No matter how well you prompt. No matter how structured your approach is. AI will overproduce. It will generate redundant code. It will lose track of what it already built. It will create noise. AI is build to always post a form of AI Verbal Diarrhea.

That 70k+ lines of wasted output? That’s not a bug. That’s the current state of Vibecoding.

And if you don’t know your system well enough to recognize what’s garbage and what’s not, you’ll ship all of it and never be able to debug even with AI. Or worse, you’ll spend personally weeks debugging a codebase you don’t understand.

The costs per 1000 credits are around 35 dollar. The development of this game cost me 210 dollar. Every company delivering Large Language Models is losing money. the code I produced should easily have costed 210 x 10 =2.100 dollar at least. I even wonder if they then even made money then.

Apart from the money, is this really worth our precious electricity? I am wondering if you pay the real price, how feasible this will be.

What people get wrong

AI isn’t magic. It’s a “smart”, fast assistant. But it doesn’t understand what matters to your business. It doesn’t grasp edge cases, constraints, risks, performance implications, or security vulnerabilities.

That’s your job!

In my case, AI helped build something real. But the AI didn’t secure the admin panel. It didn’t design the mail queue logic. It didn’t decide when to debounce async writes. I did. By being critical and asking the right questions due to build experience in the past.

Without that direction, the entire thing would have collapsed.

And I see this pattern everywhere. Teams building AI products without ownership or direction, just to say they’ve “done something with AI.” It’s shallow. It’s fragile. It doesn’t scale.

“You’re not building products. You’re laundering prompts into deployments”

How many times don’t you hear from higher hand: We need to do something with AI. We are falling behind. Behind of what?

The customer service trap

Then there’s also the AI customer service hype. Companies claim it’s perfect for 24/7 support, faster responses, fewer humans needed.

But if you need a bot to talk to your customers, your real problem is probably somewhere else. AI doesn’t fix broken processes. It hides them. Temporarily.

If you think less human contact is a win, you’ve already lost. If replying to questions takes a long time you might have to look at other things why people have so many questions and improve on that first.

The actual point

Vibecoding is just another tool. That’s all it is. A very cool tool in the right hands tho, but still a tool. And mainly good for prototyping, but with a good dosis of knowledge you can maybe produce standalone apps that actually work, and are at least testable by you and the outcome is traceable to the code written in a transparant way.

It doesn’t replace thinking. It should amplify it, and it shouldn’t make you think less.

Used right, it can be powerful. But you still need to know what you want to build. You need to own the structure. You need to filter the noise. You need to recognize AI spew when you see it. You need to lead the process.

Otherwise, you’re not building a prototype. You’re just generating guesses.

The future isn’t AI-first. The companies that matter stay human-first. They understand the problem before picking the tool. And they keep asking the only question that counts before reaching out for AI to help them:

What are we actually trying to solve?

Artificial Intelligence is here to stay, that’s not the topic here.

The problem is that everyone’s talking about AI, but few are actually thinking clearly about how to really use it. Somewhere along the way, “we use AI” became a marketing tagline, not a technical decision. Everyone wants to be part of the AI wave otherwise, you don’t belong. If you have to believe big tech the way they are forcing it down everyone’s throat.

But when you take a small step back, you’ll notice something strange:

Companies are solving problems with AI that they shouldn’t have in the first place.

Poor system designs, missing ownership, and spaghetti infrastructures are now being “fixed” by throwing LLMs on top of this pile of shit. And as a result, systems are becoming even harder to maintain, more expensive, and harder to reason about.

This blog isn’t about tearing down AI.
It’s about clear thinking, technical ownership, and using the right tools for the job.

OpenAI investigation research paper

Recent research by OpenAI and Georgia Tech highlights a core truth that often gets ignored: AI hallucinations, those moments where models make things up aren’t bugs. They’re a mathematical consequence of how these systems are built.

Even with perfect training data, the current architecture of generative AI rewards confident answers over admitting uncertainty. That means hallucinations are not only common, they’re inevitable, unless we fundamentally change how we train and evaluate these models.

As the researchers put it: “Hallucinations are mathematically inevitable.” (source)

How people abuse AI to solve the wrong problems.

Let’s take a step back.

If you’re paying $1,200/month for GPT-4 to parse email text, extract phone numbers, reformat JSON, and uppercase strings… you’re not doing innovation. You’re outsourcing common sense to an expensive black box.

Yes, someone recently did that, and realized the cost effect too late. They then moved to GPT-4o-mini and brought the bill down to $200…. But even then it still is a shit solution:

Those tasks didn’t need AI at all. Just basic software skills.

This isn’t an isolated case. It’s happening to much sadly.

People assume AI is their fix to everything, while it only shows the persons incompetence because:

• It increases costs
• Obscures what’s actually happening
• Introduces hidden dependencies
• Makes systems almost impossible to debug and maintain

AI is used to cover up a lack of system design, not to improve it.

AI as a Natural Language interface, powerful but misused

But let’s be fair: LLMs truely are powerful when used right.
They allow you to describe what you want in natural language, and get a result, without understanding the underlying logic.

This makes AI a fantastic tool for:

• Prototyping
• Interacting with large knowledge bases
• Summarizing unstructured data
• Performing fuzzy matching at scale

LLMs are essentially a new kind of interface, just like the command line, the GUI, or APIs were before.

But they’re not magic. And they are definitely not for free. And they’re almost always not the best way to solve a problem.

The more critical the function, the more important it becomes to understand:

• What is being executed?
• Where is the data flowing?
• Who owns the outputs?
• What are the real costs?

Conversational AI, the modern chatbot, but still a chatbot

There is a path forward for conversational AI. It’s just not the revolutionary leap people think it is with self thinking systems that wont hallucinate and always provide the right answer.

Yes, LLMs can provide smarter, more human-like responses then before LLM’s became mainstream.
Yes, they can automate support flows that were previously a lot more complex to script.

But let’s not fool ourselves:

People still want real, human contact with your business especially when something goes wrong.

If you outsource your entire customer service to an AI assistant, you’re sending a clear message (but the wrong one in my opinion) :

• “We don’t care enough to talk to you ourselves.”
• “You’re not worth the time of a real person.”
• “We only sell our product. dont come complaining to us please.”

This probably saves costs in the short term.
But long-term? You’re destroying the trust with your brand. Especially because people typically contact a customer service when:

• They’re frustrated
• They don’t understand something
• They have a complaint

That is not the time to offer them a chatbot, no matter how “smart” it sounds.

Great customer service is a relationship, not a cost center.

So yes, use conversational AI to assist your teams. But don’t let it replace human contact where empathy is needed. Dont make AI the interface of talking with your company for customers if you care about your product.

What you should be using AI for

Let’s flip it around in this section. What is a good use case for AI?

You should use AI when:

• It helps you do things that were previously too expensive, too slow, or technically too complex
• You’ve already optimized your architecture, but still have edge cases worth automating
• You’re enriching processes, not hiding technical debt

For example:

• Prototyping difficult functionalities so you can test outcomes quicker
• Enhancing product search with semantic understanding
• If you can make the hallucination visible in data they are perfect for BI (business intelligence)
• Generating tailored summaries for large internal reports
• Transcribing phone calls and make summaries
• Image to text so you can better semantic search on images in large image databases.

These are tasks where AI really shines, because they were even more expensive, hard or impossible to solve in traditional systems.

But again: always weigh the costs.
Not just in dollars, but in clarity, control, and customer experience. Always try to solve it without using AI first.

Tech maturity will determine who wins

Here’s the core of it all:

Companies with modern, understandable, well-documented infrastructure and data will win.
Those who keep layering AI on top of their chaos… won’t.

It’s not about how early you adopted AI. It’s about:

• Whether your stack is maintainable
• Whether your teams really understands it
• Whether you can reason about cost, performance, and failure
• Whether you own your data, your architecture, and your roadmap

The companies that treat AI as a new algorithm/interface to solve problems they could not solve before, will get the most long-term value in my opinion. Don’t use AI for it’s marketing value big tech wants you to.

Offloading data ownership is a strategic risk

Let me end with another real example I recently came across.

A company in the travel sector built an app for end-users to access their trip details. They are reselling this to other travel agencies and tour operators. That sounds feasible looking at the bad shape travel is in regarding IT.

So the travelagency can offer the app with all vouchers and itineraries for the trip of their customer. Nothing weird so far except you are offloading all your documents to a third party.

So now this itinerary app provider has access to a lot of your crucial business information. And now the customer buys extra services (e.g. insurance, transfers, upgrades) in the app. You should think this is upsell for the travel agency owning the customer.

Here comes the culprit. The platform suddenly gives commission to the touroperator… So is the app provider now also providing extra services to your customer??

Who owns the customer now?

It’s a major shift and a dangerous one. The more infrastructure you outsource, the more you lose control of your customer relationships and your first-party data.

Stay in control of the customer journey of your own customer. Protect your added value. dont become unity. And keep it personal.

This will become a huge issue! And AI will accelerate it if travel companies aren’t paying attention.

Conclusion: AI is a tool, not a strategy for most companies

AI is impressive. But it doesn’t replace fundamentals.

Don’t use it to paper over bad architecture.
Don’t use it as an excuse to avoid technical decisions.
Don’t let it obscure what your systems are doing, or who your customer really is.

Instead:

• Own your stack
• Understand your logic
• Use AI where it creates real, measurable value
• And never stop thinking for yourself

The real innovation isn’t in the tool. It’s in how clearly you think about the problems to solve.

For decades, businesses have adapted to the evolution of our internet. First came Web 1.0, a read-only internet where companies built static websites. Then Web 2.0 ushered in the interactive era—social media, user-generated content, and APIs that enabled apps to connect with services. Now, we are living in the Web 3.0 age, where AI enhances personalization, automates tasks, and understands context.

But this is just the beginning.

The next phase, Web 4.0, will introduce something even more transformative: AI as an autonomous digital agent. Instead of just assisting humans, AI will actively engage with businesses, making decisions, negotiating deals, and executing transactions—all with minimal human intervention.

I call this the autonomous web: a world where AI assistants don’t just retrieve information but act as independent digital consumers serving their Humans. And businesses that fail to prepare for this shift are at risk of becoming invisible in this generation of digital commerce.

The shift to AI-driven interactions

For years, businesses have focused on optimizing content for human users. Websites, apps, and APIs were all designed with the assumption that a person would be making the final decision. That assumption is now breaking down.

The evolution of digital interaction:

• Web 1.0: Static websites (businesses provide information, users consume it).
• Web 2.0: Interactive web (social media, APIs, mobile apps).
• Web 3.0: Intelligent web (AI-driven personalization, automation, and semantic search).
• Web 4.0: Autonomous web (AI agents acting as customers, negotiating, transacting, and making independent decisions).

We are now transitioning into Web 4.0, where AI agents will handle complex, real-time interactions—like booking trips, negotiating prices, and even dynamically adjusting services based on user behavior. Businesses that fail to adapt to this AI-first interaction model will simply be left out of AI-driven decision-making.

Why APIs alone are not enough

Many businesses might assume they are going to be prepared for AI-driven interactions because they have APIs. But traditional APIs were built for structured, predefined requests—designed for apps, not for AI agents that need context-aware, multi-step interactions.

This is where developments like the Model Context Protocol (MCP) comes in.

MCP will be one of the missing links that allows AI assistants to interact with businesses as if they were human customers. Unlike traditional APIs that return static responses, MCP allows for:

• Dynamic service requests (AI can ask for specific customizations).
• Negotiation and refinement (AI can modify criteria based on preferences and constraints).
• Autonomous decision-making (AI can finalize a transaction without human confirmation).

The companies that adopt AI-native interfaces will dominate the Web 4.0 economy. Those that don’t will be bypassed—not by people, but by AI-driven agents making purchase decisions for their humans.

A fully AI-driven transaction at 333Travel

At 333Travel, we don’t sell pre-packaged vacations. Every trip is tailor-made, requiring back and forth interaction to fine-tune your perfect trip. Today, this process happens through web forms, email exchanges, and phone calls. But in Web 4.0, this will look very different. and all these channels will be combined together with talking to an AI assistant as an option.

A new kind of travel interactions

Imagine a traveler, Lisa, who wants a customized trip to Thailand. Instead of visiting a website, filling in a form, or calling an agent, she simply tells her AI assistant:

“I need a vacation, Find me a two-week tailor-made trip to Thailand with a mix of luxury and boutique hotels, some adventure activities, and time for relaxation. somewhere mid june.”

Lisa never opens a website. Her AI assistant does the work for her.

Step 1: The AI makes an intelligent request

Lisa’s AI contacts 333Travel’s MCP-enabled interface through a search, sending a structured request with her preferences, past travel history, and budget constraints.

Step 2: AI and 333Travel’s system negotiate and refine the trip {#step-2-ai-and-333travel%E2%80%99s-system-negotiate-and-refine-the-trip}

Since 333Travel specializes in tailor-made experiences, a real-time negotiation process begins between Lisa’s AI and 333Travel’s MCP integration:

1. AI sends initial request
   • “User prefers Thailand, mix of luxury and boutique hotels, some adventure, some relaxation.”
2. 333Travel’s MCP responds dynamically
   • “Recommended itinerary: Bangkok, Chiang Mai, Koh Samui.”
   • “Luxury options: The Siam Bangkok, Anantara Chiang Mai, Four Seasons Koh Samui.”
   • “Boutique options: Sala Rattanakosin, Tamarind Village, The Library Koh Samui.”
3. Lisa’s AI refines the request
   • “She has stayed at Four Seasons before and rated it highly—prioritize that.”
   • “She prefers cultural experiences—add a food tour in Bangkok and a temple visit in Chiang Mai.”
   • “Budget cap: €5,000—adjust accordingly.”
4. 333Travel then creates a personalized proposal based on all the information normally received in a phone call.
   • Three itinerary options are discussed with Lisa’s AI, ranked by price, luxury level, and activity balance.

Step 3: The AI confirms and secures the booking

Lisa herself checks the summary and says:

“Book the second option and add an airport transfer.”

Her AI assistant:
✔️ Finalizes the booking with 333Travel
✔️ Secures payment automatically
✔️ Sends confirmation to her email and calendar

Lisa never interacted with a website. Never sent an email. Never called a travel agent.

The autonomous web can handle everything, but Lisa can step in at any moment and decide to make that phone call.

The Web 4.0 shift will completely change how businesses engage with customers.

What this means for businesses

The key takeaways:

1. AI-driven transactions can become the new norm

If a business isn’t optimized for AI-first interactions, it won’t be part of AI-driven purchase decisions.

2. AI optimization will replace traditional SEO

Just as businesses optimized for Google search rankings, they will now have to optimize for AI-driven discovery—ensuring their services are structured for AI agents to understand and interact with.

3. AI-native interfaces will be the next competitive advantage

Companies that create AI-ready endpoints using MCP-like protocols will likely dominate the Web 4.0 landscape in the early stage.

4. Customers will expect frictionless experiences

If an AI can request an entire itinerary in seconds with detailed information, why wouldn’t you make use of it, and combine it together with the normal human interaction. You just created an extra sales funnel next to your ability to Mail, send a text message, visit a website or make a phone call.

The autonomous web is here. Are you ready?

Web 4.0 isn’t something happening in the distant future—it’s happening right now. AI is no longer just assisting customers; it’s acting on their behalf.

The businesses that prepare today will be ready at the start of the Web 4.0 economy. Those that ignore it will simply be bypassed—not by human customers, but by their AI-driven agents.

The question is no longer if AI-driven customers will reshape business—it’s whether you’ll be ready to serve them when they do.

Why 333travel sets the travel standard in an AI-driven world

The world is changing rapidly. With the rise of AI models like ChatGPT, ClaudeAI, and Gemini, access to data and insights is easier than ever. Even companies with no real expertise in travel try to enter the market by leveraging the product of someone else with technology only. But what sets 333travel apart from these “digital dropshippers”? Simple: our own technical knowledge, our passion, and FULL control over our OWN product, as you are going to read below.

We’re not just a travel agency; we are also an IT company built on expertise and reliability.

At 333travel, we combine people and technology, innovation and authenticity, and remain the undeniable choice—even in a world where AI is becoming more visible every day.

Generic AI vs. self-trained AI

Not all AI is the same. Public AI models, like the ones mentioned above, provide standard solutions trained on publicly available data. This makes them accessible and allows them to function as a new sales channel. We ensure that these generic AI solutions ultimately come to us for their travel needs—because of our proven expertise.

But we already took it quite a few steps further. We’ve already been exploring extensively how self-trained AI, powered by our unique company data, can optimize and accelerate our internal processes. One thing is clear to us: AI does not replace our DNA, personal touch, or customer care. It’s a fantastic tool, a “smart notepad” that supports our specialists. Whether it’s streamlining administrative tasks, performing faster analyses, or assisting with complex inquiries—it allows our travel experts to focus on what truly matters: personal service and customer satisfaction.

Because the balance between AI and human interaction is crucial to us, we will always make sure to keep the human touch at the heart of our customer relationships. Travel is about emotions, passion, and real connections—something no AI will ever fully replace for us.

AI as a catalyst, not a replacement

AI is a powerful tool, but it wil not replace our expertise and experience. While some companies leverage AI to create generic travel offerings on the data of others, we remain the trusted point of contact where both humans and AI assistants book their trips. Why? Because even AI, when aiming for the best outcome for its user, recognizes that 333travel sets the standard for complete, worry-free travel experiences and unique up to date information.

What makes us stand out:

Up-to-date knowledge: Our specialists have personally visited the destinations, knowing more than what’s just available online.

**Local Partnerships:**In each country we offer, we have our boots on the actual ground. We work closely together with our local partners to stay on top of our game. It wont matter which country you visit with us. We will always know best how to assist you due to the fact the whole trip is also handled by us.

Full control: Our in-house IT capabilities allow us to operate independently. We manage our own data, develop our own systems, and can even train our own AI models with our unique expertise. IT is deeply embedded in our DNA—unlike many other travel companies that rely on third parties.

**Passion for travel:**Our trips are not just logically put together; they are designed to make dreams come true.

We make sure you can enjoy your journey without worries

The risks of “digital dropshipping” {#the-risks-of-%E2%80%9Cdigital-dropshipping%E2%80%9D}

In this AI-driven world, some companies offer travel products without truly understanding them, simply by smartly combining data. This approach is similar to dropshipping—selling third-party products without real knowledge or control. But how can you create real value without deep expertise in your product?

Imagine assembling a kitchen by ordering each part from a different supplier. It might sound cheap and efficient, but what happens if the parts don’t fit? Who is responsible when something goes wrong? At 333travel, we ensure that these kinds of issues never arise. We provide a seamless, complete product with full accountability and peace of mind, so you can enjoy your journey without worries.

Why 333travel is the best choice—even for AI {#why-333travel-is-the-best-choice%E2%80%94even-for-ai}

In the future, AI assistants will increasingly recommend trips based on data. But even AI will soon recognize that a successful trip depends on more than just raw data. At 333travel, we deliver an experience that AI can recognize—but cannot create itself. AI needs us to handle it and book it for them.

Unique destination knowledge: We understand the nuances and details that no database contains.
**End-to-end service:**From the first inquiry to post-trip support, we are always there for our customers.
Reliable data and systems: Our own cutting edge IT solutions and data ensure everything is accurate, up-to-date, and executable.

AI will not book a trip with a company that poses risks or offers generic solutions.

Even in the future, AI will choose security, quality, and reliability—which means choosing 333travel.

How AI strengthens us

We don’t see AI as a competitor, but as a tool. At 333travel, we use technology to streamline our processes and enhance our service, while keeping our focus on human expertise. Some examples:

Efficient processes: AI helps us recognize patterns and trends faster, allowing us to work smarter and provide better answers to our customers.
Enhanced travel proposals: Our own data and knowledge ensure every proposal is personal and unique—AI speeds up the process but always relies on our self-written content.
**Improved communication:**Chatbots and AI assistants support our team in the background, giving us more time for direct, personal contact with travelers.

What makes us unique is our passion, authenticity, and human approach—combined with our fully in-house IT backbone, which now more than ever sets us apart from the rest.

To be truly innovative, you need full control over your own business-critical processes.

The future of travel: 333travel as the undisputed choice

The travel industry is evolving. Technology will play an even greater role, but the core remains the same: the desire for unique, unforgettable experiences. At 333travel, we combine the best of both worlds—the power of technology with the warmth of human expertise.

The biggest challenge in the travel sector lies with companies that depend on third parties for AI implementations, or those that claim to “own” data while merely repackaging generic AI results.

We position ourselves as the central contact point—for both people and AI assistants—by consistently delivering the highest quality. Whether you dream of a tropical beach or a cultural adventure, you can trust that with us, you’re in good hands. We manage the entire chain, from data analysis to personalized service, ensuring that your trip is not only perfect but also worry-free and unforgettable.

Conclusion

AI is changing the game, but at 333travel, we continue to stand out. Our unique combination of up-to-date knowledge, passion, and full control over our product makes us the go-to choice for any traveler. Whether you reach us through personal contact or via an AI assistant, one thing is certain: with 333travel, you’re not just booking a trip—you’re securing a complete experience.

So, how do you stay unique in a world increasingly driven by technology? We do it by continuously improving our data, leveraging AI in smart ways, and ensuring that travelers still come to us to turn their journeys into reality—just as before.

In a world with DHCP address, being able to access your home PC or server remotely is a Pita. So, if you’re are “blessed"with an ISP that assigns a dynamic public IP address, you face a significant challenge: your IP changes periodically, breaking your ability to connect to your home device using a static domain.

This is where this free Dynamic DNS (DDNS) updater script for TransIP comes to the rescue. By seamlessly updating your DNS records whenever your external IP changes, this solution ensures your domain (e.g., vpn.example.com) always points to the correct IP. Here’s why this script is a must for anyone leveraging TransIP’s DNS services.

Key Features of the Script

1. Automated DNS Updates:
   • Automatically checks your current external IP and compares it to the DNS A-record associated with your domain.
   • Creates or updates the A-record to match your latest external IP.
2. Dynamic DNS for Dynamic IPs:
   • Perfect for environments where your public IP changes frequently, such as home internet connections with dynamic DHCP assignments.
3. Secure Authentication:
   • Leverages TransIP’s API with a private key for secure communication.
   • Generates short-lived tokens for added security (valid for only 10 minutes per session).
4. Customizable Settings:
   • Easily configure domain, subdomain, TTL (time-to-live), and logging options.
   • Adjust the script to match your specific requirements.
5. Logging and Error Handling:
   • Comprehensive logging ensures you can track every step of the update process.
   • Built-in error handling for smoother execution and debugging.

How It Works

1. Initial Setup:
   • Set up a TransIP account and generate a private key for API access at https://www.transip.nl/cp/account/api/
   • Configure the script with your domain name, subdomain, and private key path.
2. Fetching External IP:
   • The script retrieves your current external IP using a public API like ipinfo.io.
3. Token Generation:
   • Generates a short-lived access token using TransIP’s authentication endpoint.
4. DNS Record Validation:
   • Checks if the current DNS A-record matches your external IP.
   • If it differs, the script updates the record; if it’s missing, it creates a new one.
5. DNS Update:
   • Uses TransIP’s secure API to update or create the DNS A-record with your current assigned external IP.
6. Logging and Notifications:
   • Logs each step and provides clear messages in case of issues.

Prerequisites

Before using this script, ensure you have:

1. A TransIP account with API access enabled.
2. A registered domain in your TransIP account.
3. A static private key for secure API authentication.
4. Dependencies installed on your system:
   • curl, openssl, and jq.

sudo apt update
sudo apt install -y curl openssl jq

Why Use This Solution?

Unlike payed third-party DDNS providers, this script gives you free control over your DNS management, ensuring privacy and avoiding vendor lock-in. It’s tailored for you.

#!/bin/bash

# Configuration settings
LOGIN_NAME="your-login-name"  # Fill in your TransIP login name
PRIVATE_KEY_PATH="/path/transip_private_key.pem"  # Path to your private key file
AUTH_URL="https://api.transip.nl/v6/auth"
TOKEN_FILE="/var/tmp/transip_token.json"  # File to store token and expiration date
DOMAIN="example.com"  # Your main or test domain
SUBDOMAIN="vpn"  # The subdomain you want to update (e.g., test.example.com)
RECORD_TYPE="A"
TTL=60  # Time-to-live for DNS records
LOG_FILE="/var/log/transip_dyndns.log"

# Function to update a log file
log_message() {
    echo "$(date '+%Y-%m-%d %H:%M:%S') - $1" > $LOG_FILE
}

# Function to log an error and stop the script
handle_error() {
    log_message "ERROR: $1"
    echo "ERROR: $1" >&2
    exit 1
}

# Function to obtain a new access token with a short validity period (10 minutes)
generate_new_token() {
    log_message "Generating a new token with a validity period of 10 minutes."
    NONCE=$(head /dev/urandom | tr -dc A-Za-z0-9 | head -c 16)
    TIMESTAMP=$(date +%s)
    LABEL="Set DNS to external DHCP $TIMESTAMP"
    REQUEST_BODY=$(printf '{"login":"%s","nonce":"%s","read_only":false,"expiration_time":"10 minutes","label":"%s","global_key":true}' "$LOGIN_NAME" "$NONCE" "$LABEL")
    log_message "Request Body: $REQUEST_BODY"
    SIGNATURE=$(echo -n "$REQUEST_BODY" | openssl dgst -sha512 -sign "$PRIVATE_KEY_PATH" | base64 | tr -d '\n')
    log_message "Generated signature: $SIGNATURE"
    RESPONSE=$(curl -s -X POST "$AUTH_URL" \
        -H "Signature: $SIGNATURE" \
        -H "Content-Type: application/json" \
        -d "$REQUEST_BODY") || handle_error "Failed to connect to TransIP API"
    log_message "API-response received: $RESPONSE"
    ACCESS_TOKEN=$(echo "$RESPONSE" | jq -r '.token // empty')
    if [ -z "$ACCESS_TOKEN" ]; then
        handle_error "The received access token is empty or invalid. Check the API-response: $RESPONSE"
    fi
    log_message "Received token: $ACCESS_TOKEN"
}

# Function to get the current DNS A-record from TransIP
get_dns_record() {
    # Get API response and store status code
    RESPONSE=$(curl -s -X GET \
        -H "Authorization: Bearer $ACCESS_TOKEN" \
        "https://api.transip.nl/v6/domains/$DOMAIN/dns")

    # Check if response contains JSON and no errors
    if echo "$RESPONSE" | jq empty 2>/dev/null; then
        RECORD_CONTENT=$(echo $RESPONSE | jq -r '.dnsEntries[] | select(.name=="'$SUBDOMAIN'" and .type=="'$RECORD_TYPE'") | .content')
    else
        handle_error "API response does not contain valid JSON or contains an error: $RESPONSE"
    fi

    # Check if we found a valid record
    if [ -z "$RECORD_CONTENT" ]; then
        log_message "No valid DNS record found for $SUBDOMAIN.$DOMAIN. Record may need to be created."
        return 1
    fi

    echo "$RECORD_CONTENT"
    return 0
}

# Function to update or create DNS record
update_or_create_dns_record() {
    # Check if record exists
    CURRENT_DNS_IP=$(get_dns_record)

    if [ -z "$CURRENT_DNS_IP" ]; then
        log_message "No DNS A-record found for $SUBDOMAIN.$DOMAIN. Creating record."
        # Create request body for new record (TTL set to 60 seconds)
        DNS_ENTRY=$(jq -n \
        --arg name "$SUBDOMAIN" \
        --arg expire "60" \
        --arg type "$RECORD_TYPE" \
        --arg content "$CURRENT_IP" \
        '{"dnsEntry": {"name": $name, "expire": ($expire | tonumber), "type": $type, "content": $content}}')

        log_message "DNS entry (create): $DNS_ENTRY"

        # Execute POST request to create new DNS entry with better status code handling
        RESPONSE=$(curl -s -w "%{http_code}" -X POST \
            -H "Authorization: Bearer $ACCESS_TOKEN" \
            -H "Content-Type: application/json" \
            -d "$DNS_ENTRY" \
            "https://api.transip.nl/v6/domains/$DOMAIN/dns")

        HTTP_STATUS_UPDATE=$(echo "$RESPONSE" | tail -n1)
        RESPONSE_BODY=$(echo "$RESPONSE" | head -n -1)

        log_message "API response for creating DNS entry: $RESPONSE_BODY"
        log_message "HTTP status code received for creation: $HTTP_STATUS_UPDATE"

        if [ "$HTTP_STATUS_UPDATE" -ne 201 ]; then
            handle_error "API call for creating DNS entry failed with status code $HTTP_STATUS_UPDATE."
        fi

        log_message "New DNS A-record successfully created."
    else
        log_message "Updating DNS A-record for $SUBDOMAIN.$DOMAIN to IP $CURRENT_IP"
        # Create request body for updating the record
        DNS_ENTRY=$(jq -n \
        --arg name "$SUBDOMAIN" \
        --arg expire "$TTL" \
        --arg type "$RECORD_TYPE" \
        --arg content "$CURRENT_IP" \
        '{"dnsEntry": {"name": $name, "expire": ($expire | tonumber), "type": $type, "content": $content}}')

        log_message "DNS entry (update): $DNS_ENTRY"

        # Execute PATCH request to update the existing DNS record
        RESPONSE=$(curl -s -w "%{http_code}" -X PATCH \
            -H "Authorization: Bearer $ACCESS_TOKEN" \
            -H "Content-Type: application/json" \
            -d "$DNS_ENTRY" \
            "https://api.transip.nl/v6/domains/$DOMAIN/dns")

        HTTP_STATUS_UPDATE=$(echo "$RESPONSE" | tail -n1)
        RESPONSE_UPDATE_BODY=$(echo "$RESPONSE" | head -n -1)

        log_message "API update response received: $RESPONSE_UPDATE_BODY"
        log_message "HTTP status code received for update: $HTTP_STATUS_UPDATE"

        # Accept both 200 and 204 as successful updates
        if [ "$HTTP_STATUS_UPDATE" -ne 200 ] && [ "$HTTP_STATUS_UPDATE" -ne 204 ]; then
            handle_error "API call for DNS update failed with status code $HTTP_STATUS_UPDATE."
        else
            log_message "DNS A-record successfully updated to $CURRENT_IP"
        fi
    fi
}

# Get the current external IP
CURRENT_IP=$(curl -s http://ipinfo.io/ip) || handle_error "Failed to get external IP"

# Log the start
log_message "---- Start new run ----"
log_message "Current external IP: $CURRENT_IP"

# Always generate a short lived new token.
generate_new_token


# Try to update or create the DNS record
update_or_create_dns_record

log_message "---- End of run ----"

Managing SSL certificates for domains hosted on TransIP can be cumbersome without automation. TransIP luckily offers a robust API for DNS management, but to integrate it seamlessly with Certbot for dns-01 validation, a custom solution is necessary. Since I couldn’t find a ready-made script for this purpose, I created one that securely handles authentication, DNS updates, and certificate issuance.

You can go to the full API documentation here:

TransIP API

This guide explains the full setup using Docker, including how to generate the API key pair in TransIP’s web interface and use it for automated certificate management.

How TransIP authentication works

TransIP uses an RSA key pair for API authentication. Here’s how to set it up:

1. Generate the Key Pair in TransIP:
   • Log in to your TransIP control panel.
   • Go to API Settings under your account and enable API access.
   • Generate a new API key pair.
   • Download/Save the private key (transip_private_key.pem) securely and create a secret from it to use in your Docker setup.

docker secret create transip_private_key transip_private_key.pem

1. Keep the Private Key Secure:
   • Store the private key somewhere safe after creating a secret from it.
   • This private key is needed to sign requests to TransIP’s API.
2. How It Works:
   • The script uses the private key to sign an authentication request.
   • TransIP’s API responds with a short-lived token (10 minutes by default), which is then used to manage the DNS records.
   • It will create a new token for convenience on every run. This made it more resilient then checking for an active token and using that which made the whole script less resilient and cumbersome.

What this setup is going to do

1. Automates SSL Certificates:
   • Automates issuance and renewal of wildcard and domain-specific certificates.
2. Handles DNS Challenges:
   • Manages dns-01 challenges by creating and deleting DNS TXT records via the TransIP API.
3. Seamless Integration:
   • Encapsulates everything in a Docker container for portability and simplicity.
   • You can remove or add domains in this script on the -d rule part. I put the examples in for a better explanation.

We will pipe a add or clean command to the bash script to take the necessary actions as defined in the script. So we use the –manual-auth-hook and the manual-cleanup-hook here. This makes sure that a fail in the add section still cleans up after itself for example.

You can also add a –force to this script. but your will run out of attempts to Letsencrypt quickly since they only allow you to do a few calls every 24 hours. But this will make sure it refreshes the certificates always when you need to test or verify something.

version: "3.8"

services:
  certbot:
    image: certbot/certbot:latest
    container_name: certbot-transip
    secrets:
      - transip_private_key  # Reference the secret you created earlier
    # adjust these volumes to point to the right .pem and script. right now this dockerfile expects the files to be in the same folder as the docker-certbot-transip.yml
    volumes:
      - "./certbot-transip.sh:/usr/local/bin/certbot-transip.sh:ro"
      - "./letsencrypt:/etc/letsencrypt"
    environment:
    #These Variables are going to be piped in our bash script.
      LOGIN_NAME: "your-transip-login"        # Replace with your TransIP login name
      PRIVATE_KEY_PATH: "/etc/transip_private_key.pem"
      DOMAIN: "example.com"                  # Replace with your domain
      TTL: "60"                              # Optional: Time-to-live for DNS challenge
      CERTBOT_EMAIL: "you@example.com"       # Replace with your valid email as requested by letsencrypt.
    entrypoint:
      - sh
      - -c
      - |
        apk update && apk add --no-cache bash curl jq bind-tools && \
        chmod +x /usr/local/bin/certbot-transip.sh && \
        certbot certonly --manual --preferred-challenges dns \
          --manual-auth-hook "/usr/local/bin/certbot-transip.sh add" \
          --manual-cleanup-hook "/usr/local/bin/certbot-transip.sh clean" \
          --cert-name "$DOMAIN" \
          -d "*.$DOMAIN" \
          -d "example2.$DOMAIN" \
          -d "example3.$DOMAIN" \
          --expand \
          --non-interactive --agree-tos \
          --email "$CERTBOT_EMAIL"

secrets:
  transip_private_key:
    external: true  # Secret must already exist in Docker

The Bash script

This is the “advanced” part that will do its magic and make everything work with TransIP. I tried to make it a bit more resilient then standard also. So this should hopefully help in creating a resilient and good working environment.

You can also use this TransIP script to run it locally creating certificates or in a Kubectl or Helm chart.

#!/bin/bash

# Configuration settings
LOGIN_NAME=${LOGIN_NAME}
TTL=${TTL:-60}  # Default value 60 if TTL is not set
PRIVATE_KEY_PATH=${PRIVATE_KEY_PATH}
DOMAIN=${DOMAIN}

AUTH_URL="https://api.transip.nl/v6/auth"
LOG_FILE="/var/log/transip_dyndns.log"

# Function to update a log file
#log_message() {
#    echo "$(date '+%Y-%m-%d %H:%M:%S') - $1" >> $LOG_FILE
#}

# Function to log messages to stdout
log_message() {
    echo "$(date '+%Y-%m-%d %H:%M:%S') - $1"
}

# Function to log an error and stop the script
handle_error() {
    log_message "ERROR: $1"
    echo "ERROR: $1" >&2
    exit 1
}

# Function to obtain a new access token with a short validity period (10 minutes)
generate_new_token() {
    log_message "Generating a new token with a validity period of 10 minutes."

    NONCE=$(head /dev/urandom | tr -dc A-Za-z0-9 | head -c 16)
    TIMESTAMP=$(date +%s)
    LABEL="Certbot DNS update $TIMESTAMP"

    REQUEST_BODY=$(printf '{"login":"%s","nonce":"%s","read_only":false,"expiration_time":"10 minutes","label":"%s","global_key":true}' "$LOGIN_NAME" "$NONCE" "$LABEL")

    log_message "Request Body: $REQUEST_BODY"

    SIGNATURE=$(echo -n "$REQUEST_BODY" | openssl dgst -sha512 -sign "$PRIVATE_KEY_PATH" | base64 | tr -d '\n')

    log_message "Generated signature: $SIGNATURE"

    RESPONSE=$(curl -s -X POST "$AUTH_URL" \
        -H "Signature: $SIGNATURE" \
        -H "Content-Type: application/json" \
        -d "$REQUEST_BODY") || handle_error "Failed to connect to TransIP API"

    log_message "API-response received: $RESPONSE"

    ACCESS_TOKEN=$(echo "$RESPONSE" | jq -r '.token // empty')
    if [ -z "$ACCESS_TOKEN" ]; then
        handle_error "The received access token is empty or invalid. Check the API-response: $RESPONSE"
    fi

    log_message "Received token: $ACCESS_TOKEN"
}

# Function to add or update a TXT record
add_or_update_txt_record() {
    local full_domain="$1"
    local value="$2"

    # Extract the subdomain part from the full domain
    local name="_acme-challenge.${full_domain%%.*}"

    if [ -z "$name" ] || [ -z "$value" ]; then
        log_message "ERROR: Name or value for TXT record is missing."
        exit 1
    fi

    # Check if the DNS record already exists
    RESPONSE=$(curl -s -X GET \
        -H "Authorization: Bearer $ACCESS_TOKEN" \
        "https://api.transip.nl/v6/domains/$DOMAIN/dns")

    # Search for all existing records with the same name and type TXT
    MATCHING_RECORDS=$(echo "$RESPONSE" | jq -r --arg name "$name" '[.dnsEntries[] | select(.name==$name and .type=="TXT")]')

    # Check if there are multiple or no matching records
    MATCHING_COUNT=$(echo "$MATCHING_RECORDS" | jq 'length')

    # Ensure MATCHING_COUNT is always an integer
    if ! [[ "$MATCHING_COUNT" =~ ^[0-9]+$ ]]; then
        MATCHING_COUNT=0
    fi

    if [ "$MATCHING_COUNT" -eq 0 ]; then
        log_message "No existing TXT record found for $name. Creating a new record."
    elif [ "$MATCHING_COUNT" -ge 1 ]; then
        log_message "Multiple or one existing TXT record found for $name. Removing all existing records."

        # Remove all existing records using the full details of each record
        echo "$MATCHING_RECORDS" | jq -c '.[]' | while read -r record; do
            existing_name=$(echo "$record" | jq -r '.name')
            existing_expire=$(echo "$record" | jq -r '.expire')
            existing_content=$(echo "$record" | jq -r '.content')

            # Log the details of the record to be removed
            log_message "Remove record: name=$existing_name, expire=$existing_expire, content=$existing_content"

            # Make the DELETE call using the full DNS-entry details
            DELETE_ENTRY=$(jq -n \
            --arg name "$existing_name" \
            --arg expire "$existing_expire" \
            --arg type "TXT" \
            --arg content "$existing_content" \
            '{"dnsEntry": {"name": $name, "expire": ($expire | tonumber), "type": $type, "content": $content}}')

            DELETE_RESPONSE=$(curl -s -X DELETE \
                -H "Authorization: Bearer $ACCESS_TOKEN" \
                -H "Content-Type: application/json" \
                -d "$DELETE_ENTRY" \
                "https://api.transip.nl/v6/domains/$DOMAIN/dns")

            # Log the response of the DELETE call
            log_message "Response after removing $existing_name: $DELETE_RESPONSE"

            # Check if the removal was successful
            if [[ "$DELETE_RESPONSE" == *"error"* ]]; then
                log_message "ERROR: The record $existing_name could not be removed."
            else
                log_message "Record $existing_name successfully removed."
            fi
        done
    fi

    # Create a new record
    DNS_ENTRY=$(jq -n \
    --arg name "$name" \
    --arg expire "$TTL" \
    --arg type "TXT" \
    --arg content "$value" \
    '{"dnsEntry": {"name": $name, "expire": ($expire | tonumber), "type": $type, "content": $content}}')

    RESPONSE=$(curl -s -X POST \
        -H "Authorization: Bearer $ACCESS_TOKEN" \
        -H "Content-Type: application/json" \
        -d "$DNS_ENTRY" \
        "https://api.transip.nl/v6/domains/$DOMAIN/dns")

    log_message "API-response for TXT record creation: $RESPONSE"

    if [[ "$RESPONSE" == *"error"* ]]; then
        handle_error "The TXT record could not be created. Check the API-response: $RESPONSE"
    else
        log_message "TXT record successfully added for $name with value $value"
    fi
}

# Function to update an existing TXT record
update_txt_record() {
    local name="$1"
    local value="$2"

    DNS_ENTRY=$(jq -n \
    --arg name "$name" \
    --arg expire "$TTL" \
    --arg type "TXT" \
    --arg content "$value" \
    '{"dnsEntry": {"name": $name, "expire": ($expire | tonumber), "type": $type, "content": $content}}')

    RESPONSE=$(curl -s -X PATCH \
        -H "Authorization: Bearer $ACCESS_TOKEN" \
        -H "Content-Type: application/json" \
        -d "$DNS_ENTRY" \
        "https://api.transip.nl/v6/domains/$DOMAIN/dns")

    log_message "API-response for TXT record update: $RESPONSE"

    if [[ "$RESPONSE" == *"error"* ]]; then
        handle_error "The TXT record could not be updated. Check the API-response: $RESPONSE"
    else
        log_message "TXT record successfully updated for $name with value $value"
    fi
}

# Function to remove a TXT record for `dns-01` validation
remove_txt_record() {
    local full_domain="$1"

    # Extract the subdomain part from the full domain
    local name="_acme-challenge.${full_domain%%.*}"

    # Check if the DNS record exists
    RESPONSE=$(curl -s -X GET \
        -H "Authorization: Bearer $ACCESS_TOKEN" \
        "https://api.transip.nl/v6/domains/$DOMAIN/dns")

    # Search for all records with the same name and type TXT
    MATCHING_RECORDS=$(echo "$RESPONSE" | jq -r --arg name "$name" '[.dnsEntries[] | select(.name==$name and .type=="TXT")]')

    # Check if there are existing records that need to be removed
    MATCHING_COUNT=$(echo "$MATCHING_RECORDS" | jq 'length')

    if [ "$MATCHING_COUNT" -eq 0 ]; then
        log_message "No existing TXT record found for $name to remove."
    else
        log_message "Removing all existing TXT records for $name."

        # Loop through all found records and remove them
        echo "$MATCHING_RECORDS" | jq -c '.[]' | while read -r record; do
            existing_name=$(echo "$record" | jq -r '.name')
            existing_expire=$(echo "$record" | jq -r '.expire')
            existing_content=$(echo "$record" | jq -r '.content')

            # Make the DELETE call using the full details
            DELETE_ENTRY=$(jq -n \
            --arg name "$existing_name" \
            --arg expire "$existing_expire" \
            --arg type "TXT" \
            --arg content "$existing_content" \
            '{"dnsEntry": {"name": $name, "expire": ($expire | tonumber), "type": $type, "content": $content}}')

            DELETE_RESPONSE=$(curl -s -X DELETE \
                -H "Authorization: Bearer $ACCESS_TOKEN" \
                -H "Content-Type: application/json" \
                -d "$DELETE_ENTRY" \
                "https://api.transip.nl/v6/domains/$DOMAIN/dns")

            # Check if the removal was successful
            if [[ "$DELETE_RESPONSE" == *"error"* ]]; then
                log_message "ERROR: The TXT record $existing_name could not be removed."
            else
                log_message "TXT record $existing_name successfully removed."
            fi
        done
    fi
}

# Always generate a new token for each run
generate_new_token

# Certbot passes these values when adding and removing the record
if [ "$1" == "add" ]; then
    add_or_update_txt_record "$CERTBOT_DOMAIN" "$CERTBOT_VALIDATION"
    sleep 25  # Wait for DNS update to propagate
elif [ "$1" == "clean" ]; then
    remove_txt_record "$CERTBOT_DOMAIN"
fi

Steps to run

1. Download the Private Key: Save the private key file (transip_private_key.pem) from TransIP’s API settings somewhere on your device you are going to run this from.
2. Set Up Docker Compose: Place the docker-compose.yml and certbot-transip.sh files in the same directory. (or edit the script)
3. Run the Container: Start the container (cd to the path where your yml is)

docker-compose up --build

Monitor the Log output: The log output of the docker container will show the progress of DNS challenges and certificate issuance.

Thats all there is to it!

I will also post a version with a working helm chart soon. This was a little bigger culprit to make sure it all works correctly and updates all certificates through different namespaces where i needed the certificates to be available on different domains.

The API: https://api.protonvpn.ch/vpn/logicals has been formally shut down according to Proton Support. The only thing it returns now is:

{"Code":0,"Error":"Invalid access token","Details":{}}

So this post is sadly obsolete now. Somebody started a petition to get this feature back.

The deprecated guide which no longer functions as of 06/2025 For historical reasons.

When ProtonVPN discontinued its command-line interface (CLI) tool, users of headless systems like Raspberry Pi were left without an easy method to manage VPN connections. Without a graphical user interface, manually switching VPN servers has become a cumbersome task. Also selecting the fastest VPN in a pool of servers has become impossible.

To solve this problem, I developed a Python script that automatically selects the fastest ProtonVPN server based on latency and load, then updates your WireGuard configuration for you. This guide will walk you through setting up WireGuard for ProtonVPN, installing the script, and automating your VPN connection to ensure your Raspberry Pi is always connected to the best server.

Why do i need this Protonss script?

This Python script automates the process of connecting your Raspberry Pi to the fastest ProtonVPN server available, using WireGuard as the VPN protocol. It checks the latency and load of multiple VPN servers from ProtonVPN’s API, selects the one with the best performance, updates your WireGuard configuration with the new server information, and reconnects your VPN only if a significantly faster server is found (nobody actually likes reconnects). This script also logs the entire process for easier troubleshooting.

1. No more manual VPN switching: Without ProtonVPN’s CLI, switching to the fastest server would require manual work. This script automates the process by regularly checking the available servers and updating the connection based on real-time performance.
2. Optimizes connection speed: The script ensures you’re always connected to the VPN server with the lowest latency and load. This is especially useful for maintaining high-speed connections in countries with fast or low-traffic ProtonVPN servers.
3. Reduces downtime: By automating the reconnection process, the script helps minimize potential downtimes caused by poor server performance or slow connections.
4. Tailored for Raspberry Pi: If you’re running a headless Raspberry Pi (without a monitor), this script is perfect for automating VPN tasks without requiring constant oversight.
5. Security & Privacy: By ensuring your Raspberry Pi always connects to a VPN, it helps protect your data and online activity, especially if you’re using the Pi as a torrenting or networking device.

In short, this script automates finding the optimal ProtonVPN server, providing faster speeds, more reliability, and convenience for headless systems like a Raspberry Pi.

Let’s dive in!

First, you need to install WireGuard, which is a fast, lightweight VPN protocol which we will use to connect to our ProtonVPN server.

# First, update your package list and install WireGuard:
sudo apt update
sudo apt install wireguard curl resolvconf

Generate a WireGuard configuration from ProtonVPN

• Log in to your ProtonVPN account on their website.
• Go to the “Downloads” section in your dashboard.
• Generate a new configuration file under “WireGuard Configurations.”
• If you’re using torrents, enable NAT-PMP.

Save this configuration file on your Raspberry Pi:

# Save the configuration file to your Raspberry Pi:
# Place it in the WireGuard directory and rename it for simplicity:
sudo nano /etc/wireguard/protonvpn.conf

Once you’ve placed the configuration file in the correct directory, verify the connection.

# Verify connection to ProtonVPN:
sudo wg-quick up protonvpn

# This should give an ip address different to the one frome your isp
curl ifconfig.me 

# Verify connection to ProtonVPN:
sudo wg-quick down protonvpn

curl ifconfig.me 

If these two ip addresses from ifconfig.me are different your are connected to a VPN. even if you dont know your ip address since its a DHCP address or other.

Installing our Python environment for Protonss.

When developing and running Python scripts on your Raspberry Pi, it’s best to use a Python virtual environment. A virtual environment allows you to install and manage Python packages in an isolated setting, ensuring that your system’s default Python configuration remains untouched.

This way, any changes made to your environment will only affect the project you’re working on—such as installing specific packages like requests—without causing conflicts with other Python projects or system-level dependencies.

Here’s how you can set up and use a virtual environment for our script:

1. Install virtual environment: First, install the python3-venv package, which allows you to create and manage virtual environments.

sudo apt install -y python3-venv python3-full

2. Create a virtual environment: Once installed, you can create a virtual environment. This will generate an isolated environment in your specified directory (in this case, ~/protonss).

python3 -m venv ~/protonss

3. Activate the virtual environment: After creating the environment, you’ll need to activate it. This tells your system to use the Python and packages from the virtual environment, instead of the system-wide Python installation.

source ~/protonss/bin/activate

• Now that the environment is active, any Python package you install (such as requests) will be installed into this isolated environment.

pip install requests

Using this virtual environment in the future: Each time you want to run Python scripts that depend on this virtual environment, you’ll need to activate it again using the source ~/venv/bin/activate command. Once activated, any Python commands or packages you use will run in this environment.

Installing our ProtonSS script to automate a connection to the fastest VPN.

We will create a configuration file to hold all the variables that the Python script will use. For example, you can define the path to your WireGuard configuration, log file, and the number of pings to test latency.

cd ~/protonss

Sample config.ini:

nano config.ini

[Settings]
# Path to your WireGuard configuration file
wg_config_path = /etc/wireguard/protonvpn.conf

# Path to your log file
log_file = /var/log/wireguard_updater.log

# Number of pings to measure latency
ping_count = 3

# Maximum score difference before reconnection 0 - 100
# Average score of servers is between 10-40 it seems. so setting this to high results in no switches from servers
score_threshold = 5.0

# Comma-separated list of preferred countries
preferred_countries = US

# Comma-separated list of preferred cities
# If specified, servers must be in these cities (unless they're in a country without any specified city)
# Preferred cities to connect to. If specified, servers must be in these cities
# (unless they're in a country without any specified city).
# Example: preferred_cities = Phoenix, Los Angeles, Seattle
# This allows you to target specific cities within your preferred countries.
# If left empty, any city within the preferred countries will be considered.
# Note: City names are case sensitive and must match ProtonVPN's naming exactly.
# Multiple cities should be comma-separated.
preferred_cities =Phoenix

# Weight for latency in the server score
# latency is subjective since this is probably measured through the vpn and only counts as server latency.
# Checking the latency without a vpn isnt recommended since the proton servers wont reply back without vpn.
# Value cant be higher then 1. 0.3 translates to 30% of the score_treshold.
latency_weight = 0.3

# Weight for server load in the server score
# Load is the best factor 0.7 translates to 70% of the weight of the score_threshold. this cant be higher then 1.
load_weight = 0.7

# Enable or disable pinging servers (True or False)
# Since ping might not be reliable enough for you to keep on switching servers this can be disabled here.
ping_enabled = True

# Specify the interface to use for ping tests. By setting this option you can ping to the fastest vpn server
# this will ensure the lowest latency VPN from your location.
ping_interface = eth0

[Features]
# Choose which type of servers you want to use
# Options: p2p, plus, secure_core, tor
# Default is p2p if not specified
feature_type = p2p

# Feature explanation:
# Each server has exactly ONE feature number:
# 1, 17 = Secure Core servers
# 2 = TOR servers
# 0, 8, 16, 24 = Plus-only servers
# 4, 12, 20, 28 = P2P servers (these are also Plus servers)

You can customize the variables as needed. For example, leaving preferred_countries empty will make the script check a LOT! of servers, which will make the script run a very very long time!

Create the Python script

Now, we are going to create the Python script that will automate your VPN connection. We first need to prepare a few steps

The first line of the script should be a shebang, which ensures the script always runs in the correct Python environment:

source ~/protonss/bin/activate
which python

This will provide you with the pad you need to put on the first rule of this script

/home/pi/venv/bin/python

Modify the script by adding a shebang: Add the following line at the very top of your Python script. Replace /home/{user}/venv/bin/python with the actual path you got when you ran the which python command earlier. Make sure to keep the #! at the beginning of the line—this tells the system which interpreter to use to run the script. we need the absolute path so ~/ wont work.

nano protonvpn_auto_connect.py

#!/home/{user}/protonss/bin/python

import requests
import subprocess
import os
import sys
import configparser
import logging
import re
from datetime import datetime
import math
import shutil

# Get the directory of the current script
script_dir = os.path.dirname(os.path.abspath(__file__))

# Load the config file from the same directory as the script
config = configparser.ConfigParser()
config.read(os.path.join(script_dir, 'config.ini'))

WG_CONFIG_PATH = config.get('Settings', 'wg_config_path')
LOG_FILE = config.get('Settings', 'log_file')
PING_COUNT = config.getint('Settings', 'ping_count')
SCORE_THRESHOLD = config.getfloat('Settings', 'score_threshold', fallback=5.0)
PREFERRED_COUNTRIES = config.get('Settings', 'preferred_countries', fallback='').split(',')
PREFERRED_CITIES = config.get('Settings', 'preferred_cities', fallback='').split(',')
PING_INTERFACE = config.get('Settings', 'ping_interface', fallback=None)

# Weighting factors for latency and load from the config
LATENCY_WEIGHT = config.getfloat('Settings', 'latency_weight', fallback=0.3)
LOAD_WEIGHT = config.getfloat('Settings', 'load_weight', fallback=0.7)

# Ping option from config
PING_ENABLED = config.getboolean('Settings', 'ping_enabled', fallback=True)

# Clean up country codes (remove spaces and convert to uppercase)
PREFERRED_COUNTRIES = [country.strip().upper() for country in PREFERRED_COUNTRIES if country.strip()]

# Clean up city names (remove spaces)
PREFERRED_CITIES = [city.strip() for city in PREFERRED_CITIES if city.strip()]

# Set up logging
logging.basicConfig(
    filename=LOG_FILE,
    level=logging.INFO,
    format='%(asctime)s [%(levelname)s] %(message)s',
    datefmt='%Y-%m-%d %H:%M:%S'
)

logging.info('--- ProtonVPN Auto Connect Script Start ---')

# Feature definitions
FEATURE_SECURE_CORE = [1, 17]  # Secure Core servers
FEATURE_TOR = [2]         # TOR servers
FEATURE_PLUS = [0, 8, 16, 24]  # Plus-only servers
FEATURE_P2P = [4, 12, 20, 28]    # P2P servers (these are also Plus servers)
# check https://api.protonmail.ch/vpn/logicals for more info. 
# be aware that the secure core and tor server are very limited so check if one is available for your country as stated in the config.ini file.
# FREE servers are not included in the api response. so they are already left out.

def get_feature_description(features):
    # If no feature type is specified in config.ini, default to P2P
    # Features is a number representing server capabilities
    if not features:  # If features is 0 or None
        return "P2P"  # Default to P2P
    elif features in FEATURE_SECURE_CORE:
        return "Secure Core"
    elif features in FEATURE_TOR:
        return "TOR" 
    elif features in FEATURE_P2P:
        return "P2P"
    elif features in FEATURE_PLUS:
        return "Plus"
    else:
        return "P2P"  # Default to P2P for any unrecognized feature

def get_wanted_features():
    """Get the wanted feature type based on config priorities"""
    feature_type = config.get('Features', 'feature_type', fallback='p2p').lower()
    
    if feature_type == 'secure_core':
        return FEATURE_SECURE_CORE
    elif feature_type == 'tor':
        return FEATURE_TOR
    elif feature_type == 'plus':
        return FEATURE_PLUS
    else:  # Default to P2P
        return FEATURE_P2P

# Function to fetch the list of servers from the ProtonVPN API
def get_servers(full_list=False):
    try:
        url = 'https://api.protonmail.ch/vpn/logicals'
        response = requests.get(url)
        response.raise_for_status()
        data = response.json()
        logging.info("API response data received.")
        return data['LogicalServers']
    except requests.RequestException as e:
        logging.error(f'Error fetching server list: {e}')
        sys.exit(1)
    except ValueError as e:
        logging.error(f'Error parsing JSON: {e}')
        sys.exit(1)

# Function to filter P2P servers and servers in preferred countries
def filter_servers(servers):
    filtered_servers = []
    
    # Get feature preferences from config (nu maar één type)
    wanted_features = get_wanted_features()
    
    logging.info(f"Filtering servers with preferences - Countries: {PREFERRED_COUNTRIES or 'Any'}, "
                f"Cities: {PREFERRED_CITIES or 'Any'}, Features: {[get_feature_description(f) for f in wanted_features]}")
    
    for server in servers:
        if not isinstance(server, dict):
            logging.error(f"Unexpected type for server: {type(server)} - Value: {server}")
            continue
            
        features = server.get('Features', 0)
        country = server['ExitCountry']
        city = server.get('City', '')

        # Check if server features match our wanted features
        if features in wanted_features:
            # Check country and city preferences
            if country in PREFERRED_COUNTRIES:
                if not PREFERRED_CITIES or city in PREFERRED_CITIES:
                    filtered_servers.append(server)
                    logging.info(f"Server matched - Country: {country}, City: {city}, "
                              f"Features: {get_feature_description(features)}")
            
    if not filtered_servers:
        logging.warning(f"No servers found matching filters. Countries: {PREFERRED_COUNTRIES}, "
                      f"Cities: {PREFERRED_CITIES}, Features: {[get_feature_description(f) for f in wanted_features]}")
    else:
        logging.info(f"Found {len(filtered_servers)} matching servers")
        
    return filtered_servers

# Function to measure latency by pinging the server (optional)
def get_latency(hostname):
    if not PING_ENABLED:
        logging.info(f"Ping disabled, skipping latency check for {hostname}.")
        return 0  # Return 0 latency if ping is disabled

    latencies = []
    for i in range(PING_COUNT):
        try:
            # Add the interface option if it's specified
            ping_command = ['ping', '-c', '1', '-W', '1', hostname]
            if PING_INTERFACE:
                ping_command.insert(1, '-I')
                ping_command.insert(2, PING_INTERFACE)

            result = subprocess.run(ping_command, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
            if result.returncode == 0:
                output = result.stdout.decode()
                latency_line = [line for line in output.split('\n') if 'time=' in line][0]
                latency = float(latency_line.split('time=')[1].split(' ms')[0])
                latencies.append(latency)
            else:
                latencies.append(float('inf'))
        except Exception as e:
            latencies.append(float('inf'))
            logging.error(f'Ping error for {hostname}: {e}')

    # Return the lowest (best) latency value
    best_latency = min(latencies)
    
    # Handle inf latency
    if math.isinf(best_latency):
        logging.warning(f"Latency for {hostname} is inf. Assigning high latency value.")
        best_latency = 1000  # Assign a high latency value

    return best_latency

# Function to select the best server based on a weighted combination of latency and load
def select_best_server(servers):
    best_server = None
    best_score = float('inf')
    
    for server in servers:
        server_load = server.get('Load')
        
        # Handle inf load
        if server_load is None or math.isinf(server_load):
            logging.warning(f"Server {server['Name']} has inf load. Assigning high load value.")
            server_load = 100  # Assign a high load value if it's inf or None
        else:
            server_load = int(server_load)
        
        for srv in server['Servers']:
            exit_ip = srv.get('ExitIP')
            hostname = exit_ip

            # Get latency, only if ping is enabled
            latency = get_latency(hostname)

            # Combine latency and load using the configured weights
            score = (latency * LATENCY_WEIGHT) + (server_load * LOAD_WEIGHT)
            
            logging.info(f"Server {server['Name']} - Latency: {latency:.2f} ms, Load: {server_load}%, Score: {score:.2f}")
            
            if score < best_score:
                best_score = score
                best_server = server
                best_server_instance = srv

    return best_server, best_server_instance, best_score

# Function to read the current server IP from the WireGuard configuration
def get_current_server_ip():
    try:
        with open(WG_CONFIG_PATH, 'r') as f:
            config_text = f.read()
        # Extract the current Endpoint IP
        match = re.search(r'Endpoint\s*=\s*([\d\.]+):\d+', config_text)
        if match:
            logging.info(f"Current server IP found in WireGuard configuration: {match.group(1)}")
            return match.group(1)
        else:
            logging.warning('No current Endpoint IP found in WireGuard configuration.')
            return None
    except Exception as e:
        logging.error(f'Error reading the WireGuard configuration: {e}')
        return None

# Function to find the current server's performance in the live server data
def get_current_server_performance(servers, current_ip):
    if current_ip is None:
        logging.warning('Current server IP is None, skipping performance check.')
        return None, None, float('inf')
    
    for server in servers:
        for srv in server['Servers']:
            if srv['ExitIP'] == current_ip:
                server_load = server.get('Load')
                if server_load is None or math.isinf(server_load):
                    logging.warning(f"Server {server['Name']} (IP: {current_ip}) has inf load. Assigning high load value.")
                    server_load = 100  # Assign a high load value
                else:
                    server_load = int(server_load)

                latency = get_latency(current_ip) if PING_ENABLED else 0
                score = (latency * LATENCY_WEIGHT) + (server_load * LOAD_WEIGHT)

                logging.info(f"Current server {server['Name']} - Latency: {latency:.2f} ms, Load: {server_load}%, Score: {score:.2f}")
                return server, srv, score
    logging.warning(f"Current server with IP {current_ip} not found in live data.")
    return None, None, float('inf')

# Function to find the current server in the full ProtonVPN list if not found in filtered data
def find_current_server_full_list(current_ip):
    servers = get_servers(full_list=True)
    return get_current_server_performance(servers, current_ip)

# Function to update the WireGuard configuration file
def update_wg_config(server, server_instance, best_score):
    try:
        endpoint_ip = server_instance['EntryIP']
        endpoint_port = 51820  # Default WireGuard port
        endpoint = f"{endpoint_ip}:{endpoint_port}"
        public_key = server_instance['X25519PublicKey']
        server_name = server['Name']

        # Read the current WireGuard configuration
        with open(WG_CONFIG_PATH, 'r') as f:
            config_text = f.read()

        # Strip comments that start with # under [Peer] before PublicKey
        config_text = re.sub(r'#.*\n', '', config_text, flags=re.M)

        # Update the Endpoint in the configuration
        config_text = re.sub(r'Endpoint\s*=.*', f'Endpoint = {endpoint}', config_text)

        # Update the PublicKey in the configuration
        config_text = re.sub(r'PublicKey\s*=.*', f'PublicKey = {public_key}', config_text)

        # Add the new comment indicating the chosen server and score
        config_text = re.sub(r'\[Peer\]\n', f'[Peer]\n# {server_name} - Auto-generated by ProtonVPN Auto Connect - Score: {best_score:.2f}\n', config_text)

        # Write the updated configuration back to the file
        with open(WG_CONFIG_PATH, 'w') as f:
            f.write(config_text)
        os.chmod(WG_CONFIG_PATH, 0o600)

        logging.info(f"WireGuard configuration updated with server: {server_name} ({server['ExitCountry']}), Endpoint: {endpoint}, PublicKey: {public_key}, Score: {best_score:.2f}")
    except Exception as e:
        logging.error(f'Error updating the WireGuard configuration: {e}')
        sys.exit(1)

# Function to restart the WireGuard interface
def is_resolvconf_available():
    """Check if resolvconf is available on the system."""
    return shutil.which("resolvconf") is not None

def is_wg_quick_available():
    """Check if wg-quick is available on the system."""
    return shutil.which("wg-quick") is not None

def restart_wg():
    interface_name = os.path.basename(WG_CONFIG_PATH).split('.')[0]
    
    # Check if wg-quick is available before proceeding
    if not is_wg_quick_available():
        logging.error("wg-quick not found. Ensure WireGuard is installed correctly.")
        sys.exit(1)

    try:
        # Check if resolvconf is available before proceeding
        if not is_resolvconf_available():
            logging.warning("resolvconf not found, DNS settings may not be updated correctly.")

        # Bring down the interface
        subprocess.run(['wg-quick', 'down', interface_name], stdout=subprocess.PIPE, stderr=subprocess.PIPE)

        # Bring up the interface
        subprocess.run(['wg-quick', 'up', interface_name], check=True)
        logging.info(f'WireGuard interface {interface_name} started.')

    except subprocess.CalledProcessError as e:
        logging.error(f'Error starting the WireGuard interface: {e}')
        sys.exit(1)

# Main function
def main():
    logging.info('--- Starting server selection process ---')
    
    # Fetch the list of servers
    servers = get_servers()
    if not isinstance(servers, list):
        logging.error(f'Unexpected type for servers: {type(servers)}')
        sys.exit(1)

    # Filter the servers
    filtered_servers = filter_servers(servers)
    if not filtered_servers:
        logging.error('No suitable servers available after filtering.')
        sys.exit(1)

    # Get the current server's IP address from the WireGuard configuration
    current_ip = get_current_server_ip()

    # Get the current server's performance if it exists
    current_server, current_server_instance, current_score = get_current_server_performance(filtered_servers, current_ip)

    # If the current server is not found, check the full list of servers
    if current_server is None:
        logging.info(f"Current server not found in filtered list, checking full server list...")
        current_server, current_server_instance, current_score = find_current_server_full_list(current_ip)

    # Select the best server based on weighted latency and load
    best_server, best_server_instance, best_score = select_best_server(filtered_servers)
    if not best_server:
        logging.error('No suitable server found.')
        sys.exit(1)

    if current_server:
        logging.info(f"Current server {current_server['Name']} (IP: {current_ip}) - Score: {current_score:.2f}")
    
    logging.info(f"Selected best server {best_server['Name']} - Score: {best_score:.2f}")
    
    if current_server and current_score <= best_score + SCORE_THRESHOLD:
        logging.info(f"Current server {current_server['Name']} is within threshold. No switch necessary.")
    else:
        logging.info(f"Switching to server {best_server['Name']} due to better performance.")
        update_wg_config(best_server, best_server_instance, best_score)
        restart_wg()

    logging.info('--- ProtonVPN Auto Connect Script End ---')

if __name__ == '__main__':
    main()

Testing the script

chmod +x protonvpn_auto_connect.py

Once the script is saved, give it execution permissions and run it

sudo ./protonvpn_auto_connect.py

cat /var/log/wireguard_updater.log
tail -f /var/log/wireguard_updater.log

you can tail or cat the log file to check its output in the logs. Tail is preferred since you can keep watching it while running your script. Just open a second ssh with the tail to keep reviewing it.

2024-09-25 22:19:10 [INFO] --- ProtonVPN Auto Connect Script Start ---
2024-09-25 22:19:10 [INFO] --- Starting server selection process ---
2024-09-25 22:19:11 [INFO] API response data received.
2024-09-25 22:19:11 [INFO] Current server IP found in WireGuard configuration: 79.135.105.20
2024-09-25 22:19:11 [WARNING] Current server with IP 79.135.105.20 not found in filtered data.
2024-09-25 22:19:11 [INFO] Current server not found in filtered list, checking full server list...
2024-09-25 22:19:12 [INFO] API response data received.
2024-09-25 22:19:13 [INFO] Current server BH#1 - Latency: 26.90 ms, Load: 10%, Score: 15.07
2024-09-25 22:19:14 [INFO] Server AO#1 - Latency: 27.10 ms, Load: 7%, Score: 13.03
2024-09-25 22:19:15 [INFO] Server AO#2 - Latency: 26.60 ms, Load: 14%, Score: 17.78
2024-09-25 22:19:16 [INFO] Server AO#3 - Latency: 27.10 ms, Load: 9%, Score: 14.43
2024-09-25 22:19:17 [INFO] Server AO#4 - Latency: 26.60 ms, Load: 16%, Score: 19.18
2024-09-25 22:19:17 [INFO] Current server BH#1 (IP: 79.135.105.20) - Score: 15.07
2024-09-25 22:19:17 [INFO] Selected best server AO#1 - Score: 13.03
2024-09-25 22:19:17 [INFO] Current server BH#1 is within threshold. No switch necessary.
2024-09-25 22:19:17 [INFO] --- ProtonVPN Auto Connect Script End ---

sudo cat /etc/wireguard/protonvpn.conf

You can also verify that everything works by checkin the protonvpn.conf file or by checking your external ip address.

[Peer]
# BH#3 - Auto-generated by ProtonVPN Auto Connect - Score: 7.74

you will see something like this above. This comment will update when it connects to another VPN and the ipaddress inside the peer also changes with it off course.

curl ifconfig.me #this wil give the external ip in the cli.

As an additional step you can also check your external ip. this should be something different then the ip address provided by your provider.

Running this script every hour.

sudo crontab -e

0 * * * * cd /home/{user}/protonss && /home/{user}/protonss/protonvpn_auto_connect.py >> /home/{user}/protonss/protonvpn_cron.log 2>&1

Here we defined it to run every hour. but can you schedule it how you want.

sudo crontab -l

Save the crontab and verify the cron is installed.

Optional: Block specific traffic if your VPN fails

If you want to ensure no traffic leaves your device if the VPN connection fails, you can use iptables to block traffic when the WireGuard interface is down.

sudo apt-get install iptables

Verify that you have installed iptables or install it with the above command.

ifconfig -a

protonvpn: flags=209<UP,POINTOPOINT,RUNNING,NOARP>  mtu 1420
        inet 10.2.1.3  netmask 255.255.255.255  destination 10.2.1.3
        unspec 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  txqueuelen 1000  (UNSPEC)
        RX packets 612  bytes 101768 (99.3 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 765  bytes 101144 (98.7 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

First check the name of your network interface. If you named your file protonvpn.conf it should be named protonvpn.

sudo iptables -A OUTPUT -p tcp --dport 51413 ! -o protonvpn -j REJECT --reject-with icmp-port-unreachable
sudo iptables -A OUTPUT -p udp --dport 51413 ! -o protonvpn -j REJECT --reject-with icmp-port-unreachable

you can add any rule to your iptables. The 2 rules below make sure this port 51413 is rejected when the network interface called protonvpn isn’t available. There is a rule for UDP and TCP traffic in place.

sudo iptables -L OUTPUT -v -n

After adding all your required rules you can verify them with this command .

sudo apt-get install iptables-persistent

And make these rules persistent so you dont have to add them on every reboot.

sudo netfilter-persistent save

Conclusion

With this Python script, your Raspberry Pi will always be connected to the fastest ProtonVPN server available, ensuring you have the best possible VPN performance without manual intervention. Whether you’re using your Pi for networking, torrenting, or general online security, this automation ensures optimal performance and reliable privacy protection.

Update - 20-12-2024

Added Preferred_cities as an option to shorten the US country select. That included over 2000 servers to check otherwise. The performance check can at least get a bit shorter this way :-).

And i found out what each Feature section for every server in the API includes. I will explain this below. I adjusted my code accordingly to work with this correctly. Be aware that the config you made earlier should reflect the type of server you are connecting to. If your Configuration isnt configured to connect to a Secure Core dont select a Secure Core in this script also. Reflect to the created ProtonVPN Configuration.

Feature explanation:

Each server has exactly ONE feature number:

• 1, 17 = Secure Core servers
• 2 = TOR servers
• 0, 8, 16, 24 = Plus-only servers
• 4, 12, 20, 28 = P2P servers (these are also Plus servers it seems)

        {
            "Name": "US-AZ#105",
            "EntryCountry": "US",
            "ExitCountry": "US",
            "Domain": "node-us-260.protonvpn.net",
            "Tier": 2,
            "Features": 12,

After some more investigation regarding questions about the Free servers from ProtonVPN, it appears that ProtonVPN has removed public API access to their free server endpoints. The /vpn/servers endpoint that previously should have provided information about free servers now returns a 404 error:

{    "Code": 404,    "Error": "Path not found",    "Details": {}}

Currently, only the Plus server endpoint is accessible via the public

API:url = ‘https://api.protonmail.ch/vpn/logicals’

This is likely a deliberate change by ProtonVPN. So sadly this script cant support the Free servers.

A lot is being written about Scrum already. Scrum is dead and so on. Well when you have a vacancy for a “Manager” a “Janitor” or some other clearly non Scrum title but decide to call it a vacancy for a Product Owner or a Scrum Master. It’s you who are fucking up. just renaming the beast isn’t going to make it work like magic.

We’ve all heard it: “Let’s do Scrum, and Poof! We’ll deliver value in no time!” (Spoiler alert) it doesn’t work like that. Scrum isn’t some magic potion that turns chaos into order just by sprinkling a few ceremonies around. It takes a lot more than following the steps in the Scrum guide to get to real results. It’s all about teamwork, discipline, and having the right people in placeand believe it or not, it’s not that different from how things are trained in the army ;-).

Sprint equals Mission

Let’s start once again with the elephant in the room, the word “sprint.” We’re not actually running a race, are we? This term still suggests rushing to me, but in reality, we’re aiming for a deliberate, well-planned mission. Like in the army, it’s not about speed it’s about precision and success. A well-executed mission isn’t just thrown together, it is strategically planned and relies on everyone playing their role and pulling their weight to the finish line.

Doing a mission requires adaptation along the way. You are going to encounter unknown’s and you need to be prepared to deal with them instead of avoid/ignoring them and fuckup your mission.

Solving unknowns as a **TEAM**is what’s going to make an Agile/Scrum approach work.

Specialized, but all-round

In both Scrum teams and military units, we all have our own specialties and advantages. But here’s the kicker. We deliver value as a team. Just like the army, where soldiers are cross-trained to fill in gaps, we’re all-rounders too. If something goes wrong, we don’t just throw our hands up and call it a day. We adapt and get it done, together (as a team!).

Scrum doesn’t work without the right mindset {#scrum-doesn%E2%80%99t-work-without-the-right-mindset}

Scrum doesn’t run itself. Just adopting Scrum won’t suddenly create value. I’ve seen it happen. People think if they run the ceremonies and just follow some Scrum related steps, magic will happen. But without your team managing the process itself, it’s like trying to run an army mission with no captain, no briefing, and everyone just winging it. Sure, you might finish, but what have you actually done?

The army knows this well. There’s a structure, leaders who guide, but also a reliance on every team member being proactive. In Scrum, you need the right mix: a Product Owner with a mandate and an actual vision that backs the company, a Scrum Master who keeps the flow going, and a team that’s not just doing tasks but thinking about the bigger picture. You will achieve this mindset only by working together and keep talking as a single team.

A team that talks about the hard stuff

As a team we need to work together, talk openly, and tackle the hard stuff. I’m not talking about easy conversations where everyone agrees. I mean difficult, uncomfortable discussions about what’s not working or clear, what needs to improve, and how to get there. Only then right kind of delivered value is going to surface.

Discuss unknowns during your mission and get rid of them. Upholding a “daily scrum” wont magically teach that by itself.

How do you think Apple created the iPhone? Jobs and Ive didn’t sit around patting each other on the back. They argued, they iterated, they pushed each other until the product was right. And that’s how Scrum teams should operate. You don’t just do what’s easy or what’s fast. You work through the problems, challenge each other, and iterate until you’ve nailed it. You also ask a lot of (WHY) questions so you understand where each person is coming from.

Briefings and debriefings are crucial, but only if you commit to them

In Scrum, we have our Planning (briefing) and our Review and Retrospective (debriefing). Sound familiar? You can’t just jump into the mission without a game plan, and you sure as hell can’t improve without looking back at what went wrong or right. But, and here’s the caveat you have to actually use these sessions for what they’re intended. Too often, they’re rushed or treated as checkboxes. In the army, a failed briefing or debrief could cost lives, in Scrum, it’ll cost you the ability to iterate and test on things for the better the next mission. Take them seriously, or they’re just empty meetings not delivering any value.

Teamwork is everything

At the end of the day, both Scrum and military missions succeed because of teamwork. You need people who are committed, who communicate, and who have each other’s backs. Without that, you’re just a bunch of people shouting the are doing Scrum, and that doesn’t deliver any value at all. Scrum’s not about heroes, it’s about the collective goal. You win together, you lose together.

Scrum isn’t a holy hammer that solves everything.

So, here’s the thing. Scrum is a framework. It’s a humanly created structure designed to help teams work together efficiently. But just doing Scrum isn’t going to magically produce value for your company. You need the right people, the right mindset, and a clear mission that everyone’s working towards. Otherwise, you’re just doing tasks without any thought behind it—and that’s a recipe for failure.

Another thing: adapt your style of working to the needs of your business. there is a structure set by scrum you can adapt to. But dont be afraid to change some things if it works for the better for your company. The goal is creating a team that likes to work in the created structure and is able to deliver value instead of features.

You as a team or even a Company need to make it work by putting the right people (with a vision and mandate) and processes in place, or you’re setting yourself up for a crash landing not delivering anything then frustration.

Dont let solo managers trying to make themself look good take over your thrive for value and actually helping the business. Not hurting the business.

Introduction

This post aims to shed light on database performance within Azure, focusing on a direct comparison between MySQL, MariaDB, and PostgreSQL. Despite abundant literature on the subject, practical tests that measure real-life performance are scarce.

Many discussions revolve around the differences between these databases rather than their actual performance. When I started this research, my priority wasn’t the differences but finding a performant database that could serve as a viable Azure-based alternative for our MariaDB Monolithic setup. Initially, we thought switching from MariaDB to MySQL was a straightforward replacement with almost no performance impact. We were wrong there!

“There are many opinions about performance, But we needed facts”

We encountered a massive regression in performance in every query. Performance is crucial to us, and it was perplexing to see MySQL lag behind MariaDB so much. Upon discovering this, we set up a MariaDB VM ourselves in Azure, which proved to be performant again but introduced a lot of additional maintenance and management challenges, such as ensuring uptime, hot standby, and backups.

This led me to explore PostgreSQL, hoping for better performance than what MySQL offered. With this introduction, let’s dive into the tests we conducted, including a complex query and a set of small queries, to test the performance differences between the three databases and their implementations within Azure. We also included Docker tests to be able to determine whether the performance discrepancies are database-related or if Azure is the contributing factor.

Testing our complex Query

To assess the databases, we ran a complex query that aggregated data from multiple tables, representing a realistic workload for our system. For confidentiality reasons, we’ve altered table and column names in this description. This query aimed to evaluate how each database handles data aggregation and complex joins.

WITH AggregatedSkateData AS (
SELECT
    e.id AS eventID,
    e.theme_cid AS themeCID,
    s.id AS skateID,
    p.firstName || ' ' || p.lastName AS participantName,
    l.label AS locationLabel,
    s.priceOffered,
    s.inspectionCompleted,
    s.registrationDate,
    s.eventStartDate,
    org.name AS organizerName,
    'https://skateeventimages.net/' || im.path AS organizerImageURL,
    s.registrationCode,
    COUNT(DISTINCT CASE WHEN pa_last.status_cid = 0 OR pa_last.id IS NULL THEN sp.id END) AS partsPending,
    COUNT(DISTINCT CASE WHEN pa_last.status_cid = 1 THEN sp.id END) AS partsRequested,
    COUNT(DISTINCT CASE WHEN pa_last.status_cid = 2 THEN sp.id END) AS partsConfirmed,
    COUNT(DISTINCT CASE WHEN pa_last.status_cid = 8 THEN sp.id END) AS partsDenied,
    MAX(CASE WHEN ti_unscanned.id IS NOT NULL THEN 1 ELSE 0 END) AS hasUnscannedTickets,
    s.isPublic
FROM
    skateEvents s
    INNER JOIN events e ON s.eventID = e.id
    INNER JOIN participants p ON p.id = e.participantID
    LEFT JOIN organizers org ON org.id = e.organizerID
    LEFT JOIN images im ON im.id = org.profileImageID
    LEFT JOIN locations l ON l.id = s.locationID
    LEFT JOIN skateParts sp ON sp.eventID = s.id
    LEFT JOIN partApprovals pa ON pa.id = sp.partApprovalID
    LEFT JOIN ticketsUnscanned ti_unscanned ON ti_unscanned.id = sp.ticketID AND ti_unscanned.scannedAt IS NULL
    LEFT JOIN partApprovalActions pa_last ON pa_last.id = (
        SELECT pa2.id
        FROM partApprovalActions pa2
        WHERE pa2.partID = pa.id
        ORDER BY pa2.id DESC
        LIMIT 1
    )
WHERE
    s.registrationDate IS NOT NULL
    AND e.cancelledAt IS NULL
    AND e.deletedAt IS NULL
    AND s.isArchived = false
    AND s.deletedAt IS NULL
GROUP BY
    e.id, e.theme_cid, s.id, p.id, p.firstName, p.lastName, org.id, im.id, l.id, l.label, s.priceOffered, s.inspectionCompleted, s.registrationDate, s.eventStartDate, org.name, im.path, s.registrationCode, s.isPublic
)
SELECT * FROM AggregatedSkateData
WHERE
    TO_CHAR(s.registrationDate, 'YYYY-MM') = '2024-03'
    OR partsDenied + partsRequested + partsPending > 0
ORDER BY s.registrationDate DESC

Azure, complicated query

We conducted these tests within the same Azure zone for containers and databases to avoid interzone latency. Below are the specifications and results for transparency. It’s worth noting that for MariaDB and MySQL, the performance specifications didn’t markedly affect outcomes, provided we avoided the first (burstable) tier. PostgreSQL was the exception, showing noticeable improvements in large query runtimes.

Using HeidiSQL via the Azure VPN, we determined our best results out of five runs.

MySQL Flexible, Zone 2 (Standard_E4ds_v5, 450 iops)
Found rows: 130
Duration for 1 query: 13,875 sec

MariaDB VM, Zone 2 (Standard B4s v2 (4 vcpus, 16 GiB memory))
Found rows: 130
Duration for 1 query: 10,625 sec
23.4% better then MySQL

Postgres Flexible, Zone 3 (Standard_E4ds_v50 p10, 500 iops)
Found rows: 130
Duration for 1 query: 3,953 sec
71.5% better then MySQL

Local Docker, complicated query

Local hardware we tested on

12th Gen Intel(R) Core(TM) i9-12900H, 2900 Mhz, 14 Core(s), 20 Logical Processor(s)
Installed Physical Memory (RAM) 32,0 GB

MySQL 8.3 Docker
Found rows: 130
Duration for 1 query: 12,437 sec

MariaDB 11.4 Docker
Found rows: 130
Duration for 1 query: 9,484 sec

Postgres 16.2 Docker
Found rows: 130
Duration for 1 query: 3,969 sec

This concludes our complicated query tests. It is awesome to see Postgres perform so well here! We used to run in MariaDB before we started our move to Azure. It was very sad to see such bad performance in MySQL the moment we started testing with this new setup.

Testing our set small Query’s

The next test is when we fire more then 20.000 transactions per second against each database which will build us a feed in JSON that takes a certain amount of time to complete. we stress our database better with a lot of query’s which will give us a average runtime for every database.
There is also some PHP logic here since this is a PHP script we run to generate the feed. But it runs on the same hardware/docker image all the time. so the difference in runtime comes from the database.
For this we moved our containers to the same zone as our database is running to also take away potential issues with zone latency.

We are running this script within our Laravel framework. We are running our PHP against Swoole for the best possible performance within PHP 8.

Azure, small query’s

In this test we also introduce container performance aspects .
We run our containers and database again in the same Zone in Azure (Western Europe). When using Azure we run with a lot less performance than we have available in our docker environment.

For this we can only use these results to compare the differences within docker and Azure itself, not between them. Comparing the two together wont mean anything due to this hardware difference. We only want to observe the difference in runtime against the databases. and Docker is a baseline to check if the differences match up with the differences we see in Azure.

Our Node Size in Azure is a standard_b2s, 2 vCPUs with 4 GiB of RAM

API, Zone 2 -> MariaDB VM, Zone 2
24.59s runtime (spot nodes)

API, Zone 2 -> MySQL Flexible, Zone 2
27.87s runtime (spot nodes)

API, Zone 3 -> Postgres Flexible, Zone 3
24.92s runtime (spot nodes)

Local Docker, small query’s

For the small query tests within local Docker we utilize a system powered by a 12th Gen Intel(R) Core(TM) i9-12900H CPU, running at 2900 MHz with 14 cores and 20 logical processors, complemented by 32 GB of installed RAM.

These Docker images are configured to be memory-bound, meaning their performance was primarily dependent on the available memory rather than being limited by the CPU. The CPU settings were configured to allow unrestricted performance, ensuring that Docker can utilize as much CPU resource as needed.

Before each test run against a Database, I made sure to restart the Docker container to refresh its connection. After restarting, I allowed a 30-second pause to ensure that the container had fully stabilized and was running idle. This pause is crucial for ensuring more consistent test conditions. Once the container was ready, I initiated the feed generator script to begin building the feed, assessing how effectively our different database setups could handle small queries under these controlled conditions.

API -> MariaDB 11.4 Docker
22.88s runtime

API -> MySQL 8.3 Docker
23.74s runtime

API -> Postgres 16.2 Docker
20.77s runtime

MariaDB query cache

In our past use of MariaDB within our monolithic setup, we made significant use of the query cache functionality. This feature played a crucial role in delivering performance for repeated queries. This efficiency persists until the cache requires rebuilding, typically triggered by updates or inserts into the cached query’s tables. Lets do a second run on our complex query with Query cache enabled.

Local MariaDB Docker
Found rows: 130
Duration for 1 query: 0,000 sec

However, the performance benefits of the query cache aren’t universally consistent. For instance, when executing an average of 20,000 transactions per second with small queries, the runtime dropped to some extent. This variance highlights that the query cache doesn’t always produce the same level of optimization, particularly under high-load conditions. but more about that topic later.

API -> Local MariaDB Docker (query cache enabled)
17.30s runtime

API, Zone 2 -> MariaDB VM, Zone 2 (query cache enabled)
22.72s runtime (spot nodes)

The query cache in MariaDB works by storing the result set of SELECT queries, along with the query text. When an identical query is executed again (with the same text and under the same database user privileges), MariaDB can bypass the query execution phase, pulling the result directly from the cache. This feature can drastically reduce query execution times for frequently run queries that do not alter data.

It’s important to approach the use of MariaDB’s query cache with caution. Despite its benefits, the query cache was deprecated and disabled by default in newer versions of MariaDB 10.1.7 (and MySQL as of version 5.7.20) due to concerns over its impact on performance in environments with high levels of concurrent transactions. Nonetheless, for specific scenarios, particularly those with a predominant read workload, the query cache can still significantly enhance performance. It is completely removed in MySQL 8 and up.

Observation on Azure’s Flexible Servers

These tests highlight that the performance of databases on Azure’s flexible servers has improved significantly, aligning closely with the performance seen in local Docker environments. This marks a decent improvement from the previous Azure database single servers, which typically exhibited lesser performance due to some form of overhead introduced by how it was it was running in the Azure cloud.

Azure Latency anomaly

As an added bonus, we conducted some additional tests to measure interzone latency when connecting from Zone 1 or 2 to a PostgreSQL database in Zone 3 within Azure.

API, Zone 2 -> Azure PostgreSQL Flexible, Zone 3
The runtime was 44.23 seconds.

Interestingly, the runtime doubled again, a phenomenon observed exclusively when communications occur between Zones 1-2 and Zone 3, or vice versa. This pattern suggests that any container requiring a database connection experiences significant latency under these conditions, whereas latency is nearly negligible when the interaction is between Zone 1 and Zone 2.

For those who might have missed it, I’ve previously discussed a latency anomaly encountered in Zone 3. The conventional wisdom suggests maintaining the database connection within the same zone to minimize latency. However, our tests show that the latency impact is minimal when operating between Zones 1 and 2, but it becomes significantly pronounced when crossing from Zones 1 or 2 to Zone 3.

This testing further confirms that the latency issue is not exclusive to MySQL Flexible services but also affects PostgreSQL Flexible services within Azure at least, indicating a broader pattern across Azure’s database offerings.

Latency Anomaly: An attempt to untangle Azure Performance issues in Western EuropeUnveiling a significant Azure performance issue: Zone 3’s unexpected latency anomaly in the Western Europe region challenges cloud deployment strategies All Subjects MatterRemco Loup.

Conclusion

From a performance related evaluation, it’s clear that choosing the right database is crucial and should match your app’s specific needs for speed and efficiency. The results show MySQL tends to be slower. MariaDB works well and has a handy query cache feature, but Azure has no support for MariaDB anymore. This makes PostgreSQL the most viable choice for performance-focused users in Azure, as it excels in handling both complex and numerous simple queries effectively as shown in the tests above.

This analysis hopefully helps in selecting the right database technology and underlines how important careful planning is for boosting your app performance. For those using Azure and looking for top tier performance, PostgreSQL is the go-to option in my test results.

There are other reasons to choose your database on. But i only wanted to focus on the performance aspect in this post. This was our main concern since we are in the proces of moving from a Monolithic server to Azure. There are more things to consider when switching databases. The advantage here is that all three databases can be integrated quite easily if you use either of them.

Postgres is the best compatible here with its strict routines. The complicated query above is written for Postgres and runs in MariaDB without a single change. For MySQL the TO_CHAR in the last line only needs to be replaced with DATE_FORMAT.

Introduction

As we embark on a transition from a traditional VM setup hosted at TransIP to the scalable cloud infrastructure of Microsoft Azure, our journey is as much about discovery as it is about performance. Guided by the expertise of Pionative, we’re not just moving our environment; we are attempting to reinvent it in Azure, ensuring that every step we take is carefully tested and analyzed. Our goal? To ensure optimal performance in this new architecture and to uncover any potential bottlenecks or hidden issues that might lurk within our new cloud architecture as early as possible.

Azure regions and Zones

Azure’s infrastructure spans the globe, organized into regions for localized services, and further divided into zones for high availability. Regions are specific geographical areas hosting Azure services, while zones are physical locations within these regions, each with independent utilities to isolate from failures.

The Western Europe Region

The Western Europe region in Azure is structured into three distinct availability zones. These zones provide an additional layer of redundancy and fault tolerance by physically separating data centers within the region. This setup is designed to ensure that even in the event of a complete zone failure, the services can continue to operate unaffected in the other zones, offering robust disaster recovery solutions for businesses operating in Europe. Where Zone 1,2 and 3 are exactly located geographically is unknown to me.

Performance anomaly in the Western Europe Region

Our testing regime led us to a anomaly that has become the focal point of our Azure exploration for this post: a distinct latency issue within Zone 3 of the Western Europe region. This discovery wasn’t immediate; it emerged from days of performance testing and a keen observation of patterns that deviated from our expectations. This all because we just started to learn the way’s of Azure. And it’s difference in performance as opposed to our current setup.

We first started to exclude our own pods, code, performance settings within the node pool. Database performance settings and such.

Methodology and findings

Our final approach was quite thorough to verify this result. We deployed two node pools across each of Microsoft Azure’s three zones within the Western Europe region, aiming to rule out any node-specific variances. All these pools had the exact same performance setup.

Through extensive testing, including a battery of database calls and running our cron jobs integral to our application’s functionality, we noticed a consistent and surprising pattern. Irrespective of the zone’s reported capacity or congestion levels.

Interactions with the database were significantly slower when either the database or the connecting pods were situated in Zone 3.

Performance results

Below is our situation running a Cronjob that takes us locally within Docker around 32 seconds to run against our current setup, which isn’t comparable to MS Azure. But it did provide us a performance Matrix and goal. The script (cronjob) we ran fires around 32,000 queries to build a feed from all the products on our website. Yes, that’s a lot, but that is not the matter in this case. We wanted to uncover the performance issue. We know this script can be optimized to perform better.

MySQL Database in Zone 2
Zone 1 – pool 1: 33 seconds
Zone 1 – pool 2: 32 seconds

Zone 2 –  pool 1:  32 seconds
Zone 2 –  pool 2:  32 seconds

Zone 3 –  pool 1: 115 seconds
Zone 3 –  pool 2: 117 seconds

Then, to put the icing on the cake, we switched the setup with a database running in Zone 3. And see below the expected results we hoped to see to conclude this search.

MySQL Database on Zone 3
Zone 1: 120 seconds
Zone 2: 122 seconds
Zone 3: 33 seconds

We started to see the same consistency every time, even when we ran less but larger database calls.

MySQL Database on zone 2
16 DB queries single threaded.

Zone 1: Total time for queries from log: 9.57 ms
Zone 2: Total time for queries from log: 9 ms
Zone 3: Total time for queries from log: 49.96 ms

Website performance in Azure

When we run our Database in Zone 2 while having the API pod in Zone 1, the initial load time for our website decreases by at least 400 milliseconds for the first page load, compared to the configuration where the API pod is in Zone 3.

This means that the setup with the database in Zone 2 and the API pod in Zone 1 results in a much faster initial response time for the website compared to having the API pod located in Zone 3 and the Database in Zone 2.

There is a margin of error here, but we can conclude from these results that when you want to run your API demographically separated in two zones, there is no way you can consistently perform when using Zone 3. Zones 1 and 2 are the way to go.

We must be honest that this performance anomaly only happens when you have a pod with a database connection. Running our website (which has no database connection) in Zone 3 measured almost no impact here when the API pod ran in Zone 1 and the database resided in Zone 2.

Diving deeper in Zone 3’s latency issues.

This performance discrepancy in Zone 3 was astonishing to us. According to Microsoft’s best practices, the physical proximity of application servers to their respective databases within the same region should minimize latency. Our findings, however, painted a different picture, suggesting that within-region zone selection could profoundly impact application performance.

Our investigation was quite comprehensive, eliminating potential variables such as our own codebase, PHP environment configurations, MySQL performance, and much more. The consistent latency issues, particularly affecting PHP’s single-threaded cron jobs, finally pointed us towards a more systemic issue within Zone 3’s network architecture or deployment somewhere.

Collaboration and learning

Our journey, backed by the collaboration with Pionative, underscores the importance of partnership in navigating the cloud’s complexity. As we dove deeper into this latency anomaly, our shared efforts aim not only to optimize our deployment but also to contribute our findings back to a broader Azure community.

This latency impact is too large to ignore in our production environment.

Implications and next steps

This anomaly in Zone 3 opens up discussions on optimal zone selection and deployment strategies within Azure’s Western Europe region. Is this the reason that Zone 3 is the least populated zone? Online there isn’t much to find about this it looks like. So i wrote this post since this cost us literally days to pinpoint this performance difference.

This will impact our production environment. We are going to run our DB in Zone 1 or 2. and the API must reside in either of these zones. Zone 3 can be used for pods that talk with the API.

This latency anomaly in Zone 3 was more then a challenge for us. We took it as a opportunity to learn, adapt, and optimize our knowledge. As we continue to collaborate, test, and refine our approach, our journey into Azure is shaping up to be a very interesting technical exploration which is broadening our toolset for the future.

This post is a follow up on GIT deployment strategies within Kubernetes environments. Make sure you read this post first:

CI/CD pipeline with Docker and Git branching strategyExplore advanced CI/CD with Docker & Git. Master deploying Docker artifacts via Git branching. Dive into strategies for DEV, STAGING, LIVE environments. Perfect your pipeline now! All Subjects MatterRemco Loup.

In a CI/CD pipeline centered around Docker artifacts with the focus on a singular branch, implementing an efficient hotfix strategy requires a small deviation to ensure rapid deployment of critical fixes. Let’s work out our approach to hotfixing in this Docker-centric environment, given the explained branching architecture in the above post.

Identifying the need for a Hotfix

Hotfixes are a critical part of maintaining the stability and reliability of any software in production. They are typically required under specific circumstances that demand immediate attention. Understanding when to implement a hotfix is key to managing your CI/CD pipeline effectively, especially when using Docker artifacts and streamlined branching strategies like focusing primarily on a Single branch. Here are our scenarios where a hotfix can be necessary:

1. Security Vulnerabilities: One of the most urgent triggers for a hotfix is the discovery of a security vulnerability within your application. Whether it’s an exposed data risk, a potential for unauthorized access, or any flaw that could be exploited maliciously, quick action is required to patch the vulnerability and protect your users and data.
2. Critical Bugs Impacting User Experience: If a significant bug slips through your testing processes and reaches production, it can severely impact the user experience. This could be anything from a broken core functionality that prevents users from using your application as intended to severe performance issues leading to timeouts or crashes. If the issue significantly affects your user base, a hotfix is warranted.
3. Compliance Issues: Certain applications may have legal or regulatory requirements. If a compliance issue is identified that could affect your legal standing or user privacy, addressing it swiftly through a hotfix becomes a priority to avoid penalties or breaches of trust.
4. Data Loss Risks: Scenarios where there’s a risk of data loss or corruption for users require immediate attention. Whether it’s a flaw in how data is saved, updated, or retrieved, a hotfix should be deployed to mitigate any potential loss or corruption of user data.
5. Payment or Transaction Failures: For applications dealing with payments or any form of transactions, issues that prevent users from completing transactions or that might lead to financial discrepancies are critical. Such problems not only affect user trust but can also have financial repercussions for both users and the service provider, necessitating a hotfix.

Deploying a Hotfix

When a critical issue is identified within the production environment, the immediate goal is to develop, test, and deploy a hotfix as quickly as possible. Given that our primary branch for development is MAIN and that we generate Docker artifacts for deployment, we need a process that allows for rapid intervention without disrupting the ongoing work.

1. Creating a Hotfix Branch: Instead of branching off from your Production branch (as LIVE is not represented anymore in the Git repository in this strategy), the hotfix branch is created from the commit currently deployed in production. This ensures that the Hotfix is based on the exact state of the version running on the production environment. The specific commit can be identified through tags or Docker image labels that correlate with the production deployment.
2. Developing and Testing the Hotfix: The critical fix is developed within this hotfix branch. The urgency of the situation necessitates a focused approach, targeting the problem with minimal changes to expedite the testing and deployment process. Despite the rush, adherence to the Definition of Done (DoD) is crucial to ensure the stability and reliability of the hotfix. This can mean a condensed version of the DoD, prioritizing critical tests and checks that ensure the fix will not introduce new issues.
3. Creating a Docker Artifact for the Hotfix: Once the hotfix meets the agreed DoD criteria, an automated process builds a Docker image from the hotfix branch. This Docker artifact is then tagged appropriately, indicating its purpose and urgency.
4. Deploying the Hotfix: The newly created Docker artifact is deployed directly into the production environment, bypassing the usual DEV to STAGING to PRODUCTION environment strategy. This deployment is only justified by the critical nature of the fix and the testing it has undergone.
5. Merging Back and Maintaining Consistency: After the hotfix is successfully running in production, it’s essential to merge the changes back into the MAIN branch and any other relevant branches or tags. Only this step ensures that the fix is incorporated into the ongoing development work and future releases, preventing regression!!

Ensuring Rapid Deployment and Minimizing Disruption

• Automated Processes: Automation plays a key role in this strategy, from the creation of the Docker artifact to its deployment. Automated tests, builds, and deployments ensure that the hotfix process is as fast and efficient as possible.
• Communication and Monitoring: Rapid deployment of hotfixes requires clear communication among team members and close monitoring of the production environment post-deployment to ensure the fix is effective and does not introduce new issues.
• Documentation and Review: Documenting the hotfix process and conducting post-mortem reviews help in refining the hotfix strategy, ensuring that the team can respond even more effectively to future issues.

Conclusion

In a CI/CD pipeline using Docker and primarily operating through Git Flow, hotfixing requires a streamlined approach that leverages automation and focuses on rapid, targeted fixes. By creating hotfix branches based on the production-deployed commit, developing and testing fixes according to an well defined DoD, and deploying Docker artifacts directly into production, teams can address critical issues swiftly while ensuring the ongoing stability and integrity of their application.

Let’s dive into managing a CI/CD pipeline using Docker with a focus on Git branching strategy that involves development (DEV), staging (STAGING), and production (LIVE) branches. Our goal is to clarify the role each branch plays in the process, particularly how Docker artifacts are created and managed within this framework.

Docker deployment using Git Flow with multiple branches

The core of our branching strategy involved three branches:

1. DEV: This is where all development work happens. Every new feature, bug fix, or enhancement starts in a feature branch, branched off from DEV. Once the work on a feature branch is complete, reviewed and meets the Definition of Done (DoD), it’s merged back into DEV.
2. STAGING: This branch mirrors a pre-production environment. It’s where integrated features from DEV get tested as a whole and accepted by stakeholders to ensure they work together seamlessly.
3. MASTER: The branch that reflects the current state of our live environment. Only thoroughly tested and stakeholder-approved changes from STAGING make it to Production.

Git Branching and Branching StrategyIn this post, we will go over why branching is required, the difference between development, staging… DEV CommunityPreetham

This was our variation of a Git Flow strategy, we used until now. When you start working with Artifacts this all changes back to a normal Git Flow structure. We still have environments running (Dev, Staging and Prod). The real difference is that code commits get done once instead of getting merged into a branch running on Staging or in Production.

Instead we now have one Branch we called it Main. This is where a runnable artifact gets created after a successful code merge using Docker images. Everything deployed into our Staging or Production environment is a specific version of an earlier created Artifact.

The biggest change is that STAGING and PROD aren’t branches anymore where you merge code into. They have instead became environments that run a specific version of a Docker image. This all has its own challenges, especially when you need to Hotfix. Hotfixing has it’s own post which you can read after this post. Lets break down all of this without introducing hotfixing step by step first. It is comprehensive enough already.

What is an Artifact?

A Docker artifact, often referred to interchangeably with a Docker image, is a lightweight, standalone, executable package that includes everything needed to run a piece of software, including the code, a runtime, libraries, environment variables, and config files. In simple terms, it’s like a snapshot of your application and its environment, bundled together, ready to be deployed and run anywhere Docker is installed, ensuring consistency across different environments.

The Importance of DoD When Merging code

A comprehensive and clear Definition of Done (DoD) plays a pivotal role in this new process, even more when merging changes into a single branch that triggers a Docker image which should be runnable and deployable:

• Quality Assurance: The DoD ensures that all changes, including new features, enhancements, and hotfixes merged back into MAIN, meet a consistent quality standard. This includes code quality checks, successful completion of automated tests, and adherence to security best practices.
• Preventing Regression: By applying the DoD to all merges into MAIN, the team can prevent regressions and ensure that new changes do not negatively affect existing functionality. This is crucial for maintaining the integrity and stability of the codebase.
• Streamlining Deployments: A clear and comprehensive DoD facilitates smoother deployments by ensuring that only fully vetted and ready-to-deploy changes are merged into MAIN. This reduces the risk of wasting resources on building Docker images that are bugged or broken, and therefore contributes to more efficient and reliable artifacts.

Creating and Managing Artifacts

The first thing for this proces to work is when a forked branch is merged back into MAIN, this triggers an automated process in our CI/CD pipeline. Here’s what happens:

1. Automated Testing: First, automated tests run to ensure the new changes meet our quality standards. This is crucial for maintaining a stable branch.
2. Docker Artifact Creation: Upon successful running al tests, a Docker image (our Artifact) is created from the MAIN branch. This is because the MAIN branch is the single branch that represents the latest state of every DONE development, This is where we generate our artifacts from that can become “shippable” products.
3. Why using MAIN for Docker Artifacts?: Creating artifacts at this stage allows us to catch any integration issue. Since every feature merged into MAIN is going to be considered ready for a release, generating an artifact here ensures that we always have a tested and potentially deployable version of our product at hand.

Transferring from Branches to Environments

In traditional Git workflows, branches like STAGING and PROD are used to represent different stages of the development process, from testing to production. However, in a Docker-centric CI/CD pipeline, the focus shifts from using these branches for code integration to using them as references for deployment environments.

Deploying to STAGING

• Creating Docker Artifacts: Once development is complete and merged into the MAIN branch, a Docker artifact is automatically created. This artifact is a snapshot of your application at that point in time, encapsulating all the changes that were merged.
• Deployment, Not Merging: The STAGING environment’s role is to test this Docker artifact in conditions that closely mimic the production environment. Instead of merging code from MAIN to a STAGING branch, you deploy the Docker artifact to a STAGING environment. This means the code itself isn’t moving between branches; rather, the compiled and containerized version of your application is being tested.
• Purpose of STAGING: The STAGING environment is essential for comprehensive testing, including performance and security assessments. It’s a final check to ensure everything works as expected in a production-like setting without directly affecting your actual users.

Going to Production

• Stakeholder Approval: Once your Docker artifact has been rigorously tested in STAGING and approved by stakeholders, it’s ready to move to production. This step is crucial for ensuring that all features, fixes, and updates meet the necessary standards and expectations.
• Deployment to Production: The transition from STAGING to PROD involves deploying the tested Docker artifact to your production environment. This deployment process ensures consistency, as the exact artifact tested in STAGING is what gets deployed to PROD, eliminating any discrepancies between tested and deployed code.

Key takeaways

• No More Branch Merging for Deployment: In this streamlined workflow, STAGING and PROD branches are no longer used for merging code. Instead, they reference specific deployment environments where Docker artifacts are deployed for testing and production.
• Focus on Artifacts: The emphasis is on creating, testing, and deploying Docker artifacts. These artifacts represent stable versions of your application ready for STAGING and, eventually, PROD deployment.
• Consistency and Reliability: By deploying the same Docker artifact from STAGING to PROD, you ensure that what has been tested is exactly what gets released, increasing the reliability and predictability of your deployments.

This approach simplifies the deployment process, reduces the risk of errors, and ensures that every release is thoroughly tested in an environment that mirrors production as closely as possible.

Are multiple Branches still needed?

Given this new workflow, the STAGING and PROD branches in Git branching serve no active purpose anymore and therefore they have became obsolete. They helped track which versions of the code have been deployed to each environment which we have solved by running a specific version of an Artifact.

Conclusion

Incorporating Docker into our Git branching strategy streamlines the development-to-deployment process by ensuring that every merge into MAIN triggers a Docker image with the potential to go to production. The key to this strategy is the creation of Docker artifacts early in the development cycle, allowing for continuous integration and delivery while maintaining high standards of quality and stability.

By understanding the changes to Git branches and the process of creating and deploying Docker artifacts, we can rely on the complexities of CI/CD with confidence, ensuring that all new features and fixes are delivered to users as efficiently and reliably as possible.

Hotfixing

We havent spoken about hotfixing yet. As i said this deserves its own Post. You can read all about Hotfixing and its caveats here.

Hotfixing with Docker Artifacts in a Single Branch StrategyMaster hotfixes in Kubernetes with Docker. Learn to rapidly deploy critical fixes in a CI/CD pipeline focusing on streamlined branching. Perfect for tackling urgent issues without disrupting flow. Dive in now. All Subjects MatterRemco Loup.

As someone interested in the ever-evolving world of technology, I’ve witnessed a lot of developments in AI, particularly in the realms of language processing and user interaction. Reflecting on these advancements, I’ve been wanting to share why, in my opinion, the recent strides in AI aren’t as surprising as you might find them to be.

I use different pronounces as i refer to the same thing. I use AI and LLM (Large Language Model) for todays take on systems like ChatGPT.

My first Encounters with “AI”

My journey with LLMs began earlier than some people realize. I had my first experiences with Flow.ai around 2018/2019. Back then, it was clear to me that we were on the brink of something. However, the general public seemed largely unaware of these undercurrents in AI development. The fact is, companies like Flow.ai (now part of Khoros) were laying the foundation for AI systems we see today like ChatGPT.

Read up on a post i wrote back in 2019 about our experiences with Flow.AI together with Stefan Samba.

2019 - Blast from the past: How 333travel is revolutionizing the Travel industryDiscover how 333travel’s AI ‘Joy’ redefines customer interaction, blending AI’s intent understanding with personal touch. A game-changing journey in AI-led travel solutions! All Subjects MatterRemco Loup.

Flow.ai: A Missed Opportunity for 333travel

My experience with Flow.ai was an eye-opener back then. It was a game-changer in the AI space for me, demonstrating the potential of neural network-based conversational AI. Looking back at it, when we tried integrating this technology at 333travel back in 2018, we might have been to far ahead of our time. Our natural style of helping people with asking for a proposal, did not gain the traction we hoped for using a LLM. Our customers (luckily?) preferred interacting with human specialists rather than bots. This was a clear indication that while the technology was promising, market readiness was another story.

People hated talking to the “oldstyle” chatbots not understanding what you meant due to the lack of intent understanding.

The Siri example

Let’s talk a bit about Siri, Apple’s virtual assistant. Since its launch in 2011, Siri has made some strides. However, from my point of view, these improvements have been modest at best in the intent learning space. Siri, along with other tools like Cortana and Google Assistant, still required very specific commands and often struggled with understanding intent. While there has been some progress there, it’s been to slow of a curve in understanding human language and mainly intent.

• 2010-2011: Siri’s initial years saw its integration into the iPhone 4S, offering basic voice recognition and command execution.
• 2012-2013: Expansion into new languages and dialects, broadening its user base.
• 2014-2015: Entry into new markets and the launch of SiriKit, allowing third-party app integration.
• 2016-2018: Introduction to macOS and the release of Shortcuts for customized voice commands.
• 2020-2021: moderate improvements in language translation, integration with Apple’s ecosystem, and enhanced speech recognition.

AI and Customer Interaction

As a technology enthusiast, I find ChatGPT impressive in many ways, but it’s essential to acknowledge that it’s no substitute for the detailed understanding that human specialists provide. Lets take a Tour Operator for example. Crafting a feasible travel plan is a whole different take. The last thing you want is to make mistakes there.

So why would we let an AI do the talking to the customer and then replace it with a human? Isn’t it better to do it the other way around? Let our specialists use an AI to create a personal travel plan and send it quicker using AI? Bonding your customer with real human interaction is most preferable when doing sales.

If you dont know what you are talking about you are not able to spot mistakes in responses from AI. This will always be a major concern. First hand knowledge is still mandatory. That is why something like ChatGPT is a tool that needs to be used by skilled people about the subject to accelerate their work.

I dont believe in separate integrations within websites that take up space pretending to be a smart AI. The search function already present on websites will solve this issue. and people that are going to use an AI (LLM) dont want to interact with a different LLM every time. they will use their own PA to answer these questions if needed.

The first step companies need to solve is implementing a search engine in their website that understands intent. By understanding intent you can give people actually a useful answer based on the content on your website.

You need a profound integration with your LLM and your rich content. Especially in the Travel Industry. Using your website to become smarter using LLM is much more logical in my opinion then a popup “person” you can talk to.

The future of AI is undoubtedly exciting, but it’s a path we must navigate with both optimism and caution.

My Take on AI’s Future

Looking ahead, I see AI as a field growing with possibilities but also filled with challenges. The journey from the early days of specific command-based systems to the more intuitive, context-aware models from today have been fascinating. However, there is still a long way to go in achieving the of AI that can really understand and interact with humans on a more profound level.

Conclusion

In summary, while AI’s recent advancements may seem sudden and surprising to many, they are part of a long, ongoing process of technological evolution. My experiences at 333travel, especially with innovative platforms like Flow.ai, have shown me the potential and limitations of AI. As we move forward, I believe it’s crucial to balance our enthusiasm for AI with a realistic understanding of its current capabilities and limitations.

You are unique by keeping it close to yourself and not by using technology to do your core business for you. Always keep a personal touch!