Why I’m Replacing Etcd with NATS to Run Single-Node Kubernetes

image

TL;DR

• The Problem: Managed Kubernetes and traditional multi-node HA clusters incur creeping cloud bandwidth fees, unnecessary compute costs, and high operational friction for workloads that don't need massive initial scale.

• The Solution (MVK2): A single-node K3s design that removes Etcd using Kine and evaluates NATS as a fast, lightweight key-value backing store (with Postgres as a potential fallback option).

• The Blueprint Stack: Minimal Debian base layer, nats-server with JetStream enabled, K3s, Traefik + Cert-Manager mapping directly to host ports 80/443, Local Path Provisioner, and Ansible/Bash automation.

• The Core Principle: Treat both K8s infrastructure and application workloads as identical ephemeral packages defined by code, adhering to the rule that no service goes live unless it is backed up, monitored, and fully recoverable.

The Cost Trap of Multi-Node HA

I see titles on YouTube like PgBouncer in 3 Minutes – The Lightweight Hero Every Dev Needs. The objection I have to this kind of wording is the lack of words like 'ops', 'sysadmin' or 'manged service'.

Title's like this are of interest to me as they confirm a personal opinion that ops is dead. Back in 2000 something this undoubtedly would be 'every sysadmin needs'. Such job titles are often repackaged as devops or sre. They have become labels rather than descriptions.

Putting cynicism aside, I believe we need to be pragmatic when seeing the end of the road.

I built MVK initially to prove to myself and the world that K8s can run in self hosted environments cheaply and affordably, without expensive 'managed Kubernetes' services that balloon in cost over time.

I know this because I have done it, before managed Kubernetes was become the thing that it is today and before technical managers started to believe that big tech managed SaaS only is 'the way'.

So try, I must and my findings ? Mostly positive. Can you run K8s self hosted ? Yes, of course you can, what a stupid question. But with caveats.

3 node hyper converged k3s is not just possible, its a realistic alternative and we have proven that to ourselves. But cloud providers soon see that you have active VM's running on their precious infrastructure and steadily, inch up bandwidth charges month by month, realising a once monthly fee around $80 to exceed $120 with no appreciable change in workload.

This is a common tactic, no where near as egregious as one I was just subject to where one of the main hyper scalers upgraded some of our static web sites with zero load to enterprise managed firewalls without our permission and costing us a 47,000% price increase. It took me nearly a month to argue this was not of my doing and was not 'acceptable use' for them to see common sense and void the bills. I have cancelled our subscription and will never go back for my own business or personal use, ever.

But you have to be prepared for this kind of price gouging from the broligarchy. They wrote the playbook on this. Resistance is futile.

Why Single-Node K3s Makes Sense

So being a pragmatist, I embrace change and move on. MVK does not necessitate there being 3 nodes all the time, just to satisfy the HA requirements of etcd.

K3s, on which MVK is based, can run in single node, edge applications. Running a single VM with docker, compose, dokku, coolify, caprover, dokploy and likely many more are examples of what can be single points of failure, though easy to deploy and manage, SaaS platforms. They are everything K8s is meant to be but with a fraction of the management overhead.

So if we find that our use case is often not to scale to the size of Google and that we can tolerate, for the short term, having a single VM, potentially with a 2nd as backup, a K3s single node Kubernetes is an option. Right ?

Wrong, in many cases, as in my view K8s is far too complicated and difficult for the average person to run in production. Which was why I built MVK, to make its deployment as easy and simple as possible.

Replacing Etcd: Why NATS via Kine?

And this brings me to MVK2. MVK is meant to be 'minimal, viable, Kubernetes. And we have already established that this can be a single instance, where this use case applies. To do this, Rancher decoupled K8s from Etcd and replaced it with Kine, which is a drop in replacement for Etcd that uses common database technology as backing store. A database can also run in single node. Since the dawn of internet, this was the way databases started their life cycle.

A minimal viable Kubernetes node can therefor be its own management plane, worker node and have its own management plane in edge applications.

I tend to favour Postgres over MySQL, post to the goings on between Oracle and MySQL in the past, so would choose this as a way forward in an MVK2 platform that we can trust to production.

Postgres, like MySql, can suffer from being designed for use with long running connections. This was the way in the past but modern use cases include for instance micro services that create many concurrent connection requests that can quickly explode to 100;s, 1000's of connections a second. This can quickly exhaust traditional database connection pools of say, up to 100, being the Postgres default. So I would typically look toward pgbouncer or similar to put in front of Postgres in order to compensate for this and prevent at least some bottle necks from occurring.

SQLite of course is an alternative but by Ranchers own words, this is only suitable for non production as they clearly state:

If you need to run a simple, short-lived cluster in your CI/CD environment, you can use the embedded SQLite database

However, I am excited to see that the once 3 databases supported have now become 4 where the Github page for Kine states:

Kine is an etcdshim that translates etcd API to:

• SQLite

• Postgres

• MySQL/MariaDB

• NATS

So NATS is a 4th possible option, which could initially run in as a single node. This could in my view be as performant as etcd, being itself a lightweight key value store.

The MVK2 Stack Blueprint

So I'm next to build something that looks similar to the below

which to be fair, represents pretty much what I already have in test but using a SQLite back end. If NATS does not work out I can fall back to Postgres. But I would like to think that NATS can work. It can certainly be permanent and is not 'stuck in the 90's as far as older pool managed approaches that literally fork entire processes to achieve long lived sessions.

The MVK way

Aside of being minimal and viable, the MVK way needs to include principles of service delivery:

I ask the following of any service or application prior to acceptance into production

• is it backed up

• is it monitored

• can it be recovered

if the answer is no to any of these, we do not proceed, we do not go live.

This is the MVK way.

So applications in MVK need to be deployed as code like any other K8s component and can use native, well proven K8s technology such as pods, services, deployments, cron jobs, Ingres, persistent volume claims and so on.

In Kubernetes land, all of these constructs are defined and embodied as manifests, code, that can be deployed, redeployed and manged in ways that are automated and reliable. The same cannot be said of the 'old ways' of managing services on a server as we did back in the 90's. Docker, or rather, containers running with docker compose, podman are ways to reduce this complexity but they are by no means adequate in the way Kubernetes can do these things.

It is hard to wrap your head round all these things, I know, I had to do this within days of building K8s for the first time back in the 2010's and be on call rota for it 24*7. I would not want to repeat that painful learning curve and neither too should you.

So the MVK way is as much for applications as it is for infrastructure so that installing MVK should be as easy as and equivalent to installing apps that are packaged in the MVK way.

We arrive at 2 categories of package

• K8s

• Apps

Where, 1. K8s infrastructure is delivered as a package to deploy Kubernetes in the form of MVK, and 2. Apps are delivered as a package of supported apps that deploy to MVK.

Kubernetes infrastructure becomes as ephemeral as the apps that run upon it.

Both Kubernetes and applications have common life cycles:

• install

• configure .or. restore both config and data

• run / backup both data and configuration

where the 3rd item repeats over time.

The data for K8s itself or'the app' consisting of both configuration and persisted user data is stored in a common place, typically S3 or compatible such that in the case of disaster recovery or routine recovery, K8s or an application running on it simply picks up where it last left off.

Ephemeral Infrastructure vs. The "Pet Cluster" Anti-Pattern

This may sound like I'm stating the obvious, however, my experience over years has shown me that most people say they do Kubernetes, only to treat each Kubernetes instance as a 'new pet'. Each customer is hoofed onto the same, non ephemeral, fragile K8s platform that is tweaked and poked with over time yet offered as Kubernetes managed solutions.

I have seen also, every new project treated to a new managed K8s cluster. Each time, be it dev, qa, or customer delivered, each new version is deployed to a new, cloud managed Kubernetes cluster. Each version consumes another monthly, managed Kubernetes. Support of each new instance is added to an ever growing list of individual managed Kubernetes subscriptions.

If you think that these examples are rare in practices you'd be wrong. If you think that this is the practice only of smaller SMEs and the like I am sorry to say, wrong again. I have seen these examples and similar in huge, wealthy organisations, big pharma, global logistics and even bigger institutions that support our every day lives.

It is with these proclivities in mind that I designed MVK for my own use case and for the betterment of others, be they individuals or organisations, wishing to take back control of their infrastructures and to control ever growing hosting costs being forced upon them by big tech.

Concluding Thoughts on NATS

We've been using NATS for a while now, initially as a replacement for beanstalkd, an extremely lightweight queue mechanism. Efficient as it is, beanstalkd does not have the tooling and functionality that NATS has. Realising NATS now is a possible backing store for Kine, K3s replacement for Etcd, I was interested to see if this at least works before dropping back to Postgres if all else failed. So far, NATS runs behind Kine just fine. All that was needed to add to K3s setup was an extra data store end point directive in the parameters to the k3s installer.

When K3s started to run, data started to flow through NATS.

Which is proof to me at least, that NATS can be an alternative backing store for Kine than SQLite, Postgres and MySQL.

I will continue to test some more, before committing to production.