Google Scale - getting started

Self hosting with a view to scaling over time can be achieved with as little as a single host. Some projects may jump straight to multiple, clustered hosts or VMs. However to start out, many simply opt for a single instance or physical server. This can be fine for small projects.

With a single host, there is however a single point of failure - should that host stop running for whatever reason, the services it runs go off line. For small businesses and start ups, the reliability of their cloud provider’s virtualization can make this tolerable and affordable. Even the cost of dedicated hosts can be such that this they are still a viable option also.

Just so long, that is, as that single host is appropriately backed up. Volatile persisted data needs to be stored appropriately so as to be recoverable. Restoration can be to the originating single host or to another one elsewhere. Recovery hosts once stood up serve as disaster recovery end points. Downtime can be limited and tolerated, perhaps not even noticed by end users.

A single server, hosting all your apps, can be the most cost effective, if not highly available, method of delivery. Indeed, the home lab, self host movement is gaining popularity where its followers seek to take control of their own Digital Sovereignty and where single hosted services are cost effective.

In most cloud services, a virtual machine ( VM ) can be ‘snap shot’ backed up by the provider, often themselves using some kind of resilient, enterprise storage layer and the user simply selects a payment plan and schedule for these ‘backups’ to take place. This can make recovery from failed deployments or other unforeseen accidental operations quick and painless.

Deploying everything onto this single VM, however, can fall into an ad-hoc, often manual process of configuration that is not reproducible. That is, unless tools that were made for this purpose were not used from the outset.

Since around 2005 we had Puppet and then Chef, Salt, Ansible and in cloud many more culminating in Terraform and OpenTofu. Many more are available achieve similar, each one often complicated and enough for folks to build their whole career path upon.

Ansible is one of the simplest to get started on and one which can work in both self host and cloud environments.

If we are to doubt this, this article evidences that even Kubernetes operators can be written with Ansible. This is illustrative to the degree to which Ansible has permeated cloud based infrastructure as code.

As well as simple to use, it can be a good starting point for standing up a single VM and all the software it needs to get things going and thereafter to support the life cycle of the project.

Ansible is often used to deliver a variety of softwares that themselves come with tooling to manage their life-cycle thereafter.

Examples of this can be for example, docker and podman which both give us container orchestration. In turn, software can be delivered then using containers and the technologies that have evolved around this. I’ve heard Docker with compose refered to as as a ‘you only live once’ ( YOLO ) approach to container orchestration. However with simple augmentation, docker itself can give us similar to blue green deploys as can swarm, Kubernetes and other orchestrators like them.

Choosing the right containers and an optimal architecture to manage ingress and networks within docker deployments is key to this being a success. A popular and reliable solution is Traefik which serves as ingress and TLS certificate management using Let’s Encrypt . Each ‘stack’ within compose configs may be served though a front facing Traefik instance. Even blue/green deploys can be orchestrated by augmenting docker to have ‘rollout’, thanks to docker rollout installed by creating a file in your local settings directory

Docker Compose Blue Green Continuous Delivery

It came as a surprise to me that implementing continuous deploy patterns is entirely possible with just Docker Compose. With some simple changes, this can be achieved without needing to resort to more sophisticated approaches, such as Docker Swarm Mode or Kubernetes.

Most developers and infra people alike want for continuous deployment, where a new version of a container may be deployed to the ‘stack’ without any noticeable breaks in service. A default installation of Docker, for example, will typically require for a container to be stopped, removed and a new one started in its place in order for this to be fulfilled. In the time that this process takes, end users will perceive there to be a break in service whilst the shut down of the old container is completed and the new one starts and is able to accept connections.

Practical Example

There are a number of solutions to this problem, however I came across docker-rollout that makes zero or near zero down time deploys possible with docker and docker compose alone.

This extension to Docker is installed following its guide by the following:

1. Create Docker cli plugins directory if it does not exist:mkdir -p ~/.docker/cli-plugins
2. Download the docker rollout script to Docker cli plugins directory:curl https://raw.githubusercontent.com/wowu/docker-rollout/master/docker-rollout -o ~/.docker/cli-plugins/docker-rollout
3. Make the plugin executable:chmod +x ~/.docker/cli-plugins/docker-rollout

Being careful to check the contents of docker-rollout that now resides in ~/.docker/cli-plugings/ lets put this to work using a typical docker compose ‘stack’.

A docker-compose.yaml file can be used to create a simple stack with a web server delivering a static web site in a locally mounted directory called ‘dist’ where dist contains the contents of a static web site created with any number of static site generators such as Astro, 11ty, Gatsby, Hugo and so on :

services:
  lighttpd:
    image: jitesoft/lighttpd:1.4.79
    restart: unless-stopped
    environment:
      - PORT=9090
    volumes:
      - ./dist:/var/www/html:ro
    labels:
      - "traefik.docker.network=traefik-net100"
      - "traefik.enable=true"
      - "traefik.http.routers.site-live.rule=Host(`www.example.com`)"
      - "traefik.http.routers.site-live.entrypoints=websecure"
      - "traefik.http.routers.site-live.tls=true"
      - "traefik.http.routers.site-live.tls.certresolver=le"
      - "traefik.http.services.site-live.loadbalancer.server.port=9090"
    networks:
      - traefik-net100
networks:
  traefik-net100:
    external: true 

We can then start this docker compose file with

docker-compose up --scale lighttpd=3

Which creates 3 running instances of our lighttpdweb service that Traefik will use as back end services.

Now the service is running, changes can be made to the compose file, for example, changing its container version tag to a later one or swapping out the contents of ./dist with a newer version of our static web site. Then the stack can be ‘rolled out’ with

docker rollout lighttpd

New instances are invoked within the docker compose stack and when running, the old containers are expired, for the new ones take over.

There is nothing special about the above compose file aside of there being no container name definition - the rollout command would break if this were to be included as it has to allocate container names on the fly.

By the nature of this implementation with Traefik, there are no port mappings as that would break things also. However, Traefik handles that for us, making exposing external ports unnecessary. This is more secure as it only requires ports 80 and 443 to work. Docker container instances, if incorrectly configured, can easily expose ports to external access. This can happen when using port mapping within compose stacks or docker run commands. Doing so is best avoided if at all possible.

This approach is by no means a one size fits all. It is only an example of what can be done without ever deploying Kubernetes or needing to enable Docker Swarm mode.

Downsides and Gotcha’s

We would not recommend that resilient solutions like Docker Swarm Mode, Kubernetes or other multi-node container orchestration platforms be replaced by single node, single point of failure, docker instances with rollout or similar, as in the above.

However rollout is an alternative for initial infrastructure patterns where such high availability is simply not a requirement.

Obvious scenarios where this could be a good idea could be:

• proof of concept and try outs
• personal ‘hobby’ projects
• small business low energy low cost services with low, intermittent use

Indicators that infrastructures in these scenarios need migrating to more resilient platforms such as 3 or more node Kubernetes or Docker Swarm Mode might be

• uptime becomes contractually binding and service level agreements come into play stating things like ‘number of 9s’ ( that is, the number of 9s in 99.9… % uptime are required - the higher this number, typically the higher the cost for the service ). Put simply, how many minutes a year downtime can you accept.
• having many ( possibly dozens, hundreds even ) of containers configured in this manner across a number of server instances all amounting to something of a ‘sprawl’ of services and configuration. This can be difficult to manage in more than just a handful of instances

It is up to the individuals involved in decision making to take charge of these issues and set thresholds appropriately.

Plans and resource need to be in place and ready should these thresholds be breached to trigger a migration to more resilient multi-node cluster aware infrastructures.