Moving to Kubernetes

Most of the infrastructure behind my projects is self-hosted. For a long time, I ran it on a single server with Docker Compose. As the stack grew, I wanted to spread services across more machines and scale game servers with demand, so I moved to Kubernetes.

I also liked having configuration I could reproduce and roll back. That’s part of what drew me to NixOS, too.

The migration

Alongside our production services, I was hosting GitLab, Jira, Confluence, and Keycloak. Almost everything was already containerized; GitLab needed some extra migration work.

I tried Kompose, but ended up writing the manifests myself. Its output needed enough cleanup that I preferred to use the migration as a chance to learn Kubernetes.

I set up the cluster, then moved services over one at a time, starting with GitLab and Keycloak. Choosing the supporting tools took a fair amount of work:

• Ingress: I kept Traefik, which I’d already been using with Compose.
• Storage: I chose RookCeph for its range of storage options. Getting it set up wasn’t easy.
• Networking: I went with Kube-OVN after considering Calico.
• Monitoring: I hadn’t settled on a solution at the time of writing.

What changed

The biggest benefit was autoscaling our game servers with Agones. I also used Elotl Kip to let the cluster burst into the cloud when needed.

Adding nodes became easier, and keeping the configuration in files made it simpler to reproduce the setup and roll back when I broke something. Which happens… a lot.

The migration took time, but it gave me the room to grow that the single-server setup was missing. And I’ve become pretty fond of kubectl.