Kubernetes: Kubifying the Stack

It’s no secret that the infrastructure of the web is undergoing constant change. It’s also no secret that most of the infrastructure behind my projects is self-hosted. Historically, everything was managed in containers running in a single server through Docker Compose. However, with the rise of Kubernetes and the increasing demand, I decided to take the leap and migrate my stack to a more scalable and manageable environment.

Kubernetes, undeniably posed many advantages over my previous setup. It allowed me to manage my applications in a more efficient and scalable way, and it also provided me with a more robust and reliable infrastructure, and to also seamlessly add new nodes to the cluster as needed. It’s declarative system was also quickly able to seduce me, as having reproductive configuration is what got me in love with NixOS in the first place.

Apart from our production servers, which were monolythic at the time, the following other services were also self-hosted: GitLab for version control, Jira/Confluence for project management and collaboration, Keycloak for identity and access management. Everything was already containerized from day one (except for GitLab, which required some migration, but honestly, it could have been worse).

Tools like Kompose offered the allure of an automatic pathway to migrate, however, I quickly found that their manifests weren’t really the prettiest things, and I could do better by writing them myself. This was also a good opportunity to learn more about Kubernetes, and how it works under the hood.

The Migration

The migration was a long and tedious process, but it was worth it. I started by setting up a Kubernetes cluster on my servers, and then I began migrating my applications one by one. I started with the most critical services, such as GitLab and Keycloak, and then I moved on to the less critical ones, such as Jira and Confluence.

Kubernetes, not being a monolythic system, required me to make choices. A lot of choices. I ended up going with the following stack:

• Ingress: I went with Traefik, as it was already being used with compose. I do plan on migrating to Nginx at some point, but I’m waiting for it’s ingress to support HTTP/3 natively
• Storage: I went with RookCeph. It wasn’t, by any means, easy to set up, but it was worth it, as it provides a wide diversity of operating modes, which meant less work in the future.
• Networking: Kube-OVN was my choice, as it was the most feature-complete and easy to set up (and it is completely OSS). I did consider Calico, but I wasn’t too keen on the licensing.
• Monitoring: This one is something I still need to look into, I’ll update this post when I settle on a solution.

The Results

The biggest perk I got from this migration was the ability to autoscale the Game Servers for my company, with a solution based on Agones. This was a huge win, as it allowed us to scale our servers based on demand, and it also provided us with a more reliable infrastructure. By making use of Elotl Kip, I was able to add the ability to burst to the Cloud without any additional monthly costs.

The migration also allowed me to add new nodes to the cluster as needed, and it also provided me with a more robust and reliable infrastructure. I was also able to manage my applications in a more efficient and scalable way, and I was able to easily add new nodes to the cluster as needed. I am in love with kubectl. The ability to manage my configuration in a declarative way was also a huge win, as it allowed me to easily reproduce my configuration across different environments, and quickly rollback to a previous state if I somehow break stuff (which happens… a lot).

Conclusion

Kubernetes is fun! Ahem, sorry. What I meant to say is that Kubernetes is a powerful tool that can help you manage your stack in a more efficient and scalable way. It provides you with a more robust and reliable infrastructure, and it allows you to easily add new nodes to the cluster as needed. It’s declarative system is also a huge win, as it makes it easy to keep track of your changes. I’m in love with it, and I’m sure you will be too.

Thank you for reading, and I hope to see you next time!