Part 1 of getting Kubernetes in LXD working.

I have a different setup for my homelab/selfhosted server. I've always been more infrastructure leaning with my take on technology and for the most part resisted using Docker with hosting in my personal stuff. I always enjoy setting up the VMs and the requirements to run the application. That's why I would normally either spin up a droplet at DigitalOcean if I need more oomph or in an LXC/LXD container on my local server.

When I started at my current role/job, I've started to fully embrace the idea and the controls that something like Docker and Kubernetes gives. Since most of the experience has been on-the-job learn-as-you-go, I'm not picking up a lot of the things I want to try and understand with Docker and Kubernetes. Some of it is from the infrastructure side, because our OpenShift cluster and straight Kubernetes are managed, and the rest of it is from wanting to learn more from the development standpoint.

I had started playing with Docker a little bit running inside a privileged container (I had managed to figure out how to make it back into a unprivileged container, more on that later once I find my notes...) to test some things for an application I’ve been trying to build on and off for the last 5 or so years. I wanted the ability to spin up and down different test databases and webserver configs. That was probably the eye opener for me and the actual realization that Docker was extremely valuable for something to understand at more than just an infrastructure support level.

There's a bunch of blogs about using LXD and Kubernetes, either using Conjure/Juju or minikube with manual steps. Conjure is great but it make the assumptions of using Ubuntu as the hosts or the guest OS. I wanted to try and make something that either fits my setup, or could be modified easily to fit someone else's configuration easily. I'll eventually play with minikube, kubeadmin and kubespray. For now, I'm going with Kubernetes the hard way.

First thing I wanted to do was leverage Ansible to build containers and create an inventory that will be used for the playbook based on the manual steps. I had a hell of a time getting the playbook to make the containers and the inventory file. Mostly this is because I was doing something new with Ansible I’ve never done before. Most of my playbooks and Ansible usage has been simple shell manipulation or moving a file.

The different ways to create variables or work with loops is incredibly new for me, outside of a normal loop.

I ended up with an unorthodox playbook to start with. The most important part of this was this initial ridiculous line to get the IPs of the containers:

"lxc info kube{{ item }} | grep -o \"10.10.10.[[:digit:]]*\""

And this ridiculous line to fix the output into something usable.

"{{ kube | json_query('results[].stdout_lines[0]')}}"

Now I realized later (mostly when I was writing this up) there's was a very simple alternative to this that would have saved me a couple hours of headaches and pain, but probably wouldn't have taught me as much about the stuff I looked up to get it working.

The easier method is quite simple in comparison. Instead of trying to force the lxc info command to give me the information I want, use the actual lxd module to run a simpler command:

ip -f inet addr show wlan0 | grep -Po 'inet \K[\d.]+'

Unfortunately, while cleaner it still leaves the inventory looking like so

[local]
localhost
[k8s]
"10.10.10.21"
"10.10.10.22"
"10.10.10.20"