A while ago, I wrote about my self-built kubernetes cluster. One interesting detail of it, was the lack of a CNI plugin, having configured the routes for pod-to-pod networking statically, via ansible.
This worked great. I just had to configure the IP ranges (v4 and v6) in the docker daemon and configure routes
interface-up scripts, which I had anyway. In the beginning, I had a
tinc tunnel between the nodes, which
I migrated to
wireguard later and even later changed to
wireguard (to allow dynamic routing with
BGP, which I use for example for
docker support is deprecated in kubernetes, so I had to change
runtime. Both real alternatives,
cri-o, depend on having a CNI network config, so changing to
CNI was required.
I started with modifying my ansible playbook to push the generated Pod network ranges for each node into the Nodes
.spec.PodCIDRs), retrieve the default CNI plugins and place them at
/opt/cni/bin. After that, I
applied the flannel manifest, which I modified to use
host-gw mode (since I already have a vxlan I use for other
purposes too (and which was initially created for k8s, so it’s not a misuse of the existing one)).
After I had flannel running on my cluster and rebooted all nodes gracefully, I migrated from
My cluster even felt more stable after that change, probably (in part) due to not having two
things running at the same time - docker and kubelet were configured for
cgroupfs, while all nodes run debian,
so systemd. With the change to
cri-o, I also configured
kubelet to use
If the simple solutions were perfect, this post could stop now - but, oh well, flannel does not yet support
dual-stack network (it’s added by now, but not yet released), so I didn’t have IPv6 anymore in my containers,
resulting in a lot of
Instance down! from my monitoring, since I have some IPv6-only services to monitor -
Also, I have my own
/48 IPv6 PI allocation (PI: provider independent, can route via any upstream provider I have
a connection to) allocation, from which I want to use a portion as LoadBalancer IPs and not having IPv6 just isn’t
an option at all anymore (did you see the prices for IPv4? I wanted to pay some money for some, but some money just
isn’t enough O_O).
Being the most calm and patient person in existence (*ahem*), I decided to build my own small program for that.
Grabbing each Node’s IPs and adding routes to the other Node’s pod networks seemed like a simple enough thing to
do. And I succeeded - I built this in (mostly) two days, which weren’t even used fully for that (after all, I’m
still in hospital and therapy right now) and deployed it - replacing
host-gw node completely.
It’s important to note, my
kube-hostgw isn’t a CNI plugin in itself, it only generates a CNI network config
based on the default plugins, namely
bridge plugin combined with
host-local for IP address allocation does what
docker does by default: create a Linux bridge interface, give
it the first IP of the given
host-local range, create a
veth pair for every container with the host-side being
added to the bridge and the container side being configured with another IP from the range given to
portmap is required for Kubernetes services of type
NodePort - I don’t (yet) know how that plugin works
exactly, where it gets its infos from - but it was listed as dependency of flannel (and the flannel CNI config
used it) and everything works.
kube-hostgw not being a CNI plugin itself, the stability of it is not very important - it only needs to be
running in a loop if Nodes get added, removed or new PodCIDRs allocated - besides that, it could be built as a
Job instead of a
DaemonSet (but sadly there isn’t a
JobSet - one
Job per Node matching some rules).
Best thing, after I deployed
kube-hostgw to my cluster (I still only have this one, so my testing environment is
sadly the same as my prod environment ^^‘), one of my girlfriends was also building a kubernetes cluster, to move
all their stuff into it. They, too, started with flannel (I think it was my recommendation) and migrated to
kube-hostgw once it was usable (so.. even before it had a version number, not even a LICENSE or README). She used
kube-hostgw in her prod cluster from the start - so it’s running in three clusters already, just doing its job :3
She also wrote about it (mostly her cluster, a bit about
kube-hostgw and quite a lot about how friendly the
kubernetes community is) - check out her post, too!
If you want to read more about
kube-hostgw, take a look at
the project in my Gitlab instance :)