Background.
After having had my Raspberry PIs in a box together with a lot of wiring spaghetti, its nice to finally have some decent rack mount enclosures to tidy up stuff.
This will be the first of two enclosures for a total of 10 nodes. I have not really decided if they are all going to be in the same cluster or two individual.
Hardware.
The nodes themselves are Raspberry PI4-B with 8 GB of memory. They each have 128 GB of internal storage via the onboard Micro SD card. I use some SANDISK high endurance cards, that are made for 24/7 operation and rated for 20.000 hours.
Between the nodes is a normal gigabit network switch with 10 gigabit fiber backbone. for now I have more than ample ports in one switch.
The enclosure model is the Rackmount RM-PI-T1. It is a 1U enclosure with room to add a fan or a HAT for the PIs if necessary.
If your rack cabinet already has fans I doubt it will be necessary.
Software.
The nodes run normal PI-OS which is a Debian derived Linux based OS specially tailored for the PI hardware. I use the lite version which is a headless version of the OS.
On this I use the K3S (Rancher) lightweight Kubernetes platform. I think some of the choices Canonical have made lately makes me doubt MicroK8s in a production environment and I think full on Kubeadm (Vanilla Kubernetes) is probably overkill for my needs.
I use no kind of hardware emulation. This is what is known as "bare metal" in Kubernetes terms. You could add this as an extra layer of abstraction. As it is here the images need to be built for the ARM architecture - and not for AMD64 which most ready made ones are.
Something that has surprised a lot of developers after they got a new Macbook I think :-P.
You could add a hw virtualization layer - but I think the performance impact would be severe.
Some of the nodes may convert to FreeBSD at a later stage. Mostly as an experiment. They will then not be part of the Kubernetes cluster but rather their own kind running the needed services via Jails on the physical host.
Images.
The Kubernetes images are built though Ainsible. The build environment hosted on Github and built via Github actions.
I plan to make a seperate more in depth post about this and also publish scripts and configs publicly on Github (once they fully do what I want).
Backend server.
For this purpose I currently have an older and much more powerful HP DL-585 FreeBSD server with storage, that attaches to the nodes for persistent storage. Also it currently runs any databases needed.
On this server each logical node is hosted in a Jail, that gives it its own IP stack and own ZFS pool area in the storage area.
The total ZFS pool size is currently 14 TB.
This has been the way the services now in the Kubernetes cluster where all hosted in my old setup before the PI nodes.
Conclusion.
These small machines are remarkably capable and have a very low power consumption footprint. The new PI5 is arguably even more desirable with its built in interface to real SSD storage.
However the idea behind Kubernetes is to have the nodes stateless and that is negated if the different nodes have persistent data, that is not mirrored to all of the nodes.
If you have your own business and require flexible computing I highly recommend the approach of using cheaper HW like this, because you will be able to size your system in a way you can have multiple nodes fail and not even know about it except for the alarm the system will give you.
It is then very easy to simply replace a faulty node with new hardware, add the needed disk image and let it rejoin the cluster.
And do not be alarmed at the prospects of having your own on site Kubernetes setup. If I can do it with of the self parts for less than 7500 DKK any company with a decent IT department can also run something similar.
It is the most obvious way of complying to any GDPR legislation in the EU, and avoiding any worries how our US "ally" treat our data. Which you will have to if you use hosting with the hosting titans.