Rackmount hardware of Rasberry PI4

Background.

After having had my Raspberry PIs in a box together with a lot of wiring spaghetti, its nice to finally have some decent rack mount enclosures to tidy up stuff.

This will be the first of two enclosures for a total of 10 nodes. I have not really decided if they are all going to be in the same cluster or two individual.

Hardware.

The nodes themselves are Raspberry PI4-B with 8 GB of memory. They each have 128 GB of internal storage via the onboard Micro SD card. I use some SANDISK high endurance cards, that are made for 24/7 operation and rated for 20.000 hours.

Between the nodes is a normal gigabit network switch with 10 gigabit fiber backbone. for now I have more than ample ports in one switch.

 

The enclosure model is the Rackmount RM-PI-T1. It is a 1U enclosure with room to add a fan or a HAT for the PIs if necessary.

 

If your rack cabinet already has fans I doubt it will be necessary.

Software.

The nodes run normal PI-OS which is a Debian derived Linux based OS specially tailored for the PI hardware. I use the lite version which is a headless version of the OS.

On this I use the K3S (Rancher) lightweight Kubernetes platform. I think some of the choices Canonical have made lately makes me doubt MicroK8s in a production environment and I think full on Kubeadm (Vanilla Kubernetes) is probably overkill for my needs.

I use no kind of hardware emulation. This is what is known as "bare metal" in Kubernetes terms. You could add this as an extra layer of abstraction. As it is here the images need to be built for the ARM architecture - and not for AMD64 which most ready made ones are. 

Something that has surprised a lot of developers after they got a new Macbook I think :-P.

You could add a hw virtualization layer - but I think the performance impact would be severe. 

Some of the nodes may convert to FreeBSD at a later stage. Mostly as an experiment. They will then not be part of the Kubernetes cluster but rather their own kind running the needed services via Jails on the physical host.

Images.

The Kubernetes images are built though Ainsible. The build environment hosted on Github and built via Github actions.

I plan to make a seperate more in depth post about this and also publish scripts and configs publicly on Github (once they fully do what I want).

Backend server.

For this purpose I currently have an older and much more powerful HP DL-585 FreeBSD server with storage, that attaches to the nodes for persistent storage. Also it currently runs any databases needed.

On this server each logical node is hosted in a Jail, that gives it its own IP stack and own ZFS pool area in the storage area. 

The total ZFS pool size is currently 14 TB.

This has been the way the services now in the Kubernetes cluster where all hosted in my old setup before the PI nodes.

Conclusion.

These small machines are remarkably capable and have a very low power consumption footprint. The new PI5 is arguably even more desirable with its built in interface to real SSD storage.

However the idea behind Kubernetes is to have the nodes stateless and that is negated if the different nodes have persistent data, that is not mirrored to all of the nodes.

If you have your own business and require flexible computing I highly recommend the approach of using cheaper HW like this, because you will be able to size your system in a way you can have multiple nodes fail and not even know about it except for the alarm the system will give you.

It is then very easy to simply replace a faulty node with new hardware, add the needed disk image and let it rejoin the cluster. 

 

And do not be alarmed at the prospects of having your own on site Kubernetes setup. If I can do it with of the self parts for less than 7500 DKK any company with a decent IT department can also run something similar. 

 

It is the most obvious way of complying to any GDPR legislation in the EU, and avoiding any worries how our US "ally" treat our data. Which you will have to if you use hosting with the hosting titans.

Read More

FreeBSD 11 - vt console debacle

Vt console in FreeBSD

Introduction

For some reason server distributions of *nix now have to have a vt based (compositing) console, so that you can get your console into the larger resolutions.

Personally I fail to see the need for this at all. When you connect to your server it is via ssh 95% of the time. Or probably more. And your client PC where off you run ssh probably has a hd display today. Using the actual console is something you do when there are problems. Usually in single user mode. This does not have to be bling bling.

What about us, that depend on the serial console? Or text mode without a compositor. Well it would seem FreeBSD is now going down the Ubuntu server road, and we have to do some fiddeling in order to make our Inside Lights out cards, serial consoles and what not working again.

At least for me, my iLO card on my HP server could no longer bridge to the console after an upgrade to FreeBSD 11. I was told the console was in an unsupported graphics mode and after plugging a monitor onto the server, it was in 640x480.

How to fix this.

Fortunately this is simple not requiring a custom kernel build or anything. In loader.conf put:

kern.vty=sc

And you are back to the old sc driver for the console. And I can again get a console via ethernet. Also in single user via the iLO.

Conclusion

Back to normal ..

Read More

NFS version 3 on FreeBSD using unfsd

Introduction


I have had some problems reading files larger than 2 gb through nfs2. It is afaik a limitation of the standard it self. Therefore it is necessary to look at some of the later versions of this protocol for file operations on larger files.

I am not sure what the current standard in FreeBSD is (version 7.x.x) since I am still using the version 6 of FreeBSD. However this setup should also work with later versions of the OS.

Software installation

You should add the unfsd daemon using the ports collection or add it via a precompiled package. If you follow the attached link to this blog entry, it links to the installation page on freshports.org.

Subsequent configuration

The installation package does not include a rc script to start and stop the daemon, so i modified the system one in /etc/rc.d/nfsd to fit unfsd in /usr/local/etc/rc.d.

This is the rc file

# PROVIDE: unfsd
# REQUIRE: mountd
# KEYWORD: nojail

. /etc/rc.subr

name="unfsd"
rcvar=`set_rcvar unfs_server`
command="/usr/local/sbin/${name}"

load_rc_config $name
command_args="${unfs_server_flags}"
exports="${unfs_server_exports}"
pidfile=/var/run/${name}.pid
start_precmd="unfsd_precmd"
stop_cmd="unfsd_stop_cmd"
start_cmd="unfsd_start_cmd"
sig_stop="USR1"

unfsd_precmd()
{
    if ! sysctl vfs.nfsrv >/dev/null 2>&1; then
        force_depend nfsserver || return 1
    fi

    if ! checkyesno rpcbind_enable  && \
        ! /etc/rc.d/rpcbind forcestatus 1>/dev/null 2>&1
    then
        force_depend rpcbind || return 1
    fi

    if ! checkyesno mountd_enable  && \
        ! /etc/rc.d/mountd forcestatus 1>/dev/null 2>&1
    then
        force_depend mountd || return 1
    fi

    if checkyesno nfs_reserved_port_only; then
        echo 'NFS on reserved port only=YES'
        sysctl vfs.nfsrv.nfs_privport=1 > /dev/null
    fi
    return 0
}

unfsd_start_cmd()
{
    $command -e $exports -i $pidfile $command_args
}

unfsd_stop_cmd()
{
    if [ -f "$pidfile" ] ; then
        kill `cat ${pidfile}`
    fi
}

run_rc_command "$1"

This is the entries needed in /etc/rc.conf

unfs_server_enable="YES"
unfs_server_exports="/etc/uexports"

You should then add a /etc/uexports file to fit the format of unfsd. It is different from the system supplied nfsd from FreeBSD, so I suggest using a seperate file for this. In that way a rollback to the old version is a matter of re enabling it in rc.conf.

If you do not specify a named export file the unfsd will parse /etc/exports, and that will not work unless you edit it to fit the unfsd format.

Man unfsd will give you some examples of the format.

Conclusion

This will make it possible for you to use 2 gig plus files with nfs. In addition you will be able to use larger read and write sizes if you so choose.

Read More