↓ Skip to main content
Shampan
  1. Blogs/

Hosting a local NixOS Cluster on a Pi 4

·1121 words·6 mins
Shampan
Author
Shampan
Security Minded FOSS Dev and Homelabber
Table of Contents

Background
#

We needed a local cluster at my old workplace to push things to our cloud infra, and we happened to have a spare Pi 4 lying around. Now the obvious choices would be something like Alpine, headless Rasbian OS, Debian/Ubuntu, etc., and then automate everything with Ansible.

However, I saw this as the perfect opportunity to try a fascination of mine, NixOS. It’s a declarative OS with rollbacks that can be reproduced from just a single configuration file. Add to that packages being built in isolation, offering shelter from dependency hell, and I just had to try it.

Here I’ll just go over getting a barebones system set up with a single cluster running.

Installation
#

Instructions derived from the NixOS Wiki.

  1. Grab the microSD you’ll use for the Pi
  2. Plug it into the slot in your laptop/adapter (use an SDXC/SDHC card if needed)
  3. Find the latest image from Hydra with a green checkmark and download it
wget <DOWNLOAD LINK>
  1. Uncompress Image
unzstd -d <FILE NAME>
  1. Flash image to sd card
sudo dd if=<FILE NAME> of=<DEVICE PATH> bs=4096 conv=fsync status=progress
  1. Once the command has finished running, insert the microSD card into the Pi

  • Device path is usually something like /dev/sda.

  • Using Impression to flash the image onto the sd cards may be easier but I haven’t tested that with SD cards.

Pi Configuration
#

Run sudo -i to enter root, and run the following to update firmware:

nix-shell -p raspberrypi-eeprom
mount /dev/disk/by-label/FIRMWARE /mnt
BOOTFS=/mnt FIRMWARE_RELEASE_STATUS=stable rpi-eeprom-update -d -a

Edit /etc/nixos/configuration.nix and add the desired specifications. Here is a starter example:

{ config, pkgs, lib, ... }:

let
user = "guest";
password = "guest";
SSID = "mywifi";
SSIDpassword = "mypassword";
interface = "wlan0";
hostname = "myhostname";
in {

boot = {
kernelPackages = pkgs.linuxKernel.packages.linux_rpi4;
initrd.availableKernelModules = [ "xhci_pci" "usbhid" "usb_storage" ];
loader = {
grub.enable = false;
generic-extlinux-compatible.enable = true;
};
};

fileSystems = {
"/" = {
device = "/dev/disk/by-label/NIXOS_SD";
fsType = "ext4";
options = [ "noatime" ];
};
};

networking = {
hostName = hostname;
wireless = {
enable = true;
networks."${SSID}".psk = SSIDpassword;
interfaces = [ interface ];
};
};

environment.systemPackages = with pkgs; [ vim ];

services.openssh.enable = true;

users = {
mutableUsers = false;
users."${user}" = {
isNormalUser = true;
password = password;
extraGroups = [ "wheel" ];
};
};

hardware.enableRedistributableFirmware = true;
system.stateVersion = "23.11";
}

To pull this config in a single command run:

curl -L https://tinyurl.com/tutorial-nixos-install-rpi4 > /etc/nixos/configuration.nix

Queue and reboot:

sudo nixos-rebuild boot
reboot

Get the latest packages:

sudo nixos-rebuild switch --upgrade

Any new changes made to configuration.nix should be applied with:

sudo nixos-rebuild switch

SSH
#

ssh user@<PI-IP> to remotely access the Pi.

Adding SSH Keys to NixOS
#

Additionally you can add an ssh key for your machine to the authorized_keys file for the admin and chore users on NixOS. This allows you to chore into the machine as those users without having to supply your password every time.

Create a new ssh key on your machine at $HOME/.ssh.

ssh-keygen -t ed25519 -C "$(id -un) for Pi" -f id_ed25519_pi

Copy this key to the authorized_keys file for the admin and chore users.

cat ~/.ssh/id_ed25519_pi.pub | ssh nix-admin@<PI-IP> "mkdir -p ~/.ssh && cat >> ~/.ssh/authorized_keys"
cat ~/.ssh/id_ed25519_pi.pub | ssh chore@<PI-IP> "mkdir -p ~/.ssh && cat >> ~/.ssh/authorized_keys"

Installing K3S
#

Instructions derived from the following README.

Add the following to configuration.nix.

{
networking.firewall.allowedTCPPorts = [
6443 # k3s: required so that pods can reach the API server (running on port 6443 by default)
# 2379 # k3s, etcd clients: required if using a "High Availability Embedded etcd" configuration
# 2380 # k3s, etcd peers: required if using a "High Availability Embedded etcd" configuration
];
networking.firewall.allowedUDPPorts = [
# 8472 # k3s, flannel: required if using multi-node for inter-node networking
];
services.k3s.enable = true;
services.k3s.role = "server";
services.k3s.extraFlags = toString [
# "--debug" # Optionally add additional args to k3s
];
}

Add clusterInit = true; to services.k3s if you want the cluster to be immediately available.

Then add the following to avoid memory errors:

boot.kernelParams = [
"cgroup_enable=cpuset" "cgroup_memory=1" "cgroup_enable=memory"
];

And that’s it! You should have a working local cluster declaratively setup with NixOS on a Pi 4.

Chore User
#

We had some custom aliases that required a ‘chore’ user for remote kubectl commands.

Add User to Configuration.nix
# 

let
userAdmin = "nix-admin";
passwordAdmin = "some-strong-password";
userChore = "chore";
passwordChore = "some-other-also-strong-password";
hostname = "nix-pi";
in {
users = {
mutableUsers = false;
users."${userAdmin}" = {
isNormalUser = true;
password = passwordAdmin;
extraGroups = [ "wheel" ];
};
users."${userChore}" = {
isNormalUser = true;
password = passwordChore;
extraGroups = [ "wheel" ];
};
};
}

Allowing the chore user sudo-less cat
#

Our kubernetes alises used cat to read the cluster configuration if it were not present on the host’s machine. In nixOS this file is locked behind root access. The aliases do not involve the user being able to supply the password for the chore user so the chore user needs to be able to read the file without requiring a password.

One way to accomplish this is to allow the ‘chore’ user to specifically be able to use sudo cat without a password.

security.sudo.extraRules = [
{
users = [ "chore" ];
commands = [
{
command = "/nix/store/62p7d4vvddd4nps6i06c3rvas08vlbk4-system-path/bin/cat" ;
options = [ "NOPASSWD" ];
}
];
}
];

Example of Configuration
#

Below is an example of our current configuration, taking together all of the steps above. The Wireless configuration is removed as it was connected through ethernet.

{ config, pkgs, lib, ... }:

let
userAdmin = "nix-admin";
passwordAdmin = COPY THIS FROM THE PASSWORD DATABASE;
userChore = "chore";
passwordChore = COPY THIS FROM THE PASSWORD DATABASE;
hostname = "nix-pi";
in {

boot = {
kernelPackages = pkgs.linuxKernel.packages.linux_rpi4;
kernelParams = [
"cgroup_enable=cpuset" "cgroup_memory=1" "cgroup_enable=memory"
];
initrd.availableKernelModules = [ "xhci_pci" "usbhid" "usb_storage" ];
loader = {
grub.enable = false;
generic-extlinux-compatible.enable = true;
};
};

fileSystems = {
"/" = {
device = "/dev/disk/by-label/NIXOS_SD";
fsType = "ext4";
options = [ "noatime" ];
};
};

networking = {
hostName = hostname;
firewall = {
allowedTCPPorts = [ 6443 ];
};
};

environment.systemPackages = with pkgs; [ vim git ];

services.openssh.enable = true;

services.k3s = {
enable = true;
role = "server";
extraFlags = toString [ "--debug" ];
};

users = {
mutableUsers = false;
users."${userAdmin}" = {
isNormalUser = true;
password = passwordAdmin;
extraGroups = [ "wheel" ];
};
users."${userChore}" = {
isNormalUser = true;
password = passwordChore;
extraGroups = [ "wheel" ];
};
};

security.sudo.extraRules = [
{
users = [ "chore" ];
commands = [
{
command = "/nix/store/62p7d4vvddd4nps6i06c3rvas08vlbk4-system-path/bin/cat" ;
options = [ "NOPASSWD" ];
}
];
}
];

hardware.enableRedistributableFirmware = true;
system.stateVersion = "23.11";
}