My NixOS Journey Part 2

Part One | Part Two | Part Three

Welcome to my journey.  I'm documenting how I make my way down the rabbit hole, failures, successes, lessons learned all in hopes that I don't do it again, and maybe you'll be able to take away something from this too.

Where to discuss this article or contact me

Matrix:
 #my-nixos-journey:beardedtek.com
 #nixnerds:jupiterbroadcasting.com
 @beardedtek:beardedtek.com
Telegram:
 @beardedtek
Twitter:
 @beardedtek
Email:
 contact@beardedtek.com

I want to use btrfs for my NixOS servers because of the flexibility of subvolumes without the complexity of ZFS.  To do this, I'll write a simple bash script that automates partitioning, creating subvolumes, and installation.

I am trying to approach all this with multiple servers in mind at once, so I'm trying to introduce as many environment variables as I can to future proof things.

Each disk may have a different primary disk.  For example if installing on a VM, the disk will most likely be /dev/vda, but on a physical machine it may be /dev/sda or /dev/nvme0.  If this is an environment variable, I can set it before I run the install script.  Instead of showing each small part, I'll be adding onto the script as I go so you can see it all come together.  That's personally what works better for me.

#!/usr/bin/env bash
# Installs NixOS with btrfs

# These are required variables, lets set them with sane defaults in case
# We forget to set them.
DISK=${DISK:-/dev/sda}
HOSTNAME=${HOSTNAME:-nixos}

Now lets partition the drive:

#!/usr/bin/env bash
# Installs NixOS with btrfs

# These are required variables, lets set them with sane defaults in case
# We forget to set them.
DISK=${DISK:-/dev/sda}
HOSTNAME=${HOSTNAME:-nixos}

# Partition the drive with 550M boot partition and the rest btrfs
 printf "label: gpt\n,550M,U\n,,L\n" | sfdisk ${DISK}

Next we have to format the partitions and setup subvolumes:

#!/usr/bin/env bash
# Installs NixOS with btrfs

# Be sure to set the following environment variables prior to install
# or you can set them here prior to installing
#
# DISK=/dev/vda
# HOSTNAME=nixos

# Partition the drive with 550M boot partition and the rest btrfs
 printf "label: gpt\n,550M,U\n,,L\n" | sfdisk ${DISK}

# Format the partitions
# /boot:
mkfs.fat -F 32 ${DISK}1

# /:
mkfs.btrfs -f ${DISK}2

# Create subvolumes
mkdir -p /mnt
mount ${DISK}2 /mnt
btrfs subvolume create /mnt/root
btrfs subvolume create /mnt/home
btrfs subvolume create /mnt/nix

Next we need to unmount /mnt, and mount our subvolumes properly:

#!/usr/bin/env bash
# Installs NixOS with btrfs

# Be sure to set the following environment variables prior to install
# or you can set them here prior to installing
#
# DISK=/dev/vda
# HOSTNAME=nixos

# Partition the drive with 550M boot partition and the rest btrfs
 printf "label: gpt\n,550M,U\n,,L\n" | sfdisk ${DISK}

# Format the partitions
# /boot:
mkfs.fat -F 32 ${DISK}1

# /:
mkfs.btrfs -f ${DISK}2

# Create subvolumes
mkdir -p /mnt
mount ${DISK}2 /mnt
btrfs subvolume create /mnt/root
btrfs subvolume create /mnt/home
btrfs subvolume create /mnt/nix
umount /mnt

# Mount partitions
# Mount /
mount -o compress=zstd,subvol=root ${DISK}2 /mnt
# Create mount points
mkdir /mnt/{home,nix,boot}
# Mount subvolumes
mount -o compress=zstd,subvol=home ${DISK}2 /mnt/home
mount -o compress=zstd,subvol=nix ${DISK}2 /mnt/nix
mount ${DISK}1 /mnt/boot

Now all of our subvolumes are mounted, we can go on to configure our NixOS install.  This now leads me to splitting up my configuration.nix file.  Instead of having one exponentially growing configuration.nix, I'll separate it into logical sections:

- install.sh
- config:
     - configuration.nix
     - disks.nix
     - boot.nix
     - network.nix
     - packages.nix
     - services.nix
     - users.nix
     - secrets:
          - pw_user
          - sshkey_public

In the root directory I have my install.sh script we are creating and a directory named config which houses all our config files and a secrets directory.

The secrets directory is just for now as I figure out how to "properly" deal with secrets.  For now user passwords will be stored in pw_<username> files as a sha-512 hash, and sshkey_public will house our ssh authorized_keys.

In order to split up the configuration.nix, you have to find a way to import each file.  We already had a clue how to do that with hardware-configuration.nix, so we will just expand on that.

configuration.nix:

{ config, pkgs, ... }:
{
  imports =
    [ # Include all config files for this host
      ./hardware-configuration.nix
      ./disks.nix
      ./boot.nix
      ./network.nix
      ./packages.nix
      ./services.nix
      ./users.nix
    ];

  # Set your time zone.
  time.timeZone = "America/Anchorage";

  # Set nix Version
  system.stateVersion = "23.05";
}

disks.nix:
 This is where we tell Nix we are using zstd compression, and /nix will be mounted with the noatime option

{
  fileSystems = {
    "/".options = [ "compress=zstd" ];
    "/home".options = [ "compress=zstd" ];
    "/nix".options = [ "compress=zstd" "noatime" ];
  };
}

boot.nix:
 This is where you define any boot specific options, for example UEFI or GRUB boot.

{
  # Use the systemd-boot EFI boot loader.
  boot.loader.systemd-boot.enable = true;
  boot.loader.efi.canTouchEfiVariables = true;
}

network.nix:
 Here we setup any specifics to the network such as hostname, nameservers, firewall, and ipv6 settings.  There are lots more options available than shown here.

{
  # Setup Networking
  networking = {
    hostName= ""; # Set to blank to enable DHCP to set hostname
    nameservers = [
      "192.168.2.10"
      "192.168.2.145"
      "9.9.9.9"
      ];
    enableIPv6 = false;
    firewall = {
      enable = true;
      allowedTCPPorts = [
        22
        80
        443
        32400
        5000
        9000
      ];
    };
  };
}

packages.nix:
 Here we will install our Nix Packages.  If you wish to vim, you savage, you should add it here.  Nano is provided by default because, you know, it's not vim. :)

You'll notice {pkgs, ...}: on the first line, this makes pkgs available to the config

{ pkgs, ... }:
{
  # Allow non-free (Unfree) packages
  nixpkgs.config.allowUnfree = true;

  # List packages installed in system profile. To search, run:
  # $ nix search wget
   environment.systemPackages = with pkgs; [
     wget
     docker-compose
     python311
   ];
}

services.nix:
 Here we will define which services need to start on boot.

{
  # Enable docker
  virtualisation.docker.enable = true;

  # Enable Services
  services = {
    openssh.enable = true;
  };
}

users.nix
 Here we will define our system's users.  I set mutableUsers to false which disables adding or removing users from the command line.  I find this useful on servers because it prevents anyone from modifying users without modifying configuration.nix

You'll notice the options for hashedPassword and openssh.authorizedKeys.keys uses builtins.readFile.  This reads the file in from our secrets directory

{ pkgs, ... }:
{
  users = {
    mutableUsers = false;
    users = {
      myuser = {
        isNormalUser = true;
        description = "My User";
        createHome = true;
        home = "/home/myuser";
        group = "myuser";
        extraGroups = [ 
          "wheel"           # Enable sudo
          "docker"          # Enable docker
          "systemd-journal"
        ];
        hashedPassword = (builtins.readFile ./secrets/pw_myuser);
        openssh.authorizedKeys.keys = [
          (builtins.readFile ./secrets/sshkey_beardedtek)
        ];
        packages = with pkgs; [ ];
        uid = 1000;
      };
      root = {
        hashedPassword = (builtins.readFile ./secrets/pw_root);
      };
    };
    groups = {
      # Setup custom groups
      beardedtek = {
        name = "myuser";
        members = [ "myuser" ];
        gid = 1000;
      };
    };
  };
}

Now that we have our configuration complete, let's finish up our install script by copying our config to /mnt/etc/nixos,  generating hardware-configuration.nix (if it isn't already there), and installing.

#!/usr/bin/env bash
# Installs NixOS with btrfs

# Be sure to set the following environment variables prior to install
# or you can set them here prior to installing
#
# DISK=/dev/vda
# HOSTNAME=nixos

# Partition the drive with 550M boot partition and the rest btrfs
 printf "label: gpt\n,550M,U\n,,L\n" | sfdisk ${DISK}

# Format the partitions
# /boot:
mkfs.fat -F 32 ${DISK}1

# /:
mkfs.btrfs -f ${DISK}2

# Create subvolumes
mkdir -p /mnt
mount ${DISK}2 /mnt
btrfs subvolume create /mnt/root
btrfs subvolume create /mnt/home
btrfs subvolume create /mnt/nix
umount /mnt

# Mount partitions
# Mount /
mount -o compress=zstd,subvol=root ${DISK}2 /mnt
# Create mount points
mkdir /mnt/{home,nix,boot}
# Mount subvolumes
mount -o compress=zstd,subvol=home ${DISK}2 /mnt/home
mount -o compress=zstd,subvol=nix ${DISK}2 /mnt/nix
mount ${DISK}1 /mnt/boot

# Copy configuration to /etc/nixos
cp -r config/* /mnt/etc/nixos/

# Generate hardware-configuration.nix
nixos-generate-config --root /mnt

nixos-install --root /mnt

And boom.  We're all set.  run that install.sh, reboot, and you should have your system up and running!