Minirack NAS Part 2

Software and other stuff

<--

I decided long ago to go all in on NixOS for my homelab, as I believe the package manager truly is the best one out there. This post is not a walkthrough or tutorial, however the info here will probably be useful if you ever consider using NixOS for your next NAS.

Everything discussed in this post relates to this file, as that is where I put everything related to the NAS.

To use or not to use Disko

I love disko, seriously, it is great. Having said that, my logic here is that I will hopefully only ever make the drive pools once. Therefore I did not bother with it this time, I just made the ZFS pools using regular ZFS commands.

Sadly I was incredibly dumb and did not save the specific commands I used. So you will have to look somewhere else for that, but ZFS is pretty well documented.

Here is the current layout of my pools, vdevs and drives, straight from zpool status:

  • asgard (ONLINE)
    • raidz2-0
      • 14TB drive
      • 14TB drive
      • 14TB drive
      • 14TB drive
      • 14TB drive
      • 14TB drive
  • scratch
    • 1TB SSD
  • yggdrasil
    • raidz2-0
      • 4TB drive
      • 4TB drive
      • 4TB drive
      • 4TB drive
      • 4TB drive
      • 4TB drive
      • 4TB drive
    • spares
      • 4TB drive (AVAIL)

The reason I have two ZFS pools is that my old NAS used the 4TB drives. It felt bad throwing them away, but a lot of them have tons of hours (50k+). I do not store any mission critical data on them, they only store some backups as the second local backup and other unimportant files like Linux ISOs.

ZFS on NixOS

This is pretty much all subject to change, so it is important to check the current wiki page for ZFS on NixOS. This can be found here.

One important thing was to ensure the kernel used is compatible with ZFS. That is one of the first things discussed on the wiki page, so just do that. Otherwise I recommend using boot.zfs.extraPools to automatically import your ZFS pool on boot. This will fail if some of your drives are not present (ask me how I know xD).

Scrubbing

This is very important, as if you do not do this you will suffer data loss! Hard drives suffer bitrot if the data is not read for a long time. This is avoided if you run a ZFS scrub on the arrays once in a while. As an added bonus, a scrub will also find data integrity problems and fix them if it can.

In Nix this is very simple to implement, as a systemd timer has already been set up in Nixpkgs. It is defined like so:

services.zfs.autoScrub = {
    enable = true;
    # Every first Monday of the month
    interval = "Mon *-*-1..7 01:00";
};

I run mine on the first Monday of the month, but you should do your own research into how often you want to run this. It basically reads all data on all drives and checks it against parity. Expect it to take many hours on large arrays with large disks.

Config values

I did not really play around too much with this, however I did change some values to fit my needs.

  • recordsize
    • This one is a bit different for each dataset, examples:
      • images = 128K
      • media = 1M
      • general storage = 128K
  • compression = lz4
    • This just seems like a good idea, as the CPU in this computer can handle the full 10 Gib with this option enabled.
  • dedup = off
    • Dedup was just too much for my CPU, it seems.

Why a scratch pool?

The single SSD pool called scratch looks suspect, and I agree. I am a big proponent of being careful with data and data integrity. Using a single SSD is not being careful.

I use this SSD to store files that are written in tiny tiny chunks. This is usually how torrent clients operate. They write each “block” of a file and commit it to disk, and these can be quite small. There is some evidence online suggesting that this causes very high fragmentation on the ZFS array. I had a 1TB SSD lying around, so files are downloaded to that and then later moved to the proper long term storage array.

I am actually lying, as this disk was intended to only be a scratch disk but it is no longer just that. I ended up using it for the storage backend of Incus on this host, therefore I now have to back it up constantly to not have to worry about data loss.

Monitoring

ZFS Event Daemon

ZFS has its own monitoring daemon that emits events happening in the ZFS pools. This is usually used in conjunction with email based notifications. However I run my own Ntfy instance, so I really wanted to send the events there.

Ntfy requires an HTTP POST request to send a notification to a topic. This is where we can “hack” ZED to allow us to use a custom script as the program to send the notification to. I use the built in writeShellScript to add a script defined in Nix to the Nix store. This script is then used as the ZED_EMAIL_PROG, which allows us to send the events to Ntfy.

{pkgs, ...}:
let
  zedNtfyScript = pkgs.writeShellScript "zed-ntfy" ''
    ${pkgs.curl}/bin/curl -s \
      -H "Title: ZFS: $1" \
      -H "Priority: high" \
      -H "Tags: zfs,warning" \
      --data-binary @- \
      https://NTFY.URL/nas
  '';
in {
    services.zfs.zed = {
        enableMail = false;
        settings = {
            ZED_EMAIL_ADDR = "root";
            ZED_EMAIL_PROG = "${zedNtfyScript}";
            ZED_EMAIL_OPTS = "";
            ZED_NOTIFY_VERBOSE = 1;
            ZED_NOTIFY_DATA = 1;
        };
    };
}

SmartD

We do something very similar with smartd to monitor the health of our drives. Smartd allows us to use another custom script to send events to Ntfy.

services.smartd = {
    enable = true;
    notifications.test = false;
    defaults.monitored =
        let
            ntfyScript = pkgs.writeShellScript "smartd-ntfy" ''
                ${pkgs.curl}/bin/curl -s \
                -H "Title: SMART alert on $SMARTD_DEVICE" \
                -H "Priority: high" \
                -H "Tags: warning" \
                -d "$SMARTD_FULLMESSAGE" \
                https://ntfy.l.polsevev.dev/nas
            '';
        in
        "-a -o on -s S/../.././02 -W 5,50,55 -m root -M exec ${ntfyScript}";
};

Essentially this smartd config will do the following:

  • -a: Monitor all SMART health
  • -o on: Instructs the drive to run its automatic offline data collection
  • -s S/../.././02: Runs a short self test every day at 02 AM
  • -W 5,50,55: This is temp monitoring, it gives an info message at a 5° change, info at 50° and a critical at 55°. This is not perfect, as it gives critical for the SSDs all the time, but I just ignore those.
  • -M exec ${ntfyScript}: Run the script to send the messages

Samba

Yes, yes I know. Why use SMB over something like NFS? Honestly it is just what I am familiar with, as we use it at work and it is what I used on my previous NAS. It also just works, I have zero issues with it.

Samba on NixOS is actually quite nice, but my setup is very very basic.

services.samba = {
    enable = true;
    openFirewall = true;
    settings = {
      global = {
        "workgroup" = "WORKGROUP";
        "server string" = "${config.hostname}";
        "security" = "user";
      };
      asgard = {
        "path" = "/asgard";
        "browseable" = "yes";
        "read only" = "no";
        "guest ok" = "no";
        "create mask" = "0644";
        "directory mask" = "0755";
        "force user" = "smbfiles";
        "force group" = "smbfiles";
        "valid users" = "user1 user2";
      };
    };
  };

To avoid any issues with permissions when different users create files, I just force the user and group of everything in the share to be owned by the smbfiles user. This means I have no control between users, but it just saves so much headache.

Users are regular Linux users, and passwords are set with smbpasswd. This setup needs some work and “nixification”, but for now it works just fine. I do not regularly add users, as it currently is only me using this NAS. If that changes I will probably make it a bit smoother.

Me being the only user is also why I currently give every user rw access. This is something I probably should change, but again, I am lazy.

Fin

Hope this gives you some interest in creating a NixOS based NAS for yourself. Having lived with this NAS for a couple of months now, I have realized I have to make a part 3 with all the small issues that have come up. Look forward to it!