Introduction

I don't think much more needs to be said today about how great ZFS is as a file system. Word has got around. Here is how new hard disks are integrated into a ZFS pool and what the first steps are so that your own data has a proper home.

NEW: For the very impatient I have a console only section. There are only commands, no explanations.

Last update:

  • 16.11.2024: Initial document.

Basic conditions

  • The Basic settings for the system have been made
  • As already outlined in main article, the data is distributed across several ZFS pools in order to store it separately from the base system:
    • The data is located on a hard disc array in raidz2 (data) and is listed here as devices ada0 - ada3.
    • The VMs and jails are located on two mirrored SSDs (work) with the devices ada4 - ada5.

Partitioning

Before the hard disks are combined in a ZFS pool, I recommend making some preparations to ensure that the hard disks are always empty, always have the same partition scheme and, above all, have a meaningful name.

Delete hard disks

With gpart destroy DEVICE the hard drive is completely cleared of everything that could interfere later.
**Important: Adapt the devices! Any data already on the hard drives will be irretrievably destroyed.

gpart destroy -F ada0 # Destroys the partition table on ada0
gpart destroy -F ada1
gpart destroy -F ada2
gpart destroy -F ada3

gpart destroy -F ada4
gpart destroy -F ada5

Initialise hard disks

With gpart create -s TYPE DEVICE the hard disks are given a certain basic structure, which is particularly important for the later case when a hard disk fails and has to be replaced. 

gpart create -s gpt ada0 # Creates a new partition table on ada0
gpart create -s gpt ada1
gpart create -s gpt ada2
gpart create -s gpt ada3

gpart create -s gpt ada4
gpart create -s gpt ada5

Read out serial number

What is already clear here is that ada0 - ada5 is not very ‘handy and comprehensible’, and the order and device names can also change from time to time. In any case, it becomes problematic if the hard disks are moved to a new server, which turns the device names completely upside down. Assignment is then difficult. It is also difficult to identify and replace a defective hard drive in a larger hard drive network. The solution: To be able to address the hard drive with a name, simply give it one. Ideally with something that is already on the hard drive anyway: The serial number. With camcontrol identify DEVICE | sed -n ‘s/.*serial number.*\(.\{4\}\)$/\1/p’ the last 4 digits of the serial number are determined and noted.

ada0: HPAS (from serial number WD-XXXXXXXXHPAS)
ada2: E9EY (from serial number WD-XXXXXXXXE9EY)
ada1: 46EE (from serial number WD-XXXXXXXX46EE)
ada3: LLAK (from serial number WD-XXXXXXXXLLAK)

ada4: 2482 (from serial number SXXXXXXX2482)
ada5: 7815 (from serial number SXXXXXXX7815)

When partitioning with gpart add -t FILESYSTEM -l LABEL -a BLOCKSIZE DEVICE, the name determined is saved directly as a label. Practical, isn't it?

gpart add -t freebsd-zfs  -l "HPAS" -a 4K "ada0" # Creates a new FreeBSD partition with the name HPAS
gpart add -t freebsd-zfs  -l "E9EY" -a 4K "ada1"
gpart add -t freebsd-zfs  -l "46EE" -a 4K "ada2"
gpart add -t freebsd-zfs  -l "LLAK" -a 4K "ada3"

gpart add -t freebsd-zfs  -l "2482" -a 4K "ada4"
gpart add -t freebsd-zfs  -l "7815" -a 4K "ada5"

Set up pools

The discs are now ready to be combined into ZFS pools with zpool create -o MOUNTPOINT POOLNAME RAIDTYPE DEVICELIST. Here it becomes clear how helpful the newly assigned names are. These new pools are mounted in the classic TrueNAS style under /mnt/.

zpool create -o /mnt/data data raidz2 /dev/gpt/HPAS /dev/gpt/E9EY /dev/gpt/46EE /dev/gpt/LLAK # Created Zpool data as raidz2
zpool create -o /mnt/work work mirror /dev/gpt/2482 /dev/gpt/7815 # Creates Zpool work as a mirror

Tuning pools

That's a bit of a matter of taste. In my opinion, however, some attributes are quite useful for storing data efficiently and reducing the load on the hard disks. This is achieved with so-called ZFS attributes, which are later inherited from the main level to the volumes and datasets created below. It therefore makes sense to set these directly during installation.

  • zfs set compression=lz4 = Compresses files during operation and thus enables more data to be stored on limited storage space
  • zfs set atime=off = Improves performance on file systems with many small files that are accessed frequently
  • zfs set xattr=sa = Stores extended attributes in inodes, resulting in fewer hard drive requirements
  • zfs set acltype=posixacl = Enables the use of getfacl, setfacl for additional access rights
zfs set compression=lz4 atime=off xattr=sa acltype=posixacl data # Writes the attributes to the pool data
zfs set compression=lz4 atime=off xattr=sa acltype=posixacl work

The result can be checked with zpool status data.

  pool: data
  state: ONLINE
  scan: scrub repaired 0B in 00:03:52 with 0 errors on Fri Nov  8 00:07:34 2024
  config:

    NAME          STATE     READ WRITE CKSUM
    data          ONLINE       0     0     0
      raidz2-0    ONLINE       0     0     0
        gpt/HPAS  ONLINE       0     0     0
        gpt/E9EY  ONLINE       0     0     0
        gpt/46EE  ONLINE       0     0     0
        gpt/LLAK  ONLINE       0     0     0

We have now set up the hard drives and the data can be stored securely.

Console

For example with ada0 - ada3 for zpool data and ada4 - ada5 for zpool work.
Important: Adapt the devices! Any data already on the hard disks will be irretrievably destroyed.

gpart destroy -F ada0
gpart destroy -F ada1
gpart destroy -F ada2
gpart destroy -F ada3
gpart destroy -F ada4
gpart destroy -F ada5
gpart create -s gpt ada0
gpart create -s gpt ada1
gpart create -s gpt ada2
gpart create -s gpt ada3
gpart create -s gpt ada4
gpart create -s gpt ada5
SNADA0=`camcontrol identify ada0 | sed -n 's/.*serial number.*\(.\{4\}\)$/\1/p'`
SNADA1=`camcontrol identify ada1 | sed -n 's/.*serial number.*\(.\{4\}\)$/\1/p'`
SNADA2=`camcontrol identify ada2 | sed -n 's/.*serial number.*\(.\{4\}\)$/\1/p'`
SNADA3=`camcontrol identify ada3 | sed -n 's/.*serial number.*\(.\{4\}\)$/\1/p'`
SNADA4=`camcontrol identify ada4 | sed -n 's/.*serial number.*\(.\{4\}\)$/\1/p'`
SNADA5=`camcontrol identify ada5 | sed -n 's/.*serial number.*\(.\{4\}\)$/\1/p'`
gpart add -t freebsd-zfs  -l "$SNADA0" -a 4K "ada0"
gpart add -t freebsd-zfs  -l "$SNADA1" -a 4K "ada1"
gpart add -t freebsd-zfs  -l "$SNADA2" -a 4K "ada2"
gpart add -t freebsd-zfs  -l "$SNADA3" -a 4K "ada3"
gpart add -t freebsd-zfs  -l "$SNADA4" -a 4K "ada4"
gpart add -t freebsd-zfs  -l "$SNADA5" -a 4K "ada5"
zpool create -o /mnt/data data raidz2 /dev/gpt/$SNADA0 /dev/gpt/$SNADA1 /dev/gpt/$SNADA2 /dev/gpt/$SNADA3
zpool create -o /mnt/work work mirror /dev/gpt/$SNADA4 /dev/gpt/$SNADA5
zfs set compression=lz4 atime=off xattr=sa acltype=posixacl data
zfs set compression=lz4 atime=off xattr=sa acltype=posixacl work
zpool status data work

Voilá