Connect datasets outside the jail with NullFS

Certain directories within a jail can be swapped out to (datasets) "outside". This allows this data to be stored independently of the jail. The goal, similar to Docker, is to depend as little as possible on the jail itself. This way, if the jail is corrupted or deleted for some reason, we are able to restore the previous configuration and data with minimal effort. Also, these directories can then be efficiently and regularly backed up via snapshot, which also simplifies further backups. Not to mention moves or migrations.

The administration in TrueNAS is managed with iocage and is preinstalled, but can also be installed and used in FreeBSD with pkg install iocage. The advantage is that either via the TrueNAS GUI or a pure shell the most important settings and adjustments can be made. Was we also do here for both.

Last update:

  • 23.01.2024: Some tweaking and configuration added to TrueNAS
  • 18.08.2023: Initial version

Terminology

  • JAILNAME = The name of the jail, ex. gitea
  • SOURCEPATH = The directory (DATASET) on the TrueNAS/FreeBSD, ex. /mnt/tank/jails_data/gitea/data
  • TARGETPATH = The directory inside the jail, ex. /mnt/tank/iocage/jails/gitea/root/var/db/gitea
  • ID = Required user ID within the jail, ex. 211
  • ZPOOLNAME = The pool name which contains the data, ex. `tank

Directory structure

Jails are usually stored in /usr/jails or under TrueNAS in /mnt/ZPOOLNAME/iocage/jails with the following directory structure:

└── jails         # Jail main directory
    └── JAILNAME  # Jail Name
        └── root  # / of the Jail

For this purpose, a new dataset named jails_data is now created, which hosts the "jail data" and mirrors the above structure somewhat, so that it remains easily traceable. For example, this could be on a completely different pool /mnt/ZPOOLNAME/jails_data, which may have much more storage space. This is very useful for large data files like pictures or file shares, which have no place in the jail. Also read only if needed.

└── jails_data     # As a counterpart to jails.
    └── JAILNAME   # The same name as in jails, ex. `gitea`
        └── DATASET  # source directories of the outsourced data, ex. `data`

Create directories

But how does that come together? Quite simply via NullFS. NullFS virtually puts the contents of directory A on directory B, even across jail boundaries. Very handy! A key factor is access rights and proper permissions. If there is a user USER with ID 123 in the jail, then the directory outside the jail MUST have the same permissions. Fortunately, the user ID is sufficient for this, so it is not necessary to create the corresponding username in the TrueNAS/FreeBSD host each time as well. Since the directories usually do not yet exist in the jail, they have to be created beforehand. Actually they are created only during the installation of the packages, but this is too late. Because during the installation the content is often already created within the destination directory.

Everything that is executed hereafter happens on the TrueNAS/FreeBSD host. NOT inside the jail!

Directly in the shell

The jail must be restarted with iocage restart JAILNAME each time to include these paths.
With iocage fstab -e JAILNAME the once set entries can be adjusted or removed afterwards.

zfs create -p ZPOOLNAME/jails_data/JAILNAME/DATASET # create SOURCEPATH (dataset) in jail_data
chown -R ID:ID SOURCEPATH # Set user, the naked ID is sufficient, e.g. `211`.
iocage start JAILNAME # start jail
iocage exec JAILNAME mkdir -p TARGETPATH # create directory in jail
iocage fstab -a JAILNAME SOURCEPATH TARGETPATH nullfs rw 0 0 # include directories
iocage restart JAILNAME # restart jail

Via the TrueNAS management interface

User creation

  • TrueNAS / Accounts / Groups / Add
    GID: ID    # ex. 211
    Name: NAME # ex. gitea
  • TrueNAS / Accounts / Users / Add
    Full Name: NAME    # ex. Gitea User
    Username: USERNAME # ex. gitea
    Password: PASSWORD # ex. 123
    User ID: ID        # ex. 211
    Primary Group: ID  # ex. 211
    Shell: nologin     # No login allowed
    Home: /nonexistent # No home directory

Create data dataset

  • TrueNAS / Storage / Pools / tank/jails_data/JAILNAME / Add Dataset
    Name: POOLNAME # ex. data

Dataset permissions

  • TrueNAS / Jails / Mount Points / tank/jails_data/JAILNAME / Edit Permissions
    User: ID  # ex. 211
    Group: ID # ex. 211

Mount dataset

Practical: The target directories in the jail are created automatically when the mount points are assigned. Also recursive!

  • TrueNAS / Jails / JAILNAME / Mount Points / Actions / Add
    data:
    └──Source: SOURCE           # ex. /mnt/tank/jails_data/gitea/data
    └──Destination: DESTINATION # ex. /mnt/tank/iocage/jails/gitea/root/var/db/gitea

So in the jail GITEA the directory /usr/local/etc/gitea points to the actually external directory /mnt/tank/jails_data/gitea/data.
Everything clear? Good!

Voilá

If you find this content valuable and useful, then I'm happy about a feedback via Matrix, follow me on Mastodon or leave a comment here.