Using an NFS Root Filesystem

Introduction

NFS (short for Network FileSystem) has been around for a long time. It is an early form of a network file sharing system, where a server can export part of its filesystem so that this can be mounted by a client on the local network. This way, changes made to files on the server side are immediately seen by the client, and vice-versa.

This is obviously an interesting solution for some embedded Linux development tasks, when the embedded target has a hardwired connection to the local network (Ethernet). Because deploying updates to files to the root filesystem is a matter of copying them to the locally (exported) directory, without a need to reflash an SD card or an e.MMC. Similarly, when debugging applications, recompiling them and redeploying them and their associated files is again just copying them to the local filesystem instead of using SCP or USB to transfer files.

This works best with a hardwired connection. We recommend hardwired connections over wireless because wireless connections are less 'stable' if you will (packets are lost more often, and the latency between the client and server can vary). Also, in many cases the WLAN radio requires a loadable kernel module and firmware files which are located in the root filesystem. Since the root filesystem is located on the network (in the case of NFS), the WLAN driver cannot be imported and firmware files cannot be access, so WLAN cannot be used.

Requirements

There are several things necessary to enable NFS mounted root filesystems.

1. First, the host computer must be set up correctly. It must have an NFS server installed and export the rootfs so that the target can locate it and mount it.
2. U-boot, which loads and executes the Kernel, must provide the kernel with the correct argument in the kernel command line so that the kernel will look for the rootfs on a particular server and server URL (a filesystem path). This is a matter of changing the boot arguments.
3. The Linux kernel must be built with the NFS enabled.

For example purposes in this article, we'll use a normal machine built with our Yocto version 3.0 (Zeus) distribution for EDM-G-IMX8M-PLUS without any customization. We'll just use a normal image. For this one, we chose tn-image-test, however any image should work, even core-image-minimal.

Host Configuration

Install NFS Server

In order to use NFS, an NFS server must running on your host. With Ubuntu, this is relatively simple to install with apt:

sudo apt install nfs-kernel-server

Unpacking the Root FS

You must decide a location in the host filesystem in which to locate the target's root filesystem. For the example in this article, the filesystem location on the host will be in /opt/nfsroot.

Into this directory, unpack the root filesystem tarball produced during the building of the Yocto image. This tarball contains all of the filesystem files, and can easily be unpacked. The location of the filesystem, as produced by our Yocto configuration, is in under the build directory in:

<BUILDDIR>/tmp/deploy/images/<MACHINE>/<IMAGE>-<MACHINE>.tar.bz2

In our example, this would be in:

<BUILDDIR>/tmp/deploy/images/edm-g-imx8mp/tn-image-text-edm-g-imx8mp.tar.bz2

It is important to note that file ownership and permissions should be preserved of all files in the rootfs. For this, run tar as root (using sudo) and use the --same-owner argument in the command.

sudo tar --same-owner -pxjf <path/to/rootfs/tarball.tar.bz2> -C <path/to/nfs/root/directory/>

For example:

sudo tar --same-owner -pxjf ~/yocto/build-wayland/tmp/deploy/images/edm-g-imx8mp/tn-image-text-edm-g-imx8mp.tar.bz2 -C /opt/nfsroot

You can easily confirm that this command succeeded by listing the directory after unpacking:

$ ls -l  /opt/nfsroot/
total 68
drwxr-xr-x  2 root root 4096 Sep 17 13:42 bin
drwxr-xr-x  2 root root 4096 Sep 17 13:40 boot
drwxr-xr-x  2 root root 4096 May  7 15:35 dev
drwxr-xr-x 64 root root 4096 Sep 17 16:07 etc
drwxr-xr-x  3 root root 4096 Sep 17 13:40 home
drwxr-xr-x 10 root root 4096 Sep 17 13:41 lib
drwxr-xr-x  2 root root 4096 May  7 15:35 media
drwxr-xr-x  2 root root 4096 May  7 15:35 mnt
drwxr-xr-x  5 root root 4096 Sep 17 13:41 opt
dr-xr-xr-x  2 root root 4096 May  7 15:35 proc
drwxr-xr-x  2 root root 4096 May  7 15:35 run
drwxr-xr-x  3 root root 4096 Sep 17 13:42 sbin
dr-xr-xr-x  2 root root 4096 May  7 15:35 sys
drwxrwxrwt  2 root root 4096 May  7 15:35 tmp
drwxr-xr-x  4 root root 4096 Sep 17 13:41 unit_tests
drwxr-xr-x 11 root root 4096 Sep 17 13:40 usr
drwxr-xr-x  8 root root 4096 Sep 17 16:07 var

Exporting the Root FS and Restarting the NFS Server

Then, edit the /etc/exports file to include a path to the rootfs.

sudo vi /etc/exports

Then add a line to the file:

/opt/nfsroot  *(rw,sync,no_subtree_check,no_root_squash)

After this, restart the NFS server:

sudo systemctl restart nfs-kernel-server

Then, confirm that the server is running:

ps -ef | grep nfs
root      357891       2  0 16:14 ?        00:00:00 [nfsd]
root      357892       2  0 16:14 ?        00:00:00 [nfsd]
root      357893       2  0 16:14 ?        00:00:00 [nfsd]
root      357894       2  0 16:14 ?        00:00:00 [nfsd]
root      357895       2  0 16:14 ?        00:00:00 [nfsd]
root      357896       2  0 16:14 ?        00:00:00 [nfsd]
root      357897       2  0 16:14 ?        00:00:00 [nfsd]
root      357898       2  0 16:14 ?        00:00:00 [nfsd]

Kernel Configuration (Needed for ARM64 platforms only)

If the kernel doesn't have NFS support built in, it will fail to find the root filesystem. This is a matter of building the kernel with the required configuration settings enabled:

CONFIG_NFS_FS=y
CONFIG_ROOT_NFS=y

In order to make this persistent with Yocto-based builds, a patch can be added to the kernel recipe. The following is for the i.MX8 default configuration.

From c29e424642a523031a41c31c084071cebb14af66 Mon Sep 17 00:00:00 2001
From: John Weber <john.weber@technexion.com>
Date: Mon, 30 Aug 2021 11:21:40 -0500
Subject: [PATCH] ARM64:configs:tn_imx8_defconfig: Add NFS rootfs support
 
---
 arch/arm64/configs/tn_imx8_defconfig | 2 ++
 1 file changed, 2 insertions(+)
 
diff --git a/arch/arm64/configs/tn_imx8_defconfig b/arch/arm64/configs/tn_imx8_defconfig
index 7ab0cde50842..2c90c60aead1 100644
--- a/arch/arm64/configs/tn_imx8_defconfig
+++ b/arch/arm64/configs/tn_imx8_defconfig
@@ -891,6 +891,8 @@ CONFIG_HUGETLBFS=y
 CONFIG_UBIFS_FS=y
 CONFIG_SQUASHFS=y
 CONFIG_PSTORE=y
+CONFIG_NFS_FS=y
+CONFIG_ROOT_NFS=y
 CONFIG_9P_FS=y
 CONFIG_NLS_CODEPAGE_437=y
 CONFIG_NLS_ISO8859_1=y
-- 
2.25.1

This file may be downloaded here:

0001-ARM64-configs-tn_imx8_defconfig-Add-NFS-rootfs-suppo.patch

This file is then stored into the meta-tn-imx-bsp metadata:

<yocto-project-root>/sources/meta-tn-imx-bsp/recipes-kernel/linux/file/0001-ARM64-configs-tn_imx8_defconfig-Add-NFS-rootfs-suppo.patch

Then add the following lines to the bbappend recipe:

<yocto-project-root>/sources/meta-tn-imx-bsp/recipes-kernel/linux/linux-tn-imx_5.4.bbappend

Add the SRC_URI_append variable so that the recipe knows about the patch and can apply it after the kernel has been fetched:

SRC_URI_append = " \
        file://0001-ARM64-configs-tn_imx8_defconfig-Add-NFS-rootfs-suppo.patch \
        "

Then, rebuild the image. If this step is taken, then the image can be reflashed into the e.MMC or microSD card. For example:

bitbake tn-image-test

Or, only the kernel needs to be rebuilt. In this case, the kernel image on the target must be replaced with the rebuilt kernel.

bitbake virtual/kernel

U-boot Configuration

It is necessary to set U-boot up so that it is booting using a kernel and device tree from a local partition (in this case the FAT filesystem), but it is then booting the kernel, via the kernel command line, such that it mounts a root filsystem from an NFS server.

The following settings enable U-boot to do this.

Set a variable to store the NFS path on the host:

setenv nfsroot <path/to/export/nfs/directory>

Example:

setenv nfsroot /opt/nfsroot

Set another variable to store the server IP address. This is the IP address of the host computer.

setenv serverip <your host IP address>

In this example:

setenv serverip 192.168.0.100

Then, set a macro to build the variable bootargs. This is the kernel command line.

setenv mmcnfsargs 'setenv bootargs ${jh_clk} console=${console} root=/dev/nfs ip=dhcp nfsroot=${serverip}:${nfsroot},v3,tcp'

Note that this refers to the nfsroot and serverip variables set above.
Finally, set a macro to boot the board by setting up the bootargs, loading the kernel image, the device trees, and then booting the system:

setenv mmcnfsboot 'run mmcnfsargs; run loadimage; run loadfdt; run loadoverlay; booti ${loadaddr} - ${fdt_addr};'

If all looks good, then run saveenv to save the environment:

saveenv

When you are ready to try this, you can boot the board manually using:

run mmcnfsboot

If you wish your settings to be persistent across reboots, you can set bootcmd variable:

setenv bootcmd run mmcnfsboot

Conclusion

If successful, the board will boot and utilize the exported NFS directory as the root filesystem. This can be confirmed by running df on the target:

root@edm-g-imx8mp:~# df
Filesystem                               1K-blocks      Used Available Use% Mounted on
192.168.0.100:/opt/nfsroot 205374136 120947344  73924752  63% /
devtmpfs                                   1500272         0   1500272   0% /dev
tmpfs                                      1992976         0   1992976   0% /dev/shm
tmpfs                                      1992976      9332   1983644   1% /run
tmpfs                                      1992976         0   1992976   0% /sys/fs/cgroup
tmpfs                                      1992976         4   1992972   1% /tmp
tmpfs                                      1992976       192   1992784   1% /var/volatile
tmpfs                                       398592         4    398588   1% /run/user/0

Further testing can be performed by creating or modifying files on the host and target in order to ensure that the opposite is able to immediately read the changes in the filesystem.

Troubleshooting

If problems are encountered, then check the following:

• Ensure that the host has exported the directory containing the root filesystem
• Ensure that the host has the NFS server running
• Ensure the target's kernel was built with NFS support and NFS root filesystem support enabled