ZFS-only installation of FreeBSD 11.0 from DVD
FreeBSD 10+ comes with a built-in ZFS-on-root support in its installer, but that one still uses the pesky “/ROOT” filesystem as well as mountpoint hacks for all top-level children like /tmp, /var, /usr and so on, all of which with no apparent benefit-or if there is one, it is horribly undocumented. This article describes an alternative setup that does without any of those.
Disclaimer: This article is about clean installation of operating system, where “clean” means that, if you follow these instructions, all existing contents of the hard drives will be destroyed. Do not follow these instructions except on an empty hard drives. I shall not be held responsible for any loss of data.
The instructions below assumes 4 disks named ada0…ada3, and creates GPT partitions named system-{1,2,3,4}-{boot,swap,zfs} where system-1-* is on ada0, system-2-* on ada1, and so on. It works for one-disk setup as well. The name system can be customized too (see step 3 below).
The swap is not on a ZFS volume (zvol) but on a separate GPT partition on each disk, as the zvol-based swap is known to cause a deadlock. Create each swap partition sized approximately to (the desired swap size / the number of disks).
Boot using the FreeBSD 11.0 DVD.
Proceed with installation until the installer asks you how to partition the disk. Choose “Shell”.
Set the desired traits (such as partition name prefix, pool name, etc.):
# zpool="system" # gpt_prefix="system"
Wipe out any existing partition table on the disks (really, here be dragons!):
# gpart destroy -F ada0 ada0 destroyed # gpart destroy -F ada1 ada1 destroyed # gpart destroy -F ada2 ada2 destroyed # gpart destroy -F ada3 ada3 destroyed
Initialize the disk with a GUID partition table (GPT):
# gpart create -s GPT ada0 ada0 created # gpart create -s GPT ada1 ada1 created # gpart create -s GPT ada2 ada2 created # gpart create -s GPT ada3 ada3 created
Install the stage 1/2 boot loaders (pmbr in the MBR and gptzfsboot in a boot loader partition 1):
# gpart add -t freebsd-boot -l "${gpt_prefix}-1-boot" -i 1 -s 481K ada0 ada0p1 added # gpart add -t freebsd-boot -l "${gpt_prefix}-2-boot" -i 1 -s 480K ada1 ada1p1 added # gpart add -t freebsd-boot -l "${gpt_prefix}-3-boot" -i 1 -s 480K ada2 ada2p1 added # gpart add -t freebsd-boot -l "${gpt_prefix}-4-boot" -i 1 -s 480K ada3 ada3p1 added # gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0 partcode written to ada0p1 bootcode written to ada0 # gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada1 partcode written to ada1p1 bootcode written to ada1 # gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada2 partcode written to ada2p1 bootcode written to ada2 # gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada3 partcode written to ada3p1 bootcode written to ada3
Create a swap partition (desired size 1GiB divided by 4 partitions = 256MiB per partition in this example) in each disk:
# gpart add -t freebsd-swap -l "${gpt_prefix}-1-swap" -s 256M ada0 ada0p2 added # gpart add -t freebsd-swap -l "${gpt_prefix}-2-swap" -s 256M ada1 ada1p2 added # gpart add -t freebsd-swap -l "${gpt_prefix}-3-swap" -s 256M ada2 ada2p2 added # gpart add -t freebsd-swap -l "${gpt_prefix}-4-swap" -s 256M ada3 ada3p2 added
Create a ZFS partition in each disk, then create a ZFS pool with those partitions (-O options to the zpool command specify filesystem options; customize to your liking, but do keep -O mountpoint=/):
# gpart add -t freebsd-zfs -l "${gpt_prefix}-1-zfs" ada0 ada0p3 added # gpart add -t freebsd-zfs -l "${gpt_prefix}-2-zfs" ada1 ada1p3 added # gpart add -t freebsd-zfs -l "${gpt_prefix}-3-zfs" ada2 ada2p3 added # gpart add -t freebsd-zfs -l "${gpt_prefix}-4-zfs" ada3 ada3p3 added # kldload zfs # sysctl vfs.zfs.min_auto_ashift=12 # cd /dev # zpool create -o altroot=/mnt -o cachefile=/tmp/zpool.cache -O mountpoint=/ -O compression=lz4 -O atime=off "${zpool}" gpt/"${gpt_prefix}"-*-zfs) # df -h /mnt Filesystem Size Used Avail Capacity Mounted on system 897G 31k 897G 0% /mnt
Create a few more filesystems (again, customize to your liking).
These are the bsdinstall defaults:
# zfs create -o setuid=off "${zpool}/tmp" # zfs create -o canmount=off "${zpool}/var" # zfs create -o setuid=off -o exec=off "${zpool}/var/audit" # zfs create -o setuid=off -o exec=off "${zpool}/var/crash" # zfs create -o setuid=off -o exec=off "${zpool}/var/log" # zfs create -o atime=on "${zpool}/var/mail" # zfs create -o setuid=off "${zpool}/var/tmp" # zfs create -o canmount=off "${zpool}/usr" # zfs create -o setuid=off "${zpool}/usr/ports"
And these are my personal recommendations:
# zfs create -o setuid=off "${zpool}/usr/src" # zfs create -o setuid=off "${zpool}/usr/obj" # zfs create "${zpool}/usr/local" # zfs create "${zpool}/home"
Copy the zpool.cache (created earlier by zfs create) into /boot/zfs 3:
# mkdir -p /mnt/boot/zfs # cp -p /tmp/zpool.cache /mnt/boot/zfs/zpool.cache
Add the swap partition:
# echo "/dev/gpt/${gpt_prefix}-1-swap none swap sw 0 0" >> /tmp/bsdinstall_etc/fstab # echo "/dev/gpt/${gpt_prefix}-2-swap none swap sw 0 0" >> /tmp/bsdinstall_etc/fstab # echo "/dev/gpt/${gpt_prefix}-3-swap none swap sw 0 0" >> /tmp/bsdinstall_etc/fstab # echo "/dev/gpt/${gpt_prefix}-4-swap none swap sw 0 0" >> /tmp/bsdinstall_etc/fstab
Return to the installer:
# exit
Continue with regular installation procedure, until it asks you: “The installation is now finished. Before exiting the installer, would you like to open a shell in the new system to make any final manual modifications?” Choose “Yes”, which will drop you into a shell again.
Now we need to do additional, ZFS-specific configuration. First, enable ZFS when booting up:
# echo 'zfs_load="YES"' >> /boot/loader.conf # echo 'zfs_enable="YES"' >> /etc/rc.conf
Enable 4K alignment on pools in /etc/rc.conf:
# echo 'vfs.zfs.min_auto_ashift=12' >> /etc/sysctl.conf
Exit the shell.
# exit
Installation is completed; the system will ask you to reboot. Do so, with your fingers crossed. :-p
1 ^ pmbr locates a GUID partition of freebsd-boot type then loads and executes the next-stage boot code from it.
2 ^ The stage-2 boot loader partition size (480KiB) == the end of real-mode (legacy) conventional memory (512KiB) - the boot loader address in memory (0x7c00), rounded down to 4K sector size of modern drives. The theoretical maximum amount of conventional memory is 640KiB (0xa0000), but the stage-1 boot loader (pmbr) itself asserts a lower bound of 576KiB (0x90000); our limit of 512KiB (0x80000) is even more conservative.
3 ^ zpool.cache contains information about system pools (i.e. pools imported without the -R option). Various stages of booting process need it in order to locate the root/boot pool.