ZFS-only installation of FreeBSD from DVD, with encrypted root pool

[astralblue]

This is a step-by-step guide, mostly for my own reference, but you might find this useful too.

Disclaimer: This article is about clean installation of operating system, where “clean” means that, if you follow these instructions, all existing contents of the hard drive will be destroyed.  Do not follow these instructions except on an empty hard drive.  I shall not be held responsible for any loss of data.

1. 
   Boot using the FreeBSD DVD.
   
2. 
   Drop into the fixit shell, using “CDROM/DVD” as the fixit media.
   
3. 
   Load necessary kernel modules:
   Fixit# chroot /mnt2 kldload nullfs zfs geom_eli
   
4. 
   Wipe out any existing partition table on the disk (da0 in this example):
   Fixit# dd if=/dev/zero of=/dev/da0 bs=1m count=128 128+0 records in 128+0 records out 134217728 bytes transferred in 3.627965 secs (36995321 bytes/sec)
   
5. 
   Initialize the disk with a GUID partition table (GPT):
   Fixit# gpart create -s GPT da0 da0 created
   
6. 
   Install GPT boot code “pmbr” into the MBR:
   Fixit# gpart bootcode -b /mnt2/boot/pmbr da0 da0 has bootcode
   
7. 
   Create the boot partition that pmbr expects 1, then install stage 2 boot code “gptzfsboot” into it:
   Fixit# gpart add -t freebsd-boot -l crimson-bootcode -s 128 da0 da0p1 added Fixit# gpart bootcode -p /mnt2/boot/gptzfsboot -i 1 da0
   
8. 
   Create a swap partition:
   Fixit# gpart add -t freebsd-swap -l crimson-swap -s 1G da0 da0p2 added
   
9. 
   Create ZFS root/boot partitions:
   Fixit# gpart add -t freebsd-zfs -l crimson-boot -s 1G da0 da0p3 added Fixit# gpart add -t freebsd-zfs -l crimson-root da0 da0p4 added
   
10. 
    Encrypt the root partition with geli, protecting the master encryption key with a passphrase:
    Fixit# geli init -b -s 4096 -B /tmp/crimson-root.bak gpt/crimson-root Enter new passphrase:
    (your own passphrase) Reenter new passphrase:
    (your own passphrase) Metadata backup can be found in /tmp/crimson-root.bak and can be restored with the following command: # geli restore /tmp/crimson-root.bak gpt/crimson-root Fixit# geli attach gpt/crimson-root Enter passphrase:
    (your own passphrase)
    
    The passphrase must be strong but not impractically long, as it needs to be entered every time the system boots.
11. 
    Create the ZFS root/boot pools:
    Fixit# zpool create -o altroot=/mnt -o cachefile=/tmp/zpool.cache crimson-root gpt/crimson-root.eli Fixit# zpool create -o altroot=/mnt -o cachefile=/tmp/zpool.cache -m /bootfs crimson-boot gpt/crimson-boot
    
12. 
    Make the ZFS boot pool available for booting 2:
    Fixit# zpool set bootfs=crimson-boot crimson-boot
    
13. 
    Now we will copy the FreeBSD installer (sysinstall) into the new filesystem and run it in a chroot-ed environment 3.  sysinstall needs a couple of things, and we have to prepare them.  First, make /dev and /dist (FreeBSD installation DVD) available inside the filesystem:
    Fixit# mkdir /mnt/dev /mnt/mnt Fixit# mount -t devfs devfs /mnt/dev Fixit# mount_nullfs /dist /mnt/mnt
    
14. 
    Copy sysinstall and friends, residing in /stand:
    Fixit# find -xd /stand -print0 | cpio -pdum0 /mnt 5172 blocks
    
15. 
    sysinstall expects a number of tools in /usr/bin; copy them:
    Fixit# mkdir -p /mnt/usr/bin Fixit# (cd /stand && find gunzip cpio -print0 | cpio -pdum0 /mnt/usr/bin) 4749 blocks
    
16. 
    We will instruct sysinstall to emit progress messages to a file named sysinstall.debug.  Start dumping its contents onto ttyv4 (accessible by pressing Alt-F5):
    Fixit# tail -Fc+0 /mnt/sysinstall.debug > /dev/ttyv4 & Fixit# stty -f /dev/ttyv4 $(stty -g)
    
17. 
    Now chroot into the filesystem and run sysinstall there:
    Fixit# env debug=YES chroot /mnt /stand/sysinstall
    
18. 
    In the main menu, go to the “Configure” menu.
    
19. 
    In the configuration menu, choose “Media.”
    
20. 
    In the media selection menu, choose “File System.”
    
21. 
    sysinstall asks you where the installation file is.  Enter “/mnt” (without quotes).
    
22. 
    Back in the configuration menu, choose “Distributions.”
    
23. 
    In the distribution selection menu, select all that you want to install, then select “Exit.”  You will at least need the base and kernels at the very minimum.
    
24. 
    sysinstall starts installation of selected distributions.  Wait until installation finishes, and you are brought back to the configuration menu.
    
25. 
    Configure the installed system further as necessary.  Common tasks include:
    
    • Setting root password;
    • Setting the system timezone;
    • Adding initial user/group;
    • Setting at least one network interface (where you will set the hostname too);
    • Enabling sshd.
26. Exit to the main menu, then select “Exit Install” to end sysinstall.
    
27. 
    Now we need to do additional, ZFS-specific configuration.  First, move the /boot directory to the boot filesystem, and symlink to it from the root filesystem:
    Fixit# chroot /mnt mv boot /bootfs/boot Fixit# ln -shf bootfs/boot /mnt/boot Fixit# chflags -h schg /mnt/boot
    
28. 
    Copy zpool.cache, which was created earlier when we ran zfs create, into /boot/zfs 4:
    Fixit# cd /mnt/boot Fixit# mkdir -p zfs Fixit# cp -p /tmp/zpool.cache zfs/zpool.cache
    
29. 
    The previous sysinstall installed the kernel not in /boot/kernel but in /boot/GENERIC.  Instruct BTX loader to look for the kernel there instead:
    Fixit# echo 'kernel="GENERIC"' >> loader.conf
    
30. 
    Load the ZFS kernel module when booting:
    Fixit# echo 'zfs_load="YES"' >> loader.conf
    
31. 
    Load geom_eli too:
    Fixit# echo 'geom_eli_load="YES"' >> loader.conf
    
32. 
    Tell geom_eli to echo each passphrase letter entered (some systems randomly eat boot-time passphrase keystrokes and need this):
    Fixit# echo 'kern.geom.eli.visible_passphrase="2"' >> loader.conf
    
33. 
    Instruct that the ZFS root pool (“crimson-root”) is also the root filesystem:
    Fixit# echo 'vfs.root.mountfrom="zfs:crimson-root"' >> loader.conf
    
34. 
    Tell ZFS not to disable prefetch:
    Fixit# echo 'vfs.zfs.prefetch_disable="0"' >> loader.conf
    
35. 
    Enable ZFS in /etc/rc.conf:
    Fixit# cd /mnt/etc Fixit# echo 'zfs_enable="YES"' >> rc.conf
    
36. 
    Add the swap partition:
    Fixit# echo '/dev/gpt/crimson-swap.eli none swap sw 0 0' >> fstab
    
37. 
    Exit the fixit shell and go back to the FreeBSD installer:
    Fixit# exit
    
38. 
    Go to the main menu, then select “Exit Install” to reboot.
    

1 ^ pmbr locates a GUID partition of freebsd-boot type then loads and executes the next-stage boot code from it.

2 ^ gptzfsboot locates a ZFS pool with the bootfs (boot filesystem) property set, then loads and executes BTX loader (a.k.a. /boot/loader) from that filesystem.

3 ^ Note that an instance of sysinstall is already running on ttyv0, accessible by pressing Alt-F1; that's where we started the fixit shell in the first place. We are going to start another instance of sysinstall from the fixit shell on ttyv3 (Alt-F4).

4 ^ zpool.cache contains information about system pools (i.e. pools imported without the -R option).  Various stages of booting process need it in order to locate the root/boot pools.