Installing Gentoo Into a LUKS-Encrypted ZFS Root
2013-12-31 14:31 - Linux
This is a continuation of my earlier explorations booting from a LUKS encrypted disk. This time, I'm booting Gentoo Linux from a LUKS encrypted ZFS volume. I won't go into why in detail (others do that at length elsewhere), but I do like the checksumming and snapshotting that ZFS provides. I'm still re-working my backup system to function through ZFS snapshots, but my progress so far is already great.
I've now done this procedure twice for real (my server at home, and the remote server in my Mom's basement to serve as the off-site backup). And of course, I learned and tweaked as I was doing so. I took notes, and I'm doing it a third time in a virtual machine to make sure I got all the details right. The first time I figured out booting from LUKS it was really helpful, even just for me to refer to again later. You'll best start with my Gentoo minimal install LiveCD patched with ZFS support. Any bootable 64-bit ZFS enabled media will do, but that's easy and familiar if you've done Gentoo before. Note however that this is a simple cookbook style reference, not a tutorial. If you don't already know ZFS, you'd do very well to read up on it. I have a collection of links at the end for lots more detail.
Work through the Gentoo Handbook until the Preparing the Disks section. My disk scheme is: three 2TB disks in a RAIDZ1 pool, with LUKS encryption underneath. They're all the same model and thus the same size. Additionally another small disk to hold the (unencrypted) boot. I started testing with a plain old USB flash drive, and later switched to a Compact Flash to IDE adapter as a cheap/small/low power fake SSD. So in my case sda, sdb, and sdc are the main three drives, while sdd is the (USB) boot drive. Setting them up looks like:
(A quick note on the examples before this first one: The shell prompts are colored red, the inputs I type are colored green, and the rest is the output. Your output will likely differ in small details; I'm trusting you to be intelligent enough to figure that out if you're following this as a guide. But I find archiving the output still makes it easier to follow along. Your inputs may differ as well, be careful to make sure you are referencing (e.g.) the proper disk!)
livecd ~ # fdisk /dev/sdd Welcome to fdisk (util-linux 2.22.2). Changes will remain in memory only, until you decide to write them. Be careful before using the write command. Device does not contain a recognized partition table Building a new DOS disklabel with disk identifier 0x14d61137. Command (m for help): p Disk /dev/sdd: 1073 MB, 1073741824 bytes, 2097152 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x14d61137 Device Boot Start End Blocks Id System Command (m for help): n Partition type: p primary (0 primary, 0 extended, 4 free) e extended Select (default p): Using default response p Partition number (1-4, default 1): Using default value 1 First sector (2048-2097151, default 2048): Using default value 2048 Last sector, +sectors or +size{K,M,G} (2048-2097151, default 2097151): Using default value 2097151 Partition 1 of type Linux and of size 1023 MiB is set Command (m for help): w The partition table has been altered! Calling ioctl() to re-read partition table. Syncing disks. livecd ~ # mkfs.ext2 /dev/sdd1 mke2fs 1.42.7 (21-Jan-2013) Filesystem label= OS type: Linux Block size=4096 (log=2) Fragment size=4096 (log=2) Stride=0 blocks, Stripe width=0 blocks 65536 inodes, 261888 blocks 13094 blocks (5.00%) reserved for the super user First data block=0 Maximum filesystem blocks=268435456 8 block groups 32768 blocks per group, 32768 fragments per group 8192 inodes per group Superblock backups stored on blocks: 32768, 98304, 163840, 229376 Allocating group tables: done Writing inode tables: done Writing superblocks and filesystem accounting information: done
A note here on encryption passphrases. I've elected to use the same passphrase on all three disks. Later, I set it up so that I only have to type this passphrase once upon boot to unlock all three. I suggest you do the same. Also note: yes, I am encrypting the entire raw disk here, on purpose. ZFS does not need partitions, just a block device to store its data. The boot disk is partitioned however, to give GRUB room to install.
livecd ~ # cryptsetup luksFormat /dev/sda WARNING! ======== This will overwrite data on /dev/sda irrevocably. Are you sure? (Type uppercase yes): YES Enter passphrase: Verify passphrase: livecd ~ # cryptsetup luksFormat /dev/sdb WARNING! ======== This will overwrite data on /dev/sdb irrevocably. Are you sure? (Type uppercase yes): YES Enter passphrase: Verify passphrase: livecd ~ # cryptsetup luksFormat /dev/sdb WARNING! ======== This will overwrite data on /dev/sdb irrevocably. Are you sure? (Type uppercase yes): YES Enter passphrase: Verify passphrase: livecd ~ # cryptsetup luksOpen /dev/sda crypt_sda Enter passphrase for /dev/sda: livecd ~ # cryptsetup luksOpen /dev/sdb crypt_sdb Enter passphrase for /dev/sdb: livecd ~ # cryptsetup luksOpen /dev/sdc crypt_sdc Enter passphrase for /dev/sdc:
So now we have three encrypted disks called crypt_sda, crypt_sdb, and crypt_sdc. We can set up a zpool with them, and data sets within that pool.
livecd ~ # zpool create -m none -o ashift=12 -o cachefile= -O atime=off -O compression=lz4 -O xattr=sa -R /mnt/gentoo rpool raidz1 crypt_sda crypt_sdb crypt_sdc
This complicated command will:
- -m none
- Not mount this pool.
- -o ashift=12
- Set the block size. (Check an e.g. fdisk -l /dev/sda. Mine says Sector size (logical/physical): 512 bytes / 4096 bytes, which shows that the disk's physical sectors are 4k. The ashift=12 means sector sizes of 212, or 4k. If your sectors are 512 bytes, you should omit this.
- -o cachefile=
- Not create a cachefile. These can speed up the zpool import (i.e. "mount") phase. With just three vdevs in the pool, I find this unnecessary, and skipping the complication is better.
- -O atime=off
- Not record access times.
- -O compression=lz4
- Turn on compression. This is a good idea. Modern processors tend to have more spare resources than modern disks. This compression algorithm specifically is lightweight, but acheives nice ratios on appropriate content.
- -O xattr=sa
- Store extended attributes in inodes rather than hidden files. Which is supposed to make Samba more performant.
- -R /mnt/gentoo
- Sets the "altroot", i.e. the temporary alternative mount point. This value is appropriate for the Handbook driven install process.
- rpool
- The name of the pool.
- raidz1
- The type of the pool.
- crypt_sda crypt_sdb crypt_sdc
- The devices making up this pool; ZFS can find them just by these short names (which are unlikely to otherwise exist) and their brevity will make other output easier to read.
And in good Unix tradition, produces no output upon success. Now to create the datasets within the pool. ZFS datasets are heirarchical. I've created two top level data sets: root and tmp. The first gets regular snapshots (via zfs-auto-snapshot), which get replicated off site. The latter does not, because it contains only files that are easy to replace (linux kernel source, portage) or are not worth backing up (large scratch files, media archives).
livecd ~ # zfs create -o mountpoint=/ rpool/root livecd ~ # zfs create -o mountpoint=/var -o devices=off -o exec=off -o setuid=off rpool/root/var livecd ~ # zfs create -o mountpoint=/home rpool/root/home livecd ~ # zfs create -o mountpoint=/home/user rpool/root/home/user livecd ~ # zfs create -o mountpoint=none rpool/tmp livecd ~ # zfs create -o mountpoint=/tmp -o devices=off -o exec=off -o setuid=off rpool/tmp/root livecd ~ # zfs create -o mountpoint=/home/user/tmp -o devices=off -o exec=off -o setuid=off rpool/tmp/user livecd ~ # zfs create -o mountpoint=/usr/src rpool/tmp/linux-src livecd ~ # zfs create -o mountpoint=/usr/portage rpool/tmp/portage livecd ~ # zfs create -o mountpoint=/var/tmp rpool/tmp/var
Note that ZFS will auto-mount these data sets at the given mount points (relative to the altroot specified at zpool creation). Fill in the actual name for "user". Also optionally (but recommended) set up swap.
livecd ~ # zfs create -o sync=always -o primarycache=metadata -o secondarycache=none -o volblocksize=4K -V 1G rpool/swap livecd ~ # mkswap -f /dev/zvol/rpool/swap Setting up swapspace version 1, size = 1048572 KiB no label, UUID=c072e84f-08bc-4fbb-9e65-0c1ba83a85bc livecd ~ # swapon /dev/zvol/rpool/swap
Continue with the mounting section of the handbook, but remember that the ZFS data sets are already mounted.
livecd ~ # mkdir /mnt/gentoo/boot livecd ~ # mount /dev/sdd1 /mnt/gentoo/boot livecd ~ # chmod 1777 /mnt/gentoo/tmp livecd ~ # cd /mnt/gentoo
We're doing a standard Gentoo install now, from the installation section. When reaching the configuring the kernel section, use genkernel. We'll need to use a custom linuxrc in order to luksOpen all three of our encrypted drives at the right point (before it tries to mount our ZFS root). We must also build a valid kernel before we can (build and) install the ZFS kernel modules. So:
(chroot) livecd ~ # echo "sys-kernel/genkernel cryptsetup" >> /etc/portage/package.use (chroot) livecd ~ # echo "sys-fs/lvm2 -thin" >> /etc/portage/package.use (chroot) livecd ~ # emerge -va genkernel gentoo-sources These are the packages that would be merged, in order: ... (chroot) livecd ~ # genkernel --makeopts=-j6 bzImage (chroot) livecd ~ # cat >> /etc/portage/package.accept_keywords # required by zfs (argument) =sys-fs/zfs-0.6.2-r3 ~amd64 # required by sys-fs/zfs-kmod-0.6.2-r3 # required by sys-fs/zfs-0.6.2-r3 # required by zfs (argument) =sys-kernel/spl-0.6.2-r2 ~amd64 # required by sys-fs/zfs-0.6.2-r3 # required by zfs (argument) =sys-fs/zfs-kmod-0.6.2-r3 ~amd64 ^D (chroot) livecd ~ # emerge -va zfs These are the packages that would be merged, in order: ... (chroot) livecd ~ # rc-update add zfs-mount boot
We also want to create a custom linuxrc for the initramfs.
(chroot) livecd ~ # cd /root (chroot) livecd ~ # cat > linuxrc.patch --- linuxrc.orig 2016-12-21 13:23:20.089289542 -0500 +++ linuxrc 2016-12-21 14:47:20.110946586 -0500 @@ -389,10 +389,29 @@ setup_btrfsctl # Setup md device nodes if they dont exist setup_md_device + +luks_open() { + DISK="$1" + PASSPHRASE="$2" + DISK_NAM="$(echo $DISK | sed -e 's@.*/@@')" + cryptsetup isLuks "$DISK" 2>/dev/null || return + good_msg "Opening $DISK ..." + echo "$PASSPHRASE" | cryptsetup luksOpen "$DISK" crypt_$DISK_NAM + test_success "luksOpen $DISK" +} +echo -n "Please enter LUKS passphrase: " +read -s PASSPHRASE +echo +for D in /dev/sd?; do + luks_open "$D" "$PASSPHRASE" +done +unset D PASSPHRASE + + # Scan volumes startVolumes setup_keymap ^D (chroot) livecd ~ # cat /usr/share/genkernel/defaults/linuxrc > linuxrc && patch < linuxrc.patch
The result is that the three encrypted disks will be unlocked immediately before the ZFS pool is imported. This technique does unfortunately read in the LUKS passphrase in a shell script, and echo it (via the command line) into cryptsetup. Putting sensitive data like a passphrase onto the command line like this is normally a big security no-no. I'm comfortable with this simply because it exists only inside the initramfs; any attacker capable of monitoring closely enough to catch the passphrase at this point (by its presence on the command line) can mount a simpler and equally effective attack. If you're uncomfortable with that, you can simply omit the reading and echoing of $PASSPHRASE (and thus be forced to type it three times instead).
Now, be sure to call genkernel like:
(chroot) livecd ~ # genkernel --makeopts=-j6 --no-clean --menuconfig --luks --zfs --linuxrc=/root/linuxrc --callback="emerge @module-rebuild" all
Some of these flags can instead be permanently set via /etc/genkernel.conf but not (as far as I know) the --zfs nor --linuxrc flags, both of which are critical. The rebuild callback is probably only necessary this first time.
Continue with the handbook. The fstab should be empty besides /boot and swap. Install grub2 into the bootloader, and configure it for booting:
(chroot) livecd ~ # grub-install /dev/sdd Installing for i386-pc platform. Installation finished. No error reported. (chroot) livecd ~ # echo 'GRUB_CMDLINE_LINUX="dozfs=force real_root=ZFS=rpool/root"' >> /etc/default/grub # (Or, edit this file as such.) (chroot) livecd ~ # cat /proc/mounts | grep -v rootfs > /etc/mtab (chroot) livecd ~ # grub-mkconfig -o /boot/grub/grub.cfg Generating grub configuration file ... Found linux image: /boot/kernel-genkernel-x86_64-4.4.26-gentoo Found initrd image: /boot/initramfs-genkernel-x86_64-4.4.26-gentoo done
Unfortunately we really do need to dozfs=force. Linux seems unable to cleanly export (i.e. unmount) the root file system partition. Forcing the import is the only way I know to make things work. Continue from handbook section configuring the system. Done!
Appendix: Recovery
Should you ever need to reboot during installation, or later boot from the livecd for recovery, it would go something like this:
livecd ~ # rmmod floppy livecd ~ # cryptsetup luksOpen /dev/sda crypt_sda livecd ~ # cryptsetup luksOpen /dev/sdb crypt_sdb livecd ~ # cryptsetup luksOpen /dev/sdc crypt_sdc livecd ~ # zpool import -fR /mnt/gentoo rpool livecd ~ # mount /dev/sdd1 /mnt/gentoo/boot livecd ~ # mount -t proc none /mnt/gentoo/proc livecd ~ # mount --rbind --make-rslave /dev /mnt/gentoo/dev livecd ~ # mount --rbind --make-rslave /sys /mnt/gentoo/sys livecd ~ # chroot /mnt/gentoo /bin/bash
The floppy module seems to get auto-loaded even though the machine has no floppy drive, and this causes zpool import to hang while trying to check if it should open a vdev on fd0. So first remove it. The luksOpen lines unlock the encrypted volumes, then we mount all the other required paths and chroot into it.
Appendix: Links
I relied on a number of existing sources to figure out how to do this. Most of this information came from Gentoo Hardened ZFS rootfs with dm-crypt/luks 0.6.2 which is quite similar to this article. I added some more details from the Funtoo wiki's ZFS Install Guide and the Gentoo wiki's ZFS page.
