Arantius.com

Installing Gentoo Into a LUKS-Encrypted ZFS Root

2013-12-31 14:31 - Linux

This is a continuation of my earlier explorations booting from a LUKS encrypted disk. This time, I'm booting Gentoo Linux from a LUKS encrypted ZFS volume. I won't go into why in detail (others do that at length elsewhere), but I do like the checksumming and snapshotting that ZFS provides. I'm still re-working my backup system to function through ZFS snapshots, but my progress so far is already great.

I've now done this procedure twice for real (my server at home, and the remote server in my Mom's basement to serve as the off-site backup). And of course, I learned and tweaked as I was doing so. I took notes, and I'm doing it a third time in a virtual machine to make sure I got all the details right. The first time I figured out booting from LUKS it was really helpful, even just for me to refer to again later. You'll best start with my Gentoo minimal install LiveCD patched with ZFS support. Any bootable 64-bit ZFS enabled media will do, but that's easy and familiar if you've done Gentoo before. Note however that this is a simple cookbook style reference, not a tutorial. If you don't already know ZFS, you'd do very well to read up on it. I have a collection of links at the end for lots more detail.

Work through the Gentoo Handbook until the Preparing the Disks section. My disk scheme is: three 2TB disks in a RAIDZ1 pool, with LUKS encryption underneath. They're all the same model and thus the same size. Additionally another small disk to hold the (unencrypted) boot. I started testing with a plain old USB flash drive, and later switched to a Compact Flash to IDE adapter as a cheap/small/low power fake SSD. So in my case sda, sdb, and sdc are the main three drives, while sdd is the (USB) boot drive. Setting them up looks like:

(A quick note on the examples before this first one: The shell prompts are colored red, the inputs I type are colored green, and the rest is the output. Your output will likely differ in small details; I'm trusting you to be intelligent enough to figure that out if you're following this as a guide. But I find archiving the output still makes it easier to follow along. Your inputs may differ as well, be careful to make sure you are referencing (e.g.) the proper disk!)

livecd ~ # fdisk /dev/sdd
Welcome to fdisk (util-linux 2.22.2).

Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0x14d61137.

Command (m for help): p

Disk /dev/sdd: 1073 MB, 1073741824 bytes, 2097152 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x14d61137

   Device Boot      Start         End      Blocks   Id  System

Command (m for help): n
Partition type:
   p   primary (0 primary, 0 extended, 4 free)
   e   extended
Select (default p):
Using default response p
Partition number (1-4, default 1):
Using default value 1
First sector (2048-2097151, default 2048):
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-2097151, default 2097151):
Using default value 2097151
Partition 1 of type Linux and of size 1023 MiB is set

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.
livecd ~ # mkfs.ext2 /dev/sdd1
mke2fs 1.42.7 (21-Jan-2013)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
65536 inodes, 261888 blocks
13094 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=268435456
8 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376

Allocating group tables: done
Writing inode tables: done
Writing superblocks and filesystem accounting information: done

A note here on encryption passphrases. I've elected to use the same passphrase on all three disks. Later, I set it up so that I only have to type this passphrase once upon boot to unlock all three. I suggest you do the same. Also note: yes, I am encrypting the entire raw disk here, on purpose. ZFS does not need partitions, just a block device to store its data. The boot disk is partitioned however, to give GRUB room to install.

livecd ~ # cryptsetup luksFormat /dev/sda

WARNING!
========
This will overwrite data on /dev/sda irrevocably.

Are you sure? (Type uppercase yes): YES
Enter passphrase:
Verify passphrase:
livecd ~ # cryptsetup luksFormat /dev/sdb

WARNING!
========
This will overwrite data on /dev/sdb irrevocably.

Are you sure? (Type uppercase yes): YES
Enter passphrase:
Verify passphrase:
livecd ~ # cryptsetup luksFormat /dev/sdb

WARNING!
========
This will overwrite data on /dev/sdb irrevocably.

Are you sure? (Type uppercase yes): YES
Enter passphrase:
Verify passphrase:
livecd ~ # cryptsetup luksOpen /dev/sda crypt_sda
Enter passphrase for /dev/sda:
livecd ~ # cryptsetup luksOpen /dev/sdb crypt_sdb
Enter passphrase for /dev/sdb:
livecd ~ # cryptsetup luksOpen /dev/sdc crypt_sdc
Enter passphrase for /dev/sdc:

So now we have three encrypted disks called crypt_sda, crypt_sdb, and crypt_sdc. We can set up a zpool with them, and data sets within that pool.

livecd ~ # zpool create -m none -o ashift=12 -o cachefile= -O atime=off -O compression=lz4 -O xattr=sa -R /mnt/gentoo rpool raidz1 crypt_sda crypt_sdb crypt_sdc

This complicated command will:

-m none

Not mount this pool.

-o ashift=12

Set the block size. (Check an e.g. fdisk -l /dev/sda. Mine says Sector size (logical/physical): 512 bytes / 4096 bytes, which shows that the disk's physical sectors are 4k. The ashift=12 means sector sizes of 2¹², or 4k. If your sectors are 512 bytes, you should omit this.

-o cachefile=

Not create a cachefile. These can speed up the zpool import (i.e. "mount") phase. With just three vdevs in the pool, I find this unnecessary, and skipping the complication is better.

-O atime=off

Not record access times.

-O compression=lz4

Turn on compression. This is a good idea. Modern processors tend to have more spare resources than modern disks. This compression algorithm specifically is lightweight, but acheives nice ratios on appropriate content.

-O xattr=sa

Store extended attributes in inodes rather than hidden files. Which is supposed to make Samba more performant.

-R /mnt/gentoo

Sets the "altroot", i.e. the temporary alternative mount point. This value is appropriate for the Handbook driven install process.

rpool

The name of the pool.

raidz1

The type of the pool.

crypt_sda crypt_sdb crypt_sdc

The devices making up this pool; ZFS can find them just by these short names (which are unlikely to otherwise exist) and their brevity will make other output easier to read.

And in good Unix tradition, produces no output upon success. Now to create the datasets within the pool. ZFS datasets are heirarchical. I've created two top level data sets: root and tmp. The first gets regular snapshots (via zfs-auto-snapshot), which get replicated off site. The latter does not, because it contains only files that are easy to replace (linux kernel source, portage) or are not worth backing up (large scratch files, media archives).

livecd ~ # zfs create -o mountpoint=/ rpool/root
livecd ~ # zfs create -o mountpoint=/var -o devices=off -o exec=off -o setuid=off rpool/root/var
livecd ~ # zfs create -o mountpoint=/home rpool/root/home
livecd ~ # zfs create -o mountpoint=/home/user rpool/root/home/user
livecd ~ # zfs create -o mountpoint=none rpool/tmp
livecd ~ # zfs create -o mountpoint=/tmp -o devices=off -o exec=off -o setuid=off rpool/tmp/root
livecd ~ # zfs create -o mountpoint=/home/user/tmp -o devices=off -o exec=off -o setuid=off rpool/tmp/user
livecd ~ # zfs create -o mountpoint=/usr/src rpool/tmp/linux-src
livecd ~ # zfs create -o mountpoint=/usr/portage rpool/tmp/portage
livecd ~ # zfs create -o mountpoint=/var/tmp rpool/tmp/var

Note that ZFS will auto-mount these data sets at the given mount points (relative to the altroot specified at zpool creation). Fill in the actual name for "user". Also optionally (but recommended) set up swap.

livecd ~ # zfs create -o sync=always -o primarycache=metadata -o secondarycache=none -o volblocksize=4K -V 1G rpool/swap
livecd ~ # mkswap -f /dev/zvol/rpool/swap
Setting up swapspace version 1, size = 1048572 KiB
no label, UUID=c072e84f-08bc-4fbb-9e65-0c1ba83a85bc
livecd ~ # swapon /dev/zvol/rpool/swap

Continue with the mounting section of the handbook, but remember that the ZFS data sets are already mounted.

livecd ~ # mkdir /mnt/gentoo/boot
livecd ~ # mount /dev/sdd1 /mnt/gentoo/boot
livecd ~ # chmod 1777 /mnt/gentoo/tmp
livecd ~ # cd /mnt/gentoo

We're doing a standard Gentoo install now, from the installation section. When reaching the configuring the kernel section, use genkernel. We'll need to use a custom linuxrc in order to luksOpen all three of our encrypted drives at the right point (before it tries to mount our ZFS root). We must also build a valid kernel before we can (build and) install the ZFS kernel modules. So:

(chroot) livecd ~ # echo "sys-kernel/genkernel cryptsetup" >> /etc/portage/package.use
(chroot) livecd ~ # echo "sys-fs/lvm2 -thin" >> /etc/portage/package.use
(chroot) livecd ~ # emerge -va genkernel gentoo-sources

These are the packages that would be merged, in order:
...
(chroot) livecd ~ # genkernel --makeopts=-j6 bzImage
(chroot) livecd ~ # cat >> /etc/portage/package.accept_keywords
# required by zfs (argument)
=sys-fs/zfs-0.6.2-r3 ~amd64
# required by sys-fs/zfs-kmod-0.6.2-r3
# required by sys-fs/zfs-0.6.2-r3
# required by zfs (argument)
=sys-kernel/spl-0.6.2-r2 ~amd64
# required by sys-fs/zfs-0.6.2-r3
# required by zfs (argument)
=sys-fs/zfs-kmod-0.6.2-r3 ~amd64
^D
(chroot) livecd ~ # emerge -va zfs

These are the packages that would be merged, in order:
...
(chroot) livecd ~ # rc-update add zfs-mount boot

We also want to create a custom linuxrc for the initramfs.

(chroot) livecd ~ # cd /root
(chroot) livecd ~ # cat > linuxrc.patch
--- linuxrc.orig        2016-12-21 13:23:20.089289542 -0500
+++ linuxrc     2016-12-21 14:47:20.110946586 -0500
@@ -389,10 +389,29 @@
 setup_btrfsctl
 
 # Setup md device nodes if they dont exist
 setup_md_device
 
+
+luks_open() {
+  DISK="$1"
+  PASSPHRASE="$2"
+  DISK_NAM="$(echo $DISK | sed -e 's@.*/@@')"
+  cryptsetup isLuks "$DISK" 2>/dev/null || return
+  good_msg "Opening $DISK ..."
+  echo "$PASSPHRASE" | cryptsetup luksOpen "$DISK" crypt_$DISK_NAM
+  test_success "luksOpen $DISK"
+}
+echo -n "Please enter LUKS passphrase: "
+read -s PASSPHRASE
+echo
+for D in /dev/sd?; do
+  luks_open "$D" "$PASSPHRASE"
+done
+unset D PASSPHRASE
+
+
 # Scan volumes
 startVolumes
 
 setup_keymap
^D
(chroot) livecd ~ # cat /usr/share/genkernel/defaults/linuxrc > linuxrc && patch < linuxrc.patch

The result is that the three encrypted disks will be unlocked immediately before the ZFS pool is imported. This technique does unfortunately read in the LUKS passphrase in a shell script, and echo it (via the command line) into cryptsetup. Putting sensitive data like a passphrase onto the command line like this is normally a big security no-no. I'm comfortable with this simply because it exists only inside the initramfs; any attacker capable of monitoring closely enough to catch the passphrase at this point (by its presence on the command line) can mount a simpler and equally effective attack. If you're uncomfortable with that, you can simply omit the reading and echoing of $PASSPHRASE (and thus be forced to type it three times instead).

Now, be sure to call genkernel like:

(chroot) livecd ~ # genkernel --makeopts=-j6 --no-clean --menuconfig --luks --zfs --linuxrc=/root/linuxrc --callback="emerge @module-rebuild" all

Some of these flags can instead be permanently set via /etc/genkernel.conf but not (as far as I know) the --zfs nor --linuxrc flags, both of which are critical. The rebuild callback is probably only necessary this first time.

Continue with the handbook. The fstab should be empty besides /boot and swap. Install grub2 into the bootloader, and configure it for booting:

(chroot) livecd ~ #  grub-install /dev/sdd
Installing for i386-pc platform.
Installation finished. No error reported.
(chroot) livecd ~ # echo 'GRUB_CMDLINE_LINUX="dozfs=force real_root=ZFS=rpool/root"' >> /etc/default/grub
# (Or, edit this file as such.)
(chroot) livecd ~ # cat /proc/mounts | grep -v rootfs > /etc/mtab
(chroot) livecd ~ # grub-mkconfig -o /boot/grub/grub.cfg
Generating grub configuration file ...
Found linux image: /boot/kernel-genkernel-x86_64-4.4.26-gentoo
Found initrd image: /boot/initramfs-genkernel-x86_64-4.4.26-gentoo
done

Unfortunately we really do need to dozfs=force. Linux seems unable to cleanly export (i.e. unmount) the root file system partition. Forcing the import is the only way I know to make things work. Continue from handbook section configuring the system. Done!

Appendix: Recovery

Should you ever need to reboot during installation, or later boot from the livecd for recovery, it would go something like this:

livecd ~ # rmmod floppy
livecd ~ # cryptsetup luksOpen /dev/sda crypt_sda
livecd ~ # cryptsetup luksOpen /dev/sdb crypt_sdb
livecd ~ # cryptsetup luksOpen /dev/sdc crypt_sdc
livecd ~ # zpool import -fR /mnt/gentoo rpool
livecd ~ # mount /dev/sdd1 /mnt/gentoo/boot
livecd ~ # mount -t proc none /mnt/gentoo/proc
livecd ~ # mount --rbind /dev /mnt/gentoo/dev
livecd ~ # mount --rbind /sys /mnt/gentoo/sys
livecd ~ # chroot /mnt/gentoo /bin/bash

The floppy module seems to get auto-loaded even though the machine has no floppy drive, and this causes zpool import to hang while trying to check if it should open a vdev on fd0. So first remove it. The luksOpen lines unlock the encrypted volumes, then we mount all the other required paths and chroot into it.

Appendix: Links

I relied on a number of existing sources to figure out how to do this. Most of this information came from Gentoo Hardened ZFS rootfs with dm-crypt/luks 0.6.2 which is quite similar to this article. I added some more details from the Funtoo wiki's ZFS Install Guide and the Gentoo wiki's ZFS page.