Installing Gentoo Into a LUKS-Encrypted ZFS Root
2013-12-31 14:31 - Linux
Note: This is a 2019 rewrite from scratch, of an article originally written in late 2013. For posterity you can find a local mirror of that older version of the article, plus one at archive.org and another at archive.is. This newer version of the article represents a bit of my knowledge gathered over the past few years, plus some new experimentation. I'm setting up a fake system in a virtual machine, just to try out all the steps, examine options, and pick my new favorite methodology.
For the past few years I've relied on ZFS for my backup system, with atomic snapshots taken regularly, plus zfs send and zfs recv making geographic redundancy an easy extra layer. I also LUKS encrypt the disks for peace of mind, especially in the case of failed disks. This document is a detailed explanation of how I set up a brand new machine from scratch with an ecrypted ZFS root file system.
Getting Started
Work through the Gentoo Handbook until the Preparing the Disks section. You may want to use my LiveCD with ZFS support as the boot medium, as I'm doing.
My most recent setup uses raidz2 across four disks, to guarantee any two drive failures does not result in any downtime nor data loss. A raidz1 might be acceptable for you; the examples below are for a three-drive raidz1.
I used to prefer a completely separate (unencrypted) boot device. I originally thought that it was "best practice" to provision ZFS on raw disks, with no partitions. I'm no longer confident that this is true, and I've come up with reasons to prefer partitions. It can be difficult to hook up enough disks. It's common for motherboards to provide four SATA ports. If that's three ZFS volumes plus one boot disk, they're all used up. Where do you put your extra disk when swapping in a replacement or an upgrade? Even if you've got the space, there's the expense of another device, plus a separate plan for it's failure? If you've got the extra connectors and devices, maybe go for an L2ARC or ZIL. I'm skipping dedicated boot devices from here on out.
So now I'm selecting a partitioned scheme. Each disk will have partition 1 occupying the first 512MB for the unencrypted boot, then partition 2 fills the rest of the disk. Set that up.
(A quick note on the examples before this first one: The shell prompts are colored red, the inputs I type are colored green, and the rest is the output. Your output will likely differ in small details; I'm trusting you to be intelligent enough to figure that out if you're following this as a guide. But I find archiving the output still makes it easier to follow along. Your inputs may differ as well, be careful to make sure you are referencing (e.g.) the proper disk at every point!)
I prefer to use GPT partitions and label the data partitions crypt_something so that "something" is meaningful. (I use the model and/or serial number of the drive, so that if/when ZFS reports a problem with a volume, I'll know which physical disk it's on!) Depending on your situation you might need a "bios boot" or "EFI System" partition (which I take out of the 512MB mentioned above).
livecd ~ # fdisk /dev/sda Welcome to fdisk (util-linux 2.37.2). Changes will remain in memory only, until you decide to write them. Be careful before using the write command. Device does not contain a recognized partition table. Created a new DOS disklabel with disk identifier 0xaa8002a1. Command (m for help): g Created a new GPT disklabel (GUID: 968404CA-3435-C844-B6FE-FBA86C98DC13). Command (m for help): n Partition number (1-128, default 1): First sector (2048-97677278, default 2048): Last sector, +/-sectors or +/-size{K,M,G,T,P} (2048-97677278, default 97677278): +256M Created a new partition 1 of type 'Linux filesystem' and of size 256 MiB. Command (m for help): n Partition number (2-128, default 2): First sector (526336-97677278, default 526336): Last sector, +/-sectors or +/-size{K,M,G,T,P} (526336-97677278, default 97677278): +128M Created a new partition 2 of type 'Linux filesystem' and of size 128 MiB. Command (m for help): t Partition number (1,2, default 2): Partition type or alias (type L to list all): 1 Changed type of partition 'Linux filesystem' to 'EFI System'. Command (m for help): n Partition number (3-128, default 3): First sector (788480-97677278, default 788480): Last sector, +/-sectors or +/-size{K,M,G,T,P} (788480-97677278, default 97677278): Created a new partition 3 of type 'Linux filesystem' and of size 46.2 GiB. Command (m for help): w The partition table has been altered. Calling ioctl() to re-read partition table. Syncing disks. livecd ~ # gdisk /dev/sda GPT fdisk (gdisk) version 1.0.8 Partition table scan: MBR: protective BSD: not present APM: not present GPT: present Found valid GPT with protective MBR; using GPT. Command (? for help): c Partition number (1-3): 1 Enter name: boot Command (? for help): c Partition number (1-3): 3 Enter name: crypt_something Command (? for help): w Final checks complete. About to write GPT data. THIS WILL OVERWRITE EXISTING PARTITIONS!! Do you want to proceed? (Y/N): y OK; writing new GUID partition table (GPT) to /dev/sda. The operation has completed successfully.
Since we're putting ZFS in a partition, it's important to make sure that the partition start is aligned with a physical disk sector. This virtual disk has 512 byte sectors, but real drives today typically have 4,096 byte sectors. The partitions are using logical sectors, so it is possible to misalign the physical sectors. I believe the best thing to do is make sure that your main partition starts on a multiple-of-eight sector, which should be 4k aligned even for 512 byte logical sectors. I did this by picking a perfect power-of-two boot partition size.
Do the same for sdb and sdc (and/or whichever drives you're using). Afterwards, we can confirm with fdisk -l that all our partitions are the same size on each drive. If so, make a nice basic ext4 partition on sda1, which will be our boot partition. (Here is also where you'll FAT format the EFI partition, if you have it.) You can set up something to back this up to e.g. sdb2 or somewhere in your redundant ZFS.
livecd ~ # mkfs.ext4 -L boot_a -T small -U random /dev/sda1 mke2fs 1.46.5 (30-Dec-2021) Discarding device blocks: done Creating filesystem with 65536 4k blocks and 65536 inodes Filesystem UUID: fa093515-03ae-483d-9dd3-bb429c097fc0 Superblock backups stored on blocks: 32768 Allocating group tables: done Writing inode tables: done Creating journal (4096 blocks): done Writing superblocks and filesystem accounting information: done
Next, use LUKS to encrypt all three data partitions:
livecd ~ # for D in a b c; do cryptsetup luksFormat /dev/sd${D}2; done WARNING! ======== This will overwrite data on /dev/sda2 irrevocably. Are you sure? (Type uppercase yes): YES Enter passphrase: Verify passphrase: System is out of entropy while generating volume key. Please move mouse or type some text in another window to gather some random events. Generating key (68% done). Generating key (87% done). Generating key (100% done). WARNING! ======== This will overwrite data on /dev/sdb2 irrevocably. Are you sure? (Type uppercase yes): YES Enter passphrase: Verify passphrase: System is out of entropy while generating volume key. Please move mouse or type some text in another window to gather some random events. Generating key (43% done). Generating key (100% done). WARNING! ======== This will overwrite data on /dev/sdc2 irrevocably. Are you sure? (Type uppercase yes): YES Enter passphrase: Verify passphrase: System is out of entropy while generating volume key. Please move mouse or type some text in another window to gather some random events. Generating key (93% done). Generating key (100% done).
I'm using the same passphrase for all three disks, and suggest you do the same. Either way, LUKS will let you change/add/remove these at any point in the future. If you get stuck waiting for randomness, a ping -f from another machine to this one will generate kernel entropy.
With the encrypted volumes initialized, open them up for use:
livecd ~ # for D in a b c; do cryptsetup luksOpen /dev/sd${D}2 crypt_sd${D}2; done Enter passphrase for /dev/sda2: Enter passphrase for /dev/sdb2: Enter passphrase for /dev/sdc2: livecd ~ # ls -l /dev/mapper total 0 crw------- 1 root root 10, 236 Feb 22 19:12 control brw------- 1 root root 253, 0 Feb 22 19:51 crypt_sda2 brw------- 1 root root 253, 1 Feb 22 19:52 crypt_sdb2 brw------- 1 root root 253, 2 Feb 22 19:52 crypt_sdc2
Initializing ZFS
So now we have three encrypted disks called crypt_sda2, crypt_sdb2, and crypt_sdc2. We can set up a zpool with them, and data sets within that pool.
livecd ~ # zpool create -m none -R /mnt/gentoo -o ashift=12 -O atime=off -O xattr=sa rpool raidz1 crypt_sda2 crypt_sdb2 crypt_sdc2
This complicated command will:
- -m none
- Not mount this pool.
- -R /mnt/gentoo
- Sets the "altroot", i.e. the temporary alternative mount point. This value is appropriate for the Handbook driven install process.
- -o ashift=12
- Set the block size to 4k, or 212. (If your sectors are 512 bytes (unlikely), you should omit this.)
- -O atime=off
- Not record access times.
- -O xattr=sa
- Store extended attributes in inodes rather than hidden files. Which is supposed to make Samba more performant.
- rpool
- The name of the pool. (I wish I had a better scheme, but I just call it the "r"oot pool.)
- raidz1
- The type of the pool, RAID like with one disk for redundancy.
- crypt_sda crypt_sdb crypt_sdc
- The devices making up this pool; ZFS can find them just by these short names (which are unlikely to otherwise exist) and their brevity will make other output easier to read.
And in good Unix tradition, produces no output upon success.
Now to create the datasets within the pool. ZFS datasets are hierarchical. I've created two top level data sets: root and tmp. The first gets regular snapshots, which get replicated off site. The latter does not, because it contains only files that are easy to replace (linux kernel source, portage) or are not worth backing up (large scratch files, media archives).
First, (create the root dataset, and within that) initialize mount points with no write permissions, to reduce the chances of accidentally putting files there. (ZFS refuses to mount into a non-empty directory, by default.)
livecd ~ # zfs create -o mountpoint=/ rpool/root livecd ~ # for D in /home /tmp /var; do mkdir "/mnt/gentoo/$D"; chmod 0111 "/mnt/gentoo/$D"; done
Now create the rest of the ZFS data sets:
livecd ~ # cd /mnt/gentoo livecd ~ # zfs create -o mountpoint=/var rpool/root/var livecd ~ # mkdir var/tmp; chmod 0111 var/tmp livecd ~ # zfs create -o mountpoint=/home rpool/root/home livecd ~ # mkdir home/USER; chmod 0111 home/USER livecd ~ # zfs create -o mountpoint=/home/USER rpool/root/home/USER livecd ~ # mkdir home/USER/tmp; chmod 0111 home/USER/tmp livecd ~ # zfs create -o mountpoint=none rpool/tmp livecd ~ # zfs create -o mountpoint=/tmp -o devices=off -o exec=off -o setuid=off rpool/tmp/root livecd ~ # zfs create -o mountpoint=/home/USER/tmp -o devices=off -o exec=off -o setuid=off rpool/tmp/USER livecd ~ # zfs create -o mountpoint=/usr/src rpool/tmp/linux-src livecd ~ # zfs create -o mountpoint=/usr/portage rpool/tmp/portage livecd ~ # zfs create -o mountpoint=/var/tmp rpool/tmp/var
Note that ZFS will auto-mount these data sets as they're created, at the given mount points (relative to the altroot specified at zpool creation). Fill in the actual name for "USER". Also optionally (but recommended) set up swap.
livecd ~ # zfs create -o sync=always -o primarycache=metadata -o secondarycache=none -o volblocksize=4K -V 1G rpool/swap livecd ~ # mkswap -f /dev/zvol/rpool/swap Setting up swapspace version 1, size = 1048572 KiB no label, UUID=c072e84f-08bc-4fbb-9e65-0c1ba83a85bc livecd ~ # swapon /dev/zvol/rpool/swap
Continue with the last few settings for our mounts:
livecd ~ # mkdir /mnt/gentoo/boot livecd ~ # chmod 0111 /mnt/gentoo/boot livecd ~ # mount /dev/sda1 /mnt/gentoo/boot livecd ~ # chmod 1777 /mnt/gentoo/tmp livecd ~ # cd /mnt/gentoo
The Kernel
We're doing a standard Gentoo install now, from the installing Stage3 section. When reaching the configuring the kernel section make sure you include ext4 file system support hard-coded in (our boot partition). With the LiveCD having loaded modules for most or all of our hardware, we can make localmodconfig to rapidly generate a tidy kernel config. There's a few key settings we'll need, to enable LUKS and ZFS support. (The Gentoo wiki calls out some of these.)
Device Drivers --->
Multiple devices driver support --->
<*> Device mapper support
<*> Crypt target support
File systems --->
[*] Miscellaneous filesystems --->
<*> Persistent store support
<*> DEFLATE (ZLIB) compression
Cryptographic API --->
<*> XTS support
(Specifying the "persistent store support" is just a roundabout way to force CONFIG_ZLIB_DEFLATE on, which you can't control directly for some reason. But compiling SPL requires it to be set.)
Be sure to also include any hardware drivers that your machine will depend on for boot (especially e.g. SATA, maybe Ethernet, maybe USB and/or mass storage if you'd like to use those in the initramfs recovery). These must be included in the kernel (y, not m), or specified as extra modules (see below). Running lspci -k will tell you the (loaded) modules that power your hardware, search for those entries in menuconfig and enable them. Install the kernel, but then we diverge at the Building an initramfs step. We'll customize ours, to handle both LUKS encryption and a ZFS root filesystem.
First install the ZFS tools (and kernel modules, via dependency).
(chroot) livecd ~ # emerge -va zfs These are the packages that would be merged, in order: ... (chroot) livecd ~ # rc-update add zfs-mount sysinit
When changing kernels in the future, be prepared to emerge @module-rebuild to make sure these out-of-tree modules are built for the kernel being used.
Boot Configuration
We've completed the Kernel section, continue with the handbook at the Configuring the system section. The fstab should look something like:
# <fs> <mountpoint> <type> <opts> <dump/pass> LABEL=boot_a /boot ext4 noauto,noatime 1 2 /dev/zvol/rpool/swap none swap sw 0 0
But once we reach the Configuring the bootloader section, again we diverge. First, simply, we're going to install the grub2 bootloader to all three disks. In case of failure of any one disk, we'll be able to boot from the others:
(chroot) livecd ~ # cat > /etc/portage/package.use/grub # No eye candy, thanks. sys-boot/grub -fonts -themes (chroot) livecd ~ # emerge -vat grub These are the packages that would be merged, in reverse order: Calculating dependencies... done! [ebuild N ] sys-boot/grub-2.02:2/2.02::gentoo ... ... (chroot) livecd ~ # for D in a b c; do grub-install /dev/sd$D; done Installing for i386-pc platform. Installation finished. No error reported. Installing for i386-pc platform. Installation finished. No error reported. Installing for i386-pc platform. Installation finished. No error reported.
Next we need an initramfs to hold enough user space tools, unencrypted, to mount our encrypted ZFS root file system. We're going to use tranquil-initramfs for this. We also need to install a few of its dependencies before it will run.
(chroot) livecd ~ # emerge -vat dev-vcs/git cryptsetup busybox These are the packages that would be merged, in reverse order: Calculating dependencies... done! ... [ebuild N ] app-crypt/gnupg-2.2.4-r2::gentoo ... ... [ebuild N ] sys-fs/cryptsetup-1.7.5::gentoo ... ... [ebuild N ] dev-vcs/git-2.16.1::gentoo ... ... (chroot) livecd ~ # git clone https://github.com/arantius/tranquil-initramfs.git ... (chroot) livecd ~ # cd tranquil-initramfs (chroot) livecd tranquil-initramfs # ./mkinitrd.py [*] Checking preliminary binaries ... [*] Creating temporary directory at /home/t-bone/opt/tranquil-initramfs/bi-947944809 ... [*] Checking required files ... [+] Using LUKS [+] Using ZFS [*] Copying binaries ... [*] Copying modules ... [*] Generating modprobe information ... [*] Copying library dependencies ... [*] Creating symlinks ... [*] Performing finishing steps ... [*] Creating the initramfs ... 98694 blocks [*] Please copy "initrd-5.10.76-gentoo-r1" to your /boot directory
This initramfs contains the ZFS modules, necessary to import the pool of course. They're tied to the kernel version, so if you ever upgrade (or downgrade), you'll need to rebuild the initramfs to include that kernel's matching modules. (After a emerge @module-rebuild, to make them available.) You might need to (create and) edit config.ini in the tranquil-initramfs directory, to include e.g. extra files and/or modules. Check the documentation at the page linked above.
Now we configure grub to boot the kernel and initramfs we just made. By hand! I don't want the complex / graphical things that the default tools do.
(chroot) livecd ~ # cp -a initrd-4.9.76-gentoo-r1 /boot (chroot) livecd ~ # cd /boot (chroot) livecd /boot # ln -s vmlinuz-$(ls /lib/modules|sort|tail -1) kernel (chroot) livecd /boot # cp -aL kernel kernel.old (chroot) livecd /boot # ln -s initrd-$(ls /lib/modules|sort|tail -1) initrd (chroot) livecd /boot # cp -aL initrd initrd.old (chroot) livecd /boot # cat > /boot/grub/grub.cfg debug all if [ -s $prefix/grubenv ]; then load_env fi menuentry 'Gentoo GNU/Linux' { insmod part_msdos insmod ext4 set root='hd0,msdos1' echo 'Loading Linux ...' linux /kernel consoleblank=0 crashkernel=64M enc_type=pass triggers=luks,zfs redetect no_cache by=/dev/mapper enc_drives=/dev/disk/by-partlabel/crypt_* echo 'Loading initial ramdisk ...' initrd /initrd } menuentry 'Gentoo GNU/Linux (old)' { insmod part_msdos insmod ext4 set root='hd0,msdos1' echo 'Loading Linux ...' linux /kernel.old consoleblank=0 crashkernel=64M enc_type=pass triggers=luks,zfs redetect no_cache by=/dev/mapper enc_drives=/dev/disk/by-partlabel/crypt_* echo 'Loading initial ramdisk ...' initrd /initrd.old } ^D
Some of the details will depend on your specific set-up.
We're just about done! Continue from handbook section configuring the system (and skipping "Configuring the bootloader").
Appendix: Recovery
Should you ever need to reboot during installation, or later boot from the livecd for recovery, it would go something like this:
livecd ~ # for D in a b c; do cryptsetup luksOpen /dev/sd${D}2 crypt_sd${D}2; done Enter passphrase for /dev/sda2: Enter passphrase for /dev/sdb2: Enter passphrase for /dev/sdc2: livecd ~ # zpool import -fR /mnt/gentoo -d /dev/mapper rpool livecd ~ # mount /dev/sda1 /mnt/gentoo/boot livecd ~ # mount --rbind --make-rslave /dev /mnt/gentoo/dev livecd ~ # mount --rbind --make-rslave /sys /mnt/gentoo/sys livecd ~ # mount -t proc none /mnt/gentoo/proc livecd ~ # mount -t devpts devpts /mnt/gentoo/dev/pts livecd ~ # chroot /mnt/gentoo /bin/bash
Appendix: Links
I relied on a number of existing sources to figure out how to do this. Most of this information came from Gentoo Hardened ZFS rootfs with dm-crypt/luks 0.6.2 which is quite similar to this article. I added some more details from the Funtoo wiki's ZFS Install Guide and the Gentoo wiki's ZFS page.
