Installing Gentoo Into a LUKS-Encrypted ZFS Root
2013-12-31 14:31 - Linux
Note: This is a 2019 rewrite from scratch, of an article originally written in late 2013. For posterity you can find a local mirror of that older version of the article, plus one at archive.org and another at archive.is. This newer version of the article represents a bit of my knowledge gathered over the past few years, plus some new experimentation. I'm setting up a fake system in a virtual machine, just to try out all the steps, examine options, and pick my new favorite methodology.
For the past few years I've relied on ZFS for my backup system, with atomic snapshots taken regularly, plus zfs send and zfs recv making geographic redundancy an easy extra layer. I also LUKS encrypt the disks for peace of mind, especially in the case of failed disks.
This document is a detailed explanation of how I set up a brand new machine from scratch with an ecrypted ZFS root file system.
Getting Started
Work through the Gentoo Handbook until the Preparing the Disks section. You may want to use my LiveCD with ZFS support as the boot medium, as I'm doing.
I have selected raidz1 for a balance of disk redundancy plus usable space. I've had a couple disks go bad over the years, but never two at once, so this means I've always been able to replace a disk before any serious catastrophe. So I assume that there are three disks.
I used to prefer a completely separate (unencrypted) boot device. I originally thought that it was "best practice" to provision ZFS on raw disks, with no partitions. I'm no longer confident that this is true, and I've come up with reasons to prefer partitions. It can be difficult to hook up enough disks. It's common for motherboards to provide four SATA ports. If that's three ZFS volumes plus one boot disk, they're all used up. Where do you put your extra disk when swapping in a replacement or an upgrade? Even if you've got the space, there's the expense of another device, plus a separate plan for it's failure? If you've got the extra connectors and devices, maybe go for an L2ARC or ZIL. I'm skipping dedicated boot devices from here on out.
So now I'm selecting a partitioned scheme. Each disk will have partition 1 occupying the first 512MB for the unencrypted boot, then partition 2 fills the rest of the disk. Set that up.
(A quick note on the examples before this first one: The shell prompts are colored red, the inputs I type are colored green, and the rest is the output. Your output will likely differ in small details; I'm trusting you to be intelligent enough to figure that out if you're following this as a guide. But I find archiving the output still makes it easier to follow along. Your inputs may differ as well, be careful to make sure you are referencing (e.g.) the proper disk at every point!)
Warning Later parts of this guide assume GPT formatting and GPT labels on the disk, but the partitioning section has not been updated. Be sure to use GPT partitions and label your data partitions crypt_something so that "something" is meaningful to you. (I use the model and/or serial number of the drive, so that if/when ZFS reports a problem with, I'll know which physical disk it is!) Depending on your situation you might need a "bios boot" partition as well.
livecd ~ # fdisk /dev/sda Welcome to fdisk (util-linux 2.30.2). Changes will remain in memory only, until you decide to write them. Be careful before using the write command. Device does not contain a recognized partition table Building a new DOS disklabel with disk identifier 0x14d61137. Command (m for help): n Partition type: p primary (0 primary, 0 extended, 4 free) e extended Select (default p): p Partition number (1-4, default 1): 1 Using default value 1 First sector (2048-2097151, default 2048): Last sector, +sectors or +size{K,M,G,T,P} (2048-41943039, default 41943039): +512M Created a new partition 1 of type 'Linux' and of size 512 MiB. Command (m for help): n Partition type p primary (1 primary, 0 extended, 3 free) e extended (container for logical partitions) Select (default p): p Partition number (2-4, default 2): 2 First sector (1050624-41943039, default 1050624): Last sector, +sectors or +size{K,M,G,T,P} (1050624-41943039, default 41943039): Created a new partition 2 of type 'Linux' and of size 19.5 GiB. Command (m for help): p Disk /dev/sda: 20 GiB, 21474836480 bytes, 41943040 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: dos Disk identifier: 0x45d9f690 Device Boot Start End Sectors Size Id Type /dev/sda1 2048 1050623 1048576 512M 83 Linux /dev/sda2 1050624 41943039 40892416 19.5G 83 Linux Command (m for help): w The partition table has been altered. Calling ioctl() to re-read partition table. Syncing disks.
Since we're putting ZFS in a partition, it's of the utmost importance to make sure that the partition start is aligned with a physical disk sector. This virtual disk has 512 byte sectors, but real drives today typically have 4,096 byte sectors. See the logical/physical size listed. The partitions are using logical sectors, so it is possible to misalign the physical sectors. I believe the best thing to do is make sure that your main partition starts on a multiple-of-eight sector, which should be 4k aligned, even for 512 byte logical sectors. Like above, 1,050,624 / 8 = 131,328. An even multiple and at 512 bytes each, that makes the 512MB we asked for, plus the 1MB of buffer before the first partition. Perfect. (Picking a power-of-two size for the first partition is a safe call.)
Do the same for sdb and sdc. Afterwards, we can confirm with fdisk -l that all our partitions are the same size. If so, make a nice basic ext2 partition on sda, which will be our boot partition.
livecd ~ # mkfs.ext2 -L boot_a -T small -U random /dev/sda1 mke2fs 1.43.6 (29-Aug-2017) Creating filesystem with 524288 1k blocks and 131072 inodes Filesystem UUID: 7721e1b8-bf88-4534-955d-6582929a7a1e Superblock backups stored on blocks: 8193, 24577, 40961, 57345, 73729, 204801, 221185, 401409 Allocating group tables: done Writing inode tables: done Writing superblocks and filesystem accounting information: done
Next, use LUKS to encrypt all three data partitions:
livecd ~ # for D in a b c; do cryptsetup luksFormat /dev/sd${D}2; done WARNING! ======== This will overwrite data on /dev/sda2 irrevocably. Are you sure? (Type uppercase yes): YES Enter passphrase: Verify passphrase: System is out of entropy while generating volume key. Please move mouse or type some text in another window to gather some random events. Generating key (68% done). Generating key (87% done). Generating key (100% done). WARNING! ======== This will overwrite data on /dev/sdb2 irrevocably. Are you sure? (Type uppercase yes): YES Enter passphrase: Verify passphrase: System is out of entropy while generating volume key. Please move mouse or type some text in another window to gather some random events. Generating key (43% done). Generating key (100% done). WARNING! ======== This will overwrite data on /dev/sdc2 irrevocably. Are you sure? (Type uppercase yes): YES Enter passphrase: Verify passphrase: System is out of entropy while generating volume key. Please move mouse or type some text in another window to gather some random events. Generating key (93% done). Generating key (100% done).
I'm using the same passphrase for all three disks, and suggest you do the same. Either way, LUKS will let you change/add/remove these at any point in the future. If you get stuck waiting for randomness, a ping -f from another machine to this will generate kernel entropy.
With the encrypted volumes initialized, open them up for use:
livecd ~ # for D in a b c; do cryptsetup luksOpen /dev/sd${D}2 crypt_sd${D}2; done Enter passphrase for /dev/sda2: Enter passphrase for /dev/sdb2: Enter passphrase for /dev/sdc2: livecd ~ # ls -l /dev/mapper total 0 crw------- 1 root root 10, 236 Feb 22 19:12 control brw------- 1 root root 253, 0 Feb 22 19:51 crypt_sda2 brw------- 1 root root 253, 1 Feb 22 19:52 crypt_sdb2 brw------- 1 root root 253, 2 Feb 22 19:52 crypt_sdc2
Initializing ZFS
So now we have three encrypted disks called crypt_sda2, crypt_sdb2, and crypt_sdc2. We can set up a zpool with them, and data sets within that pool.
livecd ~ # zpool create -m none -R /mnt/gentoo -o ashift=12 -O atime=off -O xattr=sa rpool raidz1 crypt_sda2 crypt_sdb2 crypt_sdc2
This complicated command will:
- -m none
- Not mount this pool.
- -R /mnt/gentoo
- Sets the "altroot", i.e. the temporary alternative mount point. This value is appropriate for the Handbook driven install process.
- -o ashift=12
- Set the block size to 4k, or 212. (If your sectors are 512 bytes (unlikely), you should omit this.)
- -O atime=off
- Not record access times.
- -O xattr=sa
- Store extended attributes in inodes rather than hidden files. Which is supposed to make Samba more performant.
- rpool
- The name of the pool. (I wish I had a better scheme, but I just call it the "r"oot pool.)
- raidz1
- The type of the pool, RAID like with one disk for redundancy.
- crypt_sda crypt_sdb crypt_sdc
- The devices making up this pool; ZFS can find them just by these short names (which are unlikely to otherwise exist) and their brevity will make other output easier to read.
And in good Unix tradition, produces no output upon success.
Now to create the datasets within the pool. ZFS datasets are hierarchical. I've created two top level data sets: root and tmp. The first gets regular snapshots, which get replicated off site. The latter does not, because it contains only files that are easy to replace (linux kernel source, portage) or are not worth backing up (large scratch files, media archives).
First, (create the root dataset, and within that) initialize mountpoints with no write permissions, to reduce the chances of accidentally putting files there. (ZFS refuses to mount into a non-empty directory, by default.)
livecd ~ # zfs create -o mountpoint=/ rpool/root livecd ~ # for D in /home /tmp /var /var; do mkdir "/mnt/gentoo/$D"; chmod 0111 "/mnt/gentoo/$D"; done
Now create the rest of the ZFS data sets:
livecd ~ # cd /mnt/gentoo livecd ~ # zfs create -o mountpoint=/var rpool/root/var livecd ~ # mkdir var/tmp; chmod 0111 var/tmp livecd ~ # zfs create -o mountpoint=/home rpool/root/home livecd ~ # mkdir home/USER; chmod 0111 home/USER livecd ~ # zfs create -o mountpoint=/home/USER rpool/root/home/USER livecd ~ # mkdir home/USER/tmp; chmod 0111 home/USER/tmp livecd ~ # zfs create -o mountpoint=none rpool/tmp livecd ~ # zfs create -o mountpoint=/tmp -o devices=off -o exec=off -o setuid=off rpool/tmp/root livecd ~ # zfs create -o mountpoint=/home/USER/tmp -o devices=off -o exec=off -o setuid=off rpool/tmp/USER livecd ~ # zfs create -o mountpoint=/usr/src rpool/tmp/linux-src livecd ~ # zfs create -o mountpoint=/usr/portage rpool/tmp/portage livecd ~ # zfs create -o mountpoint=/var/tmp rpool/tmp/var
Note that ZFS will auto-mount these data sets as they're created, at the given mount points (relative to the altroot specified at zpool creation). Fill in the actual name for "USER". Also optionally (but recommended) set up swap.
livecd ~ # zfs create -o sync=always -o primarycache=metadata -o secondarycache=none -o volblocksize=4K -V 1G rpool/swap livecd ~ # mkswap -f /dev/zvol/rpool/swap Setting up swapspace version 1, size = 1048572 KiB no label, UUID=c072e84f-08bc-4fbb-9e65-0c1ba83a85bc livecd ~ # swapon /dev/zvol/rpool/swap
Continue with the last few settings for our mounts:
livecd ~ # mkdir /mnt/gentoo/boot livecd ~ # chmod 0111 /mnt/gentoo/boot livecd ~ # mount /dev/sda1 /mnt/gentoo/boot livecd ~ # chmod 1777 /mnt/gentoo/tmp livecd ~ # cd /mnt/gentoo
The Kernel
We're doing a standard Gentoo install now, from the installing Stage3 section. When reaching the configuring the kernel section make sure you include ext2 file system support hard-coded in (our boot partition). With the LiveCD having loaded modules for most or all of our hardware, we can make localmodconfig to rapidly generate a tidy kernel config. There's a few key settings we'll need, to enable LUKS and ZFS support. (The Gentoo wiki calls out some of these.)
Device Drivers --->
Multiple devices driver support --->
<*> Device mapper support
<*> Crypt target support
File systems --->
[*] Miscellaneous filesystems --->
<*> Persistent store support
<*> DEFLATE (ZLIB) compression
Cryptographic API --->
<*> XTS support
(Specifying the "persistent store support" is just a roundabout way to force CONFIG_ZLIB_DEFLATE on, which you can't control directly for some reason. But compiling SPL requires it to be set.)
Be sure to also include any hardware drivers that your machine will depend on for boot (especially e.g. SATA, maybe Ethernet, maybe USB and/or mass storage if you'd like to use those in the initramfs recovery). These must be included in the kernel (y, not m), or specified as extra modules (see below). Running lspci -k will tell you the (loaded) modules that power your hardware, search for those entries in menuconfig and enable them. Install the kernel, but then we diverge at the Building an initramfs step. We'll customize ours, to handle both LUKS encryption and a ZFS root filesystem.
First install the ZFS tools (and kernel modules, via dependency).
(chroot) livecd ~ # emerge -va zfs These are the packages that would be merged, in order: ... (chroot) livecd ~ # rc-update add zfs-mount sysinit
When changing kernels in the future, be prepared to emerge @module-rebuild to make sure these out-of-tree modules are built for the kernel being used.
Boot Configuration
We've completed the Kernel section, continue with the handbook at the Configuring the system section. The fstab should look something like:
# <fs> <mountpoint> <type> <opts> <dump/pass> LABEL=boot_a /boot ext2 noauto,noatime 1 2 /dev/zvol/rpool/swap none swap sw 0 0
But once we reach the Configuring the bootloader section, again we diverge. First, simply, we're going to install the grub2 bootloader to all three disks. In case of failure of any one disk, we'll be able to boot from the others:
(chroot) livecd ~ # cat > /etc/portage/package.use/grub # No eye candy, thanks. sys-boot/grub -fonts -themes (chroot) livecd ~ # emerge -vat grub These are the packages that would be merged, in reverse order: Calculating dependencies... done! [ebuild N ] sys-boot/grub-2.02:2/2.02::gentoo ... ... (chroot) livecd ~ # for D in a b c; do grub-install /dev/sd$D; done Installing for i386-pc platform. Installation finished. No error reported. Installing for i386-pc platform. Installation finished. No error reported. Installing for i386-pc platform. Installation finished. No error reported.
Next we need an initramfs to hold enough user space tools, unencrypted, to mount our encrypted ZFS root file system. We're going to use tranquil-initramfs for this. We also need to install a few of its dependencies before it will run.
(chroot) livecd ~ # emerge -vat dev-vcs/git cryptsetup gnupg busybox These are the packages that would be merged, in reverse order: Calculating dependencies... done! ... [ebuild N ] app-crypt/gnupg-2.2.4-r2::gentoo ... ... [ebuild N ] sys-fs/cryptsetup-1.7.5::gentoo ... ... [ebuild N ] dev-vcs/git-2.16.1::gentoo ... ... (chroot) livecd ~ # git clone https://github.com/arantius/tranquil-initramfs.git ... (chroot) livecd ~ # cd tranquil-initramfs (chroot) livecd tranquil-initramfs # echo "Kernel name: $(ls /lib/modules|sort|tail -1)" Kernel name: 4.9.76-gentoo-r1 (chroot) livecd tranquil-initramfs # ./mkinitrd.py [*] Checking preliminary binaries ... [*] Creating temporary directory at /home/t-bone/opt/tranquil-initramfs/bi-947944809 ... [*] Checking required files ... [+] Using LUKS [+] Using ZFS [*] Copying binaries ... [*] Copying modules ... [*] Generating modprobe information ... [*] Copying library dependencies ... [*] Creating symlinks ... [*] Performing finishing steps ... [*] Creating the initramfs ... 98694 blocks [*] Please copy "initrd-5.10.76-gentoo-r1" to your /boot directory
This initramfs contains the ZFS modules, necessary to import the pool of course. They're tied to the kernel version, so if you ever upgrade (or downgrade), you'll need to rebuild the initramfs to include that kernel's matching modules. (After a emerge @module-rebuild, to make them available.) You might need to (create and) edit config.ini in the tranquil-initramfs directory, to include e.g. extra files and/or modules. Check the documentation at the page linked above.
Now we configure grub to boot the kernel and initramfs we just made. By hand! I don't want the complex / graphical things that the default tools do.
(chroot) livecd ~ # cp -a initrd-4.9.76-gentoo-r1 /boot (chroot) livecd ~ # cd /boot (chroot) livecd /boot # ln -s vmlinuz-$(ls /lib/modules|sort|tail -1) kernel (chroot) livecd /boot # cp -aL kernel kernel.old (chroot) livecd /boot # ln -s initrd-$(ls /lib/modules|sort|tail -1) initrd (chroot) livecd /boot # cp -aL initrd initrd.old (chroot) livecd /boot # cat > /boot/grub/grub.cfg debug all if [ -s $prefix/grubenv ]; then load_env fi menuentry 'Gentoo GNU/Linux' { insmod part_msdos insmod ext2 set root='hd0,msdos1' echo 'Loading Linux ...' linux /kernel consoleblank=0 crashkernel=64M enc_type=pass triggers=luks,zfs redetect no_cache by=/dev/mapper enc_drives=/dev/disk/by-partlabel/crypt_* echo 'Loading initial ramdisk ...' initrd /initrd } menuentry 'Gentoo GNU/Linux (old)' { insmod part_msdos insmod ext2 set root='hd0,msdos1' echo 'Loading Linux ...' linux /kernel.old consoleblank=0 crashkernel=64M enc_type=pass triggers=luks,zfs redetect no_cache by=/dev/mapper enc_drives=/dev/disk/by-partlabel/crypt_* echo 'Loading initial ramdisk ...' initrd /initrd.old } ^D
Some of the details will depend on your specific set-up.
We're just about done! Continue from handbook section configuring the system (and skipping "Configuring the bootloader").
Appendix: Recovery
Should you ever need to reboot during installation, or later boot from the livecd for recovery, it would go something like this:
livecd ~ # for D in a b c; do cryptsetup luksOpen /dev/sd${D}2 crypt_sd${D}2; done Enter passphrase for /dev/sda2: Enter passphrase for /dev/sdb2: Enter passphrase for /dev/sdc2: livecd ~ # zpool import -fR /mnt/gentoo -d /dev/mapper rpool livecd ~ # mount /dev/sda1 /mnt/gentoo/boot livecd ~ # mount --rbind --make-rslave /dev /mnt/gentoo/dev livecd ~ # mount --rbind --make-rslave /sys /mnt/gentoo/sys livecd ~ # mount -t proc none /mnt/gentoo/proc livecd ~ # mount -t devpts devpts /mnt/gentoo/dev/pts livecd ~ # chroot /mnt/gentoo /bin/bash
Appendix: Links
I relied on a number of existing sources to figure out how to do this. Most of this information came from Gentoo Hardened ZFS rootfs with dm-crypt/luks 0.6.2 which is quite similar to this article. I added some more details from the Funtoo wiki's ZFS Install Guide and the Gentoo wiki's ZFS page.
