Arantius.com

Installing Gentoo Into a LUKS-Encrypted ZFS Root

2013-12-31 14:31 - Linux

Note: This is a 2019 rewrite from scratch, of an article originally written in late 2013. For posterity you can find a local mirror of that older version of the article, plus one at archive.org and another at archive.is. This newer version of the article represents a bit of my knowledge gathered over the past few years, plus some new experimentation. I'm setting up a fake system in a virtual machine, just to try out all the steps, examine options, and pick my new favorite methodology.

For the past few years I've relied on ZFS for my backup system, with atomic snapshots taken regularly, plus zfs send and zfs recv making geographic redundancy an easy extra layer. I also LUKS encrypt the disks for peace of mind, especially in the case of failed disks.

This document is a detailed explanation of how I set up a brand new machine from scratch with an ecrypted ZFS root file system.

Getting Started

Work through the Gentoo Handbook until the Preparing the Disks section. You may want to use my LiveCD with ZFS support as the boot medium, as I'm doing.

I have selected raidz1 for a balance of disk redundancy plus usable space. I've had a couple disks go bad over the years, but never two at once, so this means I've always been able to replace a disk before any serious catastrophe. So I assume that there are three disks.

I used to prefer a completely separate (unencrypted) boot device. I originally thought that it was "best practice" to provision ZFS on raw disks, with no partitions. I'm no longer confident that this is true, and I've come up with reasons to prefer partitions. It can be difficult to hook up enough disks. It's common for motherboards to provide four SATA ports. If that's three ZFS volumes plus one boot disk, they're all used up. Where do you put your extra disk when swapping in a replacement or an upgrade? Even if you've got the space, there's the expense of another device, plus a separate plan for it's failure? If you've got the extra connectors and devices, maybe go for an L2ARC or ZIL. I'm skipping dedicated boot devices from here on out.

So now I'm selecting a partitioned scheme. Each disk will have partition 1 occupying the first 512MB for the unencrypted boot, then partition 2 fills the rest of the disk. Set that up.

(A quick note on the examples before this first one: The shell prompts are colored red, the inputs I type are colored green, and the rest is the output. Your output will likely differ in small details; I'm trusting you to be intelligent enough to figure that out if you're following this as a guide. But I find archiving the output still makes it easier to follow along. Your inputs may differ as well, be careful to make sure you are referencing (e.g.) the proper disk at every point!)

livecd ~ # fdisk /dev/sda
Welcome to fdisk (util-linux 2.30.2).

Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0x14d61137.

Command (m for help): n
Partition type:
   p   primary (0 primary, 0 extended, 4 free)
   e   extended
Select (default p): p
Partition number (1-4, default 1): 1
Using default value 1
First sector (2048-2097151, default 2048):
Last sector, +sectors or +size{K,M,G,T,P} (2048-41943039, default 41943039): +512M

Created a new partition 1 of type 'Linux' and of size 512 MiB.

Command (m for help): n
Partition type
   p   primary (1 primary, 0 extended, 3 free)
   e   extended (container for logical partitions)
Select (default p): p
Partition number (2-4, default 2): 2
First sector (1050624-41943039, default 1050624):
Last sector, +sectors or +size{K,M,G,T,P} (1050624-41943039, default 41943039):

Created a new partition 2 of type 'Linux' and of size 19.5 GiB.

Command (m for help): p
Disk /dev/sda: 20 GiB, 21474836480 bytes, 41943040 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x45d9f690

Device     Boot   Start      End  Sectors  Size Id Type
/dev/sda1          2048  1050623  1048576  512M 83 Linux
/dev/sda2       1050624 41943039 40892416 19.5G 83 Linux

Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.

Since we're putting ZFS in a partition, it's of the utmost importance to make sure that the partition start is aligned with a physical disk sector. This virtual disk has 512 byte sectors, but real drives today typically have 4,096 byte sectors. See the logical/physical size listed. The partitions are using logical sectors, so it is possible to misalign the physical sectors. I believe the best thing to do is make sure that your main partition starts on a multiple-of-eight sector, which should be 4k aligned, even for 512 byte logical sectors. Like above, 1,050,624 / 8 = 131,328. An even multiple and at 512 bytes each, that makes the 512MB we asked for, plus the 1MB of buffer before the first partition. Perfect. (Picking a power-of-two size for the first partition is a safe call.)

Do the same for sdb and sdc. Afterwards, we can confirm with fdisk -l that all our partitions are the same size. If so, make a nice basic ext2 partition on sda, which will be our boot partition.

livecd ~ # mkfs.ext2 -L boot_a -T small -U random /dev/sda1
mke2fs 1.43.6 (29-Aug-2017)
Creating filesystem with 524288 1k blocks and 131072 inodes
Filesystem UUID: 7721e1b8-bf88-4534-955d-6582929a7a1e
Superblock backups stored on blocks:
        8193, 24577, 40961, 57345, 73729, 204801, 221185, 401409

Allocating group tables: done
Writing inode tables: done
Writing superblocks and filesystem accounting information: done

Next, use LUKS to encrypt all three data partitions:

livecd ~ # for D in a b c; do cryptsetup luksFormat /dev/sd${D}2; done

WARNING!
========
This will overwrite data on /dev/sda2 irrevocably.

Are you sure? (Type uppercase yes): YES
Enter passphrase:
Verify passphrase:
System is out of entropy while generating volume key.
Please move mouse or type some text in another window to gather some random events.
Generating key (68% done).
Generating key (87% done).
Generating key (100% done).

WARNING!
========
This will overwrite data on /dev/sdb2 irrevocably.

Are you sure? (Type uppercase yes): YES
Enter passphrase:
Verify passphrase:
System is out of entropy while generating volume key.
Please move mouse or type some text in another window to gather some random events.
Generating key (43% done).
Generating key (100% done).

WARNING!
========
This will overwrite data on /dev/sdc2 irrevocably.

Are you sure? (Type uppercase yes): YES
Enter passphrase:
Verify passphrase:
System is out of entropy while generating volume key.
Please move mouse or type some text in another window to gather some random events.
Generating key (93% done).
Generating key (100% done).

I'm using the same passphrase for all three disks, and suggest you do the same. Either way, LUKS will let you change/add/remove these at any point in the future.

With the encrypted volumes initialized, open them up for use:

livecd ~ # for D in a b c; do cryptsetup luksOpen /dev/sd${D}2 crypt_sd${D}2; done
Enter passphrase for /dev/sda2:
Enter passphrase for /dev/sdb2:
Enter passphrase for /dev/sdc2:
livecd ~ #  ls -l /dev/mapper
total 0
crw------- 1 root root  10, 236 Feb 22 19:12 control
brw------- 1 root root 253,   0 Feb 22 19:51 crypt_sda2
brw------- 1 root root 253,   1 Feb 22 19:52 crypt_sdb2
brw------- 1 root root 253,   2 Feb 22 19:52 crypt_sdc2

Initializing ZFS

So now we have three encrypted disks called crypt_sda2, crypt_sdb2, and crypt_sdc2. We can set up a zpool with them, and data sets within that pool.

livecd ~ # zpool create -m none -R /mnt/gentoo -o ashift=12 -O atime=off -O compression=lz4 -O xattr=sa rpool raidz1 crypt_sda2 crypt_sdb2 crypt_sdc2

This complicated command will:

-m none

Not mount this pool.

-R /mnt/gentoo

Sets the "altroot", i.e. the temporary alternative mount point. This value is appropriate for the Handbook driven install process.

-o ashift=12

Set the block size to 4k, or 2¹². (If your sectors are 512 bytes (unlikely), you should omit this.)

-O atime=off

Not record access times.

-O compression=lz4

Turn on compression. In modern systems, the disk is a worse bottleneck than the CPU. Compression makes everything work faster by causing less I/O for the same data transfer. See this general comparison plus this reference because the first doesn't mention LZ4 -- the second shows it's better than LZJB which the first does include. Almmost anything else adds too much compression overhead to be useful for a general-purpose system.

-O xattr=sa

Store extended attributes in inodes rather than hidden files. Which is supposed to make Samba more performant.

rpool

The name of the pool. (I wish I had a better scheme, but I just call it the "r"oot pool.)

raidz1

The type of the pool, RAID like with one disk for redundancy.

crypt_sda crypt_sdb crypt_sdc

The devices making up this pool; ZFS can find them just by these short names (which are unlikely to otherwise exist) and their brevity will make other output easier to read.

And in good Unix tradition, produces no output upon success.

Now to create the datasets within the pool. ZFS datasets are heirarchical. I've created two top level data sets: root and tmp. The first gets regular snapshots, which get replicated off site. The latter does not, because it contains only files that are easy to replace (linux kernel source, portage) or are not worth backing up (large scratch files, media archives).

First, (create the root dataset, and within that) initialize mountpoints with no write permissions, to reduce the chances of accidentally putting files there. (ZFS refuses to mount into a non-empty directory, by default.)

livecd ~ # zfs create -o mountpoint=/ rpool/root
livecd ~ # for D in /home /tmp /var /var; do mkdir "/mnt/gentoo/$D"; chmod 0111 "/mnt/gentoo/$D"; done

Now create the rest of the ZFS data sets:

livecd ~ # cd /mnt/gentoo
livecd ~ # zfs create -o mountpoint=/var rpool/root/var
livecd ~ # mkdir var/tmp; chmod 0111 var/tmp
livecd ~ # zfs create -o mountpoint=/home rpool/root/home
livecd ~ # mkdir home/USER; chmod 0111 home/USER
livecd ~ # zfs create -o mountpoint=/home/USER rpool/root/home/USER
livecd ~ # mkdir home/USER/tmp; chmod 0111 home/USER/tmp
livecd ~ # zfs create -o mountpoint=none rpool/tmp
livecd ~ # zfs create -o mountpoint=/tmp -o devices=off -o exec=off -o setuid=off rpool/tmp/root
livecd ~ # zfs create -o mountpoint=/home/USER/tmp -o devices=off -o exec=off -o setuid=off rpool/tmp/USER
livecd ~ # zfs create -o mountpoint=/usr/src rpool/tmp/linux-src
livecd ~ # zfs create -o mountpoint=/usr/portage rpool/tmp/portage
livecd ~ # zfs create -o mountpoint=/var/tmp rpool/tmp/var

Note that ZFS will auto-mount these data sets as they're created, at the given mount points (relative to the altroot specified at zpool creation). Fill in the actual name for "USER". Also optionally (but recommended) set up swap.

livecd ~ # zfs create -o sync=always -o primarycache=metadata -o secondarycache=none -o volblocksize=4K -V 1G rpool/swap
livecd ~ # mkswap -f /dev/zvol/rpool/swap
Setting up swapspace version 1, size = 1048572 KiB
no label, UUID=c072e84f-08bc-4fbb-9e65-0c1ba83a85bc
livecd ~ # swapon /dev/zvol/rpool/swap

Continue with the last few settings for our mounts:

livecd ~ # mkdir /mnt/gentoo/boot
livecd ~ # chmod 0111 /mnt/gentoo/boot
livecd ~ # mount /dev/sda1 /mnt/gentoo/boot
livecd ~ # chmod 1777 /mnt/gentoo/tmp
livecd ~ # cd /mnt/gentoo

The Kernel

We're doing a standard Gentoo install now, from the installing Stage3 section. When reaching the configuring the kernel section make sure you include ext2 file system support hard-coded in (our boot partition). With the LiveCD having loaded modules for most or all of our hardware, we can make localmodconfig to rapidly generate a tidy kernel config. There's a few key settings we'll need, to enable LUKS and ZFS support. (The Gentoo wiki calls out some of these.)

Device Drivers --->
  Multiple devices driver support --->
    <*> Device mapper support
      <*> Crypt target support
File systems --->
  [*] Miscellaneous filesystems  --->
    <*>   Persistent store support
    <*>     DEFLATE (ZLIB) compression
Cryptographic API --->
  <*> XTS support

(Specifying the "persistent store support" is just a roundabout way to force CONFIG_ZLIB_DEFLATE on, which you can't control directly for some reason. But compiling SPL requires it to be set.)

Be sure to also include any hardware drivers that your machine will depend on for boot (especially e.g. SATA, maybe Ethernet, maybe USB and/or mass storage if you'd like to use those in the initramfs recovery). These must be included in the kernel (y, not m), or specified as extra modules (see below). Running lspci -k will tell you the (loaded) modules that power your hardware, search for those entries in menuconfig and enable them. Install the kernel, but then we diverge at the Building an initramfs step. We'll customize ours, to handle both LUKS encryption and a ZFS root filesystem.

First install the ZFS tools (and kernel modules, via dependency).

(chroot) livecd ~ # emerge -va zfs

These are the packages that would be merged, in order:
...
(chroot) livecd ~ # rc-update add zfs-mount sysinit

When changing kernels in the future, be prepared to emerge @module-rebuild to make sure these out-of-tree modules are built for the kernel being used.

Boot Configuration

We've completed the Kernel section, continue with the handbook at the Configuring the system section. The fstab should look something like:

# <fs>                  <mountpoint>    <type>  <opts>          <dump/pass>
LABEL=boot_a            /boot           ext2    noauto,noatime  1 2
/dev/zvol/rpool/swap    none            swap    sw              0 0

But once we reach the Configuring the bootloader section, again we diverge. First, simply, we're going to install the grub2 bootloader to all three disks. In case of failure of any one disk, we'll be able to boot from the others:

(chroot) livecd ~ # cat > /etc/portage/package.use/grub
# No eye candy, thanks.
sys-boot/grub -fonts -themes

(chroot) livecd ~ # emerge -vat grub

These are the packages that would be merged, in reverse order:

Calculating dependencies... done!
[ebuild  N     ] sys-boot/grub-2.02:2/2.02::gentoo ...
...
(chroot) livecd ~ # for D in a b c; do grub-install /dev/sd$D; done
Installing for i386-pc platform.
Installation finished. No error reported.
Installing for i386-pc platform.
Installation finished. No error reported.
Installing for i386-pc platform.
Installation finished. No error reported.

Next we need an initramfs to hold enough user space tools, unencrypted, to mount our encrypted ZFS root file system. We're going to use bliss-initramfs for this. We also need to install a few of its dependencies before it will run.

(chroot) livecd ~ # emerge -vat dev-vcs/git cryptsetup gnupg

These are the packages that would be merged, in reverse order:

Calculating dependencies... done!
...
[ebuild  N     ] app-crypt/gnupg-2.2.4-r2::gentoo ...
...
[ebuild  N     ] sys-fs/cryptsetup-1.7.5::gentoo ...
...
[ebuild  N     ] dev-vcs/git-2.16.1::gentoo ...
...
(chroot) livecd ~ # git clone https://github.com/fearedbliss/bliss-initramfs.git
...
(chroot) livecd ~ # cd bliss-initramfs
(chroot) livecd bliss-initramfs # echo "Kernel name: $(ls /lib/modules|sort|tail -1)"
Kernel name: 4.9.76-gentoo-r1
(chroot) livecd bliss-initramfs # ./mkinitrd.py
------------------------------
Bliss Initramfs - v7.1.2
Jonathan Vasquez 
2-BSD
------------------------------

Which initramfs features do you want? (Separated by a comma):

[>] 1. ZFS
[>] 2. LVM
[>] 3. RAID
[>] 4. LUKS
[>] 5. Basic

Features [1]: 1,4

Do you want to use the current kernel: 4.9.72-gentoo [Y/n]: n

Please enter the kernel name: 4.9.76-gentoo-r1

[*] Checking preliminary binaries ...
[*] Creating temporary directory at /root/735506687 ...
[*] Checking required files ...
[+] Using LUKS
[+] Using ZFS
[*] Copying binaries ...
[*] Copying modules ...
[*] Generating modprobe information ...
[*] Copying library dependencies ...
[*] Creating symlinks ...
[*] Performing finishing steps ...
[*] Creating the initramfs ...
64978 blocks
[*] Please copy "initrd-4.9.76-gentoo-r1" to your /boot directory

This initramfs contains the ZFS modules, necessary to import the pool of course. They're tied to the kernel version, so if you ever upgrade (or downgrade), you'll need to rebuild the initramfs to include that kernel's matching modules. (After a emerge @module-rebuild, to make them available.)

Now we configure grub to boot the kernel and initramfs we just made. By hand! I don't want the complex / graphical things that the default tools do.

(chroot) livecd ~ # cp -a initrd-4.9.76-gentoo-r1 /boot
(chroot) livecd ~ # cd /boot
(chroot) livecd /boot # ln -s vmlinuz-$(ls /lib/modules|sort|tail -1) kernel
(chroot) livecd /boot # cp -aL kernel kernel.old
(chroot) livecd /boot # ln -s initrd-$(ls /lib/modules|sort|tail -1) initrd
(chroot) livecd /boot # cp -aL initrd initrd.old
(chroot) livecd /boot # cat > /boot/grub/grub.cfg
debug all

if [ -s $prefix/grubenv ]; then
  load_env
fi

menuentry 'Gentoo GNU/Linux' {
  insmod part_msdos
  insmod ext2
  set root='hd0,msdos1'

  echo    'Loading Linux ...'
  linux   /kernel root=rpool/root enc_drives=/dev/sda2,/dev/sdb2,/dev/sdc2 enc_targets=crypt_sda2,crypt_sdb2,crypt_sdc2 enc_type=pass triggers=luks,zfs
  echo    'Loading initial ramdisk ...'
  initrd  /initrd
}

menuentry 'Gentoo GNU/Linux (old)' {
  insmod part_msdos
  insmod ext2
  set root='hd0,msdos1'

  echo    'Loading Linux ...'
  linux   /kernel.old root=rpool/root enc_drives=/dev/sda2,/dev/sdb2,/dev/sdc2 enc_targets=crypt_sda2,crypt_sdb2,crypt_sdc2 enc_type=pass triggers=luks,zfs
  echo    'Loading initial ramdisk ...'
  initrd  /initrd.old
}
^D

Some of the details will depend on your specific set-up.

We're just about done! Continue from handbook section configuring the system (and skipping "Configuring the bootloader").

Appendix: Recovery

Should you ever need to reboot during installation, or later boot from the livecd for recovery, it would go something like this:

livecd ~ # for D in a b c; do cryptsetup luksOpen /dev/sd${D}2 crypt_sd${D}2; done
Enter passphrase for /dev/sda2:
Enter passphrase for /dev/sdb2:
Enter passphrase for /dev/sdc2:
livecd ~ # zpool import -fR /mnt/gentoo -d /dev/mapper rpool
livecd ~ # mount /dev/sda1 /mnt/gentoo/boot
livecd ~ # mount --rbind --make-rslave /dev /mnt/gentoo/dev
livecd ~ # mount --rbind --make-rslave /sys /mnt/gentoo/sys
livecd ~ # mount -t proc none /mnt/gentoo/proc
livecd ~ # mount -t devpts devpts /mnt/gentoo/dev/pts
livecd ~ # chroot /mnt/gentoo /bin/bash

Appendix: Links

I relied on a number of existing sources to figure out how to do this. Most of this information came from Gentoo Hardened ZFS rootfs with dm-crypt/luks 0.6.2 which is quite similar to this article. I added some more details from the Funtoo wiki's ZFS Install Guide and the Gentoo wiki's ZFS page.