arantius.com

Building a Gentoo Minimal LiveCD with ZFS support

2013-12-18 21:51 - Linux

Background

I've been using Gentoo for 20 years or more (I mention it in a post from 2004). I've been using encrypted drives for at least 15 of those years, and ZFS almost as long. I first wrote this post in 2013 (as the date above shows), before ZFS native encryption was a thing. (See earlier versions of this post at archive.org.)

So for the last decade plus, ZFS-on-LUKS is just how I've done things. I want to have a live ISO I can use to boot the ZFS-on-LUKS machines I already have, for recovery scenarios. (E.g. "Whoops, that new kernel I built doesn't boot the machine!") I have large existing data sets that I don't want to migrate. This is how I make the Gentoo live ISOs I use, for recovery of (and more rarely anymore, installation of) my Gentoo systems using LUKS encryption to host a root ZFS drive. Since early 2023 (per the wiki history) the Catalyst/Custom Media Image page has documented almost exactly this process. You might want to follow that. Or just use the gentoo LiveGUI ISO instead, which does (last I checked) come with ZFS built in. (But is over 3GiB.)

This post is oriented around making my live ISO. As slim as possible to get the job done, for me and my hardware. Being able to quickly attach my ISO as a virtual drive to a remote server, via slow IPMI, is of value to me. So I trim out everything I don't need. (And add in a few small helpful tools.)

How To

We need the releng (release engineering) tools, which are used by Catalyst. Catalyst is the tool that builds several low level Gentoo artifacts including the bootable ISO.

The catalyst process for building the ISO is split into two stages or phases. Let's prepare some prerequisites, and start stage one.

# emerge-webrsync
# echo 'dev-util/catalyst ~amd64' > /etc/portage/package.accept_keywords/catalyst
# emerge --autounmask-write dev-vcs/git catalyst pixz
# dispatch-conf
# emerge dev-vcs/git catalyst pixz
# cd /root
# git clone https://github.com/gentoo/releng.git

Due do the dependency tree involved, this step takes at least 15 minutes. (I wrote this from commit 6c41968d6bd47c99498f82c263feb9a37de9f431. If you're playing along, consider starting there.)

We need a stage file for a starting point, just like when installing Gentoo. Open Gentoo's downloads page. Use the amd64 stage3 URL found there in the wget step.

# mkdir -p /var/tmp/catalyst/builds/23.0-default
# wget https://.../stage3-amd64-....tar.xz -O /var/tmp/catalyst/builds/23.0-default/stage3-amd64-openrc.tar.xz

Before we start the long running parts, to make this as fast as is practical be sure to:

• Edit /etc/catalyst/catalyst.conf to specify jobs and load-average to emerge multiple packages at once.
• Optionally, edit /etc/catalyst/catalyst.conf to specify binhost, to use binary packages and skip compilation altogether. (I find that (due to USE differences?) this doesn't actually help much.)

If you're using a small VM like me, picking values here that don't cause OOM crashes might be tricky. In the past, I've found that found 4G ram plus 1G swap was not enough. This depends on the settings you used above for parallelism and which packages you include/exclude.

Additionally we create a portage snapshot, which is based on the emerge-webrsync from above.

# catalyst --snapshot stable
# TREEISH="$(ls -t /var/tmp/catalyst/snapshots/*.sqfs | head -n 1 | cut -b 36-75)"
# echo $TREEISH
e64612f87c9646c567552eebde09a0395e171a66
# DATESTAMP="$(date '+%Y%m%d')"
# echo $DATESTAMP
20241218

Catalyst Stage 1

# cat > common.sed <<EOF
> s#@REPO_DIR@#/root/releng#g
/version_stamp:/s#:.*#: $DATESTAMP.zfs#
s/@TIMESTAMP@/$DATESTAMP/
s/@TREEISH@/$TREEISH/
EOF
# cat > stage1.sed <<EOF
> # Match the downloaded stage name we used.
s#source_subpath: .*#source_subpath: 23.0-default/stage3-amd64-openrc#

# Skip use flags.
/	alsa/d
/	portaudio/d
/	socks5/d
/	xml/d
# Skip packages I don't need/plan to use.
/app-accessibility/d
/app-admin/d
/app-crypt/d
/app-editors/d  # Add vim instead below.
/amm-misc.tmux/d
/app-text.wgetpaste/d
/media-/d
/net-dialup/d
/net-fs/d
/net-irc/d
/net-misc.vconfig/d
/net-proxy/d
/net-wireless/d
/sys-block.partimage/d
/sys-firmware/d
/sys-fs.b/d
/sys-fs.dmraid/d
/sys-fs.jfs/d
/sys-fs.mac/d
/sys-fs.mdadm/d
/sys-fs.multipath/d
/sys-fs.ntfs/d
/sys-fs.reiser/d
/sys-fs.xfs/d
/sys-kernel.linux-firmware/d

# Add extra packages useful in recovery scenarios.
/livecd.packages/ {
  a	app-editors/vim
  a	net-analyzer/netcat
  a	net-analyzer/mtr
  a	net-analyzer/nmap
  a	net-analyzer/tcpdump
  a	net-dns/bind-tools
  a	sys-apps/hwinfo
  a	sys-apps/smartmontools
  a	sys-process/htop
}
EOF
# cp releng/releases/specs/amd64/installcd-stage1.spec .
# sed -f stage1.sed -i installcd-stage1.spec
# sed -f common.sed -i installcd-stage1.spec

We're going to need lvm2 (included by default in the stage 1 spec). It's where the device mapper tools come from, which we need for interacting with the LUKS wrapper disks. However by default the thin use flag is enabled which causes a tree of dependencies that includes (at time of writing) both llvm-core/llvm-18.1.8-r6 and llvm-core/llvm-19.1.4. Each take a long long time to compile, and tend to OOM my little VM. Since I don't want the features this provides, before running catalyst (so before it copies this file into the chroot) I edit releng/releases/portage/isos/package.use/lvm to specify -thin instead.

# catalyst -f installcd-stage1.spec 

This step creates a new chroot and then effectively sets up an entire Gentoo install there. This will emerge about 125 (all changes above) to nearly 200 packages (by default), it takes quite a while. (My time: just over 100 minutes.)

Catalyst stage 2

# cat > stage2.sed  <<EOF
> s/.iso$/.zfs.iso/
/source_subpath/s/livecd-stage1-amd64-latest/livecd-stage1-amd64-@TIMESTAMP@.zfs/
/livecd.volid/s/:.*/: gentoo-@TIMESTAMP@-zfs/

# Rename the iso.
/livecd.iso/s/install/gentoo/

# Add the ZFS package!  (Replacing the broadcom firmware package.)
/boot.kernel.gentoo.packages:/s/:.*/: sys-fs/zfs/

# Remove dracut options we've excluded in stage1.
/dracut_args/{
  s/-a mdraid //
  s/-o btrfs //
}
EOF
# cp releng/releases/specs/amd64/installcd-stage2-minimal.spec .
# sed -f stage2.sed -i installcd-stage2-minimal.spec
# sed -f common.sed -i installcd-stage2-minimal.spec
# catalyst -f installcd-stage2-minimal.spec

This extracts the image file built in stage one, builds the kernel, cleans up to reduce ISO size, then creates the final ISO image (my time: around 95 minutes), which will be located at /var/tmp/catalyst/builds/23.0-default/gentoo-amd64-minimal-$DATESTAMP.zfs.iso. You're done!

Finally, I'm making the fully built ISO image I created available for download. See this salient note on licensing. See comments below for the most up-to-date version.