php site, templating and partials, faster index generation

1 files changed, 306 insertions, 0 deletions

diff --git a/posts/2021-01-15-root-on-zfs-a-zpool-of-mirror-vdevs-the-easy-way.php b/posts/2021-01-15-root-on-zfs-a-zpool-of-mirror-vdevs-the-easy-way.php
new file mode 100644
index 0000000..d67f582
--- /dev/null
+++ b/posts/2021-01-15-root-on-zfs-a-zpool-of-mirror-vdevs-the-easy-way.php
@@ -0,0 +1,306 @@
+<?php
+$title = "Root on ZFS: A ZPool of Mirror VDEVs";
+if (isset($early) && $early) {
+	return;
+}
+include($_SERVER['DOCUMENT_ROOT'] . '/includes/head.php');
+?>
+
+<p class="description">
+	I wanted/needed to make a root on ZFS pool out of multiple mirror VDEVs,
+	and since I'm not a ZFS expert, I took a little shortcut.
+</p>
+
+<p>
+	I recently got a new-to-me server (yay!) and I wanted to do a
+	root-on-ZFS setup on it. I've really enjoyed using ZFS for my data
+	storage pools for a long time. I've also enjoyed the extra functionality
+	that comes with having a bootable system installed on ZFS on my laptop
+	and decided with this upgrade it's time to do the same on my server.
+	Historically I've used RAIDZ for my storage pools. RAIDZ functions
+	almost like a RAID10 but at the ZFS level. It gives you parity so that a
+	certain number of disks can die from your pool and you won't lose any
+	data. It does have a few tradeoffs however*, and for personal
+	preferences I've decided that for the future I would like to have a
+	single ZPool over top of multiple mirror VDEVs. In other words, my main
+	root+storage pool will be made up of two-disk mirrors and can be
+	expanded to include any number of new mirrors I can fit into the
+	machine.
+</p>
+
+<p>
+	This did present some complications. First of all,
+	<code>bsdinstall</code> won't set this up for you automatically (and
+	sure enough,
+	<a href="https://www.freebsd.org/doc/handbook/bsdinstall-partitioning.html">in the handbook</a>
+	it mentions the guided root on ZFS tool will only create a single,
+	top-level VDEV unless it's a stripe). It will happily let you use RAIDZ
+	for your ZROOT but not the more custom approach I'm taking. I did
+	however use
+	<code>bsdinstall</code> as a shortcut so I wouldn't have to do all of
+	the partitioning and pool setup manually, and that's what I'm going to
+	document below. Because I'm totally going to forget how this works the
+	next time I have to do it.
+</p>
+
+<p>
+	In my scenario I have an eight-slot, hot-swappable PERC H310 controller
+	that's configured for AHCI passthrough. In other words, all FreeBSD sees
+	is as many disks as I have plugged into the backplane. I'm going to fill
+	it with 6x2TB hard disks which, as I said before, I want to act as three
+	mirrors (two disks each) in a single, bootable, growable ZPool. For
+	starters, I shoved the FreeBSD installer on a flash drive and booted
+	from it. I followed all of the regular steps (setting hostname, getting
+	online, etc.) until I got to the guided root on ZFS disk partitioning
+	setup.
+</p>
+
+<p>
+	Now here's where I'm going to take the first step on my shortcut. Since
+	there is no option to create the pool of arbitrary mirrors I'm just
+	going to create a pool from a single mirror VDEV of two disks. Later I
+	will expand the pool to include the other two mirrors I had intended
+	for. My selections were as follows:
+</p>
+
+<ul>
+	<li>Pool Type/Disks: mirror mfisyspd0 mfisyspd1</li>
+	<li>Pool Name: zroot</li>
+	<li>Partition Scheme: GPT (EFI)</li>
+	<li>Swap Size: 4g</li>
+</ul>
+
+<p>
+	Everything else was left as a default. Then I followed the installer to
+	completion. At the end, when it asked if I wanted to drop into a shell
+	to do more to the installation, I did.
+</p>
+
+<p>
+	The installer created the following disk layout for the two disks that I
+	selected.
+</p>
+
+<pre>
+<code>
+atc@macon:~ % gpart show
+=>        40  3907029088  mfisyspd0  GPT  (1.8T)
+          40      409600          1  efi  (200M)
+      409640        2008             - free -  (1.0M)
+      411648     8388608          2  freebsd-swap  (4.0G)
+     8800256  3898228736          3  freebsd-zfs  (1.8T)
+  3907028992         136             - free -  (68K)
+
+=>        40  3907029088  mfisyspd1  GPT  (1.8T)
+          40      409600          1  efi  (200M)
+      409640        2008             - free -  (1.0M)
+      411648     8388608          2  freebsd-swap  (4.0G)
+     8800256  3898228736          3  freebsd-zfs  (1.8T)
+  3907028992         136             - free -  (68K)
+</code>
+</pre>
+
+<p>
+	The installer also created the following ZPool from my single mirror
+	VDEV.
+</p>
+
+<pre>
+<code>
+atc@macon:~ % zpool status
+  pool: zroot
+ state: ONLINE
+  scan: none requested
+config:
+
+	NAME             STATE     READ WRITE CKSUM
+	zroot            ONLINE       0     0     0
+	  mirror-0       ONLINE       0     0     0
+	    mfisyspd0p3  ONLINE       0     0     0
+	    mfisyspd1p3  ONLINE       0     0     0
+
+errors: No known data errors
+</code>
+</pre>
+
+<p>
+	There are a couple of things to take note of here. First of all,
+	<em>both</em> disks in the bootable ZPool have an EFI boot partition.
+	That means they're both a part of (or capable of?) booting the pool.
+	Second, they both have some swap space. Finally, they both have a third
+	partition which is dedicated to ZFS data, and that partition is what got
+	added to my VDEV.
+</p>
+
+<p>
+	So where do I go from here? I was tempted to just
+	<code>zpool add mirror ... ...</code> and just add my other disks to the
+	pool (actually, I <em>did</em> do this but it rendered the volume
+	unbootable for a very important reason), but then I wouldn't have those
+	all-important boot partitions (using whole-disk mirror VDEVS). Instead,
+	I need to manually go back and re-partition four disks exactly like the
+	first two. Or, since all I want is two more of what's already been done,
+	I can just clone the partitions using <code>gpart backup</code> and
+	<code>restore</code>! Easy! Here's what I did for all four remaining
+	disks:
+</p>
+
+<pre>
+<code>
+root@macon:~ # gpart backup mfisyspd0 | gpart restore -F mfisyspd2`
+</code>
+</pre>
+
+<p>
+	Full disclosure, I didn't even think of this as a possibility
+	<a href="ihttps://unix.stackexchange.com/questions/472147/replacing-disk-when-using-freebsd-zfs-zroot-zfs-on-partition#472175">until I read this Stack Exchange post</a>. This gave me a disk layout like this:
+</p>
+
+<pre>
+<code>
+atc@macon:~ % gpart show
+=>        40  3907029088  mfisyspd0  GPT  (1.8T)
+          40      409600          1  efi  (200M)
+      409640        2008             - free -  (1.0M)
+      411648     8388608          2  freebsd-swap  (4.0G)
+     8800256  3898228736          3  freebsd-zfs  (1.8T)
+  3907028992         136             - free -  (68K)
+
+=>        40  3907029088  mfisyspd1  GPT  (1.8T)
+          40      409600          1  efi  (200M)
+      409640        2008             - free -  (1.0M)
+      411648     8388608          2  freebsd-swap  (4.0G)
+     8800256  3898228736          3  freebsd-zfs  (1.8T)
+  3907028992         136             - free -  (68K)
+
+=>        40  3907029088  mfisyspd2  GPT  (1.8T)
+          40      409600          1  efi  (200M)
+      409640        2008             - free -  (1.0M)
+      411648     8388608          2  freebsd-swap  (4.0G)
+     8800256  3898228736          3  freebsd-zfs  (1.8T)
+  3907028992         136             - free -  (68K)
+
+=>        40  3907029088  mfisyspd3  GPT  (1.8T)
+          40      409600          1  efi  (200M)
+      409640        2008             - free -  (1.0M)
+      411648     8388608          2  freebsd-swap  (4.0G)
+     8800256  3898228736          3  freebsd-zfs  (1.8T)
+  3907028992         136             - free -  (68K)
+
+=>        40  3907029088  mfisyspd4  GPT  (1.8T)
+          40      409600          1  efi  (200M)
+      409640        2008             - free -  (1.0M)
+      411648     8388608          2  freebsd-swap  (4.0G)
+     8800256  3898228736          3  freebsd-zfs  (1.8T)
+  3907028992         136             - free -  (68K)
+
+=>        40  3907029088  mfisyspd5  GPT  (1.8T)
+          40      409600          1  efi  (200M)
+      409640        2008             - free -  (1.0M)
+      411648     8388608          2  freebsd-swap  (4.0G)
+     8800256  3898228736          3  freebsd-zfs  (1.8T)
+  3907028992         136             - free -  (68K)
+</code>
+</pre>
+
+<p>
+	And to be fair, this makes a lot of logical sense. You don't want a
+	six-disk pool to only be bootable by two of the disks or you're
+	defeating some of the purposes of redundancy. So now I can extend my
+	ZPool to include those last four disks.
+</p>
+
+<p>
+	This next step may or may not be a requirement. I wanted to overwrite
+	where I assumed any old ZFS/ZPool metadata might be on my four new
+	disks. This could just be for nothing and I admit that, but I've run
+	into trouble in the past where a ZPool wasn't properly
+	exported/destroyed before the drives were removed for another purpose
+	and when you use those drives in future
+	<code>zpool import</code>s, you can see both the new and the old, failed
+	pools. And, in the previous step I cloned an old ZFS partition many
+	times! So I did a small <code>dd</code> on the remaining disks to help
+	me sleep at night:
+</p>
+
+<pre>
+<code>
+root@macon:~ # dd if=/dev/zero of=/dev/mfisyspd2 bs=1M count=100
+</code>
+</pre>
+
+<p>
+	One final, precautionary step is to write the EFI boot loader to the new
+	disks. In
+	<a href="https://www.freebsd.org/doc/handbook/zfs-zpool.html">zpool admin handbook</a>
+	it mentions you should do this any time you <em>replace</em> a zroot
+	device, so I'll do it just for safe measure on all four additional
+	disks:
+</p>
+
+<pre>
+<code>
+root@macon:~ # gpart bootcode -p /boot/boot1.efifat -i 1 mfisyspd2
+</code>
+</pre>
+
+<p>
+	Don't forget that the command is different for UEFI and a traditional
+	BIOS. And finally, I can add my new VDEVs:
+</p>
+
+<pre>
+<code>
+root@macon:~ # zpool zroot add mirror mfisyspd2p3 mfisyspd3p3
+root@macon:~ # zpool zroot add mirror mfisyspd4p3 mfisyspd5p3
+</code>
+</pre>
+
+<p>And now my pool looks like this:</p>
+
+<pre>
+<code>
+atc@macon:~ % zpool status
+  pool: zroot
+ state: ONLINE
+  scan: none requested
+config:
+
+	NAME             STATE     READ WRITE CKSUM
+	zroot            ONLINE       0     0     0
+	  mirror-0       ONLINE       0     0     0
+	    mfisyspd0p3  ONLINE       0     0     0
+	    mfisyspd1p3  ONLINE       0     0     0
+	  mirror-1       ONLINE       0     0     0
+	    mfisyspd2p3  ONLINE       0     0     0
+	    mfisyspd3p3  ONLINE       0     0     0
+	  mirror-2       ONLINE       0     0     0
+	    mfisyspd4p3  ONLINE       0     0     0
+	    mfisyspd5p3  ONLINE       0     0     0
+
+errors: No known data errors
+</code>
+</pre>
+
+<p>
+	Boom. A growable, bootable zroot ZPool. Is it easier than just
+	configuring the partitions and root on ZFS by hand? Probably not for a
+	BSD veteran. But since I'm a BSD layman, this is something I can live
+	with pretty easily. At least until this becomes an option in
+	<code>bsdintall</code> maybe? At least now I can add as many more
+	mirrors as I can fit into my system. And it's just as easy to replace
+	them. This is better for me than my previous RAIDZ, where I would have
+	to destroy and re-create the pool in order to add more disks to the
+	VDEV. Now I just create another little mirror and grow the pool and all
+	of my filesystems just see more storage. And of course, having ZFS for
+	all of my data makes it super easy to create filesystems on the fly,
+	compress or quota them, and take snapshots (including the live ZROOT!)
+	and send those snapshots over the network. Pretty awesome.
+</p>
+
+<p>
+	* I'm not going to explain why here, but
+	<a href="http://www.openoid.net/zfs-you-should-use-mirror-vdevs-not-raidz/">this is a pretty well thought out article</a>
+	that should give you an idea about the pros and cons of RAIDZ versus
+	mirror VDEVs so you can draw your own conclusions.
+</p>