FreeBSD Notes: 13-Release and bectl

We’ve had several release candidates (at least one more than expected) of FreeBSD-13, of which I’ve tested the later ones as VMs on a 12.2-p6 host and found them to be lightweight and sprightly. So, when the final release was announced last week, I didn’t have any particular qualms about updating said 12.2 system to release 13.0. Having said that, FreeBSD makes it particularly easy to add belt-and-braces insurance to your updates nowadays, using “bectl“.

If you haven’t come across bectl before, I would heartily recommend that you check it out and read the manual page before you start any future upgrades. The “be” part of the name stands for “Boot Environment” and this extremely useful tool helps us to manage our ZFS filesystems to allow virtually risk-free changes to our root filesystem without needing to get deeply into the nitty-gritty of snapshots, rollbacks and boot menus. Yes, it only works through the magic of ZFS, but it’s just another reason why you definitely should be using this filesystem everywhere (even on your laptop).

bectl” will enable us to quickly create a snapshot of our current root filesystem, add that snapshot to our “beastie” boot menu and, importantly, then allow us to create a completely new clone of the existing snapshot (which we can then upgrade or change in whatever way we want) and optionally make that updated filesystem available as the default boot for our machine. This means that we can install an upgrade (in my case, release 13.0) and boot it immediately, but still have the option of selecting our original root from the boot menu, if things don’t quite go to plan.

Before we start the actual upgrade, just a couple of words of warning…

  • As noted above, the system must be using root on ZFS.
  • It should be running the GENERIC kernel.
  • We should be starting from a recently updated version of the previous major release (ie:- 12.2-p6 if updating to 13.0).

How do we prepare our system in advance of the actual upgrade? It couldn’t be much easier. The “back-up” boot environment will be the root filesystem which we have now. We will create a new clone of this existing environment to work on during the 13.0 install (leaving the original untouched).

First, we check our current boot environment with:-

root#   bectl list -D

BE              Active Mountpoint Space Created
default         NR     /          10.2G 2020-02-16 18:02

The bectl list command shows the currently available boot environments and the “-D” argument shows the space used. The “Active” column shows an “N” for the boot environment which is in use right Now and an “R” for the boot environment which will be activated on the next Reboot. In our case, we only have one boot environment, so it is both active and selected to be used on the next reboot.

To create the cloned (work) boot environment, we do:-

root#   bectl create 13.0-RELEASE

root#   bectl activate 13.0-RELEASE

The “create” is obvious. The “activate” command tells the system that we want to use the new, “13.0-RELEASE” boot environment at the next reboot. When we use the “list” command again now, we should see something similar to this:-

root#   bectl list -D
BE              Active Mountpoint Space Created
13.0-RELEASE    R      -          369K  2021-04-22 08:32
default         N      /          10.2G 2020-02-16 18:02

…indicating that “default” is currently in use, but that “13.0-RELEASE” will be used on the next reboot. If everything looks okay (and there were no error messages from the bectl commands), we can go ahead and reboot our system. It should boot as normal and look exactly the same as before (we haven’t actually changed anything on the filesystem yet, so we’re still running 12.2-p6). However, if we were to do another bectl list -D we’d see that the “Active” column would now show “NR” for 13.0-RELEASE and just “-” for “default”.

Do not proceed with the upgrade steps below unless you see the “NR” active status for the “13.0-RELEASE” boot environment.

Having checked that we’re using the cloned root filesystem, we can now go ahead and use that other wonderfully useful utility, freebsd-update, to start the actual upgrade process:-

root#   freebsd-update -r 13.0-RELEASE

Note that “13.0-RELEASE” in this command is specifying which release we want freebsd-update to upgrade to, not where we want to install it. We simply follow the instructions which freebsd-update gives as it goes along (we will need to reboot a couple of times to complete the full upgrade process).

If everything goes as planned, after the final reboot we should have a working FreeBSD 13.0 system.

My upgrade process went smoothly; all system utilities came back up and the VMs ran with no problems. Everything looked fine …until the first, automated, weekly reboot rolled around. The machine didn’t come back up. The console was complaining that there was a fatal ZFS error with the main user-data pool (zstore00 – not the root filesystem). I had to cycle power on the system to have it reboot and, when it came back up it did indeed show a checksum error on that pool. I cleared the error and initiated a scrub and once everything was stable and had been running for a couple of days, did a manual reboot. It failed again, in the same way.

The actual error shown was;-

uhub2: detached
uhub1: detached
umass0: detached
uhub3:detached
ubt0:detached
Solaris: WARNING: Pool zstore00 has encountered an uncorrectable I/O failure and has been suspended.

At that point the actual shutdown process hung solid and, as before, I needed to power cycle the system to have it come back up. I also tried updating the /boot/loader.conf file (as suggested in Graham Perrin’s very useful guide) and rebooted a couple of times manually, but to no avail.

Note that I had not run zfs upgrade on the pools, since this would have made it very difficult to roll back.

So, at this point I needed to get the system back up and running and decided to do that roll-back to 12.2. Here’s how you can do that…

Power cycle the machine (if you’re stuck with a hung system, as above) and once the beastie menu appears, wait a second for the additional options to appear at the bottom (it takes the boot process a moment or two to collect the boot environment listings) and then select the “Select Boot [E]nvironment…” option. You’ll get another, short menu where you can use #2 to cycle through the boot environment names until you see “default” displayed. At this point you can simply hit Enter to have the system select and boot into this saved environment.

Your system will now boot into the original 12.2-p6 root filesystem and work as before. The only minor issue that you have is that booting from the beastie menu only sets the boot environment for this one-time boot, so you must use bectl activate default to have your 12.2-p6 environment set as the permanent boot environment (if you do a bectl list -D at this point, “default” should now have “NR” showing in the Active column).

Looking a little closer at the listing you’ve just done, you’ll also notice that because we’ve actually got a complete 13.0 install on the alternate “13.0-RELEASE”, it now shows a much greater space-used than before (in my case around 12GB).

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s