Grub bites man. Man stamps on grub. ‡

‡ — No invertebrates were harmed during the production of this article.

Almost exactly two weeks ago I got a (normal) pop-up message on my screen, letting me know that there were updates available from Ubuntu for my 21.10 version of Mate and that, by the way, 22.04 was now available if I wanted to upgrade.  I politely refused the upgrade offer, but accepted the updates.  Once the updates were completed I was prompted to reboot (and I noted that the initramfs had been updated).  This is all normal stuff for the millions of us running the various versions of Ubuntu, so after finishing up what I was working on (and washing my hands) I hit the reboot button.

It didn’t.

It hasn’t for the last two weeks (not without a recovery image or an external disk plugged in, anyway).

At first I wasn’t too concerned.  This (as the ‘mericans say) isn’t my first rodeo and, if everything else failed, I’m using ZFS, so I just had to roll back to the last good snapshot, right?

Wrong!

As I wasted more and more time futzing around with things, I just got more and more confused. The EFI boot was working and grub was being run, but it insisted that it couldn’t find a workable filesystem.  Booting into a live-CD image said there were two perfectly good ZFS pools on the disk (actually a 512GB NVME SSD) and a “zfs scrub” passed with flying colours on both of them (bpool and rpool).  Running grub-probe always came back with a message along the lines of “Nope, ain’t nuthin there that I recognize, pinhead!”.

By this time I had convinced myself that someone had broken into Ubuntu’s update servers and slipped a Trojan into my disk headers, or maybe someone was slipping magic mushrooms into my breakfast tea, or aliens had abducted my real ZFS pools and replaced them with zombies.  Anyway, I needed to get some work done, so I found an unused external USB disk, fired up a live-CD again and, because this is ZFS, did a ZFS send-receive between the original disk and the external.  When it came time to run the grub install (on the external disk) it once again insisted that there was nothing at all there which it recognised as any sort of valid filesystem.  Duh!

Okay, nuts to this.  I put a 22.04 live-CD image in, booted it up (“Oh look, you have two perfectly good ZFS pools on this external disk!  Are you really sure you want me to scribble all over them?”) and installed a brand spanking new version of 22.04 onto the external, followed by a ZFS send-receive of just the USERDATA filesystem from the original NVME drive (I’d like to say that I rebooted and everything sprang into life on the external disk, but of course it didn’t.  You can’t easily remove an NVME “disk” from a laptop and of course EFI and grub just kept right on trying to access it).

Eventually, I did manage to broker a truce between the laptop BIOS, EFI and grub, to the point that I could reboot the system successfully, as long as I was actually there to hit the F9 key and manually select the EFI entry for the external disk  …otherwise it would stubbornly continue with a blinky dance of reset, spin disk up, flash something on the screen for 10ms, spin the disk down and repeat.  I never did manage to read what that 10ms micro message was (something from those dang aliens again).

But glory hallelujah!!  It was fantastic!  Not only did I have Mate back again, but the stupidly useless ELAN touchpad on the HP S15 laptop actually worked properly for the first time ever!!  Banzai!!  Well done 22.04!  I’m sorry that I refused your offer of an upgrade at the start of all of this.

Well, long story longer, that’s the way that things have stayed for the past two weeks.  Not that there hasn’t been great gnashing of teeth (not recommended at my advanced age) and thrashing of Gewgull searches (and even Ubuntu’s bug database) for any hints of what could be causing my problems. Searches including both “grub” and “aliens” turned up some interesting stuff, but it seems that I’m the only semi-sentient being-man-thingy in the universe who is having problems with those things in connection with computers.

Eventually, the third neuron in the second bank started firing occasionally and I began to have flashbacks of the horrible problems I had a while back, trying to do ZFS back-ups between my Linux laptop and the mirrored disks on my FreeBSD servers.  Although the FreeBSD servers could send data to a receiving  Linux system, the other way around just would not work at all.  It turned out that there were incompatibilities between the ZFS properties used between the two systems, even though the ZFS versions were meant to be the same (upgrading the FreeBSD servers to version 13.1 seemed to cure that issue).  Now it seemed probable that there was some similar incompatibility between grub and ZFS  …and, finally, it clicked. 

One of those property updates which hit FreeBSD a while back was the introduction of “zstd” compression (Ubuntu introduced it a short while later).  I had tried changing the compression to “zstd” on FreeBSD and noticed a worthwhile increase in compression-ratio with no apparent difference in speed, so I’d made the changes on my laptop, too.  The back-ups kept on working and I was saving more disk space …what’s not to like?

Double-duh!  Pinhead at work (stand clear!).

Okay, so I have (I suspect, anyway) a philosophical disconnect between grub and ZFS about what constitutes a filesystem.  In my book, ZFS wins (whatever grub says, I’m an ex-Sun guy and old loyalties die hard).  What to do now?  Well,back on Gewgull I replace “aliens” with “alternative” and get a few hits on something called “ZFSBootMenu”.  I go to their GitHub repository and start reading.  Nope, I must have left “aliens” in there somewhere, ‘coz I can’t understand this stuff.  It starts off more or less okay (I understand their reference to FreeBSD’s boot environments …a very handy feature,  sign me up), but then they start wandering off into the weeds with things like “fzf” and “dracut” (which sounds like a bad day at the vasectomy clinic) and I want to stop reading so badly that my teeth hurt (told you not to gnash).  I briefly leave the README page and head over to the pre-built releases instead. 

Ah, this is interesting.  “ZFSBootMenu v2.0.0  …New features – Dracut is now optional;”.  Phew!  Well that’s a relief.  Unfortunately, there are four different files (and additional source tarballs), but no information on what the differences are.  A couple have “release” in their names and the others have “recovery”, but no actual information in the v2.0.0 section about what the differences are or how to use them.  Two are compressed tar files (okay, I know what those are, anyway) but the two others have a “.EFI” suffix, which I haven’t come across before, but it’s not too much of a stretch to suspect that they should be used in the Extensible Firmware Interface (that’s the BIOS to you, sonny!).  But how?

Back to the README:-

“Each release includes pre-generated images (both a monolithic UEFI applications as well as separate kernel and initramfs components suitable for both UEFI and BIOS systems) based on Void Linux. Building a custom image is known to work in the following configurations…”

Well there you go then.  C’mon!  Chop-chop!  Get on with it!

I won’t bore you with the rest of it, but despite the rambling (or partly missing) documentation, it turns out that ZFSBootMenu is really quite a nifty bootloader (grub alternative) specifically for ZFS filesystems.  It does have a couple of quirks, but after a few false starts, I did manage to bring my 21.10 NVME SSD back from zombieville and ended up with a laptop which I could once again disconnect from all external cables and disks and still have it boot reliably.  To do ZFSBootMenu full justice I need to put together another post with a step-by-step guide of how to recover a stuffed ZFS machine, but just in case you’re stuck up this same creek without a paddle, here are just a couple of essential hints to getting ZFSBootMenu to work:-

  • Use the .EFI  file (I used the “release”, but “recovery” should also work okay).
  • Remember– When you’re booting using ZFSBootMenu, it assumes that the kernel and initramfs are on the root partition, so you’re going to be booting from “rpool”, not “bpool” (and so your kernel and initramfs need to be present on “rpool”, which is not standard in Ubuntu).
  • Disconnect any other disks which you may have added to the system in previous attempts to fix this boot problem (ie:- external USB disks with copies of your original bpool/rpool filesystems).  They will just confuse EFI, grub and you.
  • Set “canmount=noauto” on all filesystems which have “mountpoint=/” set (this one is really important).
  • Make sure that your ZFS pools can be imported without having to use “import -f” (use a live-CD or ZFSBootMenu itself to import the pools manually, fix any problems and then export them again, before you reboot).
  • Lastly, for those others of you out there with non-US keyboard layouts, the “Alt” key for command mode selection within ZFSBootMenu will probably not be the “Alt” key.  For my Japanese keyboard, the magic key turned out to be the one key which I had never before pressed on my keyboard …the (shudder!)  “Windows” key.

And the bottom line… Just don’t change your bpool to zstd compression.  Okay?!?    †  ∇


†  —  From a cursory check of the Grub2 code in the upstream Debian repository, it looks as though the zstd compression library was added in November of 2018.  Unfortunately it seems to have been implemented only for BTRFS, not for ZFS.

  —  “cursory”  Adjective.  Causes the reader to leap out of their chair and shout “G’dammit!!”  (this answer is sure to get you an “A” in GCSE English).

∇  –  Grub Compatibility  —  As usual, the folks at OpenZFS/Debian/Ubuntu are way ahead of me.  If you need to create a new pool which will work with Grub2, there is already an existing compatibility file (they’re stored in /usr/share/zfs/compatibility.d).  You can use it like this:-  

   zpool create -o compatibility=grub2  <POOL NAME>

If anyone is desperate enough to have read all the way down here to the bottom, I’ll just note that this problem hasn’t been submitted as an actual bug, because the auto-submission procedure won’t let you submit a bug against a program which isn’t installed on your system and, of course, I’d removed the grub package and used ZFSBootMenu to actually get my laptop working again. However, the secondary suggested method of opening the issue as a question in LaunchPad is available here if you’re interested (and I’d like to thank Manfred Hampl for trying to help me out there).

Update — 27th July 2022 – I have submitted bug request #1982897 with a condensed description of this issue and a request for an upstream addition of “zstd” support for ZFS in the Grub2 code.

Update #2 — 27th July 2022 – Submitted bug #62821 as a feature request for the addition of “zstd” support to the Grub2 ZFS code.

Update #3 — 13th August 2022 – Updated the Grub2 bug report with a little more detail on the severity of the problem, as there hasn’t been any acknowledgement of the original submission as yet.

New Kid on the BLEck

For anyone who hasn’t already heard yet, yesterday saw the release of yet another contender for the ESP’s cheap, small and connected WiFi device crown. This one comes from the Raspberry Pi stable and is a revamp of the RP2040 “Pico” which, in this new Pico-W variant, now sports a shiny, new Infineon  CYW43439 WiFi module to provide 802.11n connectivity.

My tongue-firmly-in-cheek title typo refers to the fact that the module also includes the hardware for both Classic Bluetooth and Bluetooth Low Energy …but unfortunately neither are actually enabled in this initial version of the Pico-W.

The great news about this new board is that it is cheap ($6 US) and actually available (unless, like me, you happen to live in Japan, in which case you can settle in for another 8-month wait while the Japanese government decide whether they’re going to “hanko” the approval or not — fun fact:- the Pi Zero 2W was released on October 28th, 2021, but only became officially available in Japan a couple of weeks ago).

The Pico-W certainly looks interesting (a lot more so than the non-WiFi original) and might be a real contender, with Eben Upton hinting in his announcement post that the pipeline is in place to produce millions of these boards (he was referring to the Pico family, not specifically the Pico-W). With the supply lines in place to get these into customer’s hands at a reasonable price across a large part of the globe (Element14, RS, Digikey and Mouser), they might well become the new, go-to board for connected projects. An SDK is already available for C/C++ and a network capable Micro-Python image also already exists.

If you’re looking to get your hands on one (and I do mean “one”), I can vouch for the fact The PiHut in the UK still had stock a few minutes ago and will ship internationally without breaking the bank.

Ah …seems like I might have snagged their last one. Sorry!

Visual Studio Code and PlatformIO

Regular readers will know that I’m a big proponent of PlatformIO as a programming platform for the ESP family; it’s just so much better than the Arduino IDE for us command-line interface die-hards. As I’ve noted before, I’m not a religious fanatic on the CLI vs GUI thing; it’s just that GUIs (with a couple of exceptions) don’t really click for me. Most of the time I just don’t “get it” and what’s on offer usually seems to limit the functionality without really adding much, if anything, in ease of use. Recently though, I’ve been searching for information on getting started with the ESP32’s built-in version of FreeRTOS (specifically, looking for more information on real-world use, rather than just a couple of sentences about the syntax of the xTaskCreate() call) and I found a couple of very useful videos by Xavier on his “Simply Explained” channel which, as the name suggests, do a very good job of explaining by example, how to use FreeRTOS. The thing which (as a doddering old git) really caught my attention was his use of Visual Studio Code. I was captivated by the pop-up prompt with info on the parameters (of which there are plenty) to feed to xTaskCreate. While I imagine that functionality can get pretty tedious when it pops up for each and every printf(), it does look like a definite winner when you’re learning something new (and don’t have much in the way of short-term memory any more).

Now, what was I saying again? Oh yes, here’s the link to one of Xavier’s videos. He’s got a ton of content on his channel on many diverse (but usually tech-related) topics. This one is not the first in the series, so if the abrupt lead-in is a bit too much you can go to his Intro on FreeRTOS instead. I highly recommend watching Xavier’s videos if you’re trying to get started with FreeRTOS.

Of course, I immediately went off and downloaded Visual Studio Code and fired it up …only to just as immediately get totally lost when I couldn’t get it to do anything that I wanted. I did watch a few more videos from a nice lady at Microsoft, but ended up suffering from information overload and Windows-itis. I went for a nice cup of tea instead. And then went back to PlatformIO in an xterm, just so that I could get something done.

More recently, one of my regular morning reads, Hackaday, had an article on making laminated artwork for front panels with an accompanying video (worth a look in itself) by Richard Langner. I was impressed by Richard’s succinct, no-nonsense style and dipped into his video listings to see what other goodies were there. Lo and behold, right at the top of the list were a couple of videos on how to get started with Visual Studio Code with PlatformIO. The first was only a minute and a half long and the second just over six minutes. Both are filled with essential information on doing just what I wanted to do (have the editor prompt me with useful info, but still be able to use PlatformIO as my programming environment). As they’re so short, I’m including both as embedded links below. Take a look. Even if you don’t like them, you won’t have wasted much time.

Many thanks to both Richard and Xavier (oh, and Al Williams and the team at Hackaday, as well as Ivan and everyone at PlatformIO) for making my life not just easier, but a lot more interesting, too.

Pre-Black-Friday Bargains, Japan Style

For anyone else in Japan, here’s a bargain you can pick up from Amazon Japan’s “Featured Deals” in Laptops this morning (24th Nov 2021). Take a look at the 16″ LG at the right (enlarged version below).

And just in case you’re not familiar with the Japanese Yen conversion rate; this morning $1 US is worth 115 Yen and one U.K. pound is worth 154 Yen. That’s a fabulous discount of a fraction of a penny.

Yup, someone at Amazon Japan has a sense of humour.

The Ubuntu ZFS boot pool problem [Part I]

Sorry, as you can see from the title, this is another “not-ESP8266” article, so you can skip this if you’re only interested in our magical little wireless modules. However, if you’re running a system (laptop, server or workstation) loaded with Ubuntu and have the root/boot partitions mounted on a ZFS filesystem, you should at least stick around long enough to read through the problem description, below.

If you landed here from a web search for “can’t upgrade Ubuntu – not enough space on bpool” (or something similar), then you’re in the right place. You can probably skip the introduction to the problem (you already know what you’re facing) and go directly to the solution.

Note that in these articles I am assuming that you are familiar with Linux and confident in your own abilities to administer your own system. Because we are dealing with filesystems, I am also going to assume that you have made and verified back-ups of your whole disk(s) before you follow any of the tips below. I am not responsible for your data. You are. The advice below is given in good faith after having been tested on several of my own machines (of different types), but I cannot be held responsible for any loss of data.

The bpool Problem

You see the familiar Software Updater pop-up, announcing that there are package updates available and prompting you to install them. You scan the list of updates and everything looks good, so you hit the “Install Now” button, but instead of a progress meter, you see a pop-up error window with the message:-

The upgrade needs a total of 127 M free space on disk '/boot'. Please free at least an additional 107 M of disk space on '/boot'.

The message continues with a couple of suggestions of how to increase available space in “/boot”, but basically at this point you are unable to install updates.

You’ve just run into the “bpool problem”.


Background

For the last few releases, Ubuntu has provided an easy method for installing root on a ZFS filesystem, giving you access to snapshots and the instantaneous rollback to previous versions which it enables (as well as the possibility of mirroring) without having to go through the long and tedious process of adding ZFS support after the actual install. It was a major move forwards and the “one-click” install process made it simple for anyone to benefit from ZFS technology (not just snapshots, rollbacks and mirrors, but raidz and ZFS send/receive back-ups, too). With all of those benefits, why wouldn’t you just use it by default?

Well, as a long time ZFS user, I can tell you that it does come with a few drawbacks. It is, for instance, somewhat more difficult to judge the available space left on any physical device than it was in the pre-ZFS world (good ole’ “df -h” is very good at giving you an instant and pretty accurate impression of the current capacity of your disks). ZFS tends to muddy that simplistic view a little, until you get used to depending more on the output from “zpool list -v” and “zfs list -o space” instead of “df”.

Fragmentation is another issue. Die-hard Unix users will remember that “fragmentation” was always a problem for people who used that other operating system, not for Unixes (well, unless you started to run out of inodes, anyway). However, with ZFS you do need to keep an eye on that “FRAG” column in the output of “zpool list -v”; once it starts climbing to around the 50~60% level it’s definitely time to start planning for an upgrade and, once it’s past 70% it’s time to take some urgent action (usually, but not always, time to add more physical disk space — we’ll touch on de-fragmentation as a side-effect in the “Solutions” presented in part-II). Once it hits 80%, you’re in big trouble as, even with available free space, the system will be struggling to find usable space.

These issues are a pain to come to terms with after all of these years of doing the odd “df” just to check that everything was okay, but they are still heavily outweighed by the advantages (did I mention copy-on-write or checksummed data yet?).

So, our simple ZFS installation method from Ubuntu is a real game changer and gives everyone the chance to ride on the ZFS bandwagon with minimal effort. Unfortunately, along with that simplicity and ease of installation comes another problem — sizing your filesystems. The installer is a one-size-fits-all deal; it looks at your disk size and makes a couple of fairly simplistic calculations to create a very simple partition table and then later adds a spread of ZFS datasets on top of those physical partitions. The “bpool” (for “boot-pool”) is a single ZFS filesystem which occupies a dedicated partition just for the kernel/initramfs and EFI/grub data required to boot the system. The “rpool” (for “root-pool”) is the ZFS filesystem which contains datasets for /, /var and /usr (as well as sub-directories) and the user home directories. It’s all very well laid out, so that a rollback to (for instance) a previous kernel version need not affect the user’s data. However, the installation script for ZFS currently has one major issue; it limits the size of the bpool physical partition to 2GB (it can be smaller, but not bigger), no matter what the size of your physical disk

The major drawback that people are finding to this installation mode comes from one of the strengths of ZFS — the snapshot functionality. Put simply, if you make a snapshot of say, your whole bpool, ZFS will never delete anything from that point in time. If you subsequently delete a kernel which was present when you took the snapshot, ZFS will keep the file itself, hidden away in the snapshot and only remove the named link to the file from the normal /boot directory. If, at some later time, you decide that you no longer need that snapshot (because it is outdated and you know you’re never going to revert to that old kernel again) you can destroy the snapshot and only then will you recover that disk space. This is one of the reasons that space management on a ZFS filesystem is a bit of a minefield for new users and is exactly why the “bpool problem” exists.

Every time Ubuntu pushes out an update (even a non-kernel update), the installer will automatically create a snapshot of your filesystem, so that you can roll back to the previous version if things go horribly wrong. In the case of actual kernel updates, the snapshot can consist of tens, or even hundreds of megabytes of combined kernel and initramfs files. Given that we have an upper limit of 2GB on the size of /boot (the bpool filesystem), that space doesn’t go very far if you don’t actively manage the snapshots. Worse, the Software Updater will start to fail when there is no longer enough free space in /boot to fit another version of the kernel and, to be clear, even non-kernel updates will fail to be installed if there is a kernel update still in the queue (even if you de-select it manually). The error messages are clear and quite specific (and even give tips on how to ameliorate the issue). However, by the time Software Updater flags the problem, it can already be too late to easily fix it (and how many of us know which system-generated snapshots are safe to remove and which aren’t, anyway?).

Short-term band-aids

These suggestions are not a true solution, they are fixes to get you up and running in order to have Software Updater working again in the shortest possible time.

Before making any changes which might permanently erase data from your filesystems, you should make a back-up of your whole disk (I’ll be repeating this at various points throughout these articles, because I really mean it — I cannot and will not be held responsible for any loss of data you might suffer while trying to follow the tips given here; I always assume that you have backed-up your data and that you have verified that those back-ups work. Batteries not included. May contain nuts.).

First, one quick fix which Software Updater lists in its output when this problem occurs — edit the /etc/initramfs-tools/initramfs.conf file and change the COMPRESS setting from “lz4” to “xz”. This won’t take effect until the next kernel update, but at that point the initramfs install will use the more efficient (compression, not speed) “xz” method. This can slice around 20MB off each of the initrd.img files in the /boot directory and so will save you about 40MB of space on each update (there are usually two versions, current and “.old”).

The results of this second tip are much more difficult to predict in terms of exact space saved — snapshot whack-a-mole.

You can list your snapshots in bpool using:-

zfs list -t snap -r -o name,used,refer,creation bpool

This will get you a listing with entries that look something like this (but probably much longer):-

NAME              USED   REFER  CREATION
@autozsys_7lzjmg  197M   197M   Wed Dec 30 11:07 2020
@autozsys_9u20hc   17K   199M   Wed Jan 20  6:56 2021
@autozsys_6c4qgp    0B   199M   Thu Jan 21  6:12 2021

[Note that I have removed the pathname before the “@” symbol to fit the text without line wraps]

As you can see, the “used” and “refer” columns vary widely between the snapshots. The bottom entry (6c4qgp) has a “used” entry of zero bytes; surely that can’t be right, can it?. Well, that’s the whack-a-mole function coming into play. In this particular case, there don’t appear to have been any changes to bpool between autozsys_9u20hc and autozsys_6c4qgp, so if we destroyed 6c4qgp, we wouldn’t get any free space released back to the filesystem. So where does the “refer” come from? That’s letting us know that even though there were no changes between the last two snapshots in our list, they both have references to 199MB of data already being held in other, previous snapshots (or the filesystem itself). What this means for us is that while destroying 6c4qgp may not free up any space, it is very likely that destroying 7lzjmg or 9u20hc will cause 6c4qgp to inherit some of that referenced data, thus causing the “used” count for 6c4qgp to increase and the filesystem not to gain back as much free space as we expected (hence “whack-a-mole”, the used data count might just pop back up elsewhere).

Now, to be fair, because we’re dealing with /boot and the turnover is usually caused by the replacement of kernel/initramfs files, it is more than likely that destroying the oldest snapshot will simply delete the oldest of those kernel-related files, returning about 140MB of free space to the filesystem. Likely, but not certain, so don’t go destroying snapshots left, right and centre without checking their contents first.

LISTING OF /boot CAPACITY AND AVAILABLE SPACE
["df -h"]
Filesystem                 Size  Used Avail Use% Mounted
bpool/BOOT/ubuntu_g1cutr   145M  117M   29M  81% /boot

["zfs list"]
NAME                       USED  AVAIL     REFER  MOUNT
bpool/BOOT/ubuntu_g1cutr  1.52G  28.5M      116M  /boot

["zfs list -o space"]
NAME                      AVAIL   USED  USEDSNAP  USEDDS
bpool/BOOT/ubuntu_g1cutr  28.5M  1.52G     1.41G    116M

You can always check the contents of any given snapshot by noting where the ZFS pool is mounted and then navigating to the .zfs/snapshot directory at that point. So, for our bpool snapshots, we can see from the output of “df” or “mount” that bpool/BOOT/ubuntu_g1cutr is mounted at /boot. Listing /boot/.zfs/snapshot will give us a listing of directories which correspond to the snapshot names in the listing above. You can list each of those directories to see what files and directories are included in the snapshot. As /boot is actually quite small, you can easily do an “ls -i” on two of the snapshot directories and see which files have the same inodes and which are different (which gives a good indication of which files are shared between different snapshots and the current, live filesystem and which are unique to a given snapshot).

Snapshots are removed using the “zfs destroy” command, by the way, not by removing the snapshot directories.

Don’t forget that destroying any snapshot restricts your ability to recover to a known point in time, so I would urge you to err on the side of caution — if you’re not 100% certain of what you’re about to remove, don’t do it!

If you’re a user who likes to create their own snapshots (before a major upgrade, for instance) you might already be able to easily target some of your own snapshots as candidates for deletion (perhaps you already know that you’re not going to roll back to that ancient, previous release?).


The [partial and not very satisfactory] Solution

The stop-gap solutions listed above are just that; short term solutions which will give you some extra breathing space, but not long term fixes. To remedy the bpool problem long term, we obviously need to add a substantial chunk of extra disk space.

One brutal, but simple way of getting back more bpool space (and a solution which is very topical with the release of 21.10 almost upon us) is to re-install Ubuntu after editing the ZFS section of the install script to bypass the problem. I’m not suggesting that this is the best or most versatile answer to the issue (in most cases it won’t be), but if you happen to be on the verge of upgrading, or perhaps have a machine where all of the application and user data is mounted from a fileserver rather than local disk, this may be an option (but, as always, I would recommend a full back-up of the system before taking such a drastic step).


Just in case you didn’t quite get that… ==WARNING== The following steps will delete -ALL- of the data on your disk. Do -NOT- proceed with the steps below if you are not prepared to have your disk(s) totally wiped of all existing data.


Booting from the Ubuntu install image will drop you into the Try-or-Install header page. Select “Try Ubuntu”, which will restart the desktop to the normal, live image. From there you can open a terminal session and become root using “sudo -s” (no password required).

Using whatever editor you’re most comfortable with, open /usr/local/share/ubiquity/zsys-setup.

On (or about) line number 267, you’ll find this code:-

[ ${bpool_size} -gt 2048 ] && bpool_size=2048

You just need to comment out that line completely for the simplest fix. Doing so will allow the bpool size calculation to grab a much larger chunk of your available disk space. Note that if your disk is 50GB or less, this isn’t going to help — you’ll still end up with roughly 2GB. Conversely, if your disk is 1TB or larger, you may end up with much more bpool space than you actually need, or want. In these cases, you might want to change line 267 to use some value other than 2GB; I found 10GB (ie:- replace “2048” with “10240”) to be satisfactory for my machines, but if you happen to be a kernel developer, you might want to bump that up even further.

Following the edit of the ubiquity file, you simply select “Install Ubuntu” from the desktop icon and proceed with the normal ZFS install.


Okay, that’s the gist of the bpool problem, along with a couple of suggestions for clawing back some disk space and the most drastic way of fixing the issue.

In the second part of this series, we’ll be looking at a couple of more reasonable methods of fixing existing installations without resorting to a full re-install (fair warning though — it will still involve booting from the install media, so is not entirely without risk).

And don’t forget …back up early and back up often!


An ESP32 project worth looking at…

…especially if you’re a farmer.

You might have noticed that it has been fairly quiet around here recently. That’s mainly because real life has been getting in the way quite a lot, but also because what little spare time I have has been taken up by a new project. I’m not ready to spill the beans on what that project is quite yet, but you can get a couple of fairly strong hints on what direction I’m going (!) by taking a look at the beautifully executed work of Matthias Hammer (and his daughter) on their autonomous equipment project. Note that Matthias isn’t limiting his work just to autonomous control of the tractor, but is also extending the project to automate the other agricultural equipment that he uses.

Matthias is using ESP32s in his project as replacements for the Arduinos used in the base AgOpenGPS project. He has a few videos up on YouTube giving a guided-tour through the equipment, as well as showing the tractor at work. If you’re at all interested in GPS/RTK and autonomous vehicles, his (short) videos, GitHub repository and the AgOpenGPS web site are all worth a visit (even if you’re not a farmer).


You could have used a 555!

Yup, the most frequent comment on Hackaday, “You could have used a 555 for that”. Well, I did. In fact, I used a pair of 555s …but probably not the sort you’re thinking of.

A pair of 555s

A few years back (see date-code photo, below), I found these batteries in a local equivalent of the “Dollar Store”. They only came in pairs, which made them more than twice as expensive as the normal, no-name AA alkaline cells, but with that branding I had to get at least one pack.

I used them (probably in one of my PIC projects back then) and then forgot about them, until I was scrambling to find some not-quite-so-dead batteries to check an ESP01S project a couple of weeks back and found them at the back of a drawer.

I popped the DVM probes on them (unloaded, of course) and was very surprised to see a healthy 1.5v reading.

Well okay, lets fire this thing up and see what happens! What happened was that the ESP01S blue LED flashed, then …nothing. Turning one of them upside-down, the date code reminded me that it was very many moons ago that I last used them.

"555 - 2013-09"

They certainly didn’t owe me anything, but I wondered what the voltage had gone down to under load. Having had lots of experience with these el-cheapo batteries, I wasn’t at all surprised to see that the voltage across the cell swung well into negative territory when I held down the “On” switch. What still surprised me was that the voltage swung almost straight back to that “healthy” 1.5v reading once the load was removed. Usually these cells are hard-pressed to register a no-load voltage of 0.9 ~ 1.1v in their fully depleted states.

Maybe these we’re just catching their “second wind” …or maybe they do have a 555 hidden inside them after all!

-1.33v displayed on DVM when load applied
…wth load
1.54v reading on DVM from single cell
No load

The ESP01S board wouldn’t stay powered with these batteries in the pack, so I had to hold down the “On” switch long enough to snap the photo.


No ESP01S modules were harmed in the production of this blog post, but it probably didn’t do my poor little 555s much good.

FreeBSD Notes: 13-Release and bectl

We’ve had several release candidates (at least one more than expected) of FreeBSD-13, of which I’ve tested the later ones as VMs on a 12.2-p6 host and found them to be lightweight and sprightly. So, when the final release was announced last week, I didn’t have any particular qualms about updating said 12.2 system to release 13.0. Having said that, FreeBSD makes it particularly easy to add belt-and-braces insurance to your updates nowadays, using “bectl“.

If you haven’t come across bectl before, I would heartily recommend that you check it out and read the manual page before you start any future upgrades. The “be” part of the name stands for “Boot Environment” and this extremely useful tool helps us to manage our ZFS filesystems to allow virtually risk-free changes to our root filesystem without needing to get deeply into the nitty-gritty of snapshots, rollbacks and boot menus. Yes, it only works through the magic of ZFS, but it’s just another reason why you definitely should be using this filesystem everywhere (even on your laptop).

bectl” will enable us to quickly create a snapshot of our current root filesystem, add that snapshot to our “beastie” boot menu and, importantly, then allow us to create a completely new clone of the existing snapshot (which we can then upgrade or change in whatever way we want) and optionally make that updated filesystem available as the default boot for our machine. This means that we can install an upgrade (in my case, release 13.0) and boot it immediately, but still have the option of selecting our original root from the boot menu, if things don’t quite go to plan.

Before we start the actual upgrade, just a couple of words of warning…

  • As noted above, the system must be using root on ZFS.
  • It should be running the GENERIC kernel.
  • We should be starting from a recently updated version of the previous major release (ie:- 12.2-p6 if updating to 13.0).

How do we prepare our system in advance of the actual upgrade? It couldn’t be much easier. The “back-up” boot environment will be the root filesystem which we have now. We will create a new clone of this existing environment to work on during the 13.0 install (leaving the original untouched).

First, we check our current boot environment with:-

root#   bectl list -D

BE              Active Mountpoint Space Created
default         NR     /          10.2G 2020-02-16 18:02

The bectl list command shows the currently available boot environments and the “-D” argument shows the space used. The “Active” column shows an “N” for the boot environment which is in use right Now and an “R” for the boot environment which will be activated on the next Reboot. In our case, we only have one boot environment, so it is both active and selected to be used on the next reboot.

To create the cloned (work) boot environment, we do:-

root#   bectl create 13.0-RELEASE

root#   bectl activate 13.0-RELEASE

The “create” is obvious. The “activate” command tells the system that we want to use the new, “13.0-RELEASE” boot environment at the next reboot. When we use the “list” command again now, we should see something similar to this:-

root#   bectl list -D
BE              Active Mountpoint Space Created
13.0-RELEASE    R      -          369K  2021-04-22 08:32
default         N      /          10.2G 2020-02-16 18:02

…indicating that “default” is currently in use, but that “13.0-RELEASE” will be used on the next reboot. If everything looks okay (and there were no error messages from the bectl commands), we can go ahead and reboot our system. It should boot as normal and look exactly the same as before (we haven’t actually changed anything on the filesystem yet, so we’re still running 12.2-p6). However, if we were to do another bectl list -D we’d see that the “Active” column would now show “NR” for 13.0-RELEASE and just “-” for “default”.

Do not proceed with the upgrade steps below unless you see the “NR” active status for the “13.0-RELEASE” boot environment.

Having checked that we’re using the cloned root filesystem, we can now go ahead and use that other wonderfully useful utility, freebsd-update, to start the actual upgrade process:-

root#   freebsd-update -r 13.0-RELEASE

Note that “13.0-RELEASE” in this command is specifying which release we want freebsd-update to upgrade to, not where we want to install it. We simply follow the instructions which freebsd-update gives as it goes along (we will need to reboot a couple of times to complete the full upgrade process).

If everything goes as planned, after the final reboot we should have a working FreeBSD 13.0 system.

My upgrade process went smoothly; all system utilities came back up and the VMs ran with no problems. Everything looked fine …until the first, automated, weekly reboot rolled around. The machine didn’t come back up. The console was complaining that there was a fatal ZFS error with the main user-data pool (zstore00 – not the root filesystem). I had to cycle power on the system to have it reboot and, when it came back up it did indeed show a checksum error on that pool. I cleared the error and initiated a scrub and once everything was stable and had been running for a couple of days, did a manual reboot. It failed again, in the same way.

The actual error shown was;-

uhub2: detached
uhub1: detached
umass0: detached
uhub3:detached
ubt0:detached
Solaris: WARNING: Pool zstore00 has encountered an uncorrectable I/O failure and has been suspended.

At that point the actual shutdown process hung solid and, as before, I needed to power cycle the system to have it come back up. I also tried updating the /boot/loader.conf file (as suggested in Graham Perrin’s very useful guide) and rebooted a couple of times manually, but to no avail.

Note that I had not run zfs upgrade on the pools, since this would have made it very difficult to roll back.

So, at this point I needed to get the system back up and running and decided to do that roll-back to 12.2. Here’s how you can do that…

Power cycle the machine (if you’re stuck with a hung system, as above) and once the beastie menu appears, wait a second for the additional options to appear at the bottom (it takes the boot process a moment or two to collect the boot environment listings) and then select the “Select Boot [E]nvironment…” option. You’ll get another, short menu where you can use #2 to cycle through the boot environment names until you see “default” displayed. At this point you can simply hit Enter to have the system select and boot into this saved environment.

Your system will now boot into the original 12.2-p6 root filesystem and work as before. The only minor issue that you have is that booting from the beastie menu only sets the boot environment for this one-time boot, so you must use bectl activate default to have your 12.2-p6 environment set as the permanent boot environment (if you do a bectl list -D at this point, “default” should now have “NR” showing in the Active column).

Looking a little closer at the listing you’ve just done, you’ll also notice that because we’ve actually got a complete 13.0 install on the alternate “13.0-RELEASE”, it now shows a much greater space-used than before (in my case around 12GB).

FUZIX (Unix-like OS) ported to the ESP8266

Well here’s some really interesting news. I have a job set-up to update “interesting” GitHub projects to local disk on a daily basis, so I can just do a quick listing every morning to see what, if anything, has changed. This morning the FUZIX project flagged changes (FUZIX is a minimal, unix-like OS for very small, resource-limited micros, which started off being targeted at the Z80 series). When I checked the repository to see what the updates actually were, I found:-

drwxrwxr-x 10 gaijin gaijin 20 Feb 18 05:38 ..
-rw-rw-r-- 1 gaijin gaijin 11536 Feb 18 05:38 Makefile
-rw-rw-r-- 1 gaijin gaijin 273 Feb 18 05:38 .gitignore
-rw-rw-r-- 1 gaijin gaijin 35670 Feb 18 05:38 filesys.c
-rw-rw-r-- 1 gaijin gaijin 252 Feb 18 05:38 dep.mk
drwxrwxr-x 2 gaijin gaijin 6 Feb 18 05:38 cpu-armm0
drwxrwxr-x 2 gaijin gaijin 24 Feb 18 05:38 platform-esp8266

Which led, in turn, to David Given’s marathon videos (there are roughly 30-hours of keyboard-bashing and puzzling-out-loud available right now, with another five videos still to be released on a daily basis) detailing his work to port FUZIX to the ESP8266. David’s web site has some further explanation of how far along the port currently is, along with some pre-compiled, loadable binaries for the ESP8266 (I haven’t had time to try them yet). All of David’s work is available from his GitHub repository.

You will need to attach an SD card to get the full FUZIX experience, but David says it is reasonably speedy on the ESP8266, with a boot time of just 4 seconds.

|�l�c|����{���B�p�n�dNn���cp��cl r$p�N�s����cN�|����B�|�N�l��l�Nl�d Nr���N��{$�o�nod���no��rdp�n���l"o�|����p��no�$�$�no$�{$or���o� �n�2NNl��|�N��lp�N��d�#n�|���b��Nn�d�l �oo$�{lor���N$�$sےNl��FUZIX version 0.4pre1
 Copyright (c) 1988-2002 by H.F.Bower, D.Braun, S.Nitschke, H.Peraza
 Copyright (c) 1997-2001 by Arcady Schekochikhin, Adriano C. R. da Cunha
 Copyright (c) 2013-2015 Will Sowerbutts will@sowerbutts.com
 Copyright (c) 2014-2020 Alan Cox alan@etchedpixels.co.uk
 Devboot
 80kB total RAM, 64kB available to processes (15 processes max)
 Enabling interrupts … ok.
 Scanning flash: 2591kB: hda: 
 SD drive 0: hdb: hdb1 hdb2 
 Mounting root fs (root_dev=18, ro): warning: mounting dirty file system, forcing r/o.
 OK
 Starting /init
 init version 0.9.0ac#1
 fsck-fuzix: 
 
 login: root
 Welcome to FUZIX.

David notes in the README (contained in the binary tar-file) that the system does have some limitations; first and foremost is that it is almost unusable running from flash, so you really do need to wire up an SD card (pinouts for this are included in the same README). Here is his ToDo list:-

  • userland binaries can’t find their error messages.
  • CPU exceptions should be mapped to signals.
  • single-tasking mode should be switched off (which would allow pipes to work).
  • someone needs to overhaul the SD SPI code who understands it.
  • not all the ROM routines are hooked up to userland binaries.

Right, I’m off to look for an SD card and a spare ESP. See you later!

Great price on an Atom Z8350-based system (plus a giggle)

As regular readers know, I occasionally take a dive into the world of mini-pcs (especially low cost models) in my quest to get the best bang for the buck/watt. On my travels through the depths of Aliexpress this morning, I came across very reasonably priced Z8350-based system. It’s not going to blow the doors off, but I have a soft spot for these boxes, simply because they do so much for what they are.

This one is a design which is new to me, sporting more USB ports than normal, as well as both VGA and HDMI video ports. You’ll need to be careful though, as the retailer seems to be a bit confused about what hardware they’re selling …in the product “overview” they state it has 2 x USB3 and 2 x USB2, but the images clearly show 1 x USB3 and 5 x USB2 ports. The header of their advert also rather vaguely claims “support 2.5 inch HDD”, but (despite the extra height of this case compared to other models) there are no obvious SATA headers on the motherboard pictures and in the overview they only mention support of “external” disks.

They’re also a bit on the boastful side about the service they provide (hence the “giggle” in the title of this post — see image, below).

The price for the 4GB/32GB model is a modest $84.45 (plus $3 shipping to my part of the world). Although these boxes are sold as “Windows 10” machines, I certainly wouldn’t advise anyone to try running Windows on the 32GB eMMC model, as the initial download of updates will overflow the available storage space.

As I’ve mentioned before though, these machines make excellent little low-cost, low-power-usage servers for anyone doing 24/7 self hosting (I’ve even got a similar box with more than 12TB of USB disks hanging off the back as a back-up server on my home network). This version also comes with 4GB of main memory, which is 2GB more than my original Z8350 based system (which has been running 24/7 since Dec 2016).

Please note that I don’t own this exact system, so I can’t verify its actual hardware configuration or performance. I haven’t ever used the supplier SZMZ before either, so I can’t vouch for their delivery times or packing. Another point to note is that although SZMZ has a very impressive 100% feedback rating, it is only based on two orders (at the time of writing). All the same, this looks like a nicely built mini-pc (with more ventilation holes than equivalent models, if nothing else).


‡  —  As to the strangely worded phrase in their advert, I’m working on the assumption that what they’re trying to convey is that different builds and versions of this model are available  …or maybe the copywriter just took this chance to broadcast their views on men in general.