A Really Neat Trick With The RPi Pico

I imagine that almost everyone knows by now that the RPi Pico is dual-core and probably you also know that the FreeRTOS stack is available to enable multitasking. What I didn’t know until this morning is that there’s a super-simple, super-easy way of taking advantage of both cores without having to go down the RTOS rabbit hole (although FreeRTOS isn’t that difficult, it can, like many other things, be a wee bit daunting to the beginner).

This is a biggie! This apparently simple change just opened up the world of multitasking, multicore programming to absolutely everyone. This neat idea, centred on the Arduino IDE and with the added popularity of the Raspberry-Pi community, is going to be a huge boon to hobbyists everywhere.

So what is this multiprocessor trick? It couldn’t be much simpler. Using the Raspberry Pi Pico Arduino core (“Arduino-Pico”), use these two functions:-

setup1(){}

loop1(){}

That’s it! That’s essentially all you need to know.

By default, Arduino-Pico will use the normal setup() call to initialize core-0 of the processor, but if it finds that a setup1() function exists, it will initialize core-1 of the processor with the contents. This happens more or less simultaneously at start-up.

Likewise, while core-0 will run code contained in the default loop() function, core-1 will run the code contained within the loop1() function.

There are also a couple of other calls in there which will help you communicate between the two cores and which are also really simple to use. Rather than trying to describe them, I’ll just leave you with a link to a demo program (which will run a WiFi network scan on core-0, while simultaneously running a fader on the on-board LED using core-1).

Hopefully, this change will be picked up on other platforms (ESP32 anyone?) and become an integral part of the Arduino IDE; in the meantime, have fun running multicore on the Pico, quickly and easily!


The Arduino-Pico core is available from Earle F. Philhower’s GitHub repository.

Excellent documentation for the core is also available.

Advertisement

The Daily Steal:- 11th Gen Jasper-Lake Bargain

Today’s daily-steal is actually a double, in that the same deal has two different models for almost the same price. These systems are currently available on Amazon U.S. and, although they’re not the absolute cheapest Mini-PCs available, they certainly are very good value for money, taking into account processor, memory, M2.SSD storage and peripherals. Both systems come with the same Intel N5095 processor and the same, basic port configuration.

The Amazon page has a selection box to choose between the 128GB and 256GB models and, as the 256GB model is only U.S. $4 more than the 128GB model, the choice seems to be pretty obvious.

The 8GB/128GB model is currently selling for U.S. $135 and the 8GB/256GB model is U.S. $139.

These are Beelink Mini-S machines, which means that they do not come with the USB-C port of the U59 models, but they still have 4 x USB-3 (type-A) ports for external disks. The RJ45 is 1GbE and there are dual HDMI ports on the back panel.

For those of us outside of the U.S., these systems can still be shipped (by Amazon) to most regions for a moderate extra cost, but you should be aware that you may have to pay additional import duties, too. Please also note that the shipping and import duties will change, depending upon which of the two models you choose.

Gemini-Lake Mini-PC

Right now, we seem to be on the cusp of a dramatic drop in PC prices (given the end of lockdown buy-ups and looming “Black Friday” sales) and we’re certainly starting to see some interesting deals appear in the mini-PC market already.

Photo courtesy of GMKtec

Here’s one attractive, pre-Black-Friday deal already available from Amazon.com. It is currently available for $100 ($129.99 listed price with a $30 discount coupon available). As far as I can tell, the coupon code is good for non-US purchasers too (there’s nothing that I can see in the Term & Conditions related to geographical location), but remember that you will be charged for transport and import fees, even assuming that Amazon will ship to your country.

If you can’t get it from Amazon in your country, GMKtec are running a “free, worldwide shipping” offer on their own web shop too, although the deal isn’t quite as sweet.

As you can see, although it’s a Gemini-Lake based system, it does have multiple USB-3 sockets, as well as both VGA and HDMI video (not to mention the snazzy, Cisco-esque ventilation hole pattern). It is also advertised as coming with 6GB of main memory and 128GB of eMMC, which is pretty decent.

Now I have to say up-front that I don’t own this particular model, but I have been running its baby-brother for a couple of years now — an N4000, 2-core system, badged by a different maker, but almost identical in the external configuration of ports. It runs FreeBSD, providing standard LAN services (unbound, nsd, NTP and DHCP) and has 20TB of disk storage attached across the multiple USB-3 ports; mainly providing back-up services to other machines on the LAN via ZFS send/receive and Jim Salter’s excellent “syncoid”. As well as all of that, it also runs multiple VMs. To date, it has provided me with sterling, trouble-free, 24/7 service.

The point is though, that my 2-core machine only came with 4GB of memory and a 64GB eMMC and it still works perfectly adequately as a headless server, so I think the 4-core model above, with its 6GB/128GB configuration should be a pretty reasonable investment for similar workloads (in case it’s not obvious, I wouldn’t advise anyone to buy this system as a desktop workstation).


The small(ish) print:-

  • I have no relationship (not even as a customer) with GMKtec, nor (as already mentioned above) do I actually own any of their products.
  • This is not the newest or best machine on the market. It does come in at an extremely attractive price, though.
  • Unlike the RPi range, it is available (I know, I’m going to get shirt for that remark).
  • You may want to wait to see what the Black Friday prices on N5105-based Jasper Lake systems are like.

Grub bites man. Man stamps on grub. ‡

‡ — No invertebrates were harmed during the production of this article.

TLDR:-  Skip straight to the updates at the bottom of the page for links to the bug reports.  The short version is “You can blow away any root-on-ZFS Linux system by enabling zstd compression on bpool (or whatever the actual boot partition is called on your particular distribution)”. Do NOT do this (not even to test it); you will regret it!

Almost exactly two weeks ago I got a (normal) pop-up message on my screen, letting me know that there were updates available from Ubuntu for my 21.10 version of Mate and that, by the way, 22.04 was now available if I wanted to upgrade.  I politely refused the upgrade offer, but accepted the updates.  Once the updates were completed I was prompted to reboot (and I noted that the initramfs had been updated).  This is all normal stuff for the millions of us running the various versions of Ubuntu, so after finishing up what I was working on (and washing my hands) I hit the reboot button.

It didn’t.

It hasn’t for the last two weeks (not without a recovery image or an external disk plugged in, anyway).

At first I wasn’t too concerned.  This (as the ‘mericans say) isn’t my first rodeo and, if everything else failed, I’m using ZFS, so I just had to roll back to the last good snapshot, right?

Wrong!

As I wasted more and more time futzing around with things, I just got more and more confused. The EFI boot was working and grub was being run, but it insisted that it couldn’t find a workable filesystem.  Booting into a live-CD image said there were two perfectly good ZFS pools on the disk (actually a 512GB NVME SSD) and a “zfs scrub” passed with flying colours on both of them (bpool and rpool).  Running grub-probe always came back with a message along the lines of “Nope, ain’t nuthin there that I recognize, pinhead!”.

By this time I had convinced myself that someone had broken into Ubuntu’s update servers and slipped a Trojan into my disk headers, or maybe someone was slipping magic mushrooms into my breakfast tea, or aliens had abducted my real ZFS pools and replaced them with zombies.  Anyway, I needed to get some work done, so I found an unused external USB disk, fired up a live-CD again and, because this is ZFS, did a ZFS send-receive between the original disk and the external.  When it came time to run the grub install (on the external disk) it once again insisted that there was nothing at all there which it recognised as any sort of valid filesystem.  Duh!

Okay, nuts to this.  I put a 22.04 live-CD image in, booted it up (“Oh look, you have two perfectly good ZFS pools on this external disk!  Are you really sure you want me to scribble all over them?”) and installed a brand spanking new version of 22.04 onto the external, followed by a ZFS send-receive of just the USERDATA filesystem from the original NVME drive (I’d like to say that I rebooted and everything sprang into life on the external disk, but of course it didn’t.  You can’t easily remove an NVME “disk” from a laptop and of course EFI and grub just kept right on trying to access it).

Eventually, I did manage to broker a truce between the laptop BIOS, EFI and grub, to the point that I could reboot the system successfully, as long as I was actually there to hit the F9 key and manually select the EFI entry for the external disk  …otherwise it would stubbornly continue with a blinky dance of reset, spin disk up, flash something on the screen for 10ms, spin the disk down and repeat.  I never did manage to read what that 10ms micro message was (something from those dang aliens again).

But glory hallelujah!!  It was fantastic!  Not only did I have Mate back again, but the stupidly useless ELAN touchpad on the HP S15 laptop actually worked properly for the first time ever!!  Banzai!!  Well done 22.04!  I’m sorry that I refused your offer of an upgrade at the start of all of this.

Well, long story longer, that’s the way that things have stayed for the past two weeks.  Not that there hasn’t been great gnashing of teeth (not recommended at my advanced age) and thrashing of Gewgull searches (and even Ubuntu’s bug database) for any hints of what could be causing my problems. Searches including both “grub” and “aliens” turned up some interesting stuff, but it seems that I’m the only semi-sentient being-man-thingy in the universe who is having problems with those things in connection with computers.

Eventually, the third neuron in the second bank started firing occasionally and I began to have flashbacks of the horrible problems I had a while back, trying to do ZFS back-ups between my Linux laptop and the mirrored disks on my FreeBSD servers.  Although the FreeBSD servers could send data to a receiving  Linux system, the other way around just would not work at all.  It turned out that there were incompatibilities between the ZFS properties used between the two systems, even though the ZFS versions were meant to be the same (upgrading the FreeBSD servers to version 13.1 seemed to cure that issue).  Now it seemed probable that there was some similar incompatibility between grub and ZFS  …and, finally, it clicked. 

One of those property updates which hit FreeBSD a while back was the introduction of “zstd” compression (Ubuntu introduced it a short while later).  I had tried changing the compression to “zstd” on FreeBSD and noticed a worthwhile increase in compression-ratio with no apparent difference in speed, so I’d made the changes on my laptop, too.  The back-ups kept on working and I was saving more disk space …what’s not to like?

Double-duh!  Pinhead at work (stand clear!).

Okay, so I have (I suspect, anyway) a philosophical disconnect between grub and ZFS about what constitutes a filesystem.  In my book, ZFS wins (whatever grub says, I’m an ex-Sun guy and old loyalties die hard).  What to do now?  Well,back on Gewgull I replace “aliens” with “alternative” and get a few hits on something called “ZFSBootMenu”.  I go to their GitHub repository and start reading.  Nope, I must have left “aliens” in there somewhere, ‘coz I can’t understand this stuff.  It starts off more or less okay (I understand their reference to FreeBSD’s boot environments …a very handy feature,  sign me up), but then they start wandering off into the weeds with things like “fzf” and “dracut” (which sounds like a bad day at the vasectomy clinic) and I want to stop reading so badly that my teeth hurt (told you not to gnash).  I briefly leave the README page and head over to the pre-built releases instead. 

Ah, this is interesting.  “ZFSBootMenu v2.0.0  …New features – Dracut is now optional;”.  Phew!  Well that’s a relief.  Unfortunately, there are four different files (and additional source tarballs), but no information on what the differences are.  A couple have “release” in their names and the others have “recovery”, but no actual information in the v2.0.0 section about what the differences are or how to use them.  Two are compressed tar files (okay, I know what those are, anyway) but the two others have a “.EFI” suffix, which I haven’t come across before, but it’s not too much of a stretch to suspect that they should be used in the Extensible Firmware Interface (that’s the BIOS to you, sonny!).  But how?

Back to the README:-

Each release includes pre-generated images (both a monolithic UEFI applications [sic] as well as separate kernel and initramfs components suitable for both UEFI and BIOS systems) based on Void Linux. Building a custom image is known to work in the following configurations…

Well there you go then.  C’mon!  Chop-chop!  Get on with it!

I won’t bore you with the rest of it, but despite the rambling (or partly missing) documentation, it turns out that ZFSBootMenu is really quite a nifty bootloader (grub alternative) specifically for ZFS filesystems.  It does have a couple of quirks, but after a few false starts, I did manage to bring my 21.10 NVME SSD back from zombieville and ended up with a laptop which I could once again disconnect from all external cables and disks and still have it boot reliably.  To do ZFSBootMenu full justice I need to put together another post with a step-by-step guide of how to recover a stuffed ZFS machine, but just in case you’re stuck up this same creek without a paddle, here are just a couple of essential hints to getting ZFSBootMenu to work:-

  • Use the .EFI  file (I used the “release”, but “recovery” should also work okay).
  • Remember– When you’re booting using ZFSBootMenu, it assumes that the kernel and initramfs are on the root partition, so you’re going to be booting from “rpool”, not “bpool” (and so your kernel and initramfs need to be present on “rpool”, which is not standard in Ubuntu).
  • Disconnect any other disks which you may have added to the system in previous attempts to fix this boot problem (ie:- external USB disks with copies of your original bpool/rpool filesystems).  They will just confuse EFI, grub and you.
  • Set “canmount=noauto” on all filesystems which have “mountpoint=/” set (this one is really important).
  • Make sure that your ZFS pools can be imported without having to use “import -f” (use a live-CD or ZFSBootMenu itself to import the pools manually, fix any problems and then export them again, before you reboot).
  • Lastly, for those others of you out there with non-US keyboard layouts, the “Alt” key for command mode selection within ZFSBootMenu will probably not be the “Alt” key.  For my Japanese keyboard, the magic key turned out to be the one key which I had never before pressed on my keyboard …the (shudder!)  “Windows” key.

And the bottom line… Just don’t change your bpool to zstd compression.  Okay?!?    †  ∇


†  —  From a cursory check of the Grub2 code in the upstream Debian repository, it looks as though the zstd compression library was added in November of 2018.  Unfortunately it seems to have been implemented only for BTRFS, not for ZFS.

  —  “cursory”  Adjective.  Causes the reader to leap out of their chair and shout “G’dammit!!”  (this answer is certain to get you an “A” in GCSE English).

∇  –  Grub Compatibility  —  As usual, the folks at OpenZFS/Debian/Ubuntu are way ahead of me.  If you need to create a new pool which will work with Grub2, there is already an existing compatibility file (they’re stored in /usr/share/zfs/compatibility.d).  You can use it like this:-  

   zpool create -o compatibility=grub2  <POOL NAME>

If anyone is desperate enough to have read all the way down here to the bottom, I’ll just note that this problem hasn’t (has …see below) been submitted as an actual bug, because the auto-submission procedure won’t let you submit a bug against a program which isn’t installed on your system and, of course, I’d removed the grub package and used ZFSBootMenu to actually get my laptop working again. However, the secondary suggested method of opening the issue as a question in LaunchPad is available here if you’re interested (and I’d like to thank Manfred Hampl for trying to help me out there).

Update — 27th July 2022 – I have submitted bug request #1982897 with a condensed description of this issue and a request for an upstream addition of “zstd” support for ZFS in the Grub2 code.

Update #2 — 27th July 2022 – Submitted bug #62821 as a feature request for the addition of “zstd” support to the Grub2 ZFS code.

Update #3 — 13th August 2022 – Updated the Grub2 bug report with a little more detail on the severity of the problem, as there hasn’t been any acknowledgement of the original submission as yet.

New Kid on the BLEck

For anyone who hasn’t already heard yet, yesterday saw the release of yet another contender for the ESP’s cheap, small and connected WiFi device crown. This one comes from the Raspberry Pi stable and is a revamp of the RP2040 “Pico” which, in this new Pico-W variant, now sports a shiny, new Infineon  CYW43439 WiFi module to provide 802.11n connectivity.

My tongue-firmly-in-cheek title typo refers to the fact that the module also includes the hardware for both Classic Bluetooth and Bluetooth Low Energy …but unfortunately neither are actually enabled in this initial version of the Pico-W.

The great news about this new board is that it is cheap ($6 US) and actually available (unless, like me, you happen to live in Japan, in which case you can settle in for another 8-month wait while the Japanese government decide whether they’re going to “hanko” the approval or not — fun fact:- the Pi Zero 2W was released on October 28th, 2021, but only became officially available in Japan a couple of weeks ago).

The Pico-W certainly looks interesting (a lot more so than the non-WiFi original) and might be a real contender, with Eben Upton hinting in his announcement post that the pipeline is in place to produce millions of these boards (he was referring to the Pico family, not specifically the Pico-W). With the supply lines in place to get these into customer’s hands at a reasonable price across a large part of the globe (Element14, RS, Digikey and Mouser), they might well become the new, go-to board for connected projects. An SDK is already available for C/C++ and a network capable Micro-Python image also already exists.

If you’re looking to get your hands on one (and I do mean “one”), I can vouch for the fact The PiHut in the UK still had stock a few minutes ago and will ship internationally without breaking the bank.

Ah …seems like I might have snagged their last one. Sorry!

Visual Studio Code and PlatformIO

Regular readers will know that I’m a big proponent of PlatformIO as a programming platform for the ESP family; it’s just so much better than the Arduino IDE for us command-line interface die-hards. As I’ve noted before, I’m not a religious fanatic on the CLI vs GUI thing; it’s just that GUIs (with a couple of exceptions) don’t really click for me. Most of the time I just don’t “get it” and what’s on offer usually seems to limit the functionality without really adding much, if anything, in ease of use. Recently though, I’ve been searching for information on getting started with the ESP32’s built-in version of FreeRTOS (specifically, looking for more information on real-world use, rather than just a couple of sentences about the syntax of the xTaskCreate() call) and I found a couple of very useful videos by Xavier on his “Simply Explained” channel which, as the name suggests, do a very good job of explaining by example, how to use FreeRTOS. The thing which (as a doddering old git) really caught my attention was his use of Visual Studio Code. I was captivated by the pop-up prompt with info on the parameters (of which there are plenty) to feed to xTaskCreate. While I imagine that functionality can get pretty tedious when it pops up for each and every printf(), it does look like a definite winner when you’re learning something new (and don’t have much in the way of short-term memory any more).

Now, what was I saying again? Oh yes, here’s the link to one of Xavier’s videos. He’s got a ton of content on his channel on many diverse (but usually tech-related) topics. This one is not the first in the series, so if the abrupt lead-in is a bit too much you can go to his Intro on FreeRTOS instead. I highly recommend watching Xavier’s videos if you’re trying to get started with FreeRTOS.

Of course, I immediately went off and downloaded Visual Studio Code and fired it up …only to just as immediately get totally lost when I couldn’t get it to do anything that I wanted. I did watch a few more videos from a nice lady at Microsoft, but ended up suffering from information overload and Windows-itis. I went for a nice cup of tea instead. And then went back to PlatformIO in an xterm, just so that I could get something done.

More recently, one of my regular morning reads, Hackaday, had an article on making laminated artwork for front panels with an accompanying video (worth a look in itself) by Richard Langner. I was impressed by Richard’s succinct, no-nonsense style and dipped into his video listings to see what other goodies were there. Lo and behold, right at the top of the list were a couple of videos on how to get started with Visual Studio Code with PlatformIO. The first was only a minute and a half long and the second just over six minutes. Both are filled with essential information on doing just what I wanted to do (have the editor prompt me with useful info, but still be able to use PlatformIO as my programming environment). As they’re so short, I’m including both as embedded links below. Take a look. Even if you don’t like them, you won’t have wasted much time.

Many thanks to both Richard and Xavier (oh, and Al Williams and the team at Hackaday, as well as Ivan and everyone at PlatformIO) for making my life not just easier, but a lot more interesting, too.

Pre-Black-Friday Bargains, Japan Style

For anyone else in Japan, here’s a bargain you can pick up from Amazon Japan’s “Featured Deals” in Laptops this morning (24th Nov 2021). Take a look at the 16″ LG at the right (enlarged version below).

And just in case you’re not familiar with the Japanese Yen conversion rate; this morning $1 US is worth 115 Yen and one U.K. pound is worth 154 Yen. That’s a fabulous discount of a fraction of a penny.

Yup, someone at Amazon Japan has a sense of humour.

The Ubuntu ZFS boot pool problem [Part I]

Sorry, as you can see from the title, this is another “not-ESP8266” article, so you can skip this if you’re only interested in our magical little wireless modules. However, if you’re running a system (laptop, server or workstation) loaded with Ubuntu and have the root/boot partitions mounted on a ZFS filesystem, you should at least stick around long enough to read through the problem description, below.


13th Oct 2022 – IMPORTANT NOTE:- Directly related to the “bpool problem”, there is a separate issue with Grub2 not supporting zstd compression for ZFS. So, if you are struggling with the bpool problem, do not, Do Not, DO NOT enable zstd compression as a “fix” on your bpool, otherwise your system will be rendered completely un-bootable (the standard quip is normally “Don’t try this at home, folks!”, but in this case it is most certainly “Don’t try this at work!”, otherwise you’ll be out of a job in short order).


If you landed here from a web search for “can’t upgrade Ubuntu – not enough space on bpool” (or something similar), then you’re in the right place. You can probably skip the introduction to the problem (you already know what you’re facing) and go directly to the solution.

Note that in these articles I am assuming that you are familiar with Linux and confident in your own abilities to administer your own system. Because we are dealing with filesystems, I am also going to assume that you have made and verified back-ups of your whole disk(s) before you follow any of the tips below. I am not responsible for your data. You are. The advice below is given in good faith after having been tested on several of my own machines (of different types), but I cannot be held responsible for any loss of data.

The bpool Problem

You see the familiar Software Updater pop-up, announcing that there are package updates available and prompting you to install them. You scan the list of updates and everything looks good, so you hit the “Install Now” button, but instead of a progress meter, you see a pop-up error window with the message:-

The upgrade needs a total of 127 M free space on disk '/boot'. Please free at least an additional 107 M of disk space on '/boot'.

The message continues with a couple of suggestions of how to increase available space in “/boot”, but basically at this point you are unable to install updates.

You’ve just run into the “bpool problem”.


Background

For the last few releases, Ubuntu has provided an easy method for installing root on a ZFS filesystem, giving you access to snapshots and the instantaneous rollback to previous versions which it enables (as well as the possibility of mirroring) without having to go through the long and tedious process of adding ZFS support after the actual install. It was a major move forwards and the “one-click” install process made it simple for anyone to benefit from ZFS technology (not just snapshots, rollbacks and mirrors, but raidz and ZFS send/receive back-ups, too). With all of those benefits, why wouldn’t you just use it by default?

Well, as a long time ZFS user, I can tell you that it does come with a few drawbacks. It is, for instance, somewhat more difficult to judge the available space left on any physical device than it was in the pre-ZFS world (good ole’ “df -h” is very good at giving you an instant and pretty accurate impression of the current capacity of your disks). ZFS tends to muddy that simplistic view a little, until you get used to depending more on the output from “zpool list -v” and “zfs list -o space” instead of “df”.

Fragmentation is another issue. Die-hard Unix users will remember that “fragmentation” was always a problem for people who used that other operating system, not for Unixes (well, unless you started to run out of inodes, anyway). However, with ZFS you do need to keep an eye on that “FRAG” column in the output of “zpool list -v”; once it starts climbing to around the 50~60% level it’s definitely time to start planning for an upgrade and, once it’s past 70% it’s time to take some urgent action (usually, but not always, time to add more physical disk space — we’ll touch on de-fragmentation as a side-effect in the “Solutions” presented in part-II). Once it hits 80%, you’re in big trouble as, even with available free space, the system will be struggling to find usable space.

These issues are a pain to come to terms with after all of these years of doing the odd “df” just to check that everything was okay, but they are still heavily outweighed by the advantages (did I mention copy-on-write or checksummed data yet?).

So, our simple ZFS installation method from Ubuntu is a real game changer and gives everyone the chance to ride on the ZFS bandwagon with minimal effort. Unfortunately, along with that simplicity and ease of installation comes another problem — sizing your filesystems. The installer is a one-size-fits-all deal; it looks at your disk size and makes a couple of fairly simplistic calculations to create a very simple partition table and then later adds a spread of ZFS datasets on top of those physical partitions. The “bpool” (for “boot-pool”) is a single ZFS filesystem which occupies a dedicated partition just for the kernel/initramfs and EFI/grub data required to boot the system. The “rpool” (for “root-pool”) is the ZFS filesystem which contains datasets for /, /var and /usr (as well as sub-directories) and the user home directories. It’s all very well laid out, so that a rollback to (for instance) a previous kernel version need not affect the user’s data. However, the installation script for ZFS currently has one major issue; it limits the size of the bpool physical partition to 2GB (it can be smaller, but not bigger), no matter what the size of your physical disk

The major drawback that people are finding to this installation mode comes from one of the strengths of ZFS — the snapshot functionality. Put simply, if you make a snapshot of say, your whole bpool, ZFS will never delete anything from that point in time. If you subsequently delete a kernel which was present when you took the snapshot, ZFS will keep the file itself, hidden away in the snapshot and only remove the named link to the file from the normal /boot directory. If, at some later time, you decide that you no longer need that snapshot (because it is outdated and you know you’re never going to revert to that old kernel again) you can destroy the snapshot and only then will you recover that disk space. This is one of the reasons that space management on a ZFS filesystem is a bit of a minefield for new users and is exactly why the “bpool problem” exists.

Every time Ubuntu pushes out an update (even a non-kernel update), the installer will automatically create a snapshot of your filesystem, so that you can roll back to the previous version if things go horribly wrong. In the case of actual kernel updates, the snapshot can consist of tens, or even hundreds of megabytes of combined kernel and initramfs files. Given that we have an upper limit of 2GB on the size of /boot (the bpool filesystem), that space doesn’t go very far if you don’t actively manage the snapshots. Worse, the Software Updater will start to fail when there is no longer enough free space in /boot to fit another version of the kernel and, to be clear, even non-kernel updates will fail to be installed if there is a kernel update still in the queue (even if you de-select it manually). The error messages are clear and quite specific (and even give tips on how to ameliorate the issue). However, by the time Software Updater flags the problem, it can already be too late to easily fix it (and how many of us know which system-generated snapshots are safe to remove and which aren’t, anyway?).

Short-term band-aids

These suggestions are not a true solution, they are fixes to get you up and running in order to have Software Updater working again in the shortest possible time.

Before making any changes which might permanently erase data from your filesystems, you should make a back-up of your whole disk (I’ll be repeating this at various points throughout these articles, because I really mean it — I cannot and will not be held responsible for any loss of data you might suffer while trying to follow the tips given here; I always assume that you have backed-up your data and that you have verified that those back-ups work. Batteries not included. May contain nuts.).

First, one quick fix which Software Updater lists in its output when this problem occurs — edit the /etc/initramfs-tools/initramfs.conf file and change the COMPRESS setting from “lz4” to “xz”. This won’t take effect until the next kernel update, but at that point the initramfs install will use the more efficient (compression, not speed) “xz” method. This can slice around 20MB off each of the initrd.img files in the /boot directory and so will save you about 40MB of space on each update (there are usually two versions, current and “.old”).

The results of this second tip are much more difficult to predict in terms of exact space saved — snapshot whack-a-mole.

You can list your snapshots in bpool using:-

zfs list -t snap -r -o name,used,refer,creation bpool

This will get you a listing with entries that look something like this (but probably much longer):-

NAME              USED   REFER  CREATION
@autozsys_7lzjmg  197M   197M   Wed Dec 30 11:07 2020
@autozsys_9u20hc   17K   199M   Wed Jan 20  6:56 2021
@autozsys_6c4qgp    0B   199M   Thu Jan 21  6:12 2021

[Note that I have removed the pathname before the “@” symbol to fit the text without line wraps]

As you can see, the “used” and “refer” columns vary widely between the snapshots. The bottom entry (6c4qgp) has a “used” entry of zero bytes; surely that can’t be right, can it?. Well, that’s the whack-a-mole function coming into play. In this particular case, there don’t appear to have been any changes to bpool between autozsys_9u20hc and autozsys_6c4qgp, so if we destroyed 6c4qgp, we wouldn’t get any free space released back to the filesystem. So where does the “refer” come from? That’s letting us know that even though there were no changes between the last two snapshots in our list, they both have references to 199MB of data already being held in other, previous snapshots (or the filesystem itself). What this means for us is that while destroying 6c4qgp may not free up any space, it is very likely that destroying 7lzjmg or 9u20hc will cause 6c4qgp to inherit some of that referenced data, thus causing the “used” count for 6c4qgp to increase and the filesystem not to gain back as much free space as we expected (hence “whack-a-mole”, the used data count might just pop back up elsewhere).

Now, to be fair, because we’re dealing with /boot and the turnover is usually caused by the replacement of kernel/initramfs files, it is more than likely that destroying the oldest snapshot will simply delete the oldest of those kernel-related files, returning about 140MB of free space to the filesystem. Likely, but not certain, so don’t go destroying snapshots left, right and centre without checking their contents first.

LISTING OF /boot CAPACITY AND AVAILABLE SPACE
["df -h"]
Filesystem                 Size  Used Avail Use% Mounted
bpool/BOOT/ubuntu_g1cutr   145M  117M   29M  81% /boot

["zfs list"]
NAME                       USED  AVAIL     REFER  MOUNT
bpool/BOOT/ubuntu_g1cutr  1.52G  28.5M      116M  /boot

["zfs list -o space"]
NAME                      AVAIL   USED  USEDSNAP  USEDDS
bpool/BOOT/ubuntu_g1cutr  28.5M  1.52G     1.41G    116M

You can always check the contents of any given snapshot by noting where the ZFS pool is mounted and then navigating to the .zfs/snapshot directory at that point. So, for our bpool snapshots, we can see from the output of “df” or “mount” that bpool/BOOT/ubuntu_g1cutr is mounted at /boot. Listing /boot/.zfs/snapshot will give us a listing of directories which correspond to the snapshot names in the listing above. You can list each of those directories to see what files and directories are included in the snapshot. As /boot is actually quite small, you can easily do an “ls -i” on two of the snapshot directories and see which files have the same inodes and which are different (which gives a good indication of which files are shared between different snapshots and the current, live filesystem and which are unique to a given snapshot).

Snapshots are removed using the “zfs destroy” command, by the way, not by removing the snapshot directories.

Don’t forget that destroying any snapshot restricts your ability to recover to a known point in time, so I would urge you to err on the side of caution — if you’re not 100% certain of what you’re about to remove, don’t do it!

If you’re a user who likes to create their own snapshots (before a major upgrade, for instance) you might already be able to easily target some of your own snapshots as candidates for deletion (perhaps you already know that you’re not going to roll back to that ancient, previous release?).


The [partial and not very satisfactory] Solution

The stop-gap solutions listed above are just that; short term solutions which will give you some extra breathing space, but not long term fixes. To remedy the bpool problem long term, we obviously need to add a substantial chunk of extra disk space.

One brutal, but simple way of getting back more bpool space (and a solution which is very topical with the release of 21.10 almost upon us) is to re-install Ubuntu after editing the ZFS section of the install script to bypass the problem. I’m not suggesting that this is the best or most versatile answer to the issue (in most cases it won’t be), but if you happen to be on the verge of upgrading, or perhaps have a machine where all of the application and user data is mounted from a fileserver rather than local disk, this may be an option (but, as always, I would recommend a full back-up of the system before taking such a drastic step).


Just in case you didn’t quite get that… ==WARNING== The following steps will delete -ALL- of the data on your disk. Do -NOT- proceed with the steps below if you are not prepared to have your disk(s) totally wiped of all existing data.


Booting from the Ubuntu install image will drop you into the Try-or-Install header page. Select “Try Ubuntu”, which will restart the desktop to the normal, live image. From there you can open a terminal session and become root using “sudo -s” (no password required).

Using whatever editor you’re most comfortable with, open /usr/local/share/ubiquity/zsys-setup.

On (or about) line number 267, you’ll find this code:-

[ ${bpool_size} -gt 2048 ] && bpool_size=2048

You just need to comment out that line completely for the simplest fix. Doing so will allow the bpool size calculation to grab a much larger chunk of your available disk space. Note that if your disk is 50GB or less, this isn’t going to help — you’ll still end up with roughly 2GB. Conversely, if your disk is 1TB or larger, you may end up with much more bpool space than you actually need, or want. In these cases, you might want to change line 267 to use some value other than 2GB; I found 10GB (ie:- replace “2048” with “10240”) to be satisfactory for my machines, but if you happen to be a kernel developer, you might want to bump that up even further.

Following the edit of the ubiquity file, you simply select “Install Ubuntu” from the desktop icon and proceed with the normal ZFS install.


Okay, that’s the gist of the bpool problem, along with a couple of suggestions for clawing back some disk space and the most drastic way of fixing the issue.

In the second part of this series, we’ll be looking at a couple of more reasonable methods of fixing existing installations without resorting to a full re-install (fair warning though — it will still involve booting from the install media, so is not entirely without risk).

And don’t forget …back up early and back up often!


An ESP32 project worth looking at…

…especially if you’re a farmer.

You might have noticed that it has been fairly quiet around here recently. That’s mainly because real life has been getting in the way quite a lot, but also because what little spare time I have has been taken up by a new project. I’m not ready to spill the beans on what that project is quite yet, but you can get a couple of fairly strong hints on what direction I’m going (!) by taking a look at the beautifully executed work of Matthias Hammer (and his daughter) on their autonomous equipment project. Note that Matthias isn’t limiting his work just to autonomous control of the tractor, but is also extending the project to automate the other agricultural equipment that he uses.

Matthias is using ESP32s in his project as replacements for the Arduinos used in the base AgOpenGPS project. He has a few videos up on YouTube giving a guided-tour through the equipment, as well as showing the tractor at work. If you’re at all interested in GPS/RTK and autonomous vehicles, his (short) videos, GitHub repository and the AgOpenGPS web site are all worth a visit (even if you’re not a farmer).


You could have used a 555!

Yup, the most frequent comment on Hackaday, “You could have used a 555 for that”. Well, I did. In fact, I used a pair of 555s …but probably not the sort you’re thinking of.

A pair of 555s

A few years back (see date-code photo, below), I found these batteries in a local equivalent of the “Dollar Store”. They only came in pairs, which made them more than twice as expensive as the normal, no-name AA alkaline cells, but with that branding I had to get at least one pack.

I used them (probably in one of my PIC projects back then) and then forgot about them, until I was scrambling to find some not-quite-so-dead batteries to check an ESP01S project a couple of weeks back and found them at the back of a drawer.

I popped the DVM probes on them (unloaded, of course) and was very surprised to see a healthy 1.5v reading.

Well okay, lets fire this thing up and see what happens! What happened was that the ESP01S blue LED flashed, then …nothing. Turning one of them upside-down, the date code reminded me that it was very many moons ago that I last used them.

"555 - 2013-09"

They certainly didn’t owe me anything, but I wondered what the voltage had gone down to under load. Having had lots of experience with these el-cheapo batteries, I wasn’t at all surprised to see that the voltage across the cell swung well into negative territory when I held down the “On” switch. What still surprised me was that the voltage swung almost straight back to that “healthy” 1.5v reading once the load was removed. Usually these cells are hard-pressed to register a no-load voltage of 0.9 ~ 1.1v in their fully depleted states.

Maybe these we’re just catching their “second wind” …or maybe they do have a 555 hidden inside them after all!

-1.33v displayed on DVM when load applied
…wth load
1.54v reading on DVM from single cell
No load

The ESP01S board wouldn’t stay powered with these batteries in the pack, so I had to hold down the “On” switch long enough to snap the photo.


No ESP01S modules were harmed in the production of this blog post, but it probably didn’t do my poor little 555s much good.