The Ubuntu ZFS boot pool problem [Part I]

Sorry, as you can see from the title, this is another “not-ESP8266” article, so you can skip this if you’re only interested in our magical little wireless modules. However, if you’re running a system (laptop, server or workstation) loaded with Ubuntu and have the root/boot partitions mounted on a ZFS filesystem, you should at least stick around long enough to read through the problem description, below.

If you landed here from a web search for “can’t upgrade Ubuntu – not enough space on bpool” (or something similar), then you’re in the right place. You can probably skip the introduction to the problem (you already know what you’re facing) and go directly to the solution.

Note that in these articles I am assuming that you are familiar with Linux and confident in your own abilities to administer your own system. Because we are dealing with filesystems, I am also going to assume that you have made and verified back-ups of your whole disk(s) before you follow any of the tips below. I am not responsible for your data. You are. The advice below is given in good faith after having been tested on several of my own machines (of different types), but I cannot be held responsible for any loss of data.

The bpool Problem

You see the familiar Software Updater pop-up, announcing that there are package updates available and prompting you to install them. You scan the list of updates and everything looks good, so you hit the “Install Now” button, but instead of a progress meter, you see a pop-up error window with the message:-

The upgrade needs a total of 127 M free space on disk '/boot'. Please free at least an additional 107 M of disk space on '/boot'.

The message continues with a couple of suggestions of how to increase available space in “/boot”, but basically at this point you are unable to install updates.

You’ve just run into the “bpool problem”.


Background

For the last few releases, Ubuntu has provided an easy method for installing root on a ZFS filesystem, giving you access to snapshots and the instantaneous rollback to previous versions which it enables (as well as the possibility of mirroring) without having to go through the long and tedious process of adding ZFS support after the actual install. It was a major move forwards and the “one-click” install process made it simple for anyone to benefit from ZFS technology (not just snapshots, rollbacks and mirrors, but raidz and ZFS send/receive back-ups, too). With all of those benefits, why wouldn’t you just use it by default?

Well, as a long time ZFS user, I can tell you that it does come with a few drawbacks. It is, for instance, somewhat more difficult to judge the available space left on any physical device than it was in the pre-ZFS world (good ole’ “df -h” is very good at giving you an instant and pretty accurate impression of the current capacity of your disks). ZFS tends to muddy that simplistic view a little, until you get used to depending more on the output from “zpool list -v” and “zfs list -o space” instead of “df”.

Fragmentation is another issue. Die-hard Unix users will remember that “fragmentation” was always a problem for people who used that other operating system, not for Unixes (well, unless you started to run out of inodes, anyway). However, with ZFS you do need to keep an eye on that “FRAG” column in the output of “zpool list -v”; once it starts climbing to around the 50~60% level it’s definitely time to start planning for an upgrade and, once it’s past 70% it’s time to take some urgent action (usually, but not always, time to add more physical disk space — we’ll touch on de-fragmentation as a side-effect in the “Solutions” presented in part-II). Once it hits 80%, you’re in big trouble as, even with available free space, the system will be struggling to find usable space.

These issues are a pain to come to terms with after all of these years of doing the odd “df” just to check that everything was okay, but they are still heavily outweighed by the advantages (did I mention copy-on-write or checksummed data yet?).

So, our simple ZFS installation method from Ubuntu is a real game changer and gives everyone the chance to ride on the ZFS bandwagon with minimal effort. Unfortunately, along with that simplicity and ease of installation comes another problem — sizing your filesystems. The installer is a one-size-fits-all deal; it looks at your disk size and makes a couple of fairly simplistic calculations to create a very simple partition table and then later adds a spread of ZFS datasets on top of those physical partitions. The “bpool” (for “boot-pool”) is a single ZFS filesystem which occupies a dedicated partition just for the kernel/initramfs and EFI/grub data required to boot the system. The “rpool” (for “root-pool”) is the ZFS filesystem which contains datasets for /, /var and /usr (as well as sub-directories) and the user home directories. It’s all very well laid out, so that a rollback to (for instance) a previous kernel version need not affect the user’s data. However, the installation script for ZFS currently has one major issue; it limits the size of the bpool physical partition to 2GB (it can be smaller, but not bigger), no matter what the size of your physical disk

The major drawback that people are finding to this installation mode comes from one of the strengths of ZFS — the snapshot functionality. Put simply, if you make a snapshot of say, your whole bpool, ZFS will never delete anything from that point in time. If you subsequently delete a kernel which was present when you took the snapshot, ZFS will keep the file itself, hidden away in the snapshot and only remove the named link to the file from the normal /boot directory. If, at some later time, you decide that you no longer need that snapshot (because it is outdated and you know you’re never going to revert to that old kernel again) you can destroy the snapshot and only then will you recover that disk space. This is one of the reasons that space management on a ZFS filesystem is a bit of a minefield for new users and is exactly why the “bpool problem” exists.

Every time Ubuntu pushes out an update (even a non-kernel update), the installer will automatically create a snapshot of your filesystem, so that you can roll back to the previous version if things go horribly wrong. In the case of actual kernel updates, the snapshot can consist of tens, or even hundreds of megabytes of combined kernel and initramfs files. Given that we have an upper limit of 2GB on the size of /boot (the bpool filesystem), that space doesn’t go very far if you don’t actively manage the snapshots. Worse, the Software Updater will start to fail when there is no longer enough free space in /boot to fit another version of the kernel and, to be clear, even non-kernel updates will fail to be installed if there is a kernel update still in the queue (even if you de-select it manually). The error messages are clear and quite specific (and even give tips on how to ameliorate the issue). However, by the time Software Updater flags the problem, it can already be too late to easily fix it (and how many of us know which system-generated snapshots are safe to remove and which aren’t, anyway?).

Short-term band-aids

These suggestions are not a true solution, they are fixes to get you up and running in order to have Software Updater working again in the shortest possible time.

Before making any changes which might permanently erase data from your filesystems, you should make a back-up of your whole disk (I’ll be repeating this at various points throughout these articles, because I really mean it — I cannot and will not be held responsible for any loss of data you might suffer while trying to follow the tips given here; I always assume that you have backed-up your data and that you have verified that those back-ups work. Batteries not included. May contain nuts.).

First, one quick fix which Software Updater lists in its output when this problem occurs — edit the /etc/initramfs-tools/initramfs.conf file and change the COMPRESS setting from “lz4” to “xz”. This won’t take effect until the next kernel update, but at that point the initramfs install will use the more efficient (compression, not speed) “xz” method. This can slice around 20MB off each of the initrd.img files in the /boot directory and so will save you about 40MB of space on each update (there are usually two versions, current and “.old”).

The results of this second tip are much more difficult to predict in terms of exact space saved — snapshot whack-a-mole.

You can list your snapshots in bpool using:-

zfs list -t snap -r -o name,used,refer,creation bpool

This will get you a listing with entries that look something like this (but probably much longer):-

NAME              USED   REFER  CREATION
@autozsys_7lzjmg  197M   197M   Wed Dec 30 11:07 2020
@autozsys_9u20hc   17K   199M   Wed Jan 20  6:56 2021
@autozsys_6c4qgp    0B   199M   Thu Jan 21  6:12 2021

[Note that I have removed the pathname before the “@” symbol to fit the text without line wraps]

As you can see, the “used” and “refer” columns vary widely between the snapshots. The bottom entry (6c4qgp) has a “used” entry of zero bytes; surely that can’t be right, can it?. Well, that’s the whack-a-mole function coming into play. In this particular case, there don’t appear to have been any changes to bpool between autozsys_9u20hc and autozsys_6c4qgp, so if we destroyed 6c4qgp, we wouldn’t get any free space released back to the filesystem. So where does the “refer” come from? That’s letting us know that even though there were no changes between the last two snapshots in our list, they both have references to 199MB of data already being held in other, previous snapshots (or the filesystem itself). What this means for us is that while destroying 6c4qgp may not free up any space, it is very likely that destroying 7lzjmg or 9u20hc will cause 6c4qgp to inherit some of that referenced data, thus causing the “used” count for 6c4qgp to increase and the filesystem not to gain back as much free space as we expected (hence “whack-a-mole”, the used data count might just pop back up elsewhere).

Now, to be fair, because we’re dealing with /boot and the turnover is usually caused by the replacement of kernel/initramfs files, it is more than likely that destroying the oldest snapshot will simply delete the oldest of those kernel-related files, returning about 140MB of free space to the filesystem. Likely, but not certain, so don’t go destroying snapshots left, right and centre without checking their contents first.

LISTING OF /boot CAPACITY AND AVAILABLE SPACE
["df -h"]
Filesystem                 Size  Used Avail Use% Mounted
bpool/BOOT/ubuntu_g1cutr   145M  117M   29M  81% /boot

["zfs list"]
NAME                       USED  AVAIL     REFER  MOUNT
bpool/BOOT/ubuntu_g1cutr  1.52G  28.5M      116M  /boot

["zfs list -o space"]
NAME                      AVAIL   USED  USEDSNAP  USEDDS
bpool/BOOT/ubuntu_g1cutr  28.5M  1.52G     1.41G    116M

You can always check the contents of any given snapshot by noting where the ZFS pool is mounted and then navigating to the .zfs/snapshot directory at that point. So, for our bpool snapshots, we can see from the output of “df” or “mount” that bpool/BOOT/ubuntu_g1cutr is mounted at /boot. Listing /boot/.zfs/snapshot will give us a listing of directories which correspond to the snapshot names in the listing above. You can list each of those directories to see what files and directories are included in the snapshot. As /boot is actually quite small, you can easily do an “ls -i” on two of the snapshot directories and see which files have the same inodes and which are different (which gives a good indication of which files are shared between different snapshots and the current, live filesystem and which are unique to a given snapshot).

Snapshots are removed using the “zfs destroy” command, by the way, not by removing the snapshot directories.

Don’t forget that destroying any snapshot restricts your ability to recover to a known point in time, so I would urge you to err on the side of caution — if you’re not 100% certain of what you’re about to remove, don’t do it!

If you’re a user who likes to create their own snapshots (before a major upgrade, for instance) you might already be able to easily target some of your own snapshots as candidates for deletion (perhaps you already know that you’re not going to roll back to that ancient, previous release?).


The [partial and not very satisfactory] Solution

The stop-gap solutions listed above are just that; short term solutions which will give you some extra breathing space, but not long term fixes. To remedy the bpool problem long term, we obviously need to add a substantial chunk of extra disk space.

One brutal, but simple way of getting back more bpool space (and a solution which is very topical with the release of 21.10 almost upon us) is to re-install Ubuntu after editing the ZFS section of the install script to bypass the problem. I’m not suggesting that this is the best or most versatile answer to the issue (in most cases it won’t be), but if you happen to be on the verge of upgrading, or perhaps have a machine where all of the application and user data is mounted from a fileserver rather than local disk, this may be an option (but, as always, I would recommend a full back-up of the system before taking such a drastic step).


Just in case you didn’t quite get that… ==WARNING== The following steps will delete -ALL- of the data on your disk. Do -NOT- proceed with the steps below if you are not prepared to have your disk(s) totally wiped of all existing data.


Booting from the Ubuntu install image will drop you into the Try-or-Install header page. Select “Try Ubuntu”, which will restart the desktop to the normal, live image. From there you can open a terminal session and become root using “sudo -s” (no password required).

Using whatever editor you’re most comfortable with, open /usr/local/share/ubiquity/zsys-setup.

On (or about) line number 267, you’ll find this code:-

[ ${bpool_size} -gt 2048 ] && bpool_size=2048

You just need to comment out that line completely for the simplest fix. Doing so will allow the bpool size calculation to grab a much larger chunk of your available disk space. Note that if your disk is 50GB or less, this isn’t going to help — you’ll still end up with roughly 2GB. Conversely, if your disk is 1TB or larger, you may end up with much more bpool space than you actually need, or want. In these cases, you might want to change line 267 to use some value other than 2GB; I found 10GB (ie:- replace “2048” with “10240”) to be satisfactory for my machines, but if you happen to be a kernel developer, you might want to bump that up even further.

Following the edit of the ubiquity file, you simply select “Install Ubuntu” from the desktop icon and proceed with the normal ZFS install.


Okay, that’s the gist of the bpool problem, along with a couple of suggestions for clawing back some disk space and the most drastic way of fixing the issue.

In the second part of this series, we’ll be looking at a couple of more reasonable methods of fixing existing installations without resorting to a full re-install (fair warning though — it will still involve booting from the install media, so is not entirely without risk).

And don’t forget …back up early and back up often!


Will it mirror?

Pic of (old) Laptop with SSD attached to lid

Here’s another silly one for you. What do you do if the latest release of your OS of choice ships with ZFS, but you don’t have space in your laptop for a second disk? …Answer:- Reach for the velcro.

This Sony Vaio has a Centrino Core-2 p8600 processor, so it’s not going to break any speed records, but it works well enough for day to day use. Courtesy of the Buffalo 500GB SSD taped to the lid, it now sports 1TB of disk space (500GB mirrored), which is probably well in excess of what the designers originally envisaged.

Centrino.2 sticker next to USB ports

This is one of those little “because I can” projects which I don’t necessarily recommend to anyone else, but at the same time, the lightness of an SSD compared to a normal hard-disk (even a 2.5″ one) means this is now an eminently practical solution if your old laptop happens to be running out of space (I do move the laptop around the house to work in different rooms at different times, but it generally doesn’t travel much further afield than a deck-chair out on the veranda).

So, will it mirror? Heck yes!

Should I mirror?

No, probably not. It’s much more sensible to use a periodic ZFS send/receive job to back up your work to an existing server, that way you don’t need to worry about the extra drain on your battery and you still have your work if your laptop is stolen or knocked off the deck of your yacht, mid-ocean (what, you mean that’s never happened to you?).

One other consideration when thinking about installing Ubuntu on ZFS — currently (as of 21.04) Ubuntu will not allow you to edit the size of the disk partitions; you must accept their optimized defaults. Unfortunately , those defaults include a /boot partition which is much too small (typically 2GB). It will work fine for a couple of months, but with every apt update, the system will automatically add a snapshot of the boot partition. When the upgrade includes a kernel update, this means that tens, or even hundreds of megabytes of storage can be used. Even when you set the system defaults to compress the kernels using “xz”, it doesn’t take too many updates before you start getting “not enough free space” messages from apt and it will refuse to continue with the update. This is not something a novice user can easily recover from (hint: deleting files on a ZFS partition doesn’t always return that free space to the system — it all depends on whether it is still being held by a snapshot).

T-bao R8 15.6″ $185 Laptop – Installing Linux

UPDATE  —  November 2019    As an updated preface to this article, I’d just like to note here that Ubuntu 19.10 was released quite recently and works really well with this laptop.  Rather than ploughing all the way through this article (which you can still do if you really want to) and jumping through the multiple hoops presented here, I’d recommend instead that you go to the download page for Ubuntu Mate and download 19.10 (or anything later).  Sound, Bluetooth, WiFi, etc will just work. You can also install root on ZFS on your eMMC (normally, I wouldn’t recommend installing ZFS on a single disk system …it’s not really what it is intended for, but in this case it can be a really great introduction to the ZFS “snapshot” functionality at no extra cost).


The T-bao R8 comes with Windows 10 preloaded on the internal, 64GB EMMC.  I’m not going to say anything more about Windows 10 as I’m completely unable to drive it (I just don’t “get” the Windows paradigm) and am therefore unqualified to give an informed opinion, one way or the other.  I intended from the outset that this would be a replacement for my existing, ailing laptop, which runs Elementary OS, a very usable version of Linux.

Anyway, the version of Elementary I’m running on my current laptop is fairly long in the tooth now (it’s based on Ubuntu 12.04 LTS), so it was time to upgrade the OS, too.  So, what we’re going to do here is:-

  • Break into the BIOS on the laptop and do some simple set-up.
  • Download a known, good ISO from the `net.
  • Boot the ISO image on our laptop.
  • Optionally back-up the pre-installed Windows-10 partition.
  • Install Linux.

What I discovered previously, with the Z8350-based Z83-II mini-PC (and the T-bao and Chuwi laptops are the same), was that the BIOS was strongly biased towards a Windows peculiarity where a native 64-bit system uses a 32-bit bootloader at switch-on.  This quirk, together with the UEFI requirement, means that many “UEFI enabled” distributions won’t actually boot on this device, as they only provide a 64-bit bootloader.  The answer to this boot problem for Atom equipped machines has been provided by Ian Morrison over at Linuxium in Australia.  Ian has developed a script, isorespin.sh, which will take the normal distribution ISO image file from several of the most popular Linux variants, unpack the ISO into a temporary directory structure, fix the bootloader, upgrade the kernel and add in required drivers for this type of machine (WiFi adapter and mouse-pad) and then roll the whole thing back up into a new ISO file which you can then dd to a USB thumb-drive to use as a combined bootable live demo and installation drive.  Ian has put a ton of work into this handy script and into re-spinning some ready-to-use example ISOs, so if you use his work, please be kind and buy him a beer.

Going back to the laptop, there are a couple of things you need to do and a couple of things you need to know to prepare for your install.  The BIOS on this laptop advertises itself as being a version from “American Megatrends”, but it has a very limited number of changeable settings.  You can get into the BIOS menu by hitting the escape key within a couple of seconds of powering on.  Do that first, go into general settings and set the “Num Lock” to “OFF” (if you don’t do this your keyboard will appear to be broken when you go into the UEFI shell, for instance).  You should also change the “Boot Delay” setting from “2” (seconds) to “5”, to give yourself a chance of reading any text displayed during the start-up process.  You may also find it helpful to change the boot messages option from “Quiet” to “Verbose” (which will give you the American Megatrends BIOS help messages at switch on).  Save those new settings and reset.


-IMPORTANT-

The boot device menu will not display any device that doesn’t have a 32-bit UEFI bootloader file.  This means that if you plug in a USB thumb-drive with (say) a standard 64-bit Ubuntu distribution on it, not only will the distribution be ignored, but the thumb-drive itself will not show up in the menu.  The same holds true for an external DVD-drive or any other bootable device.


Here’s where I wasted a lot of time during my initial attempts at building my chosen distribution.  I downloaded Ian’s script and ran it on my old Inspiron in an effort to build a bootable ISO for the new laptop.  I gained a lot of experience of running isorespin.sh from the command line, but never managed to produce an image which was bootable on the new laptop.  Strangely, all of Ian’s pre-built ISOs which I tried booted perfectly, so there was something about my ancient version of Linux (or just my ancient laptop) which didn’t cut the mustard in this case.

Anyway, to cut a tedious experience short, what I ended up doing was using one of Ian’s pre-built Ubuntu images to do a quick install on the T-bao R8 and then using the laptop itself to run isorespin.sh on the latest Elementary OS ISO to produce a bootable, working USB thumb-drive to do the “real” install.  This method worked perfectly first time. You don’t need to do this unless you want to install Elementary or some other version which Ian doesn’t have a pre-built ISO for.  The instructions below assume you’re doing a simple, straight install from Ian’s pre-built Lubuntu image (find the Lubuntu section and link to the image about 1/3 of the way down this page).

Load the image onto a USB thumb-drive (using dd) and connect it to your T-bao (L/H side connector is the USB-3.0 socket).  Power on and hit to get the boot selection menu and boot from USB.  Select the USB thumb-drive (if it’s not on the menu then, as per the info above, there’s something wrong with your download or copy procedure) and hit , followed by “Try Lubuntu without installing”.  Your machine should boot into the GUI within a couple of minutes.  At this point you should be able to play around with Lubuntu and get a feeling for how the keyboard and mouse perform before starting the actual install.

Once you’re comfortable with the keyboard and mouse, you can select “Install Lubuntu” either from the “System Tools” menu or by clicking on the install icon on the desktop.

A small warning here before you start the install …if you’ve chosen one of Ian’s other image files which has a “persistent” save area enabled on the thumb-drive and then have finger trouble when you enter (say) disk partitioning information, that same (bad) information will show up again the when you reboot from the thumb-drive.  It’s generally easier to reload the original, clean ISO image to the thumb-drive again, rather than try to work around it.

Note:-  The “disk” in the R8 is actually a 64GB eMMC memory module.  This is not the same as a SSD and is just one more of the compromises made when producing a budget laptop.  I found the eMMC on the Z83-II mini-PC to be quite slow and, although the T-bao unit isn’t a walking-through-waist-deep-treacle-nightmare, it still doesn’t break any speed records.

The Lubuntu install ISO which I linked to above will do a very good job of installing Linux onto the eMMC with the existing Windows-10 intact, without needing any special input from you.  In that case though, you’ll only end up with about 20GB of disk space dedicated to Linux.  You can choose to delete the Windows data and dedicate the whole disk to Lubuntu to get the whole 57-odd-GB of actual available disk space.

One other option you have while booted from the live ISO on the thumb-drive and before starting a Linux install, is to do a full copy of the eMMC drive using “dd” to another machine across the network (or to a local, USB disk drive), thus allowing you the option of restoring the machine to its default (Windows-10) state at any time in the future if, for instance,  you should decide to give it to your nephew for his school work.  This will take a bit of time (probably a couple of hours, depending upon your network and the target machine) and will use roughly 6GB of disk space on the remote machine if you use “xz” as the compression method (more for gzip, bzip2 or others).  If you have a very powerful remore server (muli-core CPU) you can compress remotely, otherwise I’d advise compressing on the laptop to get the added advantage of passing less data over your network.  A sample command (local compression on the laptop) might look something like this:-

dd   if=/dev/mmcblk0   bs=4M   |    xz   -T2   -c   |   ssh    UserName@SmallServer   "(   cd   /BigDisk/LotsaSpace   &   cat   ->   T-bao-Laptop_Windows-10_bkup_mmcblk0_dd.xz   )"

If you have some major iron on the server side and want to do the compression there instead, it might look like this instead:-

dd   if=/dev/mmcblk0   bs=4M   |   ssh    UserName@BigServer   "(   xz   -T4   -c   |   cd   /BigDisk/LotsaSpace   &   cat   ->   T-bao-Laptop_Windows-10_bkup_mmcblk0_dd.xz   )"

In the first example, the “-T2” option to the xz command tells it to use two threads for compression and the xz command itself comes before the ssh section and so is running on the laptop (a four core machine).  In the second version the xz command has moved into the ssh section and is being executed on the remote server.  Note that in this case the option has changed to “T4” to use four threads on that (imaginary) eight core machine.  In both cases the command is run as root on the laptop and the remote user “UserName” needs to have write permission in the /BigDisk/LotsaSpace directory on the remote server.

Installing from the live image on the thumb-drive after this doesn’t really require any explanation (as the prompts are informative and I’m also pretty sure you’ve already installed Linux once or twice yourself already).  The only place you need to pause and consider your answer is at the disk partitioning prompt; I don’t have any qualms about wiping Windows completely, but you might.

The actual install progresses very quickly and, as far as I can tell, selecting “download updates during install” doesn’t have any untoward effects on the (already updated) kernel and driver files from Ian’s respin (but please do let us all know in the comments if your experience differs from mine).

I had planned to limit the eMMC use on this machine purely to the initial boot of the kernel and have the laptop mount the live filesystems via NFS from one of my servers.  However, to date I’ve been unable to get the system to run reliably with even just /home mounted via NFS (the window manager appears to freeze, but shells on alternative console devices with, for instance, CTRL-ALT-F2, seem to carry on working).  This is obviously directly related to NFS traffic, as triggering multiple read/write operations on the /home filesystem with a non-trivial compile will reliably cause the hang.  “nfsstat” on the server shows an initial burst of activity which immediately gives way to just a single read operation (by the client) every couple of seconds.  Right now I’m stalled on this issue.


For anyone interested, the output from “dmesg” on the T-bao R8 running ElementaryOS 4.1 is available here.


FreeBSD Hint  —  If you’re trying to boot FreeBSD on one of these boxes and experience a hang at (or very shortly after) the point where the kernel discovers the keyboard (atkbd0),do a reset and then select option #3 (“Escape to loader prompt”) from the FreeBSD boot menu and type in these lines:-


set hint.uart.0.disabled=1
set hint.uart.1.disabled=1
boot

Once the system boots up, add the same fix into /boot/device.hints (without the “set”) to make it permanent:-


hint.uart.0.disabled="1"
hint.uart.1.disabled="1"


More issues (as of 27th Oct 2017) — Well, I’ve wasted more hours, done more Linux installs and tried out more different distros  over the past month than I care to remember.  I have a couple of major problems which I can’t, so far, solve.  The first is to get Bluetooth working reliably and the second is to get sound working at all.  These sound fairly trivial, but together mean that radio, video and on-line communications channels like Line or Skype are all completely useless.

Important Note  —  The audio chipset on this (T-Bao R8) laptop is the bytcr-rt5651 (as opposed to the es8316, which seems to be common on the earlier Chuwi versions), which currently has very little support, even in the latest 14-rc6 kernel.  You can, however, stop the syslog file from filling up with millions of “no audio driver available” messages by downloading the UCM files for this chipset from https://github.com/plbossart/UCM/tree/master/bytcr-rt5651 and following Pierre’s straightforward installation instructions (in the README.md).  This will also cause the rt5651 to show up in the audio control panel (under settings->sound), although the actual sound output will still not work.

In addition to sound and Bluetooth, there are a host of other issues which make life with this laptop not just frustrating, but actually quite unpleasant when trying to use it on a daily basis:-

  • The WiFi reception is a lot less sensitive that all of the other devices we use in the house.  WiFi drops out completely in places which provide a “strong” signal on other laptops or tablets.
  • WiFi regularly just stops completely (even when there’s a signal shown in the toolbar) and usually won’t come back unless it is toggled off and back on again.  This is exacerbated by heavy network traffic (like downloading yet another distro).
  • YouTube and other videos play at several multiples of normal speed, so that it’s actually impossible to read subtitles (which you have to use, because there’s no sound).  It’s interesting that some people have reported that fixing the sound also fixes the frame-rate problem with their Chuwi laptops.
  • The SD card slot driver seems to be slightly weird, in that plugging in a card produces an error message (“SD card error -110 on initialization”) and the card remain inaccessible.  Various cards with various formats all produced exactly the same error.
  • The trackpad gets very confused with two-finger scrolling and regularly interprets a scroll as a zoom command, or as a one-finger button touch.
  • The trackpad isn’t accurate enough to allow scrolling on windows which have thin scroll-bars (which seems to be the rage, nowadays).
  • The trackpad will not respond to setting changes such as “side scroll enable”, “natural scroll disable” or “tap disable”, regardless of distribution.
  • Occasionally, the laptop goes into sleep/hibernate (on inactivity, or closing the cover) and then won’t restart, requiring a power-cycle to recover.

Of course I’m aware that some of these problems may be distribution related, but the ones mentioned above seem to be fairly consistent across multiple different versions of Debian/Ubuntu and Arch based distros.

On the plus side, I’m really grateful that the machine boots so quickly (roughly 30-seconds for a distro loaded on the internal eMMC).