Grub bites man. Man stamps on grub. ‡

‡ — No invertebrates were harmed during the production of this article.

TLDR:-  Skip straight to the updates at the bottom of the page for links to the bug reports.  The short version is “You can blow away any root-on-ZFS Linux system by enabling zstd compression on bpool (or whatever the actual boot partition is called on your particular distribution)”. Do NOT do this (not even to test it); you will regret it!

Almost exactly two weeks ago I got a (normal) pop-up message on my screen, letting me know that there were updates available from Ubuntu for my 21.10 version of Mate and that, by the way, 22.04 was now available if I wanted to upgrade.  I politely refused the upgrade offer, but accepted the updates.  Once the updates were completed I was prompted to reboot (and I noted that the initramfs had been updated).  This is all normal stuff for the millions of us running the various versions of Ubuntu, so after finishing up what I was working on (and washing my hands) I hit the reboot button.

It didn’t.

It hasn’t for the last two weeks (not without a recovery image or an external disk plugged in, anyway).

At first I wasn’t too concerned.  This (as the ‘mericans say) isn’t my first rodeo and, if everything else failed, I’m using ZFS, so I just had to roll back to the last good snapshot, right?

Wrong!

As I wasted more and more time futzing around with things, I just got more and more confused. The EFI boot was working and grub was being run, but it insisted that it couldn’t find a workable filesystem.  Booting into a live-CD image said there were two perfectly good ZFS pools on the disk (actually a 512GB NVME SSD) and a “zfs scrub” passed with flying colours on both of them (bpool and rpool).  Running grub-probe always came back with a message along the lines of “Nope, ain’t nuthin there that I recognize, pinhead!”.

By this time I had convinced myself that someone had broken into Ubuntu’s update servers and slipped a Trojan into my disk headers, or maybe someone was slipping magic mushrooms into my breakfast tea, or aliens had abducted my real ZFS pools and replaced them with zombies.  Anyway, I needed to get some work done, so I found an unused external USB disk, fired up a live-CD again and, because this is ZFS, did a ZFS send-receive between the original disk and the external.  When it came time to run the grub install (on the external disk) it once again insisted that there was nothing at all there which it recognised as any sort of valid filesystem.  Duh!

Okay, nuts to this.  I put a 22.04 live-CD image in, booted it up (“Oh look, you have two perfectly good ZFS pools on this external disk!  Are you really sure you want me to scribble all over them?”) and installed a brand spanking new version of 22.04 onto the external, followed by a ZFS send-receive of just the USERDATA filesystem from the original NVME drive (I’d like to say that I rebooted and everything sprang into life on the external disk, but of course it didn’t.  You can’t easily remove an NVME “disk” from a laptop and of course EFI and grub just kept right on trying to access it).

Eventually, I did manage to broker a truce between the laptop BIOS, EFI and grub, to the point that I could reboot the system successfully, as long as I was actually there to hit the F9 key and manually select the EFI entry for the external disk  …otherwise it would stubbornly continue with a blinky dance of reset, spin disk up, flash something on the screen for 10ms, spin the disk down and repeat.  I never did manage to read what that 10ms micro message was (something from those dang aliens again).

But glory hallelujah!!  It was fantastic!  Not only did I have Mate back again, but the stupidly useless ELAN touchpad on the HP S15 laptop actually worked properly for the first time ever!!  Banzai!!  Well done 22.04!  I’m sorry that I refused your offer of an upgrade at the start of all of this.

Well, long story longer, that’s the way that things have stayed for the past two weeks.  Not that there hasn’t been great gnashing of teeth (not recommended at my advanced age) and thrashing of Gewgull searches (and even Ubuntu’s bug database) for any hints of what could be causing my problems. Searches including both “grub” and “aliens” turned up some interesting stuff, but it seems that I’m the only semi-sentient being-man-thingy in the universe who is having problems with those things in connection with computers.

Eventually, the third neuron in the second bank started firing occasionally and I began to have flashbacks of the horrible problems I had a while back, trying to do ZFS back-ups between my Linux laptop and the mirrored disks on my FreeBSD servers.  Although the FreeBSD servers could send data to a receiving  Linux system, the other way around just would not work at all.  It turned out that there were incompatibilities between the ZFS properties used between the two systems, even though the ZFS versions were meant to be the same (upgrading the FreeBSD servers to version 13.1 seemed to cure that issue).  Now it seemed probable that there was some similar incompatibility between grub and ZFS  …and, finally, it clicked. 

One of those property updates which hit FreeBSD a while back was the introduction of “zstd” compression (Ubuntu introduced it a short while later).  I had tried changing the compression to “zstd” on FreeBSD and noticed a worthwhile increase in compression-ratio with no apparent difference in speed, so I’d made the changes on my laptop, too.  The back-ups kept on working and I was saving more disk space …what’s not to like?

Double-duh!  Pinhead at work (stand clear!).

Okay, so I have (I suspect, anyway) a philosophical disconnect between grub and ZFS about what constitutes a filesystem.  In my book, ZFS wins (whatever grub says, I’m an ex-Sun guy and old loyalties die hard).  What to do now?  Well,back on Gewgull I replace “aliens” with “alternative” and get a few hits on something called “ZFSBootMenu”.  I go to their GitHub repository and start reading.  Nope, I must have left “aliens” in there somewhere, ‘coz I can’t understand this stuff.  It starts off more or less okay (I understand their reference to FreeBSD’s boot environments …a very handy feature,  sign me up), but then they start wandering off into the weeds with things like “fzf” and “dracut” (which sounds like a bad day at the vasectomy clinic) and I want to stop reading so badly that my teeth hurt (told you not to gnash).  I briefly leave the README page and head over to the pre-built releases instead. 

Ah, this is interesting.  “ZFSBootMenu v2.0.0  …New features – Dracut is now optional;”.  Phew!  Well that’s a relief.  Unfortunately, there are four different files (and additional source tarballs), but no information on what the differences are.  A couple have “release” in their names and the others have “recovery”, but no actual information in the v2.0.0 section about what the differences are or how to use them.  Two are compressed tar files (okay, I know what those are, anyway) but the two others have a “.EFI” suffix, which I haven’t come across before, but it’s not too much of a stretch to suspect that they should be used in the Extensible Firmware Interface (that’s the BIOS to you, sonny!).  But how?

Back to the README:-

“Each release includes pre-generated images (both a monolithic UEFI applications [sic] as well as separate kernel and initramfs components suitable for both UEFI and BIOS systems) based on Void Linux. Building a custom image is known to work in the following configurations…“

Well there you go then.  C’mon!  Chop-chop!  Get on with it!

I won’t bore you with the rest of it, but despite the rambling (or partly missing) documentation, it turns out that ZFSBootMenu is really quite a nifty bootloader (grub alternative) specifically for ZFS filesystems.  It does have a couple of quirks, but after a few false starts, I did manage to bring my 21.10 NVME SSD back from zombieville and ended up with a laptop which I could once again disconnect from all external cables and disks and still have it boot reliably.  To do ZFSBootMenu full justice I need to put together another post with a step-by-step guide of how to recover a stuffed ZFS machine, but just in case you’re stuck up this same creek without a paddle, here are just a couple of essential hints to getting ZFSBootMenu to work:-

• Use the .EFI  file (I used the “release”, but “recovery” should also work okay).
• –Remember– When you’re booting using ZFSBootMenu, it assumes that the kernel and initramfs are on the root partition, so you’re going to be booting from “rpool”, not “bpool” (and so your kernel and initramfs need to be present on “rpool”, which is not standard in Ubuntu).
• Disconnect any other disks which you may have added to the system in previous attempts to fix this boot problem (ie:- external USB disks with copies of your original bpool/rpool filesystems).  They will just confuse EFI, grub and you.
• Set “canmount=noauto” on all filesystems which have “mountpoint=/” set (this one is really important).
• Make sure that your ZFS pools can be imported without having to use “import -f” (use a live-CD or ZFSBootMenu itself to import the pools manually, fix any problems and then export them again, before you reboot).
• Lastly, for those others of you out there with non-US keyboard layouts, the “Alt” key for command mode selection within ZFSBootMenu will probably not be the “Alt” key.  For my Japanese keyboard, the magic key turned out to be the one key which I had never before pressed on my keyboard …the (shudder!)  “Windows” key.

And the bottom line… Just don’t change your bpool to zstd compression.  Okay?!?    †  ∇

---

†  —  From a cursory≅ check of the Grub2 code in the upstream Debian repository, it looks as though the zstd compression library was added in November of 2018.  Unfortunately it seems to have been implemented only for BTRFS, not for ZFS.

≅  —  “cursory”  Adjective.  Causes the reader to leap out of their chair and shout “G’dammit!!”  (this answer is certain to get you an “A” in GCSE English).

∇  –  Grub Compatibility  —  As usual, the folks at OpenZFS/Debian/Ubuntu are way ahead of me.  If you need to create a new pool which will work with Grub2, there is already an existing compatibility file (they’re stored in /usr/share/zfs/compatibility.d).  You can use it like this:-  

   zpool create -o compatibility=grub2  <POOL NAME>

If anyone is desperate enough to have read all the way down here to the bottom, I’ll just note that this problem hasn’t (has …see below) been submitted as an actual bug, because the auto-submission procedure won’t let you submit a bug against a program which isn’t installed on your system and, of course, I’d removed the grub package and used ZFSBootMenu to actually get my laptop working again. However, the secondary suggested method of opening the issue as a question in LaunchPad is available here if you’re interested (and I’d like to thank Manfred Hampl for trying to help me out there).

Update — 27th July 2022 – I have submitted bug request #1982897 with a condensed description of this issue and a request for an upstream addition of “zstd” support for ZFS in the Grub2 code.

Update #2 — 27th July 2022 – Submitted bug #62821 as a feature request for the addition of “zstd” support to the Grub2 ZFS code.

Update #3 — 13th August 2022 – Updated the Grub2 bug report with a little more detail on the severity of the problem, as there hasn’t been any acknowledgement of the original submission as yet.

Z83-II Mini-PC (Quad core 8350)

Another departure from the ESP8266, but for something that ESP8266-hackers might find interesting.

---

Update Mar 12th 2020  —  The most recent deals that I’ve found on Intel-based mini-pc boxes are here.

Update Aug 23rd 2019 —  Tanix (the TV-box maker) has just started selling its own version of the 4GB/64GB Z8350, designating it the “TX85”.  With both HDMI and VGA connectors and what seems to be an improved ventilation system, it has started showing up on most of the common techie retail sites.  I just bought two from the Shenzhen BoxKing Technology Company for $87 each, plus shipping.  The seller took a few days to mark them as shipped, but they arrived just four days after that.

Update Aug 6th 2019 —  As Z8350 afficionados will know, the “Z85” version of this popular little box is generally regarded as the “big brother” of the version described in the body of this article, as it comes with an added VGA port and (usually) 4GB of RAM and a 64GB eMMC.  Well you can currently get the Z85 from this seller on Aliexpress in the 2GB/32GB configuration (definitely –not– recommended for Windows 10) for just $78.16 including free shipping (or $86.19 for the 4GB/64GB version).

Update June 22nd 2019 —  !!EUROPE ONLY!! —  A very similar box (Z8350 but with 4GB/64GB and VGA/HDMI video connectors) is now selling for €65.99 (~$74) at Cdiscount (France).   If you’re not in Europe and don’t have a plug-adapter to hand, you can still get an updated 64GB eMMC version with dual video outputs (one VGA, one HDMI) for $99.99 (plus shipping). 

Update #2 June 22nd 2019  —  If you’re interested, a Celeron J3160 system (that’s the Z8350’s big brother) is currently on sale, too.

Update May 22nd 2019  —  !!EUROPE ONLY!!  —  …and if you’re interested in a laptop (admittedly, not a very good one) based on this same quad-core chip, CDiscount currently have another one of their insanely cheap offers on the “Yepo” version of the T-Bao R8 (which I played with a while back:-  I-Initial_Impressions  II-Installing_Linux).  Again, available only to residents of the countries listed above, but for a mere €67.80 (about $76).

P.S. – Change your GearBest password now!.

---

Over the past couple of years I’ve been ditching most of the big, ugly, power-hungry desk-top and server machines which used to run my mighty empire (aka “the house”) and replacing them with machines that are a lot smaller and a great deal less power-hungry.  I’ve got a few SBCs of various sorts; all ARM based, mostly multi-core and almost all still shamefully naked and gathering dust, even though they’re alive and kicking and running various essential services on our home network.  In addition to the lack of a case, almost all of them have other drawbacks of some sort (not well supported by the operating systems which I prefer to run, missing critical hardware, etc), so I keep looking for the perfect mini system.

I thought I’d found it a while back when I splashed out on a SolidRun CuBox-i4.  It’s small (very small, in fact), has a quad-core processor, 2GB of memory, GbE, USB ports, HDMI and (a major plus) comes in a diddy little black-plastic case.  Unfortunately, it turned out to be a major pain in the proverbial trying to install and run a reasonable operating system and lots of hoops needed to be jumped through (in the correct order) to even start the OS installation.  It has been running FreeBSD for quite some time now, but it suffers greatly from the lack of a real-time-clock (and unlike a Raspberry-Pi, for instance, it’s not easy to add one), has limited USB functionality and tends to fall over in a messy heap at reboots if the moon happens to be full or the wind is coming from the east.

So the quest goes on for a truly great (cheap) mini system and, right in the middle of the purple Thursday (or was it cerise Saturday?) sales, I found a new system on GearBest that seemed too good to miss.  Because it was on sale, it was a few dollars more expensive than normal (???) , but it was still a pretty good price, given the specs.

It’s an Intel quad-core Atom X5-Z8350 based system, with 2GB of main memory and 32GB of eMMC.  It has GbE, two USB-2.0 ports and one USB-3.0 port.  There’s also an SD-card slot and an HDMI video connector.  For those who care (and I don’t), it also comes pre-loaded with Windows-10 (don’t ask me for more details on what specific version, because I don’t know).  It’s a 64-bit system (but read on for the limitation on that) and it also comes with built in Bluetooth and a/b/g/n/ac WiFi (neither of which I’ve used).  More importantly for me, it’s also a fanless system.  It is advertised as coming with an “English User Manual”, but you can save yourself a few seconds of “I don’t believe this crap!” frustration by throwing said “manual” straight into the bin; there is nothing in it which is even remotely useful.  It does come with a fairly substantial looking mounting bracket (to attach it to a monitor, or wall) and two HDMI cables — one short and one long(er).  A power adapter is also included (but you need to make sure that you order the correct version for your particular country).  The shipping box is also very sturdy and should hold up well if a passing elephant accidentally treads on it.

I would have been quite happy to boot into some sensible operating system and immediately wipe Windows from the disk (eMMC), but it was not to be.  The one, big gotcha with this system is the BIOS.  At first glance, the BIOS menus are very sparse indeed and you’re not going to get very far at all with this system unless you know the magic incantations (it may also help to have a spare chicken to sacrifice …I haven’t actually tried this yet, though).  I did get through quite a few blank CDs and DVDs while trying to get various distributions to boot from a USB-connected DVD.  No matter what I did though, the dang machine just kept on booting into Windows (which was quite disconcerting at first, as I couldn’t see how to shut it down).  Some ‘net searches turned up suggestions for adding BIOS passwords (there are two in the version of BIOS which my machine has) to trigger extra BIOS menu items — Nope, doesn’t work.  You can forget that.  There was also one quite long diversion where some folks (seemingly correctly) noted that some functions which affect BIOS start-up are only available from within Windows, despite how stupidly chicken-and-egg-ish this seems to be (for those who want to try it, you need to select the shutdown function from the start menu and then hold down the shift key while selecting “shutdown”.  This will bring up a whole scad of extra menus which will change the UEFI boot settings …and then it will just merrily boot straight back up into Windows again).

It turns out that the UEFI boot  (which stands for “Unified ‘Effing Firmware Interface”, by the way) is the key to all of this.  It’s a versatile system which is replacing the outmoded BIOS of years gone by with an updated and secure method of booting you into Windows …and booting you into Windows …and booting you into Windows (you get the picture), no matter what you actually want to do.   On some systems (and yes, the Z83-II is one of them) it has the added twist of limiting a 64-bit system to booting from 32-bit code (you’d better go back and read that sentence again …I wrote it and I still can’t quite grok  it!).  For some reason (lost in the mists of Redmont),  Windows was/is limited to booting from 32-bit bootloaders, so because everyone-and-his-dog only ever runs Windows, some machines (like the 64-bit, Z83-II) have a UEFI which will –only– handle 32-bit bootloaders.  Now before you start jumping up and down with your hand in  the air shouting “Teacher!  Teacher! I know!  Just  run a 32-bit OS!” you should also note that, because UEFI is new-ish and 32-bit systems are just so last decade, very, very few 32-bit distributions of any sort actually have any UEFI boot capability bundled into them at all.

I know the mantra is “never give up!”, but by this stage my fingers were twitching and the old grey-matter was sending signals to my hand to grab hold of that handy, 3kg lump-hammer and send the Z83-II and its UEFI back to meet its maker in very, very small pieces.  It was only the nagging annoyance of those extra couple of dollars for the maroon Monday “sale” price that held my arm in check.

By this point I’d read so many posts and articles on the UEFI and booting systems that I’d actually stumbled across one piece of related (and essential) information which actually worked with the Z83-II.  I’d discovered already that hitting ESC during initial, power-on boot would bring up the BIOS menu, but in a couple of posts related to other mini-pc systems people mentioned that F7 would bring up the boot-device selection menu.  It worked, too.  Not only did I get a device menu (what the system had detected at boot up, rather than the BIOS preferred-device list), but it also gave a tiny bit of extra information about what the system thought it had detected on each particular device.  For the most part, to begin with anyway, that information was limited to “Windows boot manager” on the eMMC device.  This turned out to useful, though.  It quickly became apparent that having a DVD in the drive with something that the BIOS (and, presumably the UEFI) could read would result in that entry in the menu changing to give the maker’s name and model, instead of just a vanilla “USB CD/DVD” entry.  Unfortunately, very few of those DVDs would boot successfully, but at least it meant I didn’t have to go through the tedious process of having the system boot into Windows, yet again,  if there was obviously no readable disk in the drive.

I’d read in a couple of places that the latest Debian versions (greater than 8.0) had solved this 32/64-bit problem, so I burned a couple more DVDs and tried them.  I could get the extra info in the menu, but they wouldn’t boot.  A couple of other mainstream Linux distributions were also flagged (in Distrowatch) as being fixed, but none of the ones I tried actually booted.  Then I came  across a reference to Ian Morrison’s Linuxium site.  Ian obviously spends a lot of time mucking about with set-top boxes, mini-PCs and PC-sticks and has done a ton of work to get Linux booting on those devices.  He has also modded some mainstream distributions to boot on these infamous 32-cum-64 bit devices.  Ian’s modified Ubuntu 16.10 was the first thing that successfully booted on the Z83-II and proved that it could successfully boot and run something other than the dreaded “W” (thanks Ian!).  If you’re looking for a stable Linux distribution you could do a lot worse than to mosey on over to Ian’s page and check out his offerings.

Finally there was light at the end of the tunnel.  The Windows reboot cycle was broken.  I went off to see what I had to do to get OpenBSD or FreeBSD onto the system.  I didn’t have to look very far.  In another one of those “Duh!” moments, it turned out that OpenBSD had also introduced the 32/64 UEFI fix and the latest snapshot (booted from a USB thumb drive this time) not only installed, but also automatically created a dedicated UEFI “i” partition and populated it with the required boot files.  Not only that, but the BIOS now “knew” about the new, bootable drive and I could easily make it the default power-on boot device.  Yay!

So, in summary …at power on, use ESC to get into the BIOS and F7 to get into the boot device selection menu and use Ian Morrison’s modified versions of Linux, or OpenBSD 6.0 or greater, to prove that you can boot and install something other than “W” on the Z83-II.

…and don’t buy anything during a turquoise Tuesday sale.

---

For anyone who’s interested, the output from “dmesg” (OpenBSD 6.0) for this system is available here.

---

FreeBSD Hint  —  If you’re trying to boot FreeBSD on one of these boxes and experience a hang at (or very shortly after) the point where the kernel discovers the keyboard (atkbd0),do a reset and then select option #3 (“Escape to loader prompt”) from the FreeBSD boot menu and type in these lines:-


set hint.uart.0.disabled=1
set hint.uart.1.disabled=1
boot


Once the system boots up, add the same fix into /boot/device.hints (without the “set”) to make it permanent:-


hint.uart.0.disabled="1"
hint.uart.1.disabled="1"


---

Update –  I wrote this original article just a few days before Christmas in 2016.  It’s now mid April 2017 and this little system has been running flawlessly for four months.  I have to say that I really like it.  It chugs along running as primary for all of the main services on our home network (DNS, DHCP, NTP, BOOT, Mosquitto, syslog, etc) and also works as the back-up hub for all of our other machines.  The smaller, single-core machines on the network send their uncompressed dump files over the network and this machine pipes the data through the “xz” compressor (running on two of the four cores) before storing it to disk (the multi-core systems compress their data before sending it).  The dump disks are on the USB3 bus and the throughput with the GbE network is a big improvement over USB2 and 10/100.  The killer months are yet to come, though.  This is a fanless system and, over the last four winter months it has been just mildly warm.  It remains to be seen how it will  deal with a hot, humid Japanese summer.  I’ll be sure to let you know, either way.

---

Update – Sept 2017 –   Well, we’re “over the hump” of the blazingly hot and steamily humid summer weather and this little system has just chugged along, right through the worst of it.  I also threw some fairly heavy-duty, very large (hundreds of gigs) compression jobs at it during the height of August and monitored the CPU temperatures fairly closely (this wasn’t any sort of benchmark; it was work that needed to be done and I was quite worried that this fanless system might shut itself down half-way through, so I was keeping a close eye on it).  With an ambient temperature hovering around 40C, the CPU reading never got above 82C, with the “xz” compression tasks limited to two cores (“xz  -T2”).  That’s hot, but not hot enough to melt anything.

---

You might also want to check out this later article on the Beelink AP35, which is a Celeron J3355 4GB/64GB system with four USB-3.0 ports and a mounting in the base of the case for a 2.5″ drive.

Implementing a wireless console using JeeLabs’ esp-link

JeeLabs have a deservedly popular ESP8266 software package, esp-link which, as with all of the best ideas, is based on a relatively simple concept, a transparent, bi-directional pass-through between serial on one side and wireless on the other.  The main thrust of their development seems to be aimed towards OTA programming of Arduinos, ARMs and even ESP8266’es, but it’s just as easy to implement a “wireless console” for a remote device and, because JeeLabs have done such a good job of creating and packaging their software, you don’t even need to compile anything; just download their package, use esptool to load it to your ESP8266 and then connect to the device over the air to configure it.  Fantastic (especially for those of us who are more comfortable with hardware than mucking around with toolchains and eagle.app.v6 files)!

Note:-  If you’re going to configure your ESP8266 device as a transparent network-to-console link, you’ll need a version with enough pins.  The ESP-12 variants are good bets for this.  You can forget trying to use the ESP-01.

What started this quest was the need to update an older headless box (a mini server without a built-in frame-buffer) and the realization that I no longer had a working machine with a serial port to connect to its console.  Like everyone else in the ESP community, I’d seen JeeLabs’ project and was impressed by their detailed write-up, so it suggested itself as both a fun project and a solution to a pressing problem.

First of all, I knew that one thing that differed from the “standard” connect-to-an-Arduino scenario was that I would be dealing with RS232 voltages, not TTL, so whatever else, I was going to need a level shifter.  Time to troll through the spares box and look for anything with “Maxim” stamped on the top.  I eventually came up with a single, unused MAX232A.  This is the version which only needs 0.1u capacitors, but has a 5v TTL interface on the inner-facing side.  Never mind, I’ll just have to throw a level shifter in there between the 3v3 ESP and the Max (this, as it turned out, would be problematic).  Other than that, I had everything else immediately to hand:-  a WROOM-02 on a project board with extra space available, the D-type connector (and even a shell) and a length of ethernet cable (saved for just such jobs after the end connectors had both lost their lock-tabs).  We were good to go!

I already had a couple of 2N7000 MOSFETs working as level shifters on the WROOM-02 side of the board (my USB-to-serial adapter is only 5v capable) and I added another couple next to the Max232 chip.  The odd “transistor” close to the centre edge of the board is actually a DS18B20, which was already connected up to the ESP, doing duty as a battery operated remote sensor.

One of the many neat aspects of the JeeLabs package is that the serial pins used to communicate with the target can be reassigned.  This is useful if you want to use the standard port for programming and debug, but critically, it also means that you don’t have to worry about the power-up garbage, which the ESP spews out on the standard serial port, corrupting your data stream.  As mentioned earlier, you also don’t need to re-compile to make this change; you just blow the firmware files they ship with their package to your ESP and then reconfigure it from a web browser once it comes up.

Here though I hit a problem (and I need to emphasize here that it was -my- problem and nothing at all to do with the JeeLabs package).  Whatever I did, when I first plugged my newly modified hardware into the USB adapter and fired up esptool, I just couldn’t get it to program.  I went back to using the Arduino IDE and a totally different software package …just the same.  Although the serial monitor showed what appeared to be the normal power-up crap at-a-baud-rate-no-one-else-in-the-world-can-use, it just would not program, “Connect failed”.  Well, duh!

After double checking my connections and verifying the same with the meter, I went back to the “Square Doughnuts” fault-finding procedure  (U.K. techies of a certain age will probably know what I’m talking about) and asked myself “exactly what has changed?”.  Well, obviously there’s a Max sharing the bed with the ESP now, and it connects to the ESP through a couple of MOSFET level shifters and, because I don’t want the power-up crap (at-a-baud, etc), I’m connecting it to GPIOs 13 and 15.  Ding!  Ah!  Everyone knows that GPIO-15 is super special.  Thou shalt tie it to ground at switch-on, or verily there wilt be tears!  If you’re familiar with the normal MOSFET level shifter, you’ll recall that the gate of the MOSFET is tied to the 3v3 rail and there’s a pull-up resistor between the gate and the source, with the source being connected to the ESP pin (GPIO-15 in our case).  Oops! Another double-Duh! moment.  By habit, I normally tie-up and tie-down using moderately low value resistors.  In this case I’d thrown in a 4k7 between GPIO-15 and ground, but there was now a 12k (“R2” in the level-shifter schematic to the left) sitting between it and 3v3.  According to our mutual friend Georg Ω, that would put the voltage on GPIO-15 very marginally into the grey area between acceptable limits for logic high and low.  It shouldn’t be enough to cause a problem, but it is borderline and obviously was causing a problem in this configuration.

One option might be to add another resistor between the gate and the 3v3 rail and decouple it with a large-ish value capacitor (and a reverse-biased diode across the new resistor for fast discharge of the cap at switch off), which ensures that the MOSFET is clamped in the off-state at switch-on and slows down the voltage rise across the (existing) voltage divider network to ensure that GPIO-15 is pulled to ground. Once C1 has charged up through R3, the level-shifter will function as it did before.  This worked fine for me, however the real answer is to not be a cheapskate and use the proper 3v3 version of the Maxim RS232 converter chip in the first place, so that you don’t need level shifters at all.  The Max3232E is a pretty good bet (and seems to be readily available), but there are now lots of other chips with 3v3-to-5v functionality, so you  might want to do a little sleuthing (and price comparisons) before choosing.

Firmware – As mentioned previously, the esp-link package comes with pre-built binaries (in the “firmware” directory), so you don’t need to go through the whole build process yourself.  One thing you probably do want to do though, is to make a very small modification to the supplied Makefile to prevent the “make flash” command from triggering a full rebuild automatically.  To do this simply remove the “all” dependency from the “flash:” target at around line 384 in the Makefile.  Look for this block of code:-

flash: all
$(Q) $(ESPTOOL) --port $(ESPPORT) --baud $(ESPBAUD) write_flash -fs $(ET_FS) -ff $(ET_FF) \
0x00000 "$(SDK_BASE)/bin/boot_v1.5.bin" 0x01000 $(FW_BASE)/user1.bin \
$(ET_BLANK) $(SDK_BASE)/bin/blank.bin


That first line, “flash: all“, changes to just “flash:“.  This tells make to use the existing files in the firmware directory and not to try and update them by recompiling in your environment.  Note that you may have to update the path to “ESPTOOL” (at around line 64, at the top of the Makefile), so that make can find your particular installations’ esptool.py script if it happens to be in a different location from the default.  After that you’re good to go.  Connect up your ESP8266 as normal for programming and just do “make flash“.

Once programming is complete you can restart your ESP in normal (non-programming) mode and it will automatically bring up the ESP in STA+AP mode (normal client plus access point), with the access point named something like “ESP_B3769F”.  You need to connect your WiFi to that new access-point to configure your ESP.  Once connected to the access point you can point a web browser at “192.168.4.1” to access the configuration menus.  The initial screen looks like this:-

 

The “Pin Assignments” section on this initial page is where we’ll change the UART assignments for the console connection to our PC  (it’s important to understand that this does not change the debug output from the ESP8266, that data will still be squirted out on the normal UART-0 pins;  what it does is to redirect the “transparent” link between your network connection to the ESP and the RS232 hardware to use UART-1 on GPIO-15 and GPIO-13).  Note that there are a few different options in the pin assignment list, but only one of them (highlighted at the bottom) implements this “swap_uart” connection on alternate pins.

The other menus and set-up for configuring the device are all explained in detail in the GitHub read-me file for the project.

The baud-rate setting for your RS232 console must match the PC that you’re connecting to and that setting is at the very top of the page under the “uC Console” tab.

The “Debug log” tab shows the output from the ESP8266 console (not the RS232 device) and can be handy for initial troubleshooting.

One extra, handy function is the ability to have the ESP8266 log information to any MQTT server you have configured under the “REST/MQTT” tab.

So, after all of that, does it actually work? Heck yes!  And rather well, actually.  To the left is a photo of the board connected to an Alix-2D3 (from PC-Engines) while I re-installed the software on the system (the Alix-2D3 is a headless router/gateway).   I connected the esp-link board to the console port and then, from a laptop, telnet-ed into the ESP’s address and had immediate console access (after struggling to remember the baud-rate on the Alix for a few minutes I even had console access that I could read!).  The initial installation and configuration were all done via esp-link (and it looks rather as though I’m going to have to keep this little board in my toolkit for future, headless connections).   [Aside – If you look very closely at the photograph above, you’ll notice the additional components for the programming modification around the rightmost 2N7000, next to the MAX232]

After completing the installation I went off and completely forgot about the esp-link board being still plugged in.  I came back to it 4 days later and found that the batteries were still okay and it was still quite happy to communicate via telnet, which quite surprised me (I assumed that the batteries would have been flat after a day, or maybe two at most).  Anyway, if you have a room full of servers and want a permanent console connection to each one, then you’re going to have to provide a reliable supply to the ESP board (and the ITEAD Studio PSU which I mentioned a couple of weeks back looks like a good candidate at only $2.50 a pop).

Bottom line…  A big “Thank You!” to JeeLabs for their esp-link software which helped me out in a very practical way as well as providing an entertaining and worthwhile hardware project.

 

Ω – “That’s Dr Ohm to thee, lad!”.

ESP-06 Module Warning

Just a quick note here to raise a flag concerning module compatability, functionality and design.

ESP-06 Bottom

I would guess that most people reading this probably know already that the company Espressif is the manufacturer of the ESP8266 chip and the producer of a couple of the reference designs to support it.  Most (though not all) of the actual modules (the PCBs with the ESP8266 chip and additional components required to support it) are produced by other companies.  The most prolific producer is “AI Thinker” and we (the hobbyist community) have a lot to thank them for  —  the low retail cost of the modules for one thing and the wide variation of different available designs for another.  Equally so, there have been some fairly big bumps along the road, with modules getting into the supply chain with magic smoke issues (the “self healing” LED current limiting resistor problem) and confusion from both the vendors and recipients of various modules when the item received was markedly different from the item ordered (to be specific, I’m talking about the ESP-12 variants here, not odd vendor miss-ships).

The lack of useful documentation has been one of the main bugbears with the ESP8266 from the very start and AI Thinker have been a fairly major player in that particular field, even if you do happen to read Chinese.

Well, unfortunately it looks rather as though it’s happening again.  Danny Von Lintzgy has left a couple of comments over at Pete Scargill’s blog noting that his product is pretty much dead in the water for the time being, as the latest batch of ESP-06 modules has different connections from all of those which he had received previously (the ESP-12 all over again).  Danny says “It seems the new variant has three of the corner pads connected to pins, where previously all four corner pads were connected to ground“.  That’s a pretty major change to have happen without any notification or hints (even the ESP12E at least had visible, extra pads).

So, if you’re using ESP-06 modules, please take note of Danny’s warning and be very careful with any new modules you purchase.  It doesn’t sound like a magic smoke issue, but if your new ESP-06 won’t boot and won’t program, this could be the reason.

Update – Danny has reported that AI Thinker support got back to him with the news that there are currently three versions of the ESP-06 out there.  The first is the older original, then (most worrying) there’s a not-quite-right version of the new design (which apparently has a problem with one of the resistors — but no news yet as to what sort of issues, if any, this errant resistor causes) and the last is the new design with the resistor problem corrected.  Apparently, the original and the newer versions have a slightly different AI Thinker logo on the outside of the RF shielding with (I believe) the newer versions having WiFi-style radio waves radiating from the head of the character which makes up the stylized “I” of the “AI”.

…and another one from AI Thinker

Hard on the heels of the weirdo “Black board T5” development board, CNXSoft just published a news byte on the new “ESP-14” module from AI Thinker.

Guess what?  The ESP-14 is just basically an ESP-12, with an STM8S003F3P6 microcontroller strapped on (and yes, that’s yet another, different microcontroller than the one which is on the “Black board T5”).

AI Thinker ESP-14 Module Pinout

The pins on the module are basically (with the exception of GPIO-00) the STM8S pins, not the ESP8266 pins.  An interesting point with the module pins is that there’s a separate VDD pin for the ESP8266, so it looks as though there’s provision for external control of the power (which makes sense for battery operated designs).

The slightly-better-than-the-previous-weirdo news is that the microcontroller on this board looks a little more sensible (in most respects) than the off-the-wall STC15L2K32S2 on the BBT5.  It (the STM8S) is a Harvard architecture device with 8KB of Flash, 1KB of RAM and 128B of EEPROM, which should look fairly familiar to AVR and PIC folks.

It still raises the question as to why AI Thinker believe we’re all going to rush out and buy a relatively unknown micro and re-tool (both hardware and software) yet again, after all of the pain we’ve gone through over the past year getting a workable ESP8266 environment in place.  If this had hit the marketplace when we were all pulling our hair out a year ago and if it had been an on-board AVR instead of the STM8S, then they might have had a winner on their hands.  As it stands … Double-Duh!, again.

‘nuther new something in town …you can safely ignore it!

AI Thinker have obviously been listening …but the question is, to whom.  They’ve just come up with something which they’re calling the “ESP8266 Black board T5” (their capitalization, not mine).  It looks really interesting at first glance.

AI Thinker “Black board T5” development board with attached battery box

[Update:-  I ended up reworking this board to make it somewhat functional.  If you’re comfortable with a soldering iron, you might be interested in these notes]

There’s a relay and a couple of screw-down connectors at one side of the board and what’s obviously one of the older  (blue) DHT11 temperature/humidity sensors on another side, sitting next to an ESP13/WROOM-02 module.  Between the relay and the DHT11 is a barrel-connector for DC input, but the unit also comes with a 3-cell (AA size) battery box attached via short flying leads (the board itself is just a little bigger than the battery box).  There’s a beeper on the board and there are three LEDs arranged in a vaguely spoke (as in bicycle wheel) shape.  In addition, there are two mini “tact” switches, a couple of jumpers, a slide-type on/off switch and, sitting like a spider in the middle of its web, a mysterious, 32-pin, QFP package chip.  Etched along the edge of the board is “ESP822_IOT_DEMO” (sic) and also “AI-CLOUD INSIDE”.

As usual with the on-line auction site vendors, there’s b’grall useful information on any of the pages which I could find (and as the auction sites advertise it as an “ESP8266 Serial Development Board” it’s also next to impossible to find any useful information on the web, either).  Anyway, just looking at the board I thought it had to be worth the $10 asking price, just in parts alone.  With a bit of luck, the mystery chip might turn out to be an I2C expander or something equally useful.  Throwing caution to the winds (poor old Caution!), I added a single unit (Caution must have crawled back home again while I was surfing dBay) to the next order I made for other odds n’ends.

Tum-de-dum-de-do.  Wait for a couple of weeks.  Odds n’ends arrive, along with da’ board.

AI Thinker “ESP8266 Black board T5”, with red LED visible just to the left of centre.

Hmmm…  Don’t recognize the number on the top of the mystery chip.  Never mind, throw in some batteries and slide the switch to “On”.  A red LED (which I hadn’t noticed in the pictures) lights up and, after a slight delay, the board starts to emit an annoying “Do something within the next few seconds, or I’ll explode” kinda’ beep.  I press one of the buttons.  It still beeps.  I press the other button.  It still beeps.  I switch it off, hold down one of the buttons and switch it back on.  It starts to beep again.  Same procedure, but other button.  Ahah!  a single beep and then blessed silence …for about a second, then it starts to beep again.  I rip the batteries out and throw the whole thing in a drawer (I was sorely tempted to throw it on the floor and stamp on it, but I thought I might damage the floor).

Back to the web (as there was b’grall documentation in the package, of course) and I resort to a Oogleg image search.  Oogleg’s image processing may be wonderful, but it seems to think that this PCB is a dead-ringer for all sorts of weird and wonderful stuff on the web that just happens to have a black PCB.  I finally find a link to AI Thinker’s forum and a thread which has some links to documents (Yay!), one of which seems to be an English PDF (double Yay!).

My yaying turns out to be a little premature.  The English documentation is limited to the schematic and it’s a fairly useless one, at that.  Many of the components evident on the board aren’t present on the schematic at all, labelling is haphazard, parts are wrongly identified (the 32-pin QFP is labelled as an STC15L2K16S on the schematic, but the part on the board is an STC15L2K32S2 [see below], the sensor is identified as a DH11, but the part on the board is definitely a DHT11) and plugs, sockets and jumper blocks are all depicted as unidentified rectangles.  One thing that does stand out immediately though, is that most of the GPIO pins on the ESP8266 are unconnected.  Uh-oh!

Original Schematic

A P-Channel MOSFET (top R/H corner of the schematic) gives another clue to what’s going on here.  The drive signal is labelled “WIFI_VCC”.  It’s a supply-side switch for the ESP8266 power.  The board is battery powered and it looks as though the ESP8266 is only powered up when a network connection is needed (and I recognized this simply because I’m doing the same thing with one of my ESP8266 projects, but using a DS3231 RTC module to drive the power switch).

So, it turns out that AI Thinker have produced a board which has a “master” microcontroller and is using the ESP8266 simply as a network interface.

Next to the on/off slide switch there’s a three-pin connector (barely visible in the photo above – it’s slightly above and to the left of the DHT11) with the silkscreen label “R-G-T”, which appears to be the serial port for the microcontroller (not for the ESP8266).  I pulled the unit out of the drawer again just long enough to connect up the pins and go through the press-beep-disconnect routine a few more times at various baud rates.  I didn’t ever get any indication of any output at all in the terminal window.  Zilch!  Nada!  Nutt’n!  At the same time I fired up a WiFi scan on one of my access points.  I could quite easily see my neighbours’ access points going up and down, but no indication at all that the ESP-13 on the board was ever powered-up.  Back in the drawer wi’thu useless burger!

I might possibly drag the thing out again and have a poke around with a multimeter, but not before I’ve de-soldered that bloody annoying beeper.

The microcontroller chip on the AI Thinker “Black board T5”

As far as I can tell from the scant information available for this microcontroller part number available on the web, the chip is an 8051 derivative, with 32K of onboard flash memory.  I haven’t used an 8051 chip in twenty years and, although I’m sure there are some folks out there who will be positively salivating at the thought of an 8051 paired with an ESP8266, there will most certainly be many thousands more who will be scratching their heads and saying “WTF?”.  I’m not (and probably never will be) an AVR/Atmel kinda’ guy, but why AI Thinker would AI Think it a good idea to produce a board for the hobby market with such an odd-ball chip is beyond me.  If they’d dropped in an ATtiny85, a Propeller, an MPS430, a PIC18F/24F, or just about any low-end ARM chip they would have had the (ESP8266) world beating that proverbial path to their door.  As it is, they’re destined to get a big, red “FAIL” stamp splattered across their collective foreheads.  Anyway, if you’re at all interested, I finally found the data sheet for the chip on STC’s web site (no thanks to their total lack of indexing or search function and the quiet inclusion in one data-sheet — with a single processor ID as the only title —  of 18 different chips).  The highlights are, yes 8051 based, 32KB of flash, 2KB of SRAM,  29KB(Eh?!?) of EEPROM, 2 x UART, 3 x PWM, internal clock and 8-channel, 10-bit A-to-D.  If all of that lights your fire, then this is the board for you (and I’ll sell it to you …cheap!).

Hold on just a second; I need to go and add a “Double Duh!” category for this one.

Update – In an effort to introduce some semblance of balance into this post (and especially to note that I’m not, generally “anti” AI-Thinker), I’d like to point you at a couple of articles on one of their other boards, which actually is something of a bargain!  Just about the only thing it has in common with the “T5” is a long and overly complicated name, but you should definitely keep an eye open for the ESP12-based “Yellow Development Board”.  Apart from anything else, it has lots of nice flashing LEDs and you don’t need to resolder anything to get them working.

SDK I2C Code. Today’s “Duh!” Story.

This entry is here as a helper (hopefully) for any other poor soul banging their head against the wall of Espressif’s I2C “master” implementation.  <TL;DR Spoiler>  When using the i2c_master.c code supplied with the IoT SDK, you must call i2c_master_gpio_init() before any other I2C functions in your code (and, obviously, your GPIOs need to be correctly defined beforehand).

ESP8266-03 with DS3231 module and DS18S20

I’ve been trying for days to get I2C working with an ESP8266-03 and a DS3231 RTC module (the el-cheapo, middle-kingdom one with the AT24C32 memory on the same PCB).  This was starting from scratch with I2C on the ESP8266 for me (I’d previously put together a trio of PICs talking to each other over I2C without too much trouble, so I naively thought that the DS3231 should be a doddle).

Hardware wise, I had the DS3231 hanging off a 4v5 supply and the ESP8266 running from a 3v3 linear regulator, supplied by the same 4v5 source.  The reason for this is that I intend to make the unit into a self-contained, battery-powered data-logger and I want the DS3231 to be able to control the power to the whole shebang through a small MOSFET driven from the alarm output.  The I2C bus is run through a MOSFET-based level shifter to ensure that the ESP8266 pins aren’t over-driven.

As normal, on the software side I started off with TuanPM’s totally excellent MQTT package (and if you’re not using this already, you should be!) as a starting point for my new project.  I simply added the i2c_master.c and i2c_master.h from the Espressif (IoT) SDK into the tree, modified the GPIO settings in i2c_master,h to point to the ESP8266 default I2C pins (GPIO2 for SDA and GPIO14 for SCL), added a tiny calling function into user/user_main.c and compiled.  I flashed the ESP-03 with the RTC module attached and… nothing!.   Okay, quick check… Duh, my SDA and SCL connections on the stripboard were the wrong way round.  Okay, resoldered the jumper wires, double checked with the meter and we’re good to go.  And… nothing.  Again!  My output data resolutely remained at all zeros, but the DS3231 calls (using Richard Burton’s ESP8266-ready code) didn’t throw any errors.  Double duh!

I certainly had no reason to doubt Richard’s code.  He’d only just recently done this himself and has a blog about his ongoing projects that is definitely recommended reading for anyone playing with the ESP8266.  I did have reason to suspect that the hardware might be a touch on the dodgy side though, as the DS3231 module already has a pull-up resistor pack on the I2C lines and the MOSFET level-converter also uses 10k pull-ups to both the 4v5 and 3v3 sides.  Thus the 4v5 side had the DS3231 pull-ups and the level-converter pull-ups in parallel and I suspected that the bus might not be being driven hard enough to a logical “0” on that side because of it.  Back to the soldering station, off with the SMD pull-up pack (and, while I was at it, off with the red-LED and the coin-cell “charging” diode, too…  I don’t need an LED sucking extra current from the 4v5 battery pack and the charge diode, as has been noted in lots of places on the ‘net, is a positive danger when using a normal, non-rechargeable CR2032 in the RTC).  Reconnected and yes, I know you’ve already guessed it, bugroll again.  Sigh!

Back to the soldering station.  Ripped out the original RTC module and slapped in another one.  Bugroll, repeated.

At this point I was considering popping one of the RTCs into a PIC-based board, just to check that I didn’t have two dodgy ones, but on reflection, decided that the quickest and easiest next test was to use another I2C implementation instead.  I had already seen Nathan Chantrell’s ESP8266-enabled OLED project and remembered reading that he’d used Zarya’s I2C implementation to drive the display.  Nathan’s blog is another one of my favourites and he certainly doesn’t seem to get things wrong as often as I do, so off I went to GitHub again and pulled Zarya’s I2C code.  It’s basically just two files, the i2c.c and i2c.h, which replace the i2c_master files from the SDK implementation.  First glitch… that missing word, “master”.  Richard’s code calls the i2c_master_whatever() functions, a couple of which are not compatible, argument wise, with Zarya’s implementation.  Okay, “Cheat” is my middle name, so taking the laziest course of action I just rolled half a dozen, one-line wrappers to encapsulate Zarya’s code into i2c_master_whatever() functions.  The only minor change was to make the ack and nack separate function calls (Zarya’s code sensibly uses a single ack function and passes either a 0 or a 1 depending upon whether it’s actually an ack or a nack).  Flashed and… Hurrah!!  Three cheers!  Two pints of Guinness and a packet of crisps, please!  The module burst into life and sent real data.

Mostly it sent really bad data, but at least not just those boring old zeros any more.  The memory module obviously wasn’t writing data at all and the RTC was returning some really weird values for everything except the seconds value, but the internal temperature sensor in the DS3231 was sending sensible, accurate temperatures (instead of the previous, monotonous “Bit chilly in ‘ere, innit”).  Hmmm, verrry interestink!

Unfortunately though, this still left me in something of a quandry.  Now I knew my hardware was capable of working to some extent, but was the bad data a result of my inept programming of the wrappers, or did I still have a hardware issue of some sort, or (perish the thought) was Zarya’s I2C implementation incompatible with the SDK (which has been updated several times since Zarya originally published his code) or Richards DS3231 driver?  Okay, there are a couple of other I2C implementations out there and one of them, by EADF, implements a replacement i2c_master with a significantly easier method of specifying the GPIO defines than the original, SDK master (EADF is the guy who also brought us “easygpio”, so this should be no surprise).  Off to GitHub again.  “https://github.com/eadf/esp8266_i2c_master”. Download. change out the files. compile, flash and… bugroll again.  Aaaargh!  Now I’m starting to doubt my own sanity.  I trust EADF like a brother.  He’s never let me down where the ESP8266 is concerned.  What’s going on here?

I spent more time looking at the two implementations (Zarya’s and EADF’s) side by side to try and work out what was different and it eventually struck me that there was one extra function in EADF’s code which, while missing from Zarya’s, was also present in the SDK version —  i2c_master_gpio_init(). So I finally check the much maligned and utterly useless SDK documentation and, yes, there it is, labelled as “Function: set GPIO in i2c master mode“, nothing else.  Looking at the code itself, it’s obvious that the function does pretty much nothing, other than the all-important job of setting the GPIO pins to the correct mode for I2C (and Espressif have even helpfully commented their code with “//open drain” on those lines), before calling the I2C initialize function.  So, the documentation, despite it’s shortcomings, isn’t so utterly useless after all (just bloody useless!) and a quick edit of my calling code to replace i2c_master_init() with i2c_master_gpio_init() caused it all to spring back into life again… this time with sensible data coming from the RTC and the AT24C32 memory chip, too.  Yip-finally-ee!

Bottom line (same as the top one)…  When using the i2c_master.c code supplied with the IoT SDK, you must call i2c_master_gpio_init() before any other I2C functions in your code (and, obviously, your GPIOs need to be correctly defined beforehand).  Failure to do so will cause hours of frustration (this seems to be my standard state when trying to do almost anything with the ESP8266 nowadays) and will also cause your significant other (and possibly the neighbours) to complain about the unrestrained use of Anglo-Saxon and unwarranted cat kicking.