Visual Studio Code and PlatformIO

Regular readers will know that I’m a big proponent of PlatformIO as a programming platform for the ESP family; it’s just so much better than the Arduino IDE for us command-line interface die-hards. As I’ve noted before, I’m not a religious fanatic on the CLI vs GUI thing; it’s just that GUIs (with a couple of exceptions) don’t really click for me. Most of the time I just don’t “get it” and what’s on offer usually seems to limit the functionality without really adding much, if anything, in ease of use. Recently though, I’ve been searching for information on getting started with the ESP32’s built-in version of FreeRTOS (specifically, looking for more information on real-world use, rather than just a couple of sentences about the syntax of the xTaskCreate() call) and I found a couple of very useful videos by Xavier on his “Simply Explained” channel which, as the name suggests, do a very good job of explaining by example, how to use FreeRTOS. The thing which (as a doddering old git) really caught my attention was his use of Visual Studio Code. I was captivated by the pop-up prompt with info on the parameters (of which there are plenty) to feed to xTaskCreate. While I imagine that functionality can get pretty tedious when it pops up for each and every printf(), it does look like a definite winner when you’re learning something new (and don’t have much in the way of short-term memory any more).

Now, what was I saying again? Oh yes, here’s the link to one of Xavier’s videos. He’s got a ton of content on his channel on many diverse (but usually tech-related) topics. This one is not the first in the series, so if the abrupt lead-in is a bit too much you can go to his Intro on FreeRTOS instead. I highly recommend watching Xavier’s videos if you’re trying to get started with FreeRTOS.

Of course, I immediately went off and downloaded Visual Studio Code and fired it up …only to just as immediately get totally lost when I couldn’t get it to do anything that I wanted. I did watch a few more videos from a nice lady at Microsoft, but ended up suffering from information overload and Windows-itis. I went for a nice cup of tea instead. And then went back to PlatformIO in an xterm, just so that I could get something done.

More recently, one of my regular morning reads, Hackaday, had an article on making laminated artwork for front panels with an accompanying video (worth a look in itself) by Richard Langner. I was impressed by Richard’s succinct, no-nonsense style and dipped into his video listings to see what other goodies were there. Lo and behold, right at the top of the list were a couple of videos on how to get started with Visual Studio Code with PlatformIO. The first was only a minute and a half long and the second just over six minutes. Both are filled with essential information on doing just what I wanted to do (have the editor prompt me with useful info, but still be able to use PlatformIO as my programming environment). As they’re so short, I’m including both as embedded links below. Take a look. Even if you don’t like them, you won’t have wasted much time.

Many thanks to both Richard and Xavier (oh, and Al Williams and the team at Hackaday, as well as Ivan and everyone at PlatformIO) for making my life not just easier, but a lot more interesting, too.

The Ubuntu ZFS boot pool problem [Part I]

Sorry, as you can see from the title, this is another “not-ESP8266” article, so you can skip this if you’re only interested in our magical little wireless modules. However, if you’re running a system (laptop, server or workstation) loaded with Ubuntu and have the root/boot partitions mounted on a ZFS filesystem, you should at least stick around long enough to read through the problem description, below.

If you landed here from a web search for “can’t upgrade Ubuntu – not enough space on bpool” (or something similar), then you’re in the right place. You can probably skip the introduction to the problem (you already know what you’re facing) and go directly to the solution.

Note that in these articles I am assuming that you are familiar with Linux and confident in your own abilities to administer your own system. Because we are dealing with filesystems, I am also going to assume that you have made and verified back-ups of your whole disk(s) before you follow any of the tips below. I am not responsible for your data. You are. The advice below is given in good faith after having been tested on several of my own machines (of different types), but I cannot be held responsible for any loss of data.

The bpool Problem

You see the familiar Software Updater pop-up, announcing that there are package updates available and prompting you to install them. You scan the list of updates and everything looks good, so you hit the “Install Now” button, but instead of a progress meter, you see a pop-up error window with the message:-

The upgrade needs a total of 127 M free space on disk '/boot'. Please free at least an additional 107 M of disk space on '/boot'.

The message continues with a couple of suggestions of how to increase available space in “/boot”, but basically at this point you are unable to install updates.

You’ve just run into the “bpool problem”.


Background

For the last few releases, Ubuntu has provided an easy method for installing root on a ZFS filesystem, giving you access to snapshots and the instantaneous rollback to previous versions which it enables (as well as the possibility of mirroring) without having to go through the long and tedious process of adding ZFS support after the actual install. It was a major move forwards and the “one-click” install process made it simple for anyone to benefit from ZFS technology (not just snapshots, rollbacks and mirrors, but raidz and ZFS send/receive back-ups, too). With all of those benefits, why wouldn’t you just use it by default?

Well, as a long time ZFS user, I can tell you that it does come with a few drawbacks. It is, for instance, somewhat more difficult to judge the available space left on any physical device than it was in the pre-ZFS world (good ole’ “df -h” is very good at giving you an instant and pretty accurate impression of the current capacity of your disks). ZFS tends to muddy that simplistic view a little, until you get used to depending more on the output from “zpool list -v” and “zfs list -o space” instead of “df”.

Fragmentation is another issue. Die-hard Unix users will remember that “fragmentation” was always a problem for people who used that other operating system, not for Unixes (well, unless you started to run out of inodes, anyway). However, with ZFS you do need to keep an eye on that “FRAG” column in the output of “zpool list -v”; once it starts climbing to around the 50~60% level it’s definitely time to start planning for an upgrade and, once it’s past 70% it’s time to take some urgent action (usually, but not always, time to add more physical disk space — we’ll touch on de-fragmentation as a side-effect in the “Solutions” presented in part-II). Once it hits 80%, you’re in big trouble as, even with available free space, the system will be struggling to find usable space.

These issues are a pain to come to terms with after all of these years of doing the odd “df” just to check that everything was okay, but they are still heavily outweighed by the advantages (did I mention copy-on-write or checksummed data yet?).

So, our simple ZFS installation method from Ubuntu is a real game changer and gives everyone the chance to ride on the ZFS bandwagon with minimal effort. Unfortunately, along with that simplicity and ease of installation comes another problem — sizing your filesystems. The installer is a one-size-fits-all deal; it looks at your disk size and makes a couple of fairly simplistic calculations to create a very simple partition table and then later adds a spread of ZFS datasets on top of those physical partitions. The “bpool” (for “boot-pool”) is a single ZFS filesystem which occupies a dedicated partition just for the kernel/initramfs and EFI/grub data required to boot the system. The “rpool” (for “root-pool”) is the ZFS filesystem which contains datasets for /, /var and /usr (as well as sub-directories) and the user home directories. It’s all very well laid out, so that a rollback to (for instance) a previous kernel version need not affect the user’s data. However, the installation script for ZFS currently has one major issue; it limits the size of the bpool physical partition to 2GB (it can be smaller, but not bigger), no matter what the size of your physical disk

The major drawback that people are finding to this installation mode comes from one of the strengths of ZFS — the snapshot functionality. Put simply, if you make a snapshot of say, your whole bpool, ZFS will never delete anything from that point in time. If you subsequently delete a kernel which was present when you took the snapshot, ZFS will keep the file itself, hidden away in the snapshot and only remove the named link to the file from the normal /boot directory. If, at some later time, you decide that you no longer need that snapshot (because it is outdated and you know you’re never going to revert to that old kernel again) you can destroy the snapshot and only then will you recover that disk space. This is one of the reasons that space management on a ZFS filesystem is a bit of a minefield for new users and is exactly why the “bpool problem” exists.

Every time Ubuntu pushes out an update (even a non-kernel update), the installer will automatically create a snapshot of your filesystem, so that you can roll back to the previous version if things go horribly wrong. In the case of actual kernel updates, the snapshot can consist of tens, or even hundreds of megabytes of combined kernel and initramfs files. Given that we have an upper limit of 2GB on the size of /boot (the bpool filesystem), that space doesn’t go very far if you don’t actively manage the snapshots. Worse, the Software Updater will start to fail when there is no longer enough free space in /boot to fit another version of the kernel and, to be clear, even non-kernel updates will fail to be installed if there is a kernel update still in the queue (even if you de-select it manually). The error messages are clear and quite specific (and even give tips on how to ameliorate the issue). However, by the time Software Updater flags the problem, it can already be too late to easily fix it (and how many of us know which system-generated snapshots are safe to remove and which aren’t, anyway?).

Short-term band-aids

These suggestions are not a true solution, they are fixes to get you up and running in order to have Software Updater working again in the shortest possible time.

Before making any changes which might permanently erase data from your filesystems, you should make a back-up of your whole disk (I’ll be repeating this at various points throughout these articles, because I really mean it — I cannot and will not be held responsible for any loss of data you might suffer while trying to follow the tips given here; I always assume that you have backed-up your data and that you have verified that those back-ups work. Batteries not included. May contain nuts.).

First, one quick fix which Software Updater lists in its output when this problem occurs — edit the /etc/initramfs-tools/initramfs.conf file and change the COMPRESS setting from “lz4” to “xz”. This won’t take effect until the next kernel update, but at that point the initramfs install will use the more efficient (compression, not speed) “xz” method. This can slice around 20MB off each of the initrd.img files in the /boot directory and so will save you about 40MB of space on each update (there are usually two versions, current and “.old”).

The results of this second tip are much more difficult to predict in terms of exact space saved — snapshot whack-a-mole.

You can list your snapshots in bpool using:-

zfs list -t snap -r -o name,used,refer,creation bpool

This will get you a listing with entries that look something like this (but probably much longer):-

NAME              USED   REFER  CREATION
@autozsys_7lzjmg  197M   197M   Wed Dec 30 11:07 2020
@autozsys_9u20hc   17K   199M   Wed Jan 20  6:56 2021
@autozsys_6c4qgp    0B   199M   Thu Jan 21  6:12 2021

[Note that I have removed the pathname before the “@” symbol to fit the text without line wraps]

As you can see, the “used” and “refer” columns vary widely between the snapshots. The bottom entry (6c4qgp) has a “used” entry of zero bytes; surely that can’t be right, can it?. Well, that’s the whack-a-mole function coming into play. In this particular case, there don’t appear to have been any changes to bpool between autozsys_9u20hc and autozsys_6c4qgp, so if we destroyed 6c4qgp, we wouldn’t get any free space released back to the filesystem. So where does the “refer” come from? That’s letting us know that even though there were no changes between the last two snapshots in our list, they both have references to 199MB of data already being held in other, previous snapshots (or the filesystem itself). What this means for us is that while destroying 6c4qgp may not free up any space, it is very likely that destroying 7lzjmg or 9u20hc will cause 6c4qgp to inherit some of that referenced data, thus causing the “used” count for 6c4qgp to increase and the filesystem not to gain back as much free space as we expected (hence “whack-a-mole”, the used data count might just pop back up elsewhere).

Now, to be fair, because we’re dealing with /boot and the turnover is usually caused by the replacement of kernel/initramfs files, it is more than likely that destroying the oldest snapshot will simply delete the oldest of those kernel-related files, returning about 140MB of free space to the filesystem. Likely, but not certain, so don’t go destroying snapshots left, right and centre without checking their contents first.

LISTING OF /boot CAPACITY AND AVAILABLE SPACE
["df -h"]
Filesystem                 Size  Used Avail Use% Mounted
bpool/BOOT/ubuntu_g1cutr   145M  117M   29M  81% /boot

["zfs list"]
NAME                       USED  AVAIL     REFER  MOUNT
bpool/BOOT/ubuntu_g1cutr  1.52G  28.5M      116M  /boot

["zfs list -o space"]
NAME                      AVAIL   USED  USEDSNAP  USEDDS
bpool/BOOT/ubuntu_g1cutr  28.5M  1.52G     1.41G    116M

You can always check the contents of any given snapshot by noting where the ZFS pool is mounted and then navigating to the .zfs/snapshot directory at that point. So, for our bpool snapshots, we can see from the output of “df” or “mount” that bpool/BOOT/ubuntu_g1cutr is mounted at /boot. Listing /boot/.zfs/snapshot will give us a listing of directories which correspond to the snapshot names in the listing above. You can list each of those directories to see what files and directories are included in the snapshot. As /boot is actually quite small, you can easily do an “ls -i” on two of the snapshot directories and see which files have the same inodes and which are different (which gives a good indication of which files are shared between different snapshots and the current, live filesystem and which are unique to a given snapshot).

Snapshots are removed using the “zfs destroy” command, by the way, not by removing the snapshot directories.

Don’t forget that destroying any snapshot restricts your ability to recover to a known point in time, so I would urge you to err on the side of caution — if you’re not 100% certain of what you’re about to remove, don’t do it!

If you’re a user who likes to create their own snapshots (before a major upgrade, for instance) you might already be able to easily target some of your own snapshots as candidates for deletion (perhaps you already know that you’re not going to roll back to that ancient, previous release?).


The [partial and not very satisfactory] Solution

The stop-gap solutions listed above are just that; short term solutions which will give you some extra breathing space, but not long term fixes. To remedy the bpool problem long term, we obviously need to add a substantial chunk of extra disk space.

One brutal, but simple way of getting back more bpool space (and a solution which is very topical with the release of 21.10 almost upon us) is to re-install Ubuntu after editing the ZFS section of the install script to bypass the problem. I’m not suggesting that this is the best or most versatile answer to the issue (in most cases it won’t be), but if you happen to be on the verge of upgrading, or perhaps have a machine where all of the application and user data is mounted from a fileserver rather than local disk, this may be an option (but, as always, I would recommend a full back-up of the system before taking such a drastic step).


Just in case you didn’t quite get that… ==WARNING== The following steps will delete -ALL- of the data on your disk. Do -NOT- proceed with the steps below if you are not prepared to have your disk(s) totally wiped of all existing data.


Booting from the Ubuntu install image will drop you into the Try-or-Install header page. Select “Try Ubuntu”, which will restart the desktop to the normal, live image. From there you can open a terminal session and become root using “sudo -s” (no password required).

Using whatever editor you’re most comfortable with, open /usr/local/share/ubiquity/zsys-setup.

On (or about) line number 267, you’ll find this code:-

[ ${bpool_size} -gt 2048 ] && bpool_size=2048

You just need to comment out that line completely for the simplest fix. Doing so will allow the bpool size calculation to grab a much larger chunk of your available disk space. Note that if your disk is 50GB or less, this isn’t going to help — you’ll still end up with roughly 2GB. Conversely, if your disk is 1TB or larger, you may end up with much more bpool space than you actually need, or want. In these cases, you might want to change line 267 to use some value other than 2GB; I found 10GB (ie:- replace “2048” with “10240”) to be satisfactory for my machines, but if you happen to be a kernel developer, you might want to bump that up even further.

Following the edit of the ubiquity file, you simply select “Install Ubuntu” from the desktop icon and proceed with the normal ZFS install.


Okay, that’s the gist of the bpool problem, along with a couple of suggestions for clawing back some disk space and the most drastic way of fixing the issue.

In the second part of this series, we’ll be looking at a couple of more reasonable methods of fixing existing installations without resorting to a full re-install (fair warning though — it will still involve booting from the install media, so is not entirely without risk).

And don’t forget …back up early and back up often!


FUZIX (Unix-like OS) ported to the ESP8266

Well here’s some really interesting news. I have a job set-up to update “interesting” GitHub projects to local disk on a daily basis, so I can just do a quick listing every morning to see what, if anything, has changed. This morning the FUZIX project flagged changes (FUZIX is a minimal, unix-like OS for very small, resource-limited micros, which started off being targeted at the Z80 series). When I checked the repository to see what the updates actually were, I found:-

drwxrwxr-x 10 gaijin gaijin 20 Feb 18 05:38 ..
-rw-rw-r-- 1 gaijin gaijin 11536 Feb 18 05:38 Makefile
-rw-rw-r-- 1 gaijin gaijin 273 Feb 18 05:38 .gitignore
-rw-rw-r-- 1 gaijin gaijin 35670 Feb 18 05:38 filesys.c
-rw-rw-r-- 1 gaijin gaijin 252 Feb 18 05:38 dep.mk
drwxrwxr-x 2 gaijin gaijin 6 Feb 18 05:38 cpu-armm0
drwxrwxr-x 2 gaijin gaijin 24 Feb 18 05:38 platform-esp8266

Which led, in turn, to David Given’s marathon videos (there are roughly 30-hours of keyboard-bashing and puzzling-out-loud available right now, with another five videos still to be released on a daily basis) detailing his work to port FUZIX to the ESP8266. David’s web site has some further explanation of how far along the port currently is, along with some pre-compiled, loadable binaries for the ESP8266 (I haven’t had time to try them yet). All of David’s work is available from his GitHub repository.

You will need to attach an SD card to get the full FUZIX experience, but David says it is reasonably speedy on the ESP8266, with a boot time of just 4 seconds.

|�l�c|����{���B�p�n�dNn���cp��cl r$p�N�s����cN�|����B�|�N�l��l�Nl�d Nr���N��{$�o�nod���no��rdp�n���l"o�|����p��no�$�$�no$�{$or���o� �n�2NNl��|�N��lp�N��d�#n�|���b��Nn�d�l �oo$�{lor���N$�$sےNl��FUZIX version 0.4pre1
 Copyright (c) 1988-2002 by H.F.Bower, D.Braun, S.Nitschke, H.Peraza
 Copyright (c) 1997-2001 by Arcady Schekochikhin, Adriano C. R. da Cunha
 Copyright (c) 2013-2015 Will Sowerbutts will@sowerbutts.com
 Copyright (c) 2014-2020 Alan Cox alan@etchedpixels.co.uk
 Devboot
 80kB total RAM, 64kB available to processes (15 processes max)
 Enabling interrupts … ok.
 Scanning flash: 2591kB: hda: 
 SD drive 0: hdb: hdb1 hdb2 
 Mounting root fs (root_dev=18, ro): warning: mounting dirty file system, forcing r/o.
 OK
 Starting /init
 init version 0.9.0ac#1
 fsck-fuzix: 
 
 login: root
 Welcome to FUZIX.

David notes in the README (contained in the binary tar-file) that the system does have some limitations; first and foremost is that it is almost unusable running from flash, so you really do need to wire up an SD card (pinouts for this are included in the same README). Here is his ToDo list:-

  • userland binaries can’t find their error messages.
  • CPU exceptions should be mapped to signals.
  • single-tasking mode should be switched off (which would allow pipes to work).
  • someone needs to overhaul the SD SPI code who understands it.
  • not all the ROM routines are hooked up to userland binaries.

Right, I’m off to look for an SD card and a spare ESP. See you later!

Will it mirror?

Pic of (old) Laptop with SSD attached to lid

Here’s another silly one for you. What do you do if the latest release of your OS of choice ships with ZFS, but you don’t have space in your laptop for a second disk? …Answer:- Reach for the velcro.

This Sony Vaio has a Centrino Core-2 p8600 processor, so it’s not going to break any speed records, but it works well enough for day to day use. Courtesy of the Buffalo 500GB SSD taped to the lid, it now sports 1TB of disk space (500GB mirrored), which is probably well in excess of what the designers originally envisaged.

Centrino.2 sticker next to USB ports

This is one of those little “because I can” projects which I don’t necessarily recommend to anyone else, but at the same time, the lightness of an SSD compared to a normal hard-disk (even a 2.5″ one) means this is now an eminently practical solution if your old laptop happens to be running out of space (I do move the laptop around the house to work in different rooms at different times, but it generally doesn’t travel much further afield than a deck-chair out on the veranda).

So, will it mirror? Heck yes!

Should I mirror?

No, probably not. It’s much more sensible to use a periodic ZFS send/receive job to back up your work to an existing server, that way you don’t need to worry about the extra drain on your battery and you still have your work if your laptop is stolen or knocked off the deck of your yacht, mid-ocean (what, you mean that’s never happened to you?).

One other consideration when thinking about installing Ubuntu on ZFS — currently (as of 21.04) Ubuntu will not allow you to edit the size of the disk partitions; you must accept their optimized defaults. Unfortunately , those defaults include a /boot partition which is much too small (typically 2GB). It will work fine for a couple of months, but with every apt update, the system will automatically add a snapshot of the boot partition. When the upgrade includes a kernel update, this means that tens, or even hundreds of megabytes of storage can be used. Even when you set the system defaults to compress the kernels using “xz”, it doesn’t take too many updates before you start getting “not enough free space” messages from apt and it will refuse to continue with the update. This is not something a novice user can easily recover from (hint: deleting files on a ZFS partition doesn’t always return that free space to the system — it all depends on whether it is still being held by a snapshot).

FreeBSD Diaries — Adding bootstrap code to ZFS root disks

When you add a new disk device to the “zroot” pool (or whatever it is that you’ve named the ZFS pool where your root partition resides) you should also add bootstrap code to that specific disk, so that the system can actually boot from it should the other disk(s) in the pool suffer a hardware failure.

Assuming that you’re using disks partitioned using “gpart” and have an EFI partition, your disk might look something like this (using “gpart show da3”, for example):-

=> 34 7814037101 da3 GPT (3.6T)
34 6 - free - (3.0K)
40 1024000 1 efi (500M)
1024040 12582912 2 freebsd-swap (6.0G)
13606952 209715200 3 freebsd-zfs (100G)
223322152 7590714976 4 freebsd-zfs (3.5T)
7814037128 7 - free - (3.5K)

You also use “gpart” to write the boostrap code to your new disk. In this example, the command would be:-

gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da3
partcode written to da3p1
bootcode written to da3

Note that “gpart” confirms that it has written to the disk.

PLEASE MAKE SURE that the you change the disk device specifier (“da3” above) to specify -your- correct target disk device. This command will quite happily destroy filesystems if you get it wrong.

Recent Updates [July 2020]

E32-915T30S LoRa Module“Xreef” (Renzo Mischianti) has just updated his excellent LoRa E32 Series Library to fix a memory leak issue.  All of his examples have also been updated to include the fix, so if you’re thinking of playing with some of the EBYTE manufactured boards and an ESP8266 or ESP32, this is the go-to library (and Renzo has also produced a couple of adapter boards to make it easier to use the E32 boards with your favourite ESP).

SixPack GRBL CNC Controller“Bdring” (Bart Dring) has updated his port of GRBL for the ESP32 with many fixes and improvements (with a merge from the dev branch), so if you’re into CNC, you should check out his repository now.  Bart also shows the rather impressive “6-Pack” (six axes, XYZABC, for Pololu/StickStep drivers) ESP32-based controller board (left) on his GitHub repository page, but it doesn’t seem to be available from his Tindie outlet, yet.

 

 

 

Support for ESP32 and Ethernet in TASMOTA

Development on TASMOTA continues apace (with the ongoing C-19 movement restrictions, possibly even more rapidly than usual) and lots of new and interesting stuff has been popping up in the code recently (for instance, if you want to connect an anemometer, to add wind-speed to your weather-station, Matteo Albinola has you covered). However, one of the most exciting recent additions has been the arrival of baked-in support for the ESP32, based on Jörg Schüler-Maroldt’s work. This first appeared in version 8.2.0.6, back at the beginning of May, after Jörg created the libesp32 compatibility library and an initial pull request back in April. Since that time there have been a lot of “#ifdef ESP32” lines added to the code.

Now (version 8.3.1.5, as of June 17th 2020) we not only have the ESP32 compatibility and compile additions but also the addition of ESP32 hardware Ethernet support with the recently added xdrv_82_ethernet.ino driver file. The new, tasty goodness doesn’t end there, though. Olimex ESP32-POE  If you take a quick look at the headers of that driver file, you’ll find the pinout defines and TASMOTA template for the Olimex ESP32-POE, so not only do we get the ESP32 and ethernet, we also get PoE thrown in (the Olimex board currently sells from their site for €17.95, but is also available from the likes of Mouser and even Amazon in some areas).

If you bought one of the exceedingly cheap ESP32-CAM boards (AIThinker/Geekcreit), you now have the option of TASMOTA enabling  it, too.  Look for a second new driver file, xdrv_81_webcam.ino, to see the pinout details and TASMOTA template for that device.

Okay, so new, sexy additions to the code, but how do we compile TASMOTA for the ESP32. Well, if you’re using PlatformIO (and you should be!), it couldn’t be much simpler. Copy the Sonoff-Tasmota code into a clean directory (or git clone the repository) and then:-

  • In that directory, copy the platformio_override_sample.ini file to platform_override.ini.
  • Edit your new platform_override.ini file and uncomment line number 29 so that “; tasmota32” becomes just “ tasmota32“.
  • Type “pio run“.

Sit back and watch for a couple of minutes while the compile runs. It will compile two different versions. The vanilla “tasmota” (ESP8266 version) will be created in .pioenvs/tasmota/firmware.bin. The tasmota32 (ESP32 version) will be created in .pioenvs/tasmota32/firmware.bin.

[ For anyone who hasn’t compiled TASMOTA before, the customizations for your local network settings and for the target device itself are made in the ./tasmota directory. The simplest method for a first time compile is to use the user_config_override.h file to update only the bare essentials. Once you’ve made sure that you can compile successfully, you can modify the (very much more complex) ./tasmota/my_user_config.h file for full customization. ]

(Blowfish) Virus Protection

Hopefully this will lift your spirits a little bit (if you’re a techie, anyway).  Here’s a quick snap of the tabletop, taken this morning while my better half was taking a break.

A section of the OpenBSD "Blowfish" T-shirt being cut up for use as a facemask

Just in case you don’t know what you’re looking at, it’s her workspace, which is currently dedicated to producing DIY face-masks and the donor material is an iconic, OpenBSD “Blowfish” T-shirt.  This shirt had a picture of Puffy on the front, along with the text:- “So long and thanks for all the passwords”.  The back was what contributed to it becoming mildly infamous at the time, as it was printed with the Blowfish algorithm and was (allegedly) illegal to export from the U.S. (although I don’t remember ever hearing of anyone being arrested for walking through customs wearing one).

So, what used to protect your ssh tunnel‡ is now helping to protect (in a small way, I know) against the SARS-CoV-2 virus and, just in case it’s still on the banned list (and we ever get to travel again), the offending algorithm will be folded into the inside of the mask, keeping our secret safe.

 


 

‡  —  If you’re still using Blowfish, you should know that Bruce Schneier, the designer, now recommends upgrading to “Twofish”.

Amazon Smart Bulb

Csndice RGBWW Smart Bulb (box, front view)

We have a little log house (which we built ourselves and used to live in) which we let out as a holiday cottage.  It’s something that is (generally) fun to do and also pays a few of the bills, but one of the big mistakes we made when we put it together our large, wooden Lego kit was to put too many light switches in a single location (if you live in a house long enough, you’ll eventually remember what most of them do, but for short-term visitors it’s just plain confusing, even with little stick-on labels).  So one of the things we find happens quite often is that one (or two …or five) of the outside lights are left on all night (and might be left on all day too, if our guests head out in a hurry in the morning).  Long ago we changed out all of the bulbs for the little CFL corkscrew types to reduce the overall power usage and we thought we might get around to replacing them all with LEDs, if the prices ever come down.

To compound the mistake with the switches, we also bought almost all of the external lights with E17 size fittings (about half the size of a normal household bulb), not because we really wanted them, but because that was the only size the maker supplied the fittings in.  This latter issue came back to bite us when I decided I could fix the lights-left-on problem by replacing the CFLs not just with LED bulbs, but with “Smart” LED bulbs (Wow-weee!).

Nope!  No smart bulbs in E17 form-factor (at the time).  So I did what any other self-respecting ESP hacker would do and, when one of the CFLs died a noisy, sparky death one day; I chopped off its head, ripped out the charred remains of the inverter and replaced it with and ESP01S, a TRIAC and one of the brightest (and cheapest) E17 LED bulbs I could find.  I stuck my little FrankenShine monster (I admit, it wasn’t particularly pretty) into the fitting right outside our kitchen door, so that we could play with it and monitor it without inconveniencing any of our guests during the initial testing and burn-in period.

(FrankenESP) Photo of DIY E17 ESP01S light assembly

The FrankenShine in all of its glory …The ESP01S and other DC components are at the extreme R/H end of the board, with the opto-isolator to the left of the programming header.  The TRIAC is closest to the camera (slightly left of centre) with the PSU sitting over at the back.  You can just see the PROG/RUN mode jumper partially hidden by the big, black electrolytic.

I set up a TASMOTA rule to switch it on at dusk and off again a few hours later and coached my better half in how to communicate her wishes to the nice lady stuck inside that tiny, pinky-orangey UFO thingy that landed on the kitchen counter a couple of years ago (…and always say “Thank you”, because we want our grandchildren to grow up to be polite).  So, (her) “Hey Gewjull!  Turn on the kitchen door”.  (GH, pertly) “I’m sorry, I don’t know how to turn on the kitchen door”.  “Hey Gewjull, turn on the effin’ kitchen door!!”.  (GH, sullen) “Okay, turning on the effin’ kitchen door”.  “Thank you!”.  (GH, stilted and mechanically) “That makes me so ‘effin happy”.  Ah, another successful foray into the world of home subjugation.

For a while, everything went along swimmingly.  In fact we got so used to coming home at night and being able to see the keyhole that the transfer of FrankenShine to the cottage was delayed, several times.  Then one evening we came back to a dim, flickering welcome …uh-oh!  A power-cycle cured it, but only for about five minutes and then it was back to flickering again.  A PSU problem?  More likely a TRIAC issue.  Gottverdeckel!  I pulled Franky out and put the LED bulb back in the fitting (yup, works okay).  Unfortunately, as usual with these intermittent problems, Franky worked fine on the test bench, with not a flicker to be seen.  I went back outside and gave the light fitting a couple of dunts with the flat of my hand to test for loose connection, but no flickering or dimming was evident.  And so everything remained as it was for (quite) a while  …until I decided to replace the CFL in the worst offending cottage light fitting with an LED bulb, anyway (Franky or no Franky).  The light itself was identical to the test fitting next to our kitchen door, but was some four years older.  I opened it up and, lo and behold, it was an E26 holder assembly.  “Oh crikey!” said I (or maybe something similar), all of this mucking around and I could have just gone out and bought an E26 “smart” bulb and flashed it with TASMOTA.  At about the same time, the E17 bulb in the kitchen door fitting couldn’t contain its hilarity any longer and started flickering and dimming intermittently again (no Franky involved). “Oh double crikey!” (or words to that effect).

Anyway, suitable chastened with my unsuccessful techie adventure and unreliable memory, I went off to Amazon (Japan) and searched for an E26 bulb with a high output (lumens) and cool-white hue, suitable for an outside fitting.  Box (side-panel, specifications)What I finally decided on was a “Csndice” branded,  RGBWW bulb that had a 900lm stated output and an adjustable colour temperature between 2700 and 6500K.  In addition, it was tagged as being compatible with Google Home and Alexa, with “no hub required” (which is usually a pretty good indication that there’s an ESP variant in there and that it can be flashed with TASMOTA).  It wasn’t particularly cheap, but I can confirm that it is easily converted to TASMOTA (with the excellent Tuya-Convert) and seems to work reliably (my better-half has started to complain about the “Geriatric disco mode” of the colour sequencer test).

This particular bulb works with the Qualitel ALS08 setting from the TASMOTA templates repository.

Photo of CSNDICE smart bulbThe light (in WW mode) is very bright and easily equals that of the CFL.  It has the added bonus of being easily adjustable (from sliders on the TASMOTA main-menu page) if you prefer a particular hue of white output (from “cool” blue through to “warm” yellow).  If the fancy takes you, you can also fiddle with the RGB settings to have a particular colour and shade, instead of that boring old white.

To help you (well, okay, to help me) test out the functionality and reliability, I’ve put up a simple, command-line exerciser on GitHub.  You’ll need to change the variables (at the top of the file) to use your TASMOTA MQTT topic name (variable: BULB_ID) and your MQTT broker (server) IP-address or name (variable: MQTT_SERV).  After that, you can just run the program from the command line with no options or arguments (to turn the bulb on in white, neutral hue mode), or use one of the following options:-

  • -c — “C”ool white. Switches on the WW LEDs in the bulb with a blue hue.
  • -w — “W”arm white. Switches on the WW LEDs with a yellow hue.
  • -n — “N”eutral white. Switches on the WW LEDs with the hue set mid-way between cool and warm.
  • -o or -0 — Switches all LEDs (RGB and WW) off (that’s a zero, by the way).
  • -s — “S”equence. Turns on various colour mixes of RGB for 2 seconds before fading to the next colour (Control-C to quit).
  • -d — “D”ebug. Fairly quiet debug output.
  • -D — “D”EBUG. Very verbose debug output.
  • -h — “H”elp. Basically this command listing.

Now you too can have fun at the geriatric disco.

Using a GPS with TASMOTA (Part II)

This is part-2 of a two part series.  If you missed it, the first part is here.


CONFIGURATION

Once you have your build of TASMOTA installed and the physical connections made, you need to configure which ESP pins TASMOTA will use to communicate with the GPS module.  There are a couple of points to make here …first, the TASMOTA pin configuration refers to the data flow with respect to the ESP8266 itself, while the labelling on the GPS module pins refers to data flow with respect to the GPS.  This means that the pin you define as GPS_TX on the ESP actually connects to “RX” on the GPS module and GPS_RX connects to “TX”.  The second point is “Don’t Panic!”, as I mentioned earlier, you can’t harm either the GPS or the ESP by connecting or configuring them the wrong way round.  The last point is that if your GPS module doesn’t support the UBX protocol, it ain’t going to work, no matter which way round you connect things.

Okay, boot your ESP8266 and connect to the TASMOTA web interface.  Go to “Configure” and then “Configure Module” and select the “Module type” to be “Generic (18)” (you’ll need to scroll right down to the bottom of the list of modules to find it).The TASMOTA pin configuration menu, showing GPS_RX highlighted  Save this change.  When the save is complete you can go back to the “Configure Module” tab and will now find all of the available pins on your module displayed.  Select GPIO12 and again, scroll right to the bottom of the pull-down listing.  You should see the options “GPS_RX (190)” and “GPS_TX (191) close to the bottom of the list (if you don’t see them, then the “#define GPS” option in the tasmota/my_user_config.h file has not been un-commented, or you’ve flashed the wrong image to your ESP).  Remembering that “GPS_RX” refers to data received by the ESP, select that option for pin GPIO12 (which is connected to the “TX” pin of the GPS module) and then “GPS_TX” for pin GPIO_13 (connected to “RX” on the GPS module).  Save those changes.

The ESP8266 will restart itself as part of the “save” process and when it comes back up again you should see several GPS status lines above the normal buttons on the main menu.

TASMOTA main menu showing GPS status lines, but all readings zero.
No Data From GPS

If your GPS module hasn’t acquired satellite data (GPS LED not flashing). or if it doesn’t support the UBX protocol, the GPS status lines will be present, but the data will be zeroed out (as shown above).  It’s also possible that your TX and RX configuration is reversed, so it’s worth trying either changing the pin configuration in TASMOTA, or just swapping the wires over (whichever is easiest).

TASMOTA main menu with live GPS data shown
GPS With Live Data

If your GPS is correctly configured, has already acquired data and supports the UBX protocol, you should see something like the screenshot above, with the data updating once per second.

There are several commands available via the TASMOTA console to manipulate the GPS data handling functions.  Commands are entered at the console and have a format of sensor60  n[n] where “n” is a number and “sensor60” specifies that this command is  for sensor number 60 (which is reserved for the GPS module).  So an example might be:-

sensor60  9   —   Start the NTP server process (using time data from the GPS module).

Here’s the complete list of commands, as of early Feb, 2020.

  • COMMAND     DESCRIPTION
  • sensor60  0  —  Write log data to all available storage, then stop
  • sensor60  1  —  Write log data to all available storage, then restart from the beginning (overwrite mode)
  • sensor60  2  —  Filter horizontal noise from GPS signal
  • sensor60  3  —  Stop horizontal noise filter
  • sensor60  4  —  Start recording log data to storage in append data mode
  • sensor60  5  —  Start recording log data to storage in overwrite data mode
  • sensor60  6  —  Stop recording log data to storage
  • sensor60  7  —  Send an MQTT update on each positional change (very noisy)
  • sensor60  8  —  Stop sending MQTT updates on positional changes
  • sensor60  9  —  Start the NTP server on port 123 (GPS web updates stop)
  • sensor60  10 — Stop the NTP server (GPS web updates start again)
  • sensor60  11 — Force a TASMOTA time update on each GPS packet
  • sensor60  12 — Stop time updates on GPS packets
  • sensor60  13 — Set TASMOTA lat/long values from GPS positional data
  • sensor60  14 — Start virtual serial TCP server (UBX data) on port 1234
  • sensor60  15 — Pause virtual serial TCP server

USES
LATITUDE/LONGITUDE

Now that we have our commands down pat, we can put the GPS data to use.  The first (and possibly the the most immediately useful) is command 13, setting the latitude and longitude values from the GPS reported values.  This will allow the TASMOTA sunrise/sunset timer functions to work without any manual intervention (that is, even if your timezone isn’t correctly set, having longitude and latitude available from the GPS allows the sunrise/sunset algorithm to calculate those values).  Note that there was an issue with this function in the initial release, so you should use TASMOTA version 8.1.0.9 or above (the link in part-I of this article is to a good, working version).

NTP SERVER

Don’t be confused by this.  TASMOTA already has NTP client capability baked-in (it will listen to other NTP servers to set its own time), but this option, command 9, starts an NTP server process which allows other systems to interrogate your ESP8266 on port 123 for time data (your ESP becomes a network time provider, instead of just being a consumer).

As you might expect with such constrained memory limits, this implementation has some limits.  The stratum level  (the measure of how many levels, or hops, we are removed from the physical source of the time signal) is hard-coded to “2”.  In a normal configuration, the time source itself counts as stratum 0 and the machine connected to it would be stratum 1, which would be treated as a preferred source by downstream clients.  However, our ESP8266 isn’t using  pulse-per-second synchronization of the time data (it’s simply publishing it on arrival), so marking it as a stratum-1 primary source could lead to problems (especially if you happen to be a day-trader).  The good news is that it is still a very accurate source of time data in certain situations.  For instance, if you have an isolated network of IoT devices, accuracy to the nearest second is probably acceptable and having an ESP8266+GPS combination providing NTP services is a low cost, low power option for a network of wireless data-loggers.

There’s another point you need to be aware of if you’re setting this up on a network with other NTP servers present; the ESP needs to be configured as a “server” in the ntp.conf file on the other machines, not as a “peer”.  This is because being a peer requires two-way communications between the machines and this implementation only has the capability of replying to a simple request for a time packet.  If you configure the ESP as a peer, the other machines in the network will ignore it and show it as being stuck in the “INIT” state.  If you configure it as a server however, the other machines will happily request time data from it …although, in my case anyway, the data from the ESP is so far off the time reported by the other systems in the NTP group (that is, local NTP servers on my network as well as “upstream” NTP sources on the internet) that the ESP is almost immediately marked as an “outlier” (with a “-” character in the first column of the “ntpq -p” output) and subsequently ignored.  It could still be quite useful, though.  If you are relying on upstream NTP servers on the internet (which is generally a good thing), then you would lose synchronization if your internet connection was down for any significant amount of time.  Your ESP though, would still be providing a local time source with one second accuracy, which is still pretty good (day-traders excepted).

The load of updating the GPS readout (as seen in the photo above) and providing time critical replies to NTP requests is also a little bit too much for the ESP8266, so you’ll see a message, “NTP Active” appear in that window and the GPS updates will stop while the NTP server process is running.  Once the “Sensor60 10” command is given to stop NTP, the GPS updates will restart and the “NTP Active” message will disappear.

One final wrinkle to the NTP implementation is worth noting (although it’s more of an “interesting characteristic” than a bug).  The “refid” field in the output of  “ntpq -p” command (on another NTP client on your network, not on the TASMOTA device) has two different modes of output.  Normally it will display the IP address of the machine to which the indicated peer is currently listening, but if the peer is a stratum-1 clock source, it will print one of a limited number of strings to let you know to what type of a clock device it is connected (ie:- radio-clock, atomic-clock, etc).  Now because our ESP is directly connected to a clock-like device (the GPS) but, as mentioned earlier, is pegged as a stratum-2 device, the refid output field displays the highly unlikely IP address of “71.80.83.0” (hint:- if you haven’t already guessed why it should do this, try “man ascii” for the answer).

LOGGING

If you happen to take your ESP8266/GPS combo with you when you’re hiking, paragliding or orienteering, then this section might be of interest.  Christian has added the option of logging GPS data to the ESP8266 flash, so you can save your experience for perpetuity (well, you can probably save some of it anyway …there’s not that much flash on an ESP8266).  This is the reason for the very oddly named “#FLOG” setting in the TASMOTA config file (nothing at all, as far as I know, to do with fifty shades of black and blue).  If that setting is un-commented, then flash-logging of GPS data is enabled and commands “sensor60 0” and “sensor60 1” set the log handling mode to “overwrite” or “stop writing when flash is full”, respectively.  Commands 4 and 5 both start the log recording, but 4 starts in append mode, while 5 starts a completely new log, overwriting whatever data was there.  When you start a recording the “Flash-Log” status line in the TASMOTA main menu window (see photo, above) changes from “ready” to “recording”.  Command number 6 stops the recording.

Once you stop a recording, a new, blue button will appear above the GPS information in the main menu.  There is no text displayed on the button (at least there isn’t in either Firefox or Chrome) until you move your mouse pointer over it, at which point the message “Download GPX-File” will appear, flashing on and off in time with the updates to the GPS data (perhaps this works as a normal button in some other browser?).  In any case, clicking on the button will download the flash-log data to your system.

VIRTUAL SERIAL PORT

You can also connect to a virtual serial port on the ESP8266 to get a live view of the (binary!) data streaming from the GPS unit.  The “sensor60 14” command will start this virtual serial connection on port 1234 and command 15 will stop it.  If you’d like to check the data, you can connect to your ESP using:-

nc  IP-ADDRESS  1234  |  od -x

Christian mentions that the intended use for this (other than trying to read hex data very quickly) is to feed into the u-blox “u-center” application, which is a debug and management application, freely available from the u-blox site (recent revisions are for  Windows only, though).

The Bottom Line

Christian has produced a very versatile and interesting application, adding an exciting new sensor option to the TASMOTA family.  He’s not finished yet, either.  The most obvious challenge is trying to get PPS working (which should significantly increase the accuracy of the NTP server function) without making the ESP8266 completely unusable for any of the other options running on it.  Even without that addition though, this is still a useful addition to TASMOTA and is certainly a lot more fun to play with than most other sensors.

If the idea of an ESP interfaced to a GPS module tickles your fancy (as it did mine), there’s also Chris Liebman’s stand-alone ESPNTPServer project, which I’ve mentioned before in these pages and which now has its own PCB and 3d-printed case.