Keeping your local ESP8266 source trees up to date

I have a burgeoning directory on my laptop which holds all of the source code for any ESP8266 project I’ve ever come across and thought “This looks interesting…”.  Through the magic of Git, it usually only takes a few seconds to clone a complete project to local disk and have a permanent local copy which you can play with.  The only problem being that the code usually gets stale fairly quickly and you never know when the original author has pushed out an update.  To this end, I knocked together a very simple script which simply lists all of the subdirectories in my ESP8266 master directory and then goes off and does a “git pull” for everything it finds …assuming that the subdirectory is actually a git structured repository and is not flagged with “Do Not Update”.

Git being git, it doesn’t actually care whether the remote, original repository is on GitHub, PasteBin or Bob’s_basement_box, it’ll happily update from the stored URL, whatever and where-ever it is.

The “Do Not Update” feature is there in case you’re lazy (like me) and tend to make working directories with local modifications (which would fail to update and cause errors), or just because you want to maintain a pristine, original snapshot of a repository at some particular point in time.  You can tell the script not to update a repository in two ways.  The first is to add the name of the subdirectory to the file Do_Not_Update.txt in the top-level ESP8266 directory.  The second is to create a file named “Do_Not_Copy” in the actual repository subdirectory itself (the file can be empty, or you can use it to store a note on why you wanted to prevent updates).

The script will keep a (fairly verbose) log of the update process (by default, it is written to /var/tmp/ESP8266_auto_update.log).

The only thing you must change for your particular installation is the path to the top-level ESP8266 directory on your machine (which can be called anything you wish, BTW), “WK_DIR”, which is right at the top of the script.  There’s a sample cron entry in the README.md file which will run the script once per week.

I’ve been running this early every morning for my ESP8266, ESP32 and a couple of other directories for well over a year now and my morning routine now includes a quick “ls -alrt” of those directories to see what’s been updated in the last 24 hours  …a good way of staying abreast of Theo’s amazing update rate on TASMOTA.

Advertisements

An interesting new Sonoff (ITEAD) product on the way

Browsing through the wiki pages on the ITEAD site is always a good way to pass a few idle minutes and usually rewards the curious reader with interesting stuff (like schematics, for instance) which ITEAD are kind enough to publish for our edification.  Sonoff 433MHz to WiFi bridge, block diagramToday’s snippet was some information on what looks like an as-yet unannounced product, a WiFi to 433MHz gateway module.  The schematic shows this as an ESP8266-based unit, but there’s no separate flash memory chip that I can see and the block diagram refers to an ESP8285 (shame!).  There are both transmitter and receiver sections on the 433MHz side and it appears to use an EFM8BB1 “Busy Bee” 8-bit microcontroller to interface between the 433MHz RX/TX section and the ESP UART, with what looks like a slide switch (S2 on the diagram) to disconnect the Busy Bee to allow for programming of the ESP.  The device itself receives external power via a micro-USB socket.

Depending upon the price (and ITEAD prices are usually pretty reasonable) and the range of the 433MHz components, this could be a neat little device to have around. Front and back views (photo courtesy of ITEAD) It’s not just all of those older 433MHz switch modules that have been available for years, but also the slew of devices which just transmit (doorbells, weather stations, window interlocks, etc).  There does seem to be a four device limit on the number of remote 433MHz modules supported by the stock firmware though, according to the User’s Guide.

Update – ITEAD have just sent out a “Mid-Year Carnival Sale” promotion which features this unit (with the photo above) but, bizarrely for a sale, without a price.

Update 8th Aug 2017 – The main sales page is available on ITEAD’s site now and, for a time anyway, the unit is available at an introductory price of $9.90 (down from $12.90).  There are some clarifications of the details too, with the supported device limits being shown as “up to sixteen 433MHz RF devices” or “up to four 1-4 button 433MHz RF Remotes” (so basically 16 addressable channels).

As expected, Theo isn’t far behind and Sonoff-TASMOTA has had support for the 433MHz RF Bridge incorporated since the 5.5.0 version (released on July 30th), with further updates to the code added in v5.5.1.

Save power by (reliably) switching the ESP WiFi on and off

-Update Oct 2017-  Doug Elliot very kindly submitted a comment (see below) to let me know that the redoubtable Erik Bakke has done a much better job at both troubleshooting and writing up this issue in a series of articles on his blog.  Rather than wasting your time with my half-hearted fumbling around, I’ll point you directly to the first article in Erik’s series.  Stay with it and read the whole bunch right through to the end; he keeps on coming up with new tweaks as he goes along.  There are a couple of extra articles (one on eliminating network scan at start-up and another on solar power for the ESP8266) which are part of the series, so don’t miss them, either.


I have an ongoing project to add an ESP8266 to a solar trickle charger for a lead-acid battery.  Really, the only set-in-concrete requirement is that the controller must switch off the feed from the solar panel before the battery voltage gets to dangerous (“gassing”) levels, but it would be nice to have remote data logging of voltages (battery and solar) and temperature, as well as some control over the load (a small impeller pump in an external sump well, used for orchard irrigation during the summer and autumn).

Because the target battery is a standard (wet plate) car battery, I have quite a bit of capacity to play with, but even so, my back-of-the-envelope calculations show that the ESP8266 is quite capable of completely discharging the battery in under a week (assuming no sunshine and an ESP8266 module with WiFi enabled and transmitting on a more or less continuous basis), so I’d been looking for a reliable way to save power, but still have the ESP8266 do real time monitoring of the battery and solar voltages and interrupt the charge process when the battery was already fully charged (I should probably note that we get a lot of sunshine here during the summer and autumn months and our usage of the pump is only very intermittent, so despite having a mere 300mA output from the solar panel, it is possible to overcharge the battery over time).

I didn’t want to use deep-sleep mode (although it’s the least power hungry), because I want the ESP to be the actual charge controller and it can’t do that if it’s off in la-la land for the best part of its day.  I still want to use the WiFi capability (otherwise there’d be little point in using the ESP over a PIC), but I don’t need to have 24/7 connectivity.  Turning the WiFi on and off has always been a bit of a hit and miss business for me, though (and with quite a few other people too, judging by the number of hits a web search on the subject generates).  So I was pleasantly surprised to find some answers in a GitHub “issue” raised against the ESP8266/Arduino project.  The thread contained some excellent answers and testing experiences from some of the well known members of the ESP8266 community (thanks guys!), so I felt fairly sure I was onto a winner, but there were one or two twists along the way.  So here is a brief rundown on how I’m doing it with those kinks ironed out and here is a program which will demonstrate the practicality and reliability of using the “Issue 644” method.

First, let me assure the nervous reader that, despite what follows, we are not putting the ESP to sleep at all, we do not need to use an interrupt or a reset to restart the WiFi and we also do not need to make any hardware changes to the ESP module for this method to work.

The crux of this method is to use WiFi.forceSleepBegin() to put the ESP into “modem sleep” mode.  It’s important to realize that this does not put the core into any form of sleep; it simply turns the modem (radio) off.  There are some other incantations in there to ensure that WiFi is shut down in  an orderly fashion:-


bool WiFi_Off() {
WiFi.disconnect();
WiFi.mode(WIFI_OFF);
WiFi.forceSleepBegin();

One of the issues at this point though, is that the WiFi can take a finite, but variable time to shut down after the final forceSleepBegin() call, so at this point I’ve just added a timeout loop, checking the status of the connection until it goes down (returning without an error) or times out (returning with an error):-


while ((WiFi.status() == WL_CONNECTED)
&& (conn_tries++ < WIFI_RETRIES)) {
delay(100);
#ifdef DEBUG
Serial.print(".");
#endif
}
if (WiFi.status() != WL_CONNECTED)
return (true);
else
return (false);

The default timeout of 3 seconds is more than enough for the WiFi_Off() function.

The WiFi_On() function is pretty much a reversal of the “off” code:-


bool WiFi_On() {
conn_tries = 0;

WiFi.forceSleepWake();
WiFi.mode(WIFI_STA);
wifi_station_connect();
WiFi.begin(STA_SSID, STA_PASS, WIFI_CHANNEL, NULL);

while ((WiFi.status() != WL_CONNECTED)
&& (conn_tries++ < WIFI_RETRIES)) {
delay(100);
#ifdef DEBUG
Serial.print(“.”);
#endif
}
if (WiFi.status() == WL_CONNECTED)
return (true);
else
return (false);
}

The call to “wifi_station_connect()” seems to be the equivalent of a sacrificial chicken for most people, though. Without it, the restart of the WiFi doesn’t always work; with it, everything is hunky-dory (whatever that means).

Unfortunately, actually getting the call to wifi_station_connect() to compile turned out to be a bit problematic for me (from within the PlatformIO build environment). Just where the heck is it? Well, some searching found a define for it in the user_interface.h file in Espressif’s SDK. Adding a simple:-

#include "user_interface.h"

…in the header of my ESP_Power_Save.ino file got me past the initial “not found” messages, but then, at the last stage, the linker failed with “undefined reference” errors. Grrrr! The magic sauce for that though, is simply to let the C++ compiler know that it’s looking for a plain C function (buried deep in the nether regions of the Espressif SDK, in this case) by modifying that include to be:-

extern "C" {
#include "user_interface.h"
}

Finally, it compiled …and worked, too.

Before you flash ESP_Power_Save.ino to your ESP8266, you’ll need to configure the IP addresses, access-point SSID and password in the user_config.h file.

After flashing, connect to the ESP8266 console as normal and you’ll find yourself in a mini test environment where you can use a few simple, single character commands:-

  • “w” — to toggle the state of the WiFi between on and off.
  • “s” — to display the current state of the WiFi connection.
  • “c” — to display a simple count on the console, just to prove that the ESP core is still operational.

I’d recommend opening a second window and using it to ping the ESP’s IP address, to verify whether it is up or not as you toggle the WiFi.

Finally, a word of caution …If you try to connect to a non-existent access-point, or one which is out of range, or you use an incorrect password, then the WiFi will not turn off. That seems pretty counter-intuitive to me and I haven’t yet worked out why it happens. At any rate, it’s obviously worth doing some testing (I use a hand-held ESP8266 rig running on three AA batteries) to make sure that you have connectivity before deploying this power-save code to your remote units.


Update — The latest version of the code now includes a cut-down version of the “FSWebServer” example (from the ESP8266WebServer library) which will server up a 750kB JPEG file from SPIFFS on the ESP.  It takes a non-trivial amount of time for the little ESP to retrieve such a substantial file from SPIFFS, chunk it up and push it out across the network, which hopefully will give you adequate time to measure the current consumption of your module while busy transmitting, for comparison with when the WiFi is toggled to the “off” state.


Update #2 –  Real world testing has brought some interesting “features” to light.  Most problematical is that the WiFi doesn’t seem to always switch off completely, which is peculiar.

First, you need to know that the ESP8266 returns its current status when interrogated with the WiFi.printDiag() request.  When running normally, the results look something like this:-


Mode: STA
PHY mode: N
Channel: 11
AP id: 0
Status: 5
Auto connect: 1
SSID (9): Mainwaring
Passphrase (10): D0ntPan1c!
BSSID set: 0

The two most important indicators are the Mode and Status lines (although the SSID and Passphrase entries will also change when the WiFi is disabled).

The current draw on  my test module is generally in the area of 75ma when WiFi is enabled, with very brief peaks when actually transmitting (so brief that they’re impossible to measure with an ordinary digital multimeter).  That value drops off to 20ma when the WiFi is disabled and at the same time, WiFi.printDiag() changes to show “Mode: NULL” and “Status: 255” (and the module stops servicing web requests or even  a simple “ping”).

Frequently, toggling the WiFi (with the “w” command) will give the same result in terms of connectivity (ie:- the unit stops responding to web requests or pings) and the result of a WiFi.printDiag() is still “Mode: NULL” and “Status: 255”, but the current draw remains in the 75ma region.  I suspected that the auto-connect or auto-reconnect settings may have been responsible, but after further testing that doesn’t seem to be the case.

So, after using the word “reliably” in the title of this post (which, admittedly, referred specifically to being able to turn WiFi back on after a modem sleep), I now find that the current reduction isn’t reliable at all.  Does anyone have any ideas?

Killer (ESP8266) Apps for the Weekend

Sometimes, while browsing GitHub for ESP8266-related projects, I come across a user’s page which just happens to “push all the right (ESP8266) buttons” for me.  One which I came across recently was from martin-ger and he has several projects which tickle my fancy.

Those three should keep you amused over the weekend and give you a sample of what Martin is capable of.  Definitely a page worth bookmarking!

 

 

ESP8266 Overclocking & Speedometer

A few days ago I posted the code for a(n almost unusable) version of Zork (or actually, the Zmachine) for the ESP8266.  It was (and is) really, really slow.  The main problem appears to be that it constantly re-writes the Zmachine stack back to SPIFFS which, in addition to causing the running-through-chest-deep-molasses effect, will also wear-out your flash in double-quick time.  The fix for this (obviously) is to shoehorn the whole thing into main memory (which is a work-in-progress), but while trying to squeeze enough performance out of the ESP8266 to make this early cut at least somewhat playable, I thought I’d take the easy option first and run the ESP at 160MHz instead of the default 80MHz.  I should warn you right now that as far as making Zork playable went, it was a total failure, but I was pleasantly surprised at just how easy the speed selection is when using PlatformIO.  I didn’t even have to sacrifice a chicken.

Here’s the simple incantation, which you just need to add to the bottom of your platformio.ini file in the project directory (the comment line is optional):-

; Set the ESP8266 clock frequency to 160MHz
board_f_cpu = 160000000L

…and then just recompile (and, because you’ve just made a change to the platformio.ini file, PlatformIO itself is smart enough to know that it needs to do a complete rebuild, not a partial).

That’s it …you’re done!


 

NOTE – You can also use the system call “system_update_cpu_freq()” to dynamically update the ESP8266 clock frequency from within the program itself, instead of using the platformio.ini “board_f_cpu” setting.  Using the system call will override the compile-time setting.

CODE – Ray Burnette’s adaptation of the Dhrystone test program is available from my repository in both static and dynamic versions.  The static version simply loops endlessly, reporting the performance of the current module (so effectively the compile-time clock setting from the platformio.ini file).  The dynamic version toggles the clock speed using the system_update_cpu_freq() system call on each iteration and displays the performance for whichever the current setting is.

 

 

A present..

…just not a very good one!   🙂

I know that all of you ESP enthusiasts of a certain age will have been waiting for this with bated breath, so without more ado, from the “Why? Because we (almost!) can” department, here’s:-

Zork for the ESP8266

This is a port of Louis Davis’ Arduino Z-Code interpreter to the ESP, but before you get too excited, I should tell you that it is slow …really, really slow.  It’s still a work-in-progress, with a ton of very, very rough edges, but it does actually work (after a fashion).

It uses SPIFFS to store the game files and the included “data” directory has a couple of other games in there, in addition to the default “minizork.z3” file.  All you need to do is change the “G_FILENAME” define in user_config.h to run one of the others.

Using PlatformIO (if you’re not, why not?!?!) SPIFFS can be initialized and uploaded with:-

platformio run -t buildfs

and

platformio run -t uploadfs

(the uploadfs command will take a couple of minutes to complete).

The code can be uploaded using:-

platformio run -t upload

Cat successfully skinned!

Regular visitors may remember a mention I made of an ESP8266/433Mhz gateway project a couple of weeks back.  Well, one of the reasons I found it so interesting was that I have a (fairly long-in-the-tooth) Oregon Scientific weather station installation which has always frustrated me with its lack of connectivity.  It does have a 9-pin, D-type serial connector on the bottom, but that assumes that you have an RS232 equipped machine within cable reach of the display unit (and that HID‡ would not object to yet another trailing cable).

I had toyed with the idea of plugging an ESP8266 into that port, with the excellent serial adapter  firmware from JeeLabs, but it doesn’t really address the HID/cable/ugliness issue.

Both of these methods also suffer from a fatal design flaw with this particular model of weather station, in that data isn’t squirted out of the serial port unless all of the sensors which this model was sold with are operational (otherwise you just get an error message along the lines of “Rain sensor not detected” and nothing else).  So I concluded that, by collecting the (433Mhz) transmitted data from the actual sensors (which are remote from the display unit), I could just use the data directly and ignore the base-station/display part of my weather station completely.  Hence my interest in the 433Mhz gateway project.

The final piece of the puzzle to drop into place for one of those light-bulb moments came when I was reading through the comments to one of Pete Scargill’s recent articles on the 433Mz RFLink project.  Commenter Paul gave a link to a GitHub repository called “rtl_433”, which is a 433Mhz decoder for SDR dongles by Benjamin Larsson.  Benjamin’s project is specifically for picking up the data from remote sensors (from many, many manufacturers) which operate in that open, 433Mhz band.

I’d recently bought an SDR dongle from a vendor on Ebay which was advertised as having an R820T tuner chip, suitable for ADS-B monitoring.  It turned out to be a bogus ad, with the actual tuner chip being a 0012, which doesn’t even cover the 1090Mhz ADS-B band.  I threw the useless dongle into the drawer and ordered a decent one directly from the manufacturer (which, incidentally, has worked perfectly from the first moment it was plugged in – Nooelec.com is the place to go), writing off the $10 Ebay one to experience.

Having seen Paul’s post, retrieved and compiled Benjamin’s “rtl_433” package and pulled out the “useless” dongle from the murky depths of the spares drawer, I had direct data from all of the Oregon Scientific sensors published to MQTT in less than five minutes after plugging it in.  One cat neatly skinned in a completely different way to that which I’d originally envisaged.

Just for your reference, here’s the simple pipe to publish your data:-

./rtl_433 -F json   |  mosquitto_pub -l -h hazeltonrig.throgmortons-bottom.org -t sensors/rtl_433

The “-F json” argument to rtl_433 is to force the data to be output in JSON format.  It will also accept “csv” (comma separated values) and “kv” (key:value pair) format arguments.

And here are a few lines of sample data from the various sensors:-

{"time" : "2017-03-21 15:52:24", "brand" : "OS", "model" : "THGR968", "id" : 204, "channel" : 1, "battery" : "OK", "temperature_C" : 8.300, "humidity" : 49}
{"time" : "2017-03-21 15:52:24", "brand" : "OS", "model" : "BHTR968", "id" : 66, "channel" : 0, "battery" : "LOW", "temperature_C" : 19.700, "temperature_F" : 67.460, "humidity" : 30, "pressure" : 946}
{"time" : "2017-03-21 15:52:25", "brand" : "OS", "model" : "BHTR968", "id" : 66, "channel" : 0, "battery" : "LOW", "temperature_C" : 19.700, "temperature_F" : 67.460, "humidity" : 30, "pressure" : 946}
{"time" : "2017-03-21 15:52:26", "brand" : "OS", "model" : "WGR968", "id" : 183, "channel" : 0, "battery" : "OK", "gust" : 1.400, "average" : 1.400, "direction" : 902.000}

Note that the “direction” readings are meant to be in degrees, so 902.000 (last line) doesn’t make much sense. Looking at the code and at the actual weather station display, it seems like there’s a nibble ordering issue with the decoding of the raw data and that should actually read “290”, instead (the fix is tested here and is now working its way backup the line).


‡  “Her InDoors”