“Xreef” (Renzo Mischianti) has just updated his excellent LoRa E32 Series Library to fix a memory leak issue. All of his examples have also been updated to include the fix, so if you’re thinking of playing with some of the EBYTE manufactured boards and an ESP8266 or ESP32, this is the go-to library (and Renzo has also produced a couple of adapter boards to make it easier to use the E32 boards with your favourite ESP).
NOTE:- This article was originally written mid 2019, but was never posted (it seems that I received a non-maskable interrupt in mid sentence and never got back to it). Prices quoted are probably no longer valid, but I note that the systems themselves are still available from the supplier linked-to below.
As noted a couple of months back in the Odd Bargains post, I’ve been experimenting with some low-cost Celeron-based systems (in addition to the original Z8350, Atom-based unit, which started me along this particular track) as cheap, complete servers for our home network. The main advantages for me are good OS support for most (but definitely not all) peripherals, RTC and battery as standard and they all come complete with a case and power-supply included in the up-front price. They are a lot cheaper to run too, as these low-end processors were originally intended for laptops and tablets rather than full blown PCs. While there’s no denying that the CPU performance is generally nothing to get too excited about, they (the quad-core units, especially) still work remarkably well as 24/7 infrastructure servers for services such as DNS, NTP, DHCP, low volume web servers and reverse proxies. Most ,but not all, come with a GbE port and are quite capable of handling significant amounts of traffic (…watch out for the low-end “ACE PC” branded models though, as they only have a 100Mb port), but all GbE chipsets are not created equal and my tests with a cheap, external USB-3 to GbE dongle (as a super-budget firewall) were a resounding failure (the internal port on the Celeron box could handle the traffic, but the dongle would give up the ghost after 2 or 3 hours).
I found along the way that there are quite a few, virtually identical systems in this price range which have completely different chipsets. Most of the very low cost machines come with Realtek chips, which research on the ‘net shows to be less than ideal for a firewall (the symptoms reported are similar to my own experience with the USB-3 dongle). This isn’t to say that the Realtek chips are to blame (there are lots of other variables in the mix), but it is fairly common to see posts recommending Intel chipsets for long term reliability under heavy load. So, after playing around with a couple of systems that I actually have and taking into account reviews and research, I eventually came down to the choice of a J3160 based system with four, Intel-based GbE ports (the J3160 because it’s a quad-core chip with slightly better performance than the Z8350 and (importantly) with AES-NI hardware cryptography support and four ports because I need to provide for a couple of “guest” networks, firewalled off from the main, home network).
I found a reputable looking supplier on Alibaba who had a lot of good feedback and decent prices (they sell under the names of “Yanling”, “Minisys” and “iWill”). The model I chose was their Nuc-C3L4. In addition to the four (Intel) GbE ports, it also comes with dual-HDMI, 2x USB-3.0 and an RS232 console port. The cost for the bare-bones unit is was $142.60, but that doesn’t include shipping (which was an additional $20 for my location). This vendor does accept PayPal, but only from a limited range of countries (and mine wasn’t one of them). Depending upon where and how you’re shopping, you can probably get 4GB of memory and a 64GB eMMC card for an additional $30, or so. And yes, because I have used this supplier and had a good experience with them and their products, I do recommend them. Communications with them (in English) were easy, fast and friendly. Their shipping was also fast and their packing is excellent (the boxes are sturdy and the units are completely surrounded by a custom, expanded polystyrene foam cushion …which may not be very environmentally friendly, but certainly is effective). The power supply, mains lead and included VESA mounting plate are separated from the system itself by a cardboard divider and all of the individual parts (including the system) are enclosed in their own plastic bags to keep moisture at bay. Once out of the bag, the unit proved to be very black, very shiny and of all metal construction (unlike the Beelink AP35 which I wrote about a couple of weeks back). It looks very nicely made and well put together and it’s obvious that someone put more than a couple of minutes of thought into this very compact design (for instance, the bottom of the unit has ventilation slots and it is secured to the body with four, asymmetrically offset screws, so that it’s actually impossible to attach it in a way which would block those slots).
I ordered memory and an mSATA SSD module from Amazon here in Japan and actually got a pretty good deal. If you do buy one of these units, it’s important that you only use the low-voltage (1.35v) variants of the DDR3 SODIMM, though; this system won’t work with higher voltage rated memory.
Here’s where I hit a very small speed-bump in the road to getting it all working. It turns out that the motherboard slots are not identical. You have a 50/50 chance of getting it right when installing the mSATA …and I got it 100% wrong.
As you can see, the mSATA module needs to be plugged into the right-hand socket to be correctly recognized by the system.
Take another look at that “This way!!” photograph again. The first point of note is the RTC battery (the yellow blob in the bottom, left-hand corner). This system comes with an RTC and battery, which means any Unix-based OS works right out of the box; just tell the OS what timezone you’re in and you’re done. Notice also the row of RJ45 sockets at the L/H side. If you click on the image to get the full-sized version (it will open in a new tab), you can easily read the MAC address assigned to each port. It’s probably worthwhile making a note of those (they’re sequential) while you have the bottom off, to help with identification later.
You can also see there’s a SATA port available on the motherboard, but with this particular model there’s no space available to fit an internal drive.
Here’s a partial dmesg output from the machine once FreeBSD was loaded, showing the CPU features for the J3160:-
Copyright (c) 1992-2018 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 12.0-RELEASE-p4 b0ff15badd(RELENG_2_5) GENERIC amd64
FreeBSD clang version 6.0.1 (tags/RELEASE_601/final) (based on LLVM 6.0.1)
VT(vga): resolution 640×480
CPU: Intel(R) Celeron(R) CPU J3160 @ 1.60GHz (1600.05-MHz K8-class CPU)
Origin=”GenuineIntel” Id=0x406c4 Family=0x6 Model=0x4c Stepping=4
Structured Extended Features=0x2282<TSCADJ,SMEP,ERMS,NFPUSG>
Structured Extended Features3=0xc000000<IBPB,STIBP>
TSC: P-state invariant, performance statistics
real memory = 4294967296 (4096 MB)
avail memory = 4002127872 (3816 MB)
Event timer “LAPIC” quality 600
ACPI APIC Table:
WARNING: L1 data cache covers fewer APIC IDs than a core (0 < 1)
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s)
Close to the end of the “Features2” line, you’ll see “AESNI” included. These are Intel’s “Advanced Encryption Standard – New Instructions” which enable faster, hardware-assisted encryption and decryption. Although this technology is now available on some other processors (including ARM), older Intel processors don’t have it (for instance, the predecessor to this mini-pc system had a Celeron J1900 processor, which doesn’t have AES-NI), so a J3160 is worth the extra few dollars if you’re planning on a VPN server, for instance.
I actually got two of these systems, one for my own use (to replace an ancient firewall box) and one for a remote site. They’ve now been running for almost exactly a year (since I began this article in early August, 2019) and have been totally reliable during that time. I did think that the heat-sink was running a little bit on the hot side when I first installed them, but the “touch test” is deceptive and even during the mid-summer heat, the processors barely register 50°C.
The systems handle the traffic we pull through them without any problem and can handle multiple firewalled VLANs, encrypted VPN traffic and multiple physical networks with ease (as well as handling the normal associated processes — NAT, DHCP, DNS, NTP, etc). I’d like to emphasize what good value these little systems are. Not only is the initial purchase price very low, but the lack of fans and the 6W (avg TDP) processor mean the power requirements (and hence the monthly electricity bill) are low, too. I was also very impressed with the quality of the (all metal) case, the general design and workmanship, as well as the packing and delivery from the vendor. They may not be as cheap as a Raspberry Pi, but the quality of the case, included PSU, cables and accessories, as well as the RTC (and battery) and four GbE ports more than make up for that. You won’t be running a 20TB database with 200 concurrent users on one of these machines, but for moderately light 24/7 operations for a good sized household or small business, I don’t thing you can go far wrong.
Nowadays I can easily forget what I’ve already done. When it comes to ZFS, that includes forgetting what pools I’d already created on a particular device. That’s the point of this “memo to self” …zpool import is your friend.
When you want to import the pools after juggling disks between machines, but can’t remember what the heck you named the pools in the first place, just do:-
…to display a list of all available pools which are not currently attached (or a message to the effect that there are none available). The important point here is that it won’t actually try to actually import any pools with this simple command; it only lists them.
If you have a pool which is shown as available for import (ie:- it exists, but is not currently attached — which might be the case if you’ve just physically moved a disk from some other machine to this system), you can temporarily have it imported and attached to a temporary mount point using the “-R /tmp/NAME” option. This can be really handy for recovering data from a “retired” drive. For instance, an old disk has a “data_tank” pool, which had the mountpoint “/store” on the old machine. You want to recover some data from that drive, but your current system already has “d_pool” mounted at “/store”, so you can’t simply do “zpool import data_tank”, because the mountpoint is already in use. Your import command should instead be:-
zpool import -R /tmp/old_store data_tank
…and all of your old data will be mounted to /tmp/old_store where you can access it normally. But wait, there’s more!
If you already have an existing pool named “data_tank” on your current system, you can have “import” rename the old pool to something different (to prevent embarrassing mistakes) by simply appending a new name to the previous command:-
Now when you do a “zpool status” or “zpool list” the pool mounted from the old disk will show up as “old_data_tank”.
When things go awry
What about if you’ve already suffered some late-night brain fade and just destroyed the live pool on your current system by mistake? Well, as long as you realize your mistake fairly promptly and haven’t already scribbled all over the disk, import can help you with that, too. The “-D” option will show you any pools which have been destroyed, but for which ZFS can still find valid metadata. So:-
zpool import -D
…will display all pools, including previously destroyed ones, which still appear to be available for importing.
…will import the original “data_tank” pool (note the -D -f options to force import of a previously destroyed pool) with the new pool name of “old_data_tank” and mount it at /tmp/old_store.
More pool recovery tricks
There are additional options to “import” which can further aid recovery of incomplete pools (see the zpool manual page entries for import and check the “-F”, “-m” and “-n” options for more information on how import can provide extra help for getting out of sticky situations).
Doing things nicely
While the import command will do it’s very best to save you from yourself, you can help things along considerably by doing the right thing and using the “export” command on any pool which you intend to re-import elsewhere at a later date. Note that this command will make the target pool unavailable on the system where you run it (that’s the whole point …to effectively shut down the pool cleanly and prevent any further modifications by marking it as still being reserved space), but the subsequent import shouldn’t have any difficulty at all when re-importing a previously exported pool.
Regular readers will probably know by now that I’m enamoured of the tiny, all-in-one, Intel-based systems generally referred to as “Mini-PCs”. They are more expensive than a Raspberry Pi, but they come with a case, a PSU, an on-board RTC, generally a decent amount of memory and sometimes built-in eMMC storage, too. Prices have increased over the past few months (the C-19 effect, again), but the bottom-end models (generally the quad-core Atom Z8350 equipped systems) are still available for around the $100 range (GearBest, FastTech, CDiscount, etc).
I already have a few of these mini systems and one which underwhelmed me on first impressions was the AP35 Beelink J3355-based box, mainly because it was advertised as “fanless”, but does have a CPU blower and because the eMMC just would not work reliably. I gave up on the eMMC and put in a small, internal SSD and since then it has been giving sterling service as a mini NAS (ZFS filesystems with TimeMachine as a backup server for various Macs) as well as running a couple of Bhyve virtual machines (doing very light duty). Once I stopped trying to use the eMMC and put it in the basement “computer room”, my opinion of it increased considerably. It’s a little work horse and, although not a speed monster, performs well enough for my requirements and has been absolutely reliable over the past year (again, since I gave up on the eMMC).
I mention that system specifically simply because I’ve been keeping an eye on prices, with the idea of adding another machine to the collection, if the price is right and the target machine has more than two USB-3 ports. I’ve noticed that there are a few other J3355-based models appearing, with a form-factor pretty much the same as the Z8350 systems and priced in the same general area (ie:- the bottom end of the price range). One thing which I’ve noticed recently though, is that almost all of the advertisments for these systems specify the J3355 as a “quad-core” CPU chip. It isn’t. Here’s an excerpt from the dmesg output of the AP35:-
FreeBSD 12.1-RELEASE-p6 GENERIC amd64
FreeBSD clang version 8.0.1 (tags/RELEASE_801/final 366581) (based on LLVM 8.0.1)
VT(efifb): resolution 800×600
Skipping TSC calibration since no legacy devices reported by FADT and CPUID works
CPU: Intel(R) Celeron(R) CPU J3355 @ 2.00GHz (1996.80-MHz K8-class CPU)
Origin=”GenuineIntel” Id=0x506c9 Family=0x6 Model=0x5c Stepping=9
Structured Extended Features=0x2294e283
Structured Extended Features3=0x2c000000
TSC: P-state invariant, performance statistics
real memory = 4294967296 (4096 MB)
avail memory = 3894444032 (3714 MB)
Event timer “LAPIC” quality 600
ACPI APIC Table:
WARNING: L1 data cache covers fewer APIC IDs than a core (0 < 1)
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
FreeBSD/SMP: 1 package(s) x 2 core(s)
Having said all of that, here’s a link to a system on FastTech’s site to an MII-V with 4GB/64GB, GbE and four USB-3 ports which, at $112.45 (with free shipping), seems to be the best, generally available deal on this class of machine at the moment …just remember, it is DUAL-core (no matter what they say).
Development on TASMOTA continues apace (with the ongoing C-19 movement restrictions, possibly even more rapidly than usual) and lots of new and interesting stuff has been popping up in the code recently (for instance, if you want to connect an anemometer, to add wind-speed to your weather-station, Matteo Albinola has you covered). However, one of the most exciting recent additions has been the arrival of baked-in support for the ESP32, based on Jörg Schüler-Maroldt’s work. This first appeared in version 18.104.22.168, back at the beginning of May, after Jörg created the libesp32 compatibility library and an initial pull request back in April. Since that time there have been a lot of “#ifdef ESP32” lines added to the code.
Now (version 22.214.171.124, as of June 17th 2020) we not only have the ESP32 compatibility and compile additions but also the addition of ESP32 hardware Ethernet support with the recently added xdrv_82_ethernet.ino driver file. The new, tasty goodness doesn’t end there, though. If you take a quick look at the headers of that driver file, you’ll find the pinout defines and TASMOTA template for the Olimex ESP32-POE, so not only do we get the ESP32 and ethernet, we also get PoE thrown in (the Olimex board currently sells from their site for €17.95, but is also available from the likes of Mouser and even Amazon in some areas).
If you bought one of the exceedingly cheap ESP32-CAM boards (AIThinker/Geekcreit), you now have the option of TASMOTA enabling it, too. Look for a second new driver file, xdrv_81_webcam.ino, to see the pinout details and TASMOTA template for that device.
Okay, so new, sexy additions to the code, but how do we compile TASMOTA for the ESP32. Well, if you’re using PlatformIO (and you should be!), it couldn’t be much simpler. Copy the Sonoff-Tasmota code into a clean directory (or git clone the repository) and then:-
In that directory, copy the platformio_override_sample.ini file to platform_override.ini.
Edit your new platform_override.ini file and uncomment line number 29 so that “; tasmota32” becomes just “ tasmota32“.
Type “pio run“.
Sit back and watch for a couple of minutes while the compile runs. It will compile two different versions. The vanilla “tasmota” (ESP8266 version) will be created in .pioenvs/tasmota/firmware.bin. The tasmota32 (ESP32 version) will be created in .pioenvs/tasmota32/firmware.bin.
[ For anyone who hasn’t compiled TASMOTA before, the customizations for your local network settings and for the target device itself are made in the ./tasmota directory. The simplest method for a first time compile is to use the user_config_override.h file to update only the bare essentials. Once you’ve made sure that you can compile successfully, you can modify the (very much more complex) ./tasmota/my_user_config.h file for full customization. ]
So, what used to protect your ssh tunnel‡ is now helping to protect (in a small way, I know) against the SARS-CoV-2 virus and, just in case it’s still on the banned list (and we ever get to travel again), the offending algorithm will be folded into the inside of the mask, keeping our secret safe.
‡ — If you’re still using Blowfish, you should know that Bruce Schneier, the designer, now recommends upgrading to “Twofish”.
We have a little log house (which we built ourselves and used to live in) which we let out as a holiday cottage. It’s something that is (generally) fun to do and also pays a few of the bills, but one of the big mistakes we made when we put it together our large, wooden Lego kit was to put too many light switches in a single location (if you live in a house long enough, you’ll eventually remember what most of them do, but for short-term visitors it’s just plain confusing, even with little stick-on labels). So one of the things we find happens quite often is that one (or two …or five) of the outside lights are left on all night (and might be left on all day too, if our guests head out in a hurry in the morning). Long ago we changed out all of the bulbs for the little CFL corkscrew types to reduce the overall power usage and we thought we might get around to replacing them all with LEDs, if the prices ever come down.
To compound the mistake with the switches, we also bought almost all of the external lights with E17 size fittings (about half the size of a normal household bulb), not because we really wanted them, but because that was the only size the maker supplied the fittings in. This latter issue came back to bite us when I decided I could fix the lights-left-on problem by replacing the CFLs not just with LED bulbs, but with “Smart” LED bulbs (Wow-weee!).
Nope! No smart bulbs in E17 form-factor (at the time). So I did what any other self-respecting ESP hacker would do and, when one of the CFLs died a noisy, sparky death one day; I chopped off its head, ripped out the charred remains of the inverter and replaced it with and ESP01S, a TRIAC and one of the brightest (and cheapest) E17 LED bulbs I could find. I stuck my little FrankenShine monster (I admit, it wasn’t particularly pretty) into the fitting right outside our kitchen door, so that we could play with it and monitor it without inconveniencing any of our guests during the initial testing and burn-in period.
The FrankenShine in all of its glory …The ESP01S and other DC components are at the extreme R/H end of the board, with the opto-isolator to the left of the programming header. The TRIAC is closest to the camera (slightly left of centre) with the PSU sitting over at the back. You can just see the PROG/RUN mode jumper partially hidden by the big, black electrolytic.
I set up a TASMOTA rule to switch it on at dusk and off again a few hours later and coached my better half in how to communicate her wishes to the nice lady stuck inside that tiny, pinky-orangey UFO thingy that landed on the kitchen counter a couple of years ago (…and always say “Thank you”, because we want our grandchildren to grow up to be polite). So, (her) “Hey Gewjull! Turn on the kitchen door”. (GH, pertly) “I’m sorry, I don’t know how to turn on the kitchen door”. “Hey Gewjull, turn on the effin’ kitchen door!!”. (GH, sullen) “Okay, turning on the effin’ kitchen door”. “Thank you!”. (GH, stilted and mechanically) “That makes me so ‘effin happy”. Ah, another successful foray into the world of home subjugation.
For a while, everything went along swimmingly. In fact we got so used to coming home at night and being able to see the keyhole that the transfer of FrankenShine to the cottage was delayed, several times. Then one evening we came back to a dim, flickering welcome …uh-oh! A power-cycle cured it, but only for about five minutes and then it was back to flickering again. A PSU problem? More likely a TRIAC issue. Gottverdeckel! I pulled Franky out and put the LED bulb back in the fitting (yup, works okay). Unfortunately, as usual with these intermittent problems, Franky worked fine on the test bench, with not a flicker to be seen. I went back outside and gave the light fitting a couple of dunts with the flat of my hand to test for loose connection, but no flickering or dimming was evident. And so everything remained as it was for (quite) a while …until I decided to replace the CFL in the worst offending cottage light fitting with an LED bulb, anyway (Franky or no Franky). The light itself was identical to the test fitting next to our kitchen door, but was some four years older. I opened it up and, lo and behold, it was an E26 holder assembly. “Oh crikey!” said I (or maybe something similar), all of this mucking around and I could have just gone out and bought an E26 “smart” bulb and flashed it with TASMOTA. At about the same time, the E17 bulb in the kitchen door fitting couldn’t contain its hilarity any longer and started flickering and dimming intermittently again (no Franky involved). “Oh double crikey!” (or words to that effect).
The light (in WW mode) is very bright and easily equals that of the CFL. It has the added bonus of being easily adjustable (from sliders on the TASMOTA main-menu page) if you prefer a particular hue of white output (from “cool” blue through to “warm” yellow). If the fancy takes you, you can also fiddle with the RGB settings to have a particular colour and shade, instead of that boring old white.
To help you (well, okay, to help me) test out the functionality and reliability, I’ve put up a simple, command-line exerciser on GitHub. You’ll need to change the variables (at the top of the file) to use your TASMOTA MQTT topic name (variable: BULB_ID) and your MQTT broker (server) IP-address or name (variable: MQTT_SERV). After that, you can just run the program from the command line with no options or arguments (to turn the bulb on in white, neutral hue mode), or use one of the following options:-
-c — “C”ool white. Switches on the WW LEDs in the bulb with a blue hue.
-w — “W”arm white. Switches on the WW LEDs with a yellow hue.
-n — “N”eutral white. Switches on the WW LEDs with the hue set mid-way between cool and warm.
-oor-0 — Switches all LEDs (RGB and WW) off (that’s a zero, by the way).
-s — “S”equence. Turns on various colour mixes of RGB for 2 seconds before fading to the next colour (Control-C to quit).
An Intel-based mini-pc (not even an Atom, but a fairly decent quad-core Celeron) with a case and power supply for less than the price of a Raspberry Pi. Can it be real?
[Update 11th March] — Looks as though these (assuming there ever was more than one) have sold out. 😦 Scroll down for some other, not quite as cheap, but still available mini-pc bargains. — …and the answer is, probably not! Check the delivery cost very carefully before clicking the order button. As far as I can tell, this seller is hoping to charge you €999.00 for express delivery of the first item and €80.00 for each additional item. It’s not clear to me whether that is the default, or whether you can have it shipped “normal delivery” for €4.99; perhaps someone more familiar with on-line shopping in French can comment on that?— Reader “raspi” has confirmed that delivery in France is €4.99 (see the comments section, below), so it does look as though this is a bargain after all (but you should still check delivery costs and final price before confirming your order).
Unfortunately, as I’ve mentioned before, Cdiscount only sell/deliver to a small number of western European countries, so the rest of us are out of luck, but if you’re in France, Spain, Germany or Belgium …get them while you can (and let us know in the comments if you manage to snag one of these at the advertised price). [IMPORTANT – If you didn’t read the update at the top of the page, go back and read it now — verify the shipping cost before you buy!!]
Silence is golden — Although the adverts are usually vaguely worded, you can safely assume that most of these thin, Celeron-based systems have a laptop-style fan hidden away somewhere inside (generally the earlier generation, Atom-based mini-pcs didn’t). The Coofun advertisment above very straightforwardly displays a photo of the heatsink and fan unit, so there’s no guesswork involved.
AMAZON N3150/J3160 BAREBONES
Currently (11th March 2020), most Amazon regions (including the U.S.) are advertising an N3150 or J3160 (virtually identical) based fanless model with 4 x USB3, 2 x USB2, 2 x HDMI and 2 x GbE. Note that this is the barebones price (no memory, no SSD/HDD). The price seems to be pretty much the same for Amazon US, Japan and UK at roughly $135 with free shipping:-
It seems that you’ll be shipped whichever CPU is at hand when your order is received (in other words, you can’t choose). It’s still very attractive, though; this is a Celeron machine at the current Atom-Z8350 price. From personal experience with similar machines, you can expect this beastie to run fairly hot; not hot enough to burn you, but hot enough to be uncomfortable to the touch.
As usual, these are not “monetized” links and I have no relationship at all with CDiscount.com (because they won’t sell or deliver to me). I am an occasional customer of Amazon.com, but I haven’t ever bought a “Coofun” branded computer (from them, or anyone else), so please do treat this as a simple pointer and not a recommendation. [ END OF THE SMALL PRINT ]
Once you have your build of TASMOTA installed and the physical connections made, you need to configure which ESP pins TASMOTA will use to communicate with the GPS module. There are a couple of points to make here …first, the TASMOTA pin configuration refers to the data flow with respect to the ESP8266 itself, while the labelling on the GPS module pins refers to data flow with respect to the GPS. This means that the pin you define as GPS_TX on the ESP actually connects to “RX” on the GPS module and GPS_RX connects to “TX”. The second point is “Don’t Panic!”, as I mentioned earlier, you can’t harm either the GPS or the ESP by connecting or configuring them the wrong way round. The last point is that if your GPS module doesn’t support the UBX protocol, it ain’t going to work, no matter which way round you connect things.
Okay, boot your ESP8266 and connect to the TASMOTA web interface. Go to “Configure” and then “Configure Module” and select the “Module type” to be “Generic (18)” (you’ll need to scroll right down to the bottom of the list of modules to find it). Save this change. When the save is complete you can go back to the “Configure Module” tab and will now find all of the available pins on your module displayed. Select GPIO12 and again, scroll right to the bottom of the pull-down listing. You should see the options “GPS_RX (190)” and “GPS_TX (191) close to the bottom of the list (if you don’t see them, then the “#define GPS” option in the tasmota/my_user_config.h file has not been un-commented, or you’ve flashed the wrong image to your ESP). Remembering that “GPS_RX” refers to data received by the ESP, select that option for pin GPIO12 (which is connected to the “TX” pin of the GPS module) and then “GPS_TX” for pin GPIO_13 (connected to “RX” on the GPS module). Save those changes.
The ESP8266 will restart itself as part of the “save” process and when it comes back up again you should see several GPS status lines above the normal buttons on the main menu.
If your GPS module hasn’t acquired satellite data (GPS LED not flashing). or if it doesn’t support the UBX protocol, the GPS status lines will be present, but the data will be zeroed out (as shown above). It’s also possible that your TX and RX configuration is reversed, so it’s worth trying either changing the pin configuration in TASMOTA, or just swapping the wires over (whichever is easiest).
If your GPS is correctly configured, has already acquired data and supports the UBX protocol, you should see something like the screenshot above, with the data updating once per second.
There are several commands available via the TASMOTA console to manipulate the GPS data handling functions. Commands are entered at the console and have a format of sensor60 n[n] where “n” is a number and “sensor60” specifies that this command is for sensor number 60 (which is reserved for the GPS module). So an example might be:-
sensor60 9 — Start the NTP server process (using time data from the GPS module).
Here’s the complete list of commands, as of early Feb, 2020.
sensor60 0 — Write log data to all available storage, then stop
sensor60 1 — Write log data to all available storage, then restart from the beginning (overwrite mode)
sensor60 2 — Filter horizontal noise from GPS signal
sensor60 3 — Stop horizontal noise filter
sensor60 4 — Start recording log data to storage in append data mode
sensor60 5 — Start recording log data to storage in overwrite data mode
sensor60 6 — Stop recording log data to storage
sensor60 7 — Send an MQTT update on each positional change (very noisy)
sensor60 8 — Stop sending MQTT updates on positional changes
sensor60 9 — Start the NTP server on port 123 (GPS web updates stop)
sensor60 10 — Stop the NTP server (GPS web updates start again)
sensor60 11 — Force a TASMOTA time update on each GPS packet
sensor60 12 — Stop time updates on GPS packets
sensor60 13 — Set TASMOTA lat/long values from GPS positional data
sensor60 14 — Start virtual serial TCP server (UBX data) on port 1234
sensor60 15 — Pause virtual serial TCP server
Now that we have our commands down pat, we can put the GPS data to use. The first (and possibly the the most immediately useful) is command 13, setting the latitude and longitude values from the GPS reported values. This will allow the TASMOTA sunrise/sunset timer functions to work without any manual intervention (that is, even if your timezone isn’t correctly set, having longitude and latitude available from the GPS allows the sunrise/sunset algorithm to calculate those values). Note that there was an issue with this function in the initial release, so you should use TASMOTA version 126.96.36.199 or above (the link in part-I of this article is to a good, working version).
Don’t be confused by this. TASMOTA already has NTP client capability baked-in (it will listen to other NTP servers to set its own time), but this option, command 9, starts an NTP server process which allows other systems to interrogate your ESP8266 on port 123 for time data (your ESP becomes a network time provider, instead of just being a consumer).
As you might expect with such constrained memory limits, this implementation has some limits. The stratum level (the measure of how many levels, or hops, we are removed from the physical source of the time signal) is hard-coded to “2”. In a normal configuration, the time source itself counts as stratum 0 and the machine connected to it would be stratum 1, which would be treated as a preferred source by downstream clients. However, our ESP8266 isn’t using pulse-per-second synchronization of the time data (it’s simply publishing it on arrival), so marking it as a stratum-1 primary source could lead to problems (especially if you happen to be a day-trader). The good news is that it is still a very accurate source of time data in certain situations. For instance, if you have an isolated network of IoT devices, accuracy to the nearest second is probably acceptable and having an ESP8266+GPS combination providing NTP services is a low cost, low power option for a network of wireless data-loggers.
There’s another point you need to be aware of if you’re setting this up on a network with other NTP servers present; the ESP needs to be configured as a “server” in the ntp.conf file on the other machines, not as a “peer”. This is because being a peer requires two-way communications between the machines and this implementation only has the capability of replying to a simple request for a time packet. If you configure the ESP as a peer, the other machines in the network will ignore it and show it as being stuck in the “INIT” state. If you configure it as a server however, the other machines will happily request time data from it …although, in my case anyway, the data from the ESP is so far off the time reported by the other systems in the NTP group (that is, local NTP servers on my network as well as “upstream” NTP sources on the internet) that the ESP is almost immediately marked as an “outlier” (with a “-” character in the first column of the “ntpq -p” output) and subsequently ignored. It could still be quite useful, though. If you are relying on upstream NTP servers on the internet (which is generally a good thing), then you would lose synchronization if your internet connection was down for any significant amount of time. Your ESP though, would still be providing a local time source with one second accuracy, which is still pretty good (day-traders excepted).
The load of updating the GPS readout (as seen in the photo above) and providing time critical replies to NTP requests is also a little bit too much for the ESP8266, so you’ll see a message, “NTP Active” appear in that window and the GPS updates will stop while the NTP server process is running. Once the “Sensor60 10” command is given to stop NTP, the GPS updates will restart and the “NTP Active” message will disappear.
One final wrinkle to the NTP implementation is worth noting (although it’s more of an “interesting characteristic” than a bug). The “refid” field in the output of “ntpq -p” command (on another NTP client on your network, not on the TASMOTA device) has two different modes of output. Normally it will display the IP address of the machine to which the indicated peer is currently listening, but if the peer is a stratum-1 clock source, it will print one of a limited number of strings to let you know to what type of a clock device it is connected (ie:- radio-clock, atomic-clock, etc). Now because our ESP is directly connected to a clock-like device (the GPS) but, as mentioned earlier, is pegged as a stratum-2 device, the refid output field displays the highly unlikely IP address of “188.8.131.52” (hint:- if you haven’t already guessed why it should do this, try “man ascii” for the answer).
If you happen to take your ESP8266/GPS combo with you when you’re hiking, paragliding or orienteering, then this section might be of interest. Christian has added the option of logging GPS data to the ESP8266 flash, so you can save your experience for perpetuity (well, you can probably save some of it anyway …there’s not that much flash on an ESP8266). This is the reason for the very oddly named “#FLOG” setting in the TASMOTA config file (nothing at all, as far as I know, to do with fifty shades of black and blue). If that setting is un-commented, then flash-logging of GPS data is enabled and commands “sensor60 0” and “sensor60 1” set the log handling mode to “overwrite” or “stop writing when flash is full”, respectively. Commands 4 and 5 both start the log recording, but 4 starts in append mode, while 5 starts a completely new log, overwriting whatever data was there. When you start a recording the “Flash-Log” status line in the TASMOTA main menu window (see photo, above) changes from “ready” to “recording”. Command number 6 stops the recording.
Once you stop a recording, a new, blue button will appear above the GPS information in the main menu. There is no text displayed on the button (at least there isn’t in either Firefox or Chrome) until you move your mouse pointer over it, at which point the message “Download GPX-File” will appear, flashing on and off in time with the updates to the GPS data (perhaps this works as a normal button in some other browser?). In any case, clicking on the button will download the flash-log data to your system.
VIRTUAL SERIAL PORT
You can also connect to a virtual serial port on the ESP8266 to get a live view of the (binary!) data streaming from the GPS unit. The “sensor60 14” command will start this virtual serial connection on port 1234 and command 15 will stop it. If you’d like to check the data, you can connect to your ESP using:-
nc IP-ADDRESS 1234 | od -x
Christian mentions that the intended use for this (other than trying to read hex data very quickly) is to feed into the u-blox “u-center” application, which is a debug and management application, freely available from the u-blox site (recent revisions are for Windows only, though).
The Bottom Line
Christian has produced a very versatile and interesting application, adding an exciting new sensor option to the TASMOTA family. He’s not finished yet, either. The most obvious challenge is trying to get PPS working (which should significantly increase the accuracy of the NTP server function) without making the ESP8266 completely unusable for any of the other options running on it. Even without that addition though, this is still a useful addition to TASMOTA and is certainly a lot more fun to play with than most other sensors.
If the idea of an ESP interfaced to a GPS module tickles your fancy (as it did mine), there’s also Chris Liebman’s stand-alone ESPNTPServer project, which I’ve mentioned before in these pages and which now has its own PCB and 3d-printed case.
Towards the end of last year, Christian Baars integrated GPS-as-a-sensor support into TASMOTA with the addition of the xsns_60_GPS.ino file and ancillary code. You can now easily attach a cheap GPS unit to your ESP8266, although just exactly how to connect it and how to communicate with it may not be immediately obvious.
In this two-part series,we’ll look at some of the issues influencing Christian’s design choices before getting down to the nitty-gritty of physical connections, TASMOTA configuration and (finally) uses.
TIP — Before we start …if you have a new GPS module‡ which you’ve never used before, plug in the antenna and connect it to a 3v3 supply while you’re reading the rest of this article (it’ll save you a little bit of time and perhaps some frustration, too). You’ll notice that the LED is off when it’s first powered-on, but after a while (and it could take up to thirty minutes) it will start flashing at 1Hz. This is letting you know that it has acquired signal from multiple satellites and is now ready to send data.
First of all, just a tiny bit of background on Christian’s implementation. He is using the u-blox binary format (“UBX”) to transfer serial data from the GPS module. GPS modules from most suppliers, including u-blox, normally come with ASCII format (“NMEA”) data transfer enabled by default, but this format produces long, fairly verbose messages which are not particularly well suited to the ESP8266’s limited memory and single core processor (remembering that the CPU must attend to WiFi housekeeping tasks on a regular basis). To give TASMOTA the space and time to run its own tasks, Christian initializes the GPS board to use the short-form UBX binary data transfer instead. Unfortunately, there’s a catch — not all GPS modules will support the “UBX” native protocol which this implementation requires. As far as I know (from reading the u-blox web site), it seems as though the majority of modules for sale on Alibaba, EBay or Aliexpress are probably fakes in one way or another (older modules being labelled as newer types, or just plain knockoffs with no u-blox chips in them), so it’s no surprise that a lot of them don’t speak the UBX protocol.
My own experience pretty much mirrors this. Across three different modules, I have one (blue PCB. four-pin connector) which works with NMEA (the most common protocol), but not with UBX, a second (red, narrow PCB, five-pin connector, with PPS¹) which just didn’t work at all and, lastly, a (black PCB, five-pin connector, with PPS) board marked as “WAVGAT NEO-6M” which does actually work using both NMEA and UBX protocols. None of these modules pass the u-blox label test . The WAVGAT module came from the WAVGAT Aliexpress shop though, so if nothing else, they may be more consistent than oddball modules from oddball suppliers (please treat this as a comment, not a recommendation). Christian mentions in the header notes of the xsns_60_GPS.ino file that in addition to a (genuine) u-blox NEO-6M, the Beitian 220 module has also been proven to work.
Okay, so now we know that why not all GPS modules are compatible with this TASMOTA addition and why that is, but how do we connect up our module to see whether it works or not? The first thing you need to know about the connection is that serial communication with the GPS module from the ESP8266 is via the TasmotaSerial library (a TASMOTA-specific software-serial implementation) and not the ESP hardware serial port. This means that, excluding the usual suspects of GPIO-15, and GPIO-02, we can connect to just about any available pin. I’d recommend GPIO-12 (D6 on most ESP8266 boards) to the TX pin on your GPS board and GPIO-13 (D7) to RX on the GPS board. Note that it is very, very easy to reconfigure your ESP8266 (through the TASMOTA web interface) to change these pin settings, so don’t worry too much about mixing up TX and RX …it won’t hurt either the ESP or the GPS. Finally, you need to supply GND and 3v3 connections to the GPS board (do be careful with these… there doesn’t seem to be too much consistency between the different GPS makers when it comes to positioning the power connections). Those four connections are all we need (the current implementation doesn’t use the PPS signal).
Next comes TASMOTA. I’m going to assume here that you’re familiar enough with it to be able to flash a binary to your ESP8266 (and if you don’t, there’s a ton of documentation on the web site). If you’re comfortable building TASMOTA from source, then you should build with your normal “my_user_configs.h” file, but uncomment:-
¹ – PPS = Pulse-Per-Second. This signal is used to synchronize the time data output – the hardware equivalent of “At the tone, it will be 3 AM, exactly“. All of the u-blox modules which I have (including a USB dongle) have PPS outputs from pin-3 of the RF module, but many modules connect it to an LED as an activity/lock indicator. However, you can easily add a PPS feed to your micro by simply adding a 220Ω resistor on the RF module side of the 1k LED current limiting resistor. Click on the photo to the left and zoom in on the bottom, L/H corner to see where the resistor is connected. Note that this module only has a four pin connector (at the top, centre of the board).
† – GPS modules can generally be configured to emit only those “sentences” of NMEA data which you specifically want, however it is still long-form, human readable data, compared to the UBX binary format.
‡ – When you first connect a new GPS module to power, nothing appears to happen. This is because the module needs to receive a certain amount of data from satellites before it can determine the current time and it’s own location. This can take up to thirty minutes in some locations. Once the module has downloaded enough data to initialize, the “PPS” LED will start flashing (even if there are no serial data connections). The data received on this initial “cold start” is automatically saved by the module itself to battery-backed memory, so on subsequent power cycles the start-up sequence will be much shorter.