Recovering data from TASMOTA config back-up files

Okay, so you have a TASMOTA-enabled ESP8266 board in a place which is difficult to access and the latest upgrade doesn’t quite go as planned.  The upgrade itself appears to work, but the configuration back-up from the previous revision doesn’t take (your ESP is missing some peripheral devices).  Unfortunately, it’s also a custom build, not a shop bought device with a pre-rolled config already in the (vast) array of choices already shipped with TASMOTA.  What to do (other than retrieve the device and open it up)?  [“Document your projects better, you pillock!”]

Well, as I found out earlier, there is help at hand in the form of the command-line Python program by Norbert Richter:-  decode-config.py.

The good news is that it does (even more than) what you’d expect and can be used for recovering configs directly from a device, or converting between input and output formats.  The bad news is that it isn’t immediately obvious how to use it and it’s also very fussy about what version of Python you have (more especially right now, with the impending doom of v2.7 rushing towards us at the speed of a new year’s party-popper).

DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade….

…etc, etc. So, before you do anything else, grab the latest version of decode-config.py from it’s own repository, as that one is already Python3-ified. You’ll probably need to install dependencies, too (check the header comments).

Anyway, here’s a really, really short intro to using decode-config.py to recover data from older config files, so that you can plug that data straight back into your latest release.

decode-config.py --full-help

Will, as you expect, give a fairly verbose listing of the options.

decode-config.py -f /path/to/saved/configs/Config_Sonoff_6.3.0.dmp

Will give you an extremely verbose (and virtually indecipherable) JSON listing of the whole configuration (better than nothing, but not by much).

decode-config.py --indent-json 4 -f /path/to/saved/configs/Config_Sonoff_6.3.0.dmp

Will give you a reasonably formatted version of that same, JSON output (we’re getting there!).

decode-config.py --indent-json 4 -f /path/to/saved/configs/Config_Sonoff_6.3.0.dmp -g rules

Will give you a much shorter, much more useful, formatted JSON output of your saved rules (assuming you have any) preceded by a bunch of housekeeping information about the decode-config.py program itself and the system you’re running it on.

The sections which you can use with the “-g” flag are listed as:- “Control, Devices, Display, Domoticz, Internal, Knx, Light, Management, Mqtt, Power, Rf, Rules, Sensor, Serial, Setoption, Shutter, System, Timer, Wifi”, most of which are self-explanatory, but not all of which seem to work as you’d expect when trying to recover data from much older revisions of TASMOTA (as an example of this, the GPIO information which I was searching for was tagged onto the end of the “Mqtt” data). So, eventually I happened on this final invocation, which seemed to produce sane (and much more readable) output from the same back-up file.

decode-config.py -T command -f /path/to/saved/configs/Config_Sonoff_6.3.0.dmp

This (the winner!) provides output which you can scroll through directly, or as I did, save to a file for future reference. The data is grouped under the same headings as listed above in plain ASCII text. Everything appears to be grouped together logically (unlike the JSON output) and the only problem I found was an apparent off-by-one error in the numbering of the GPIOs (starting from GPIO1, rather than GPIO0) …which was fixed before I even finished this article (world record support times!).

So, that’s my brief introduction to decode-config.py, a bacon-saver if ever there was one.


Late News  —  Norbert (the author of decode-config) has just let me know that binary versions of the utility are also available for multiple operating systems, so there’s no longer any need to update Python.

First steps with the W600 (on Linux)

I ordered a couple of these W600 dev boards a few weeks ago, as they were reported to be the easiest ones to use when introducing yourself to the ThingsTurn Dev BoardW600 chip (these “ThingsTurn” boards don’t have any second processor to grapple with, but do have a UART chip and USB connector for easy power/comms).

It turned out that while the power connection was easy, the comms were anything but and I wasted far too much time messing around trying to get it to accept an upload or even just a terminal connection.  So today’s post mainly consists of a series of short tips on how to get started with the W600 “ThingsTurn” dev board without pulling out all of your hair.  Tips 1 through 4 deal with getting MicroPython onto your W600, while those from #5 onwards are examples of what you can do with the board from the MicroPython REPL (command line).

For my purposes, I just wanted to upload MicroPython as a quick, functional test, without having to download and install a toolchain and/or SDK, or really do anything at all that took more than a couple of lines of typing or 5 minutes of my time.  Two days later, I was -still- getting nowhere very fast; my blood pressure was rising and I was itching to use my magic, solves-everything tool  …the 1.5kg lump hammer.

So, tip #1 is  —  Don’t believe everything you read (including this!).  The W600 firmware and applications are still under heavy development.  Most (but not all, see tip #2) of that work is taking place in China.  The translation and maintenance of the manuals seems to be quite well down the TODO stack (something that most of us are guilty of).  Specifically, don’t bother trying to use their “download.py” (if you can find it) for anything, most especially not for uploading to ThingsTurn boards.

Tip #2  —  Volodymyr Shymanskyy has recently posted a great tool for uploading to the W600.  It knows about the two available bootloader methods and will try to do the right thing, depending upon which type of file you’re attempting to upload.  Don’t bother with anything else; go straight to his repository and start using it now.

Tip #3  —  Don’t assume that your favourite terminal program(s) will work with the ThingsTurn boards, even if it has worked with everything else you’ve ever tried it with.  It turns out the the the ThingsTurn boards (note that I’m being very specific here, as I don’t believe that any of the other W600 modules suffer from this issue) have a requirement for the RTS signal to be held at “0” when connecting, otherwise the W600 will be forced into a permanent reset condition.  I was unable to get either “screen” or “minicom” (my usual goto choices for comms terminals) to play with the ThingsTurn boards.

By this time I knew that the boards worked, as they communicated with “w600tool” easily and loaded MicroPython (to displace the simple, one-second flash on/off program which comes as default on the boards).  Connecting with either screen or minicom would very occasionally produce a random, brief flash from one of the blue LEDs, but there was never any further indication of life and definitely nothing coming from the boards.  A major step forward was finding that sending a “Hang-Up” request (CTRL-A H) from Minicom produced the MicroPython header and a REPL prompt from the board, before it lapsed back into its previous, comatose state.  This confirmed that the upload had indeed worked and that, given the right conditions, MicroPython would run.  At this point I reverted back to my 1980’s RS232 self and started writing a simple C program to toggle the control lines…  before remembering that this was the glorious future and somebody else on the ‘net had undoubtedly already done it (and undoubtedly made a far better job of it than I ever did).  So I knuckled myself on the side of the head a couple of times and turned back to Volodymyr Shymanskyy, as he’d already helped me out once and seemed to have already cracked this particular problem.  That particular rabbit-hole led me, in very short order, to Python.

Tip #4  —  Use the force, Luke!  Or at least, don’t be as stupid as me and refuse to see what’s right under your nose.  You wanted to get running with something quickly:-  MicroPython.  MicroPython =~ Python.  W600tool =~ Python.  There’s a bit of a pattern emerging here.  Yup, those clever Python dev guys have already written the solution to our problem and it is called “miniterm”.  A quick check of the on-line manual pages showed that it was quite capable of toggling selected hardware control lines and holding them in the selected state, as well as all of the normal port and baud selection stuff.  So here’s the command line to get you connected, up and running using miniterm:-


python3 -m serial.tools.miniterm --rts 0 /dev/ttyUSB0 115200

[…and just a reminder here that this is from Linux, you’ll likely have to change at least the device name if you’re using a different OS].

Once you’ve typed that in you’ll see something like:-

— forcing RTS inactive
— Miniterm on /dev/ttyUSB0 115200,8,N,1 —
— Quit: Ctrl+] | Menu: Ctrl+T | Help: Ctrl+T followed by Ctrl+H —

…followed by an ASCII-art representation of the Winner-Micro (stacked “W” and “M”) logo (which really confuses WordPress formatting, so is not included here), followed by:-

WinnerMicro W600
MicroPython v1.10-279-g3c60627-dirty on 2019-05-24; WinnerMicro module with W600
Type “help()” for more information.
>>>

Yay!  We’re in business!


Tip #5  —  Import the machine-specific hardware module.  Configure a W600 pin (PB14 in this case) as an output (with no pull-up or pull-down) and assign name “led14” to it.  Use the assigned name to toggle the value of the W600 pin (note that the blue LEDs on the ThingsTurn boards have their anodes connected to VCC via a 1k resistor and the cathodes are connected to the GPIO pins, so the output value of “1” turns the LED off and “0” turns it on:-


>>> import machine
>>> led14=machine.Pin(machine.Pin.PB_14,machine.Pin.OUT,machine.Pin.PULL_FLOATING)
>>> led14.value(1)
>>> led14.value(0)


Tip #6  —  Connect to an existing Access Point.  To do this, we first import the network module and then initialize as a station and scan for available networks.  We connect to an available network (SSID “BigHouse”) with whatever the password is (“Th3Pa55word” in this example), check for a valid, live connection and finally, print out the IP address.


>>> import network
>>> sta_if = network.WLAN(network.STA_IF)
>>> sta_if.active(True)
True
>>> sta_if.scan()
[(b'BigHouse', b'0Z:\xc7\x82\x10', 6, -67, 32, False)]
>>> sta_if.connect("BigHouse", "Th3Pa55word")
>>> sta_if.isconnected()
True
print("Connected, ip is: " + sta_if.ifconfig()[0])
Connected, ip is: 192.168.4.3


Tip #7  — Create an access-point on the W600  …and then shut it down again.  First we need to load the easyw600 module, which contains Winner Micro’s own helper functions to make this easy.  There’s just a single call to easyw600.createap(), with your choice of SSID.  You’ll find that the  new access point pops up immediately and is visible and accessible (but not very useful) from any nearby device equipped with WiFi:-

>>> import easyw600
>>> easyw600.createap(ssid="W600_softAP")
softap working, ip is 192.168.43.1

When you’re finished, there’s just a single call to easyw600.closeap() to shut it all down again:-

>>> easyw600.closeap()


Tip #8  –  Enable an FTP server process.  This will allow us to manipulate files on the W600, so that longer MicroPython programs can be written elsewhere and then debugged on the device itself.   Before you start with the FTP server, you need to make sure that your W600 is already on-line and associated with an existing AP (see tip #6, above).

The FTP server is included in yet another module, this time named simply “w600”, so to begin with, we need to make sure that it is imported.  After that, there’s just a one line command to start the FTP server with a given username and password (note that in the given example, the username and password are both set literally to the word “None”):-

>>> import w600
>>> w600.run_ftpserver(port=21,username="None",password="None")
ftpserver is running.

[After writing this tip, I found that the “easyw600” module has a wrapper for run_ftpserver() which is even easier to use, just:-  “easyw600.ftpserver()” ]

You can now access your W600 from any local machine and you’ll see something like this:-

Connected to 192.168.4.3.
220-= welcome on W600 FTP server =-
220
Name (192.168.4.3:guest): None
331 Password required for None
Password:
230 User logged in
Remote system type is UNIX.
ftp> ls
200 Port Command Successful.
150 Opening Binary mode connection for file list.
drwxrwxrwx 0 root root 0 Jan 1 2018 sys
drwxrwxrwx 0 root root 0 Jan 1 2018 lib
drwxrwxrwx 0 root root 0 Jan 1 2018 cert
-rwxrwxrwx 0 root root 139 Jan 1 2018 boot.py
-rwxrwxrwx 0 root root 34 Jan 1 2018 main.py
-rwxrwxrwx 0 root root 40 Jan 1 2018 easyw600.py
226 Transfert Complete.
ftp>

So, we can now create a file, “fc.py” containing Python code on our local Linux machine:-

def wibble(FC="dog"):
    print("Furry creature: " + FC)

wibble("cat")
wibble()
wibble("ferret")

Now upload it to the W600 using FTP’s “put” command and then go back to the W600 console and type:-

>>> import fc

…(note:- there’s no “.py” on the end) and you’ll see the output from your new code.

If you repeat the “import fc” command, there’s no further output, but you can now call the “wibble” function from the REPL command line.  Try:-

>>> fc.wibble()

and

>>> fc.wibble("rabbit")

Note that it isn’t really possible to un-import a file (or module) once it has been cached, so -don’t- edit your file remotely and then expect to see the changes reflected when it is re-uploaded. MicroPython generally ignores an import request for something which it already has marked as existing in cache. The only bullet-proof way around this is to reboot the W600.


Tip #9  –  How do I reboot the W600 (without pulling the power cord)?

>>> import machine
>>> machine.reset()

Tip #10 – And finally, here’s a replacement main.py file which you can now upload to your W600 to have it automatically connect to your WiFi network and start the FTP server (both as described above) on reboot. You -will- need to set the SSID and password for your access point before installing it.

##
## main.py file content. Used to automatically start
## the network, join an access-point and then start
## an FTP server on the W600 at initial boot.
##
## *** You must set an SSID and password ***
##
import network
import w600
import time

print(“Starting network and FTP server…”)

##
## Join Existing Access-Point (Local Wireless Network).
##
sta_if = network.WLAN(network.STA_IF)
sta_if.active(True)
sta_if.scan()
sta_if.connect(“MY_SSID”, “MY_PASSWORD”)

##
## Start built-in FTP Server.
##
w600.run_ftpserver(port=21,username=”None”,password=”None”)

##
## …and let the user know what our address is.
##
print(“Waiting for DHCP…”)
time.sleep(5)
sta_if.isconnected()
print(“Connected, ip is: ” + sta_if.ifconfig()[0])