Mosquitto “invalid protocol MQTT in CONNECT” errors (and ESP8266 continuously trying to reconnect to broker service, without success)

[Edit: See “Stop Press” note at bottom of article before embarking on this fix]

This is a problem which is common to Ubuntu based Linux systems running the “mosquitto” broker.

Unfortunately, the default MQTT package available from the Ubuntu Software Centre is an ancient revision, v0.15, which equally unfortunately, shows up as being “3.1/3.1.1 compatible” …when it isn’t.

For us ESP8266 esp_mqtt users, that translates to a constant stream of “reconnecting…” messages on the ESP8266 console and “Invalid protocol MQTT” messages in the broker’s log (probably /var/log/syslog on your system, but check your mosquitto.conf file to verify).  The end result is that the ESP8266 fails to establish an MQTT session with the broker and is therefore unable to publish or subscribe.

What changed and why did it suddenly stop working?  Well, when Tuan (who has done an absolutely fantastic job on producing this package and on responding to issues, including this one) updated esp_mqtt to run with the nodemca LUA package, he updated (as I understand it) esp_mqtt to be fully 3.1.1 compliant.  Unfortunately, because the stock Ubuntu package is so long in the tooth, this bit most Ubuntu users (and users of Ubuntu derived distributions, such as ElementaryOS) in the arse (so blame the Ubuntu package maintainers, not Tuan).

Because there hasn’t been an update to the official Ubuntu package for so long, Roger Light, the author and maintainer of the mosquitto package, has gone to the trouble of providing a local repository for the package which the Ubuntu Update Manager can be configured to poll and download from (it will show up in the Update Manager just the same as any other third-party package, such as the Google “Chrome” browser).  You can configure the Update Manager to use Roger’s repository (and to automatically update your stock, outdated version of mosquitto) in just a few simple steps.  Please note that these steps, described below, add a non-Canonical repository to your Update Manager, which means you are explicitly giving Roger Light and the mosquitto development team permission to change package contents on your system; if you are uncomfortable with this then you should not follow these instructions and should probably remove the existing (now useless) mosquitto package, too (your call).

Okay, now we’ve got the small-print out of the way, lets get going.  Before starting the upgrade, you should stop your existing mosquitto daemon:-

  service mosquitto stop

and back-up your existing mosquitto.conf file:-

  cp /etc/mosquitto/mosquitto.conf /etc/mosquitto/mosquitto.conf_ORIG

and then start the Update Manager (icon, menu, cli, whatever your particular Ubuntu variant uses).  Leave the Update Manager to sort itself out (it takes a few seconds to start up) and go back to your web-browser and go to Roger’s excellent update page, at:-

(it will open in a new tab).  Click on the green “Technical details about this PPA” close to the top of the page.  This will expose a pull-down menu,  “Choose your Ubuntu version”.  Once you’ve selected your version, say 12.04, the text box below will change to display the package manager entry for that specific version of Ubuntu.  Two lines will be displayed.  One starting with just “deb”, which is the main package repository and the other starting with “deb-src” which is (duh!) the package source-code repository.  We need to use these addresses to configure the Update Manager to use Roger’s repository, so select the first, “deb” line and then swap back from the browser to the Update Manager window.

In the Update Manager window, click on the “settings” button to bring up the configuration window and select the “Other Software” tab.  Near the bottom of the settings window you should now see an “add” button and clicking on that will bring up another window with the natty heading “Enter the complete APT line of the repository that you want to add as source“.  Paste the “deb” line into the text box, making sure there are no trailing characters after the last word, “main” and then hit the “Add Source” button to add it.  Optionally, you can now go back and repeat the same procedure with the “deb-src” line, if you want to download the source-code, too (again, your call).

Going back once again to Roger’s page, take a note of the number displayed in the “Signing key” line.  Flip back to the Update Manager settings window and select the “authentication” tab.  Scroll down the list of Trusted Software Suppliers and somewhere close to the bottom you should find an entry for “Launchpad mosquitto“.  Check that it is the same key number as the one you just noted.  The Update Manager will automatically download the public key specified by this number from Roger’s site to enable secure updates.

Okay, we’re done with the heavy lifting.  You can now close the “Software Sources” (settings) window.  If your mosquitto update has been successful, the close will trigger an automatic update of the available software cache (so click the “reload” button, when prompted).  The Update Manager should (again, automatically) go off and check for new software and find the newly available mosquitto package.  Select the package in the normal way and click “Install Updates” to have the Update Manager remove the old version of mosquitto and install the latest available version from Roger’s site.

One thing you need to be aware of is that the current version (as of Feb 2015) of the mosquitto package has a very minimal configuration file, so before starting the new daemon you should check your original mosquitto.conf file for settings specific to your site and import them into that new file.  A simple method for pulling those settings out of your old file (you did back it up, didn’t you!?!) is to use this command pipe:-

grep -v "^#" /etc/mosquitto/mosquitto.conf_ORIG | cat -s

The `-v “^#”‘ sequence tells grep not to match any comment lines and the “cat -s” suppresses multiple blank output lines.

Make sure you put the settings from the old configuration file into the new, minimal file (especially any “user” or authentication settings) before you restart the new daemon.  You can use man mosquitto.conf to get help with the available settings.

Stop Press: Literally a few minutes before I pressed the “publish” button on this missive, Tuan rolled out an update (told you he was good :-)) to esp_mqtt, adding a protocol define option to the include/user_config.h file:-

#define PROTOCOL_NAMEv31      /* MQTT version 3.1 compatible with Mosquitto v0.15 */
//#define PROTOCOL_NAMEv311   /* MQTT version 3.11 compatible with Paho & Mosquitto > v0.15 */

…so now you don’t have to upgrade if you don’t want to.  Yay!