Distribute an ESP binary for Thinger to noobs

1

Following problem was to solve:
Distributing an ESP32 or ESP8266 Thinger solution without requiring the user to compile a version with his own credentials.

I don’t want use the usual way of a temporary website to enter the credentials.
That is a very clumsy solution requiring a lot of code and program space (all the HTML code must be in program space)
Even for the user it is absolutely not straight forward…
SmartConfig is a nice solution, but only provides SSID and Password, you need th Thinger.io credentials as well.

For Thinger, you need at least: SSID, WiFiPassword, Device Name, Thinger Account, Device Credentials (the rest of the options and parameters can then be gathered from the Thinger properties)

My solution is quite simple, elegant and requires practically no additional ESP code, no library and nothing to install at user’s side. (excepted Python, which he needs anyway to upload the binary).

It is a Python script that patches the credentials into your compiled binary.
It works that way:

In your ESP code, you define placeholders for your credentials and make them long enough to potentially fit any user’s content.
Here:

  #define DEVICE_NAME             "DEVCNAME        " 
  #define WIFI_SSID               "WIFISSID        "
  #define WIFI_PASS               "WIFIPASS                "
  #define THINGER_USERNAME        "CLOUDNAM        "
  #define DEVICE_CREDENTIALS      "DEVCCRED        "

The placeholders MUST match exactly those defined in the provided PythonPatcher.py Python script.
You then compile your code and distribute the raw binary together with PythonPatcher.py
The user places the two files on his desktop and runs the PythonPatcher.py.
The script will ask for the file to patch, and the credentials and patch the binary accordingly, saving the patched file on the destop too:

%Run PythonPatcher.py
    Enter binfile to patch:PythonTest.bin
    Enter SSID:GW-****
    Enter Password:*******
    Enter Device Name:Steroids
    Enter Cloud User Name:Thing4
    Enter Device Credentials:*******
    File PythonTest_patched.bin saved

You then upload the patched file to the ESP and you are online at Thinger !

Soo easy…

Caveats:
This solution is not suitable to distribute an ESP device including code, the user must patch and upload himself.

You must take care to use the credentials only at one place of your code (if you need it a several places, copy it to a String-variable).
In my example, the placeholders are 16 chars long, the placeholder for WiFI password is 24 chars long
You might use longer strings in your code and match them in the PythonPacher.

The Python script is on Github:

2

Hi @rin67630,

Thank you for sharing your approach to distributing an ESP32 or ESP8266 Thinger firmware! It looks awesome. I’m happy you found a way to streamline the device flashing process.

Another option could be to create something on the web, perhaps using esptool-js. While I don’t know exactly how it works or if it can be forked to modify the binary directly in the browser, it seems like a possibility worth exploring. Here are the resources for esptool-js:

However, at Thinger.io, we should aim for a better way for device auto-provisioning. Ideally, we could have the same firmware for all devices, with the devices negotiating the credentials by themselves. This would simplify OTA updates and device startup, using methods like Esp Touch, SmartConfig, etc.

3

[quote=“alvarolb, post:2, topic:5171”]

That is very true. SmartConfig is the way to go for pre-flashed commercial devices.
Maybe you could use your business influence to convince Espressif to extend SmartConfig to include a provision for three additional generic cloud service credentials?
Being only online at the local WiFi is not enough if you use a cloud service.

My PythonPatcher is intended for those, who flash themselves stock ESP devices.
It can also be used to flash small series of devices:
Compile once, run several times the PythonPatcher and all devices are online and connected !
Ways faster than over a configuration website.

By the way: OTA and later WiFi reconfigurations with Smartconfig are still possible.

4

An interesting way, that I was not aware of.
It has however yet some showstoppers:
It works only on Chrome-based browsers: on Firefox or Safari, it was not functional
The console can only handle 115200 baud, the very common 9600 Baud is not supported.

It were however really interesting, if it could work everywhere without installing anything.
My PythonPatcher just requires Python, that comes built-in on macOS and on most Linux systems.
On Windows, you need to get it for free from the store, but that remains an easy step.

And now, since i have the drive on Python to manipulate files, I imagine, once I have entered the Thinger credentials and uploaded the ESP program, to subsequently directly patch provided template json dashboard developer files with the device name and username to immediately give to the newbie some powerful ready configured dashboards.
He would just have to cut and paste the patched developer files to get his own adjusted awesome Thinger dashboard ready configured.
This is of course not a process for commercial key accounts who provide one dashboard to many users, and use shared dashboards on one account, but for new users who, will create new accounts and water-mouth them with Thinger…

5

It will soon be working that way:

Welcome to PythonPatcher Thinger-Edition for ESP devices 
from RIN67630 @ https://github.com/rin67630/ESP_Binary_patcher
Please read the instruction file to pre-configure a new account/device at Thinger.io: PreConfigureThinger.txt

Please select a file from: 
[1] ZSP_SwissArmyKnife_Rev06_2024 .bin
[2] ASP_SwissArmyKnife_Rev06_2024  2.bin
[3] ESP_SwissArmyKnife_Rev06_2024.bin
Select .bin file[1-3]: 3
working on ESP_SwissArmyKnife_Rev06_2024.bin, let's begin to patch !
Enter SSID:GW-FM-7390
Enter Password:*********
Enter Device Name:Witty
Enter Cloud User Name:Thinger***
Enter Device Credentials:*****************
Patched binary ESP_SwissArmyKnife_Rev06_2024_patched.bin saved
Flashing it now to the ESP device on the first valid serial port?y

esptool.py v3.3
Found 2 serial ports
Serial port /dev/cu.usbserial-1410
Connecting....
Detecting chip type... Unsupported detection protocol, switching and trying again...
Connecting....
Detecting chip type... ESP8266
Chip is ESP8266EX
Features: WiFi
Crystal is 26MHz
MAC: bc:dd:c2:7b:14:76
Uploading stub...
Running stub...
Stub running...
Configuring flash size...
Flash will be erased from 0x00000000 to 0x0007efff...
Compressed 517216 bytes to 375390...
Writing at 0x00000000... (4 %)
Writing at 0x00005dc1... (8 %)
Writing at 0x0000b801... (13 %)
Writing at 0x000118cb... (17 %)
Writing at 0x000178de... (21 %)
Writing at 0x0001d7c0... (26 %)
Writing at 0x00022b65... (30 %)
Writing at 0x000285fc... (34 %)
Writing at 0x0002dafb... (39 %)
Writing at 0x000331cf... (43 %)
Writing at 0x00038670... (47 %)
Writing at 0x0003daee... (52 %)
Writing at 0x00042cdf... (56 %)
Writing at 0x00047e9e... (60 %)
Writing at 0x0004ce73... (65 %)
Writing at 0x00051e21... (69 %)
Writing at 0x00056ff0... (73 %)
Writing at 0x0005c94a... (78 %)
Writing at 0x0006214b... (82 %)
Writing at 0x0006751e... (86 %)
Writing at 0x0006c99c... (91 %)
Writing at 0x00072f80... (95 %)
Writing at 0x00078f09... (100 %)
Wrote 517216 bytes (375390 compressed) at 0x00000000 in 36.4 seconds (effective 113.6 kbit/s)...
Hash of data verified.

Leaving...
Hard resetting via RTS pin...
Enjoy your ESP device on-line ! 

I found these dashboards templates: ['z_Thinger.Stats.h', 'z_Thinger.Min.h', 'z_Thinger.Sec.h']
Personalising these templates to your credentials ?y
Opening each file and replacing b'Cloudname' with b'Thinger***', replace b'Device' with b'Witty' ...
Done!
Please read now the instructions for post-configuring the dashboards @ Thinger.io: PostConfigureThinger.txt

Enjoy your gorgeous Thinger Dashboards !
6

Nice work!! :rocket:

Can I ask for an improvement?

I would like to see the option to enter Cloud Hostname, i.e., with a default value for the community servers (iot.thinger.io), so the users from the private servers can benefit also from the tool :wink:

7

@rin67630
This project uses an installer for ESP32 and Browser:

The prefered way is the Web Installer and Console which is a browser-based tool to flash the ESP32 and extract the log over USB:

@alvarolb

Something similar to Over-The-Air-Activation (OTAA) of LoraWAN?

  • Over-The-Air-Activation (OTAA) - the most secure and recommended activation method for end devices. Devices perform a join procedure with the network, during which a dynamic device address is assigned and security keys are negotiated with the device.
  • Activation By Personalization (ABP) - requires hardcoding the device address as well as the security keys in the device. ABP is less secure than OTAA and also has the downside that devices can not switch network providers without manually changing keys in the device.
    End Device Activation | The Things Network
8

Could it be that a flow based on the MAC of the WiFi board and a second stage of manual authorization and recording of non-NVS data would be possible?
Ex:

  1. The user saves the firmware inside the ESP32, only by informing the name of the HOST, User and a certificate (AppKey/root key), an AES-128 bit secret key, generated by the user’s Thinger Server (a token key created on the display screen of the server Token keys);
  2. The ESP32, upon connecting to the internet, sends an authentication request to the Thinger Server;
  3. The user views the authorization request on a screen, where he is informed of the MAC of the WiFi board. (The information that would differentiate each device would be its MAC board. This information would be a temporary “Device Id”, until the device is confirmed as safe or legitimate by the user and, then, defined the “Device Id” and “Device Credentials” in the next stage);
  4. The user can: Confirm or Deny the connection by clicking a button;
  5. If the user confirms, the attribution of a “Device Id” and “Device Credentials” (and “Device Name”, “Device Description”, “Asset Type”, “Asset Group”… in the web page) to the device will be requested. ;
  6. Next, the device stores/records the “Device Id” and “Device Credentials” inside its NVS (preferences.h library for the ESP32);
9

I found this library (Preferences.h developed by Volodymyr Shymanskyy, creator of the TinyGSM library) that extends the functionalities NVS for the devices:

  • ESP8266 using LittleFS
  • RP2040 boards with Pico core
  • Realtek boards with Ameba Arduino SDK
  • Arduino Nano 33 IoT, MKR1010, MKR VIDOR using WiFiNINA storage
  • Particle Gen3 devices: Argon, Boron, Xenon, Tracker, BSOM
10

Of course…
Coming soon.

11

That given solution GitHub - jomjol/AI-on-the-edge-device: Easy to use device for connecting "old" measuring units (water, power, gas, ...) to the digital world. is for ESP32 only devices that have a SD card to read from… nor really the vanilla ESP8266. :thinking:
Then the flashing is done with a Javascript that works only on Chrome-based browsers. :thinking:

I would prefer concentring ourselves on my current way before considering other approaches.
I will soon provide an interesting demo incl, tutorials that everyonone can flash on a bare ESP device in a few minutes and almost immediately get a set of gorgeous Thinger dashboards on his own account.

12

With this flow, it would be possible to perform updates via OTA, without the risk of losing the information on “Device Id” and “Device Credentials”, because they would be stored in NVS, which is not affected when updating via OTA.

13

I know that one.
With my approach, I dont need any Non Volatile Memory and can even use it to extend my scarce programing memory space.
I am also using Thinger’s device properties to store stuff that must survive resets (important for long-term integrations and statistics).
This has following tremendous advantages over NVM:

  • No wear of NVM memory (the ESP NVM has a very limited writes tolerance)
  • You can replace devices without losing your information
    That is a fantastic feature of Thinger !
14

This user visualization web page is where the “magic of flux” would occur.
As such, we would not need to record the “Device Id” and “Device Credentials” when first recording the firmware.
Isso would allow the same firmware to be recorded on hundreds of devices, just informing the: 1) HOST URL of the Server; or 2) User ID; e 3) a certificate generated and managed by the Thinger Server (as is usually the case with tokens).

The attribution of “Device Id” and “Device Credentials” is only necessary for the user to confirm that the device is trustworthy/legitimate. On the screen, the user could view some information such as the MAC address of the device.
After confirmation, open a screen to inform the “Device Id” and “Device Credentials” of the device. Next, the device records the “Device Id” and “Device Credentials” on your NVS.
Otherwise, access would be denied and this non-dispositive status would be confirmed that access was not permitted.

Wow, it had the potential to work perfectly!

15

There is no wear and tear on NVS when operating and only reading. Wear only occurs during the recording operation.
The ESP32 saves the SSID and the WiFi password in NVS. If the NVS is corrupted, the device will not be able to connect to the internet, it will be offline.

Your approach is fantastic, congratulations!!! I can wait for the final launch. However, apparently, your approach does not allow you to perform OTA updates on a scale, so it would be necessary to inform “Device Id” and “Device Credentials” in the code.

16

My current setup routine takes a three step connection approach.
a) reconnect with the credentials stored in ESP’s internal system NVME, if if fails:
b) connect using the default credentials stored in the programm (incl PythonPatcher), if if fails:
c) SmartConfig, if if fails:
d) Run offl-line with Serial dialog :grimacing:

17

I mentioned wear on NVS in the context of using it for a reset-proof persistance of value integrations
You need to write periodically to update the values existing before the reset.
That is where Thinger has an ace in hands with device properties.

18

@alvarolb , What do you say about this flow? Is he promising?

19

I have been thinking a lot about this topic: What is the best way to store the device data on “Device Id” and “Device Credentials”?
Then, as options were:

  • Previously save “Device Id” and “Device Credentials” numa Macro ’ #define DEVICE_ID “xxxxxxxxx” '. The problem to be addressed is the need to inform the “Device Id” and “Device Credentials” in the source code. Also, it does not allow mass OTA updates.
  • Previously record “Device Id” and “Device Credentials” in the Device File System. The inconvenient of this approach is that two stages are necessary to be carried out. First save the firmware and then the file (JSON, TXT…) in the File System. Also, there is a problem that many libraries manipulate the Device Archive System.
  • Previously record “Device Id” and “Device Credentials” not NVS. The same problems as above, since it will be necessary to record the “Device Id” and “Device Credentials” in the device code and, therefore, it will not be possible to perform OTA in mass, since the “Device Id” and “Device Credentials” ” would have to be previously in code.

Also, these previous options, the user must enter the “Device Id” and “Device Credentials” data in two places: code (either NVS or File System) and register these data on the Thinger Server. Ou seja, we have two manual etadas that are duplicated.
It is necessary to manually store the “Device Id” and “Device Credentials” data in the code or NVS or File System. And we still need to register the “Device Id” and “Device Credentials” on the Thinger Server.

For example, when performing an OTA update on a device, I have to enter the “Device Id” and “Device Credentials” data in the code, so follow the “architecture” to set up the “Device Id” and “Device Credentials” in the file. “.h”. But this approach is inconvenient when you need to perform updates via OTA, because you have to inform the “Device Id” and “Device Credentials” in the “.h” file. Is there a manual error risk, you can make an error and report the wrong “Device Id” and “Device Credentials” code to perform the update via OTA.
With the new flow approach described above, there is no manual error risk when defining “Device Id” and “Device Credentials”, this is just one stage.

But now, based on the discussions on this topic, the idea of ​​assigning “Device Id” and “Device Credentials” to the user on a specific web page would apparently solve everything! :bulb: :bulb: :bulb: :bulb: :bulb: :bulb: :bulb: :bulb: :boom: :boom: :boom: :boom: :boom:It will guarantee a safe flow for the user to validate the devices as they trust a specific web page for that stage. However, the user would only need to register/enter the “Device Id” and “Device Credentials” only once, without needing to write them down in another notepad, which is much simpler than the above options. Also, the user does not need to set up the “Device Id” and “Device Credentials” information to perform OTA updates on the devices. Also, it would be possible to automate the process like NodeRed, defining “Device Id” and “Device Credentials” automatically.
If for some reason the NVS becomes corrupted, the device would return to the previous stage and request new access permission from the user. In this case, there may still be a signal/flag/notice that the device with the correct MAC and Token was already linked to a previous registry on the Thinger Server. This option, the user will only restore the link between the device that is requesting access and the “Device Id” and “Device Credentials” that were previously registered.

20

Hi Alvaro,
OTA makes nothing than uploading a binary file into the device’s program memory.
Currently OTA is started from the IDE.
Technically the IDE compiles the binary and then runs either the OTA process or the esptool to upload over serial.
I found also a solution to flash OTA directly:

Once my PythonPatcher and the PythonPatcher (Thinger.Edition) are both running over Serial and are documented, I will look after that work to include the OTA version.
But anyway, there is no OTA possible before the ESP device has been initally programmed to listen to OTA requests.

If I get a way to upload my patched binary over OTA as well, we have won.
And if my already flashed device moves to another WiFi (what happens frequently when the ISP delivers a new router) just reboot the device, it will accept new credentials from SmartConfig.