Firmware update via OTA fails frequently

1

We noticed that after recent updates (both in the VS Code Extension and in the Thinger-Arduino library), the OTA update fails frequently.
We have a device with Thinger-Arduino version 2.25.2 (ESP8266) connected to a router with rssi -77 (on average).
We are trying to perform an OTA update, but the process fails frequently in the middle of the update. On some devices we managed to update on the fourth or fifth attempt. As for the current one, we have been trying several times without success.
The previous version of the VS Code Extension was more stable.

UPDATE: The attempts below were using the Notebook with cable/ethernet/fiber optic internet.

Processing nodemcuv2 (platform: espressif8266@4.2.1; board: nodemcuv2; framework: arduino)
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Verbose mode can be enabled via `-v, --verbose` option
CONFIGURATION: https://docs.platformio.org/page/boards/espressif8266/nodemcuv2.html
PLATFORM: Espressif 8266 (4.2.1) > NodeMCU 1.0 (ESP-12E Module)
HARDWARE: ESP8266 80MHz, 80KB RAM, 4MB Flash
PACKAGES:
 - framework-arduinoespressif8266 @ 3.30102.0 (3.1.2)
 - tool-esptool @ 1.413.0 (4.13)
 - tool-esptoolpy @ 1.30000.201119 (3.0.0)
 - toolchain-xtensa @ 2.100300.220621 (10.3.0)
LDF: Library Dependency Finder -> https://bit.ly/configure-pio-ldf
LDF Modes: Finder ~ chain, Compatibility ~ soft
Found 52 compatible libraries
Scanning dependencies...
Dependency Graph

.....
|-- thinger.io @ 2.31.0
.......
Building in release mode
Retrieving maximum program size .pio\build\nodemcuv2\firmware.elf
Checking size .pio\build\nodemcuv2\firmware.elf
Advanced Memory Usage is available via "PlatformIO Home > Project Inspect"
RAM:   [====      ]  44.8% (used 36740 bytes from 81920 bytes)
Flash: [=====     ]  51.8% (used 540847 bytes from 1044464 bytes)
=============================================================================== [SUCCESS] Took 4.50 seconds ===============================================================================




2024-09-10T00:32:12.826Z [info] OTA update process started. Device: XXXXXXXXXXXXXX, Firmware Version: , Environment: nodemcuv2
2024-09-10T00:32:34.302Z [info] ████░░░░░░░░░░░░░░░░░░░░░░ 15.78% - Device: XXXXXXXXXXXXXX, Time spent: 21 s 476 ms, ETA: 1 m 54 s 594 ms
2024-09-10T00:33:04.422Z [error] Device: XXXXXXXXXXXXXX. Status: FAILURE. Duration: 51 s 584 ms. Description: Error while writing to device: Error: socket hang up
2024-09-10T00:33:04.422Z [info] OTA update process completed. Total duration: 51 s 596 ms. Success: 0, Failures: 1, Already Updated: 0



2024-09-10T00:33:51.910Z [info] OTA update process started. Device: XXXXXXXXXXXXXX, Firmware Version: , Environment: nodemcuv2
2024-09-10T00:34:06.017Z [info] ███░░░░░░░░░░░░░░░░░░░░░░░ 10.15% - Device: XXXXXXXXXXXXXX, Time spent: 14 s 107 ms, ETA: 2 m 4 s 930 ms
2024-09-10T00:34:36.135Z [error] Device: XXXXXXXXXXXXXX. Status: FAILURE. Duration: 44 s 196 ms. Description: Error while writing to device: Error: socket hang up
2024-09-10T00:34:36.135Z [info] OTA update process completed. Total duration: 44 s 225 ms. Success: 0, Failures: 1, Already Updated: 0



2024-09-10T00:35:30.910Z [info] OTA update process started. Device: XXXXXXXXXXXXXX, Firmware Version: , Environment: nodemcuv2
2024-09-10T00:35:38.079Z [info] █░░░░░░░░░░░░░░░░░░░░░░░░░ 5.26% - Device: XXXXXXXXXXXXXX, Time spent: 7 s 169 ms, ETA: 2 m 9 s 98 ms
2024-09-10T00:36:08.199Z [error] Device: XXXXXXXXXXXXXX. Status: FAILURE. Duration: 37 s 285 ms. Description: Error while writing to device: Error: socket hang up
2024-09-10T00:36:08.199Z [info] OTA update process completed. Total duration: 37 s 289 ms. Success: 0, Failures: 1, Already Updated: 0



2024-09-10T00:36:36.328Z [info] OTA update process started. Device: XXXXXXXXXXXXXX, Firmware Version: , Environment: nodemcuv2
2024-09-10T00:37:10.094Z [info] ███████░░░░░░░░░░░░░░░░░░░ 25.55% - Device: XXXXXXXXXXXXXX, Time spent: 33 s 766 ms, ETA: 1 m 38 s 372 ms
2024-09-10T00:37:40.214Z [error] Device: XXXXXXXXXXXXXX. Status: FAILURE. Duration: 1 m 3 s 881 ms. Description: Error while writing to device: Error: socket hang up
2024-09-10T00:37:40.214Z [info] OTA update process completed. Total duration: 1 m 3 s 886 ms. Success: 0, Failures: 1, Already Updated: 0



2024-09-10T00:38:26.657Z [info] OTA update process started. Device: XXXXXXXXXXXXXX, Firmware Version: , Environment: nodemcuv2
2024-09-10T00:38:41.786Z [info] ███░░░░░░░░░░░░░░░░░░░░░░░ 11.27% - Device: XXXXXXXXXXXXXX, Time spent: 15 s 129 ms, ETA: 1 m 59 s 69 ms
2024-09-10T00:39:11.905Z [error] Device: XXXXXXXXXXXXXX. Status: FAILURE. Duration: 45 s 244 ms. Description: Error while writing to device: Error: socket hang up
2024-09-10T00:39:11.905Z [info] OTA update process completed. Total duration: 45 s 248 ms. Success: 0, Failures: 1, Already Updated: 0


2024-09-10T00:39:35.110Z [info] OTA update process started. Device: XXXXXXXXXXXXXX, Firmware Version: , Environment: nodemcuv2
2024-09-10T00:40:13.718Z [info] ████████░░░░░░░░░░░░░░░░░░ 31.19% - Device: XXXXXXXXXXXXXX, Time spent: 38 s 608 ms, ETA: 1 m 25 s 174 ms
2024-09-10T00:40:43.834Z [error] Device: XXXXXXXXXXXXXX. Status: FAILURE. Duration: 1 m 8 s 718 ms. Description: Error while writing to device: Error: socket hang up
2024-09-10T00:40:43.834Z [info] OTA update process completed. Total duration: 1 m 8 s 724 ms. Success: 0, Failures: 1, Already Updated: 0




2024-09-10T00:41:06.777Z [info] OTA update process started. Device: XXXXXXXXXXXXXX, Firmware Version: , Environment: nodemcuv2
2024-09-10T00:41:46.004Z [info] ████████░░░░░░░░░░░░░░░░░░ 31.57% - Device: XXXXXXXXXXXXXX, Time spent: 39 s 227 ms, ETA: 1 m 25 s 42 ms
2024-09-10T00:42:16.125Z [error] Device: XXXXXXXXXXXXXX. Status: FAILURE. Duration: 1 m 9 s 334 ms. Description: Error while writing to device: Error: socket hang up
2024-09-10T00:42:16.125Z [info] OTA update process completed. Total duration: 1 m 9 s 348 ms. Success: 0, Failures: 1, Already Updated: 0


2024-09-10T00:43:21.936Z [info] OTA update process started. Device: XXXXXXXXXXXXXX, Firmware Version: , Environment: nodemcuv2
2024-09-10T00:44:17.754Z [info] ████████████░░░░░░░░░░░░░░ 44.34% - Device: XXXXXXXXXXXXXX, Time spent: 55 s 818 ms, ETA: 1 m 10 s 60 ms
2024-09-10T00:44:47.877Z [error] Device: XXXXXXXXXXXXXX. Status: FAILURE. Duration: 1 m 25 s 936 ms. Description: Error while writing to device: Error: socket hang up
2024-09-10T00:44:47.877Z [info] OTA update process completed. Total duration: 1 m 25 s 941 ms. Success: 0, Failures: 1, Already Updated: 0


2024-09-10T00:45:08.057Z [info] OTA update process started. Device: XXXXXXXXXXXXXX, Firmware Version: , Environment: nodemcuv2
2024-09-10T00:45:49.479Z [info] █████████░░░░░░░░░░░░░░░░░ 32.69% - Device: XXXXXXXXXXXXXX, Time spent: 41 s 422 ms, ETA: 1 m 25 s 276 ms
2024-09-10T00:46:19.597Z [error] Device: XXXXXXXXXXXXXX. Status: FAILURE. Duration: 1 m 11 s 538 ms. Description: Error while writing to device: Error: socket hang up
2024-09-10T00:46:19.597Z [info] OTA update process completed. Total duration: 1 m 11 s 540 ms. Success: 0, Failures: 1, Already Updated: 0


2024-09-10T00:46:46.375Z [info] OTA update process started. Device: XXXXXXXXXXXXXX, Firmware Version: , Environment: nodemcuv2
2024-09-10T00:47:21.642Z [info] ███████░░░░░░░░░░░░░░░░░░░ 28.56% - Device: XXXXXXXXXXXXXX, Time spent: 35 s 267 ms, ETA: 1 m 28 s 218 ms
2024-09-10T00:47:51.760Z [error] Device: XXXXXXXXXXXXXX. Status: FAILURE. Duration: 1 m 5 s 381 ms. Description: Error while writing to device: Error: socket hang up
2024-09-10T00:47:51.760Z [info] OTA update process completed. Total duration: 1 m 5 s 385 ms. Success: 0, Failures: 1, Already Updated: 0
2

Allow us to install version 1.0.6 so that we can carry out comparative tests with version 1.1.1

image
image933×436 32.3 KB

3

Is there any relation with the timeout set to 60000 in the VS Code extension source code?

4

After about 15 attempts, we managed to update the device!!!

The new attempts below were using the Notebook with GSM-4G internet (Hotspot Smartphone) / WiFi (Notebook). Apparently, the results were “better”.
It may be necessary to adopt an approach that the process tolerates intermittent internet.


2024-09-10T14:37:52.310Z [info] OTA update process started. Device: XXXXXXXXXXXXXXX, Firmware Version: null, Environment: nodemcuv2
2024-09-10T14:39:02.243Z [info] ███████████░░░░░░░░░░░░░░░ 41.23% - Device: XXXXXXXXXXXXXXX, Time spent: 1 m 9 s 933 ms, ETA: 1 m 39 s 676 ms
2024-09-10T14:39:32.457Z [error] Device: XXXXXXXXXXXXXXX. Status: FAILURE. Duration: 1 m 40 s 133 ms. Description: Error while writing to device: Error: socket hang up
2024-09-10T14:39:32.457Z [info] OTA update process completed. Total duration: 1 m 40 s 147 ms. Success: 0, Failures: 1, Already Updated: 0


2024-09-10T14:40:31.313Z [info] OTA update process started. Device: XXXXXXXXXXXXXXX, Firmware Version: null, Environment: nodemcuv2
2024-09-10T14:42:34.175Z [info] ██████████████████░░░░░░░░ 69.22% - Device: XXXXXXXXXXXXXXX, Time spent: 2 m 2 s 862 ms, ETA: 54 s 622 ms
2024-09-10T14:43:04.333Z [error] Device: XXXXXXXXXXXXXXX. Status: FAILURE. Duration: 2 m 32 s 984 ms. Description: Error while writing to device: Error: socket hang up
2024-09-10T14:43:04.333Z [info] OTA update process completed. Total duration: 2 m 33 s 20 ms. Success: 0, Failures: 1, Already Updated: 0


2024-09-10T14:45:53.952Z [info] OTA update process started. Device: XXXXXXXXXXXXXXX, Firmware Version: null, Environment: nodemcuv2
2024-09-10T14:46:06.064Z [info] ██░░░░░░░░░░░░░░░░░░░░░░░░ 7.19% - Device: XXXXXXXXXXXXXXX, Time spent: 12 s 112 ms, ETA: 2 m 36 s 410 ms
2024-09-10T14:46:36.247Z [error] Device: XXXXXXXXXXXXXXX. Status: FAILURE. Duration: 42 s 287 ms. Description: Error while writing to device: Error: socket hang up
2024-09-10T14:46:36.247Z [info] OTA update process completed. Total duration: 42 s 295 ms. Success: 0, Failures: 1, Already Updated: 0



2024-09-10T14:47:08.872Z [info] OTA update process started. Device: XXXXXXXXXXXXXXX, Firmware Version: null, Environment: nodemcuv2
2024-09-10T14:47:38.028Z [info] ████░░░░░░░░░░░░░░░░░░░░░░ 17.02% - Device: XXXXXXXXXXXXXXX, Time spent: 29 s 156 ms, ETA: 2 m 22 s 125 ms
2024-09-10T14:48:08.241Z [error] Device: XXXXXXXXXXXXXXX. Status: FAILURE. Duration: 59 s 351 ms. Description: Error while writing to device: Error: socket hang up
2024-09-10T14:48:08.241Z [info] OTA update process completed. Total duration: 59 s 369 ms. Success: 0, Failures: 1, Already Updated: 0


2024-09-10T14:51:09.574Z [info] OTA update process started. Device: XXXXXXXXXXXXXXX, Firmware Version: null, Environment: nodemcuv2
2024-09-10T14:53:09.989Z [info] █████████████████████░░░░░ 79.06% - Device: XXXXXXXXXXXXXXX, Time spent: 2 m 415 ms, ETA: 31 s 895 ms
2024-09-10T14:53:40.168Z [error] Device: XXXXXXXXXXXXXXX. Status: FAILURE. Duration: 2 m 30 s 578 ms. Description: Error while writing to device: Error: socket hang up
2024-09-10T14:53:40.168Z [info] OTA update process completed. Total duration: 2 m 30 s 594 ms. Success: 0, Failures: 1, Already Updated: 0


2024-09-10T14:54:18.358Z [info] OTA update process started. Device: XXXXXXXXXXXXXXX, Firmware Version: null, Environment: nodemcuv2
2024-09-10T14:54:41.899Z [info] ████░░░░░░░░░░░░░░░░░░░░░░ 15.51% - Device: XXXXXXXXXXXXXXX, Time spent: 23 s 541 ms, ETA: 2 m 8 s 246 ms
2024-09-10T14:55:12.045Z [error] Device: XXXXXXXXXXXXXXX. Status: FAILURE. Duration: 53 s 675 ms. Description: Error while writing to device: Error: socket hang up
2024-09-10T14:55:12.045Z [info] OTA update process completed. Total duration: 53 s 687 ms. Success: 0, Failures: 1, Already Updated: 0

2024-09-10T14:55:48.402Z [info] OTA update process started. Device: XXXXXXXXXXXXXXX, Firmware Version: null, Environment: nodemcuv2
2024-09-10T14:58:16.136Z [info] ██████████████████████████ 100.00% - Device: XXXXXXXXXXXXXXX, Time spent: 2 m 27 s 734 ms, ETA: 0 ms
2024-09-10T14:58:16.712Z [info] Device: XXXXXXXXXXXXXXX. Status: SUCCESS. Duration: 2 m 28 s 272 ms. Description: OK
2024-09-10T14:58:16.712Z [info] OTA update process completed. Total duration: 2 m 28 s 310 ms. Success: 1, Failures: 0, Already Updated: 0
5

Hi @alvarolb

We tried to update the same device again. This time, the device has version 2.31.0 of the Arduino-Library/Thinger library.

We tried 10 times, and all attempts failed. We tried to perform the OTA process on other devices, and the failures in the middle of the process are frequent.

Apparently, this version of the OTA (perhaps the VS Code extension) is less tolerant to connection instability. Apparently, it would be necessary for the VS Code extension to be more tolerant to internet connection instability.

2024-09-10T23:24:53.978Z [info] OTA update process started. Device: XXXXXXXXXXXXXX, Firmware Version: , Environment: nodemcuv2
2024-09-10T23:25:33.471Z [info] ███████░░░░░░░░░░░░░░░░░░░ 27.81% - Device: XXXXXXXXXXXXXX, Time spent: 39 s 493 ms, ETA: 1 m 42 s 526 ms
2024-09-10T23:26:03.681Z [error] Device: XXXXXXXXXXXXXX. Status: FAILURE. Duration: 1 m 9 s 692 ms. Description: Error while writing to device: Error: socket hang up
2024-09-10T23:26:03.681Z [info] OTA update process completed. Total duration: 1 m 9 s 703 ms. Success: 0, Failures: 1, Already Updated: 0


2024-09-10T23:26:26.298Z [info] OTA update process started. Device: XXXXXXXXXXXXXX, Firmware Version: , Environment: nodemcuv2
2024-09-10T23:27:05.434Z [info] ████████░░░░░░░░░░░░░░░░░░ 29.31% - Device: XXXXXXXXXXXXXX, Time spent: 39 s 136 ms, ETA: 1 m 34 s 382 ms
2024-09-10T23:27:35.611Z [error] Device: XXXXXXXXXXXXXX. Status: FAILURE. Duration: 1 m 9 s 300 ms. Description: Error while writing to device: Error: socket hang up
2024-09-10T23:27:35.611Z [info] OTA update process completed. Total duration: 1 m 9 s 313 ms. Success: 0, Failures: 1, Already Updated: 0


2024-09-10T23:28:19.248Z [info] OTA update process started. Device: XXXXXXXXXXXXXX, Firmware Version: , Environment: nodemcuv2
2024-09-10T23:28:37.231Z [info] ███░░░░░░░░░░░░░░░░░░░░░░░ 11.65% - Device: XXXXXXXXXXXXXX, Time spent: 17 s 983 ms, ETA: 2 m 16 s 386 ms
2024-09-10T23:29:07.440Z [error] Device: XXXXXXXXXXXXXX. Status: FAILURE. Duration: 48 s 187 ms. Description: Error while writing to device: Error: socket hang up
2024-09-10T23:29:07.440Z [info] OTA update process completed. Total duration: 48 s 192 ms. Success: 0, Failures: 1, Already Updated: 0


2024-09-10T23:29:46.092Z [info] OTA update process started. Device: XXXXXXXXXXXXXX, Firmware Version: , Environment: nodemcuv2
2024-09-10T23:31:09.819Z [info] ███████████████░░░░░░░░░░░ 58.62% - Device: XXXXXXXXXXXXXX, Time spent: 1 m 23 s 727 ms, ETA: 59 s 96 ms
2024-09-10T23:31:40.041Z [error] Device: XXXXXXXXXXXXXX. Status: FAILURE. Duration: 1 m 53 s 947 ms. Description: Error while writing to device: Error: socket hang up
2024-09-10T23:31:40.041Z [info] OTA update process completed. Total duration: 1 m 53 s 949 ms. Success: 0, Failures: 1, Already Updated: 0


2024-09-10T23:32:00.422Z [info] OTA update process started. Device: XXXXXXXXXXXXXX, Firmware Version: , Environment: nodemcuv2
2024-09-10T23:33:41.591Z [info] ███████████████████░░░░░░░ 72.53% - Device: XXXXXXXXXXXXXX, Time spent: 1 m 41 s 169 ms, ETA: 38 s 323 ms
2024-09-10T23:34:11.815Z [error] Device: XXXXXXXXXXXXXX. Status: FAILURE. Duration: 2 m 11 s 390 ms. Description: Error while writing to device: Error: socket hang up
2024-09-10T23:34:11.815Z [info] OTA update process completed. Total duration: 2 m 11 s 393 ms. Success: 0, Failures: 1, Already Updated: 0


2024-09-10T23:35:20.899Z [info] OTA update process started. Device: XXXXXXXXXXXXXX, Firmware Version: , Environment: nodemcuv2
2024-09-10T23:37:13.836Z [info] ████████████████████░░░░░░ 76.28% - Device: XXXXXXXXXXXXXX, Time spent: 1 m 52 s 937 ms, ETA: 35 s 110 ms
2024-09-10T23:37:44.075Z [error] Device: XXXXXXXXXXXXXX. Status: FAILURE. Duration: 2 m 23 s 162 ms. Description: Error while writing to device: Error: socket hang up
2024-09-10T23:37:44.075Z [info] OTA update process completed. Total duration: 2 m 23 s 176 ms. Success: 0, Failures: 1, Already Updated: 0



2024-09-10T23:38:52.249Z [info] OTA update process started. Device: XXXXXXXXXXXXXX, Firmware Version: , Environment: nodemcuv2
2024-09-10T23:40:46.130Z [info] █████████████████████░░░░░ 80.79% - Device: XXXXXXXXXXXXXX, Time spent: 1 m 53 s 881 ms, ETA: 27 s 71 ms
2024-09-10T23:41:16.294Z [error] Device: XXXXXXXXXXXXXX. Status: FAILURE. Duration: 2 m 24 s 42 ms. Description: Error while writing to device: Error: socket hang up
2024-09-10T23:41:16.294Z [info] OTA update process completed. Total duration: 2 m 24 s 45 ms. Success: 0, Failures: 1, Already Updated: 0


2024-09-10T23:41:49.908Z [info] OTA update process started. Device: XXXXXXXXXXXXXX, Firmware Version: , Environment: nodemcuv2
2024-09-10T23:43:18.353Z [info] █████████████████░░░░░░░░░ 65.01% - Device: XXXXXXXXXXXXXX, Time spent: 1 m 28 s 445 ms, ETA: 47 s 601 ms
2024-09-10T23:43:48.476Z [error] Device: XXXXXXXXXXXXXX. Status: FAILURE. Duration: 1 m 58 s 560 ms. Description: Error while writing to device: Error: socket hang up
2024-09-10T23:43:48.476Z [info] OTA update process completed. Total duration: 1 m 58 s 568 ms. Success: 0, Failures: 1, Already Updated: 0


2024-09-10T23:44:08.947Z [info] OTA update process started. Device: XXXXXXXXXXXXXX, Firmware Version: , Environment: nodemcuv2
2024-09-10T23:44:50.738Z [info] ███████░░░░░░░░░░░░░░░░░░░ 28.56% - Device: XXXXXXXXXXXXXX, Time spent: 41 s 791 ms, ETA: 1 m 44 s 537 ms
2024-09-10T23:45:20.979Z [error] Device: XXXXXXXXXXXXXX. Status: FAILURE. Duration: 1 m 12 s. Description: Error while writing to device: Error: socket hang up
2024-09-10T23:45:20.979Z [info] OTA update process completed. Total duration: 1 m 12 s 32 ms. Success: 0, Failures: 1, Already Updated: 0


2024-09-10T23:54:08.914Z [info] OTA update process started. Device: XXXXXXXXXXXXXX, Firmware Version: , Environment: nodemcuv2
2024-09-10T23:55:53.824Z [info] ████████████████████░░░░░░ 76.63% - Device: XXXXXXXXXXXXXX, Time spent: 1 m 44 s 910 ms, ETA: 31 s 988 ms
2024-09-10T23:56:24.027Z [error] Device: XXXXXXXXXXXXXX. Status: FAILURE. Duration: 2 m 15 s 111 ms. Description: Error while writing to device: Error: socket hang up
2024-09-10T23:56:24.027Z [info] OTA update process completed. Total duration: 2 m 15 s 113 ms. Success: 0, Failures: 1, Already Updated: 0
6

Apparently, the failures are related to the signal strength (RSSI) of the Device (which is related to the quality of the WiFi connection and the intermittency of the internet).

We tested the OTA update on a device with RSSI of -55 (on average) and the update was successful on the first attempt.

However, other devices with RSSI signal strength of -75 and -82 fail during the update process.

In other words, the VS Code extension has to be more tolerant of devices that have WiFi internet connection with low RSSI between -70 and -90.

7

Hi,

RSSI is related with a physic value (received signal strength indicator), not a tolerance limit that is established at the plugin.

Having low RSSI (below -80) is pretty close to floor noise, so basically the communication is unreliable, and therefore, highly probably to have packet loss due poor signal, that is why the socket hangs, because the plugin sends the package and waits for confirmation, receiver does not receive the packet and does not send that confirmation, plugin hits timeout and “socket hang” message is delviered.

Solution is improve device’s wifi coverage.

Hope this helps,

8

I found out, that heavily using Serial impedes OTA on an ESP8266.
Why? I don’t know, but there is a clear correlation.