Cannot upload OTA with ESP8266OTA and EspMQTTClient library

Hi all,

Maybe you could help me explain why this happen. I use VSCode + PlatformIO + ThingerIO. I am using Wemos ESP8266 D1 Mini board.

Vanilla example ESP8266OTA from ThingerIO was working. I added simple led blink to that example also was working – I could update the blink interval using OTA update.

Next, I put EspMQTTClient arduino/platformio library from Patrick Lapointe in the sketch. Once the firmware is running … I can no longer do OTA update. The serial debug showed “Cannot Read Socket” and in VSCode there was a notification of “timeout 60000ms”.

I use my own MQTT server in my local network, and I want to see if OTA from ThingerIO can be used for this project.

Below is the code which is no longer able to handle OTA after I added MqttClient.loop() in the loop(){}.

I hope someone can help me explain what happened, and how should I do it to make it work.

By the way, it is working fine with ESP32 (NodeMCU-32s).

Cheers,
Fahmy

/* ThingerIO */
#define THINGER_SERIAL_DEBUG
#include <ThingerESP8266.h>
#include <ThingerESP8266OTA.h>
#include "arduino_secrets.h"

ThingerESP8266 thing(USERNAME, DEVICE_ID, DEVICE_CREDENTIAL);

/***
 * Initialize ESP8266 OTA
 * use Thinger.io VSCode Studio extension + Platformio to upgrade the device remotelly
 */
ThingerESP8266OTA ota(thing);

/* MQTT */
#include <EspMQTTClient.h>

/* Init MQTT Client */
EspMQTTClient MqttClient(
    MQTT_ADDRESS, // MQTT Broker server ip
    MQTT_PORT,    // The MQTT port, default to 1883. this line can be omitted
    MQTT_USER,    // mqtt user, Can be omitted if not needed
    MQTT_PW,      // mqtt password, Can be omitted if not needed
    MODULE_NAME   // Client name that uniquely identify your device
);

void onConnectionEstablished();

void setup()
{
  // open serial for monitoring
  Serial.begin(115200);

  // set builtin led as output
  pinMode(LED_BUILTIN, OUTPUT);

  // add WiFi credentials
  thing.add_wifi(SSID, SSID_PASSWORD);

  // digital pin control example (i.e. turning on/off a light, a relay, configuring a parameter, etc)
  thing["led"] << digitalPin(LED_BUILTIN);

  // resource output example (i.e. reading a sensor value)
  thing["millis"] >> outputValue(millis());

  // more details at http://docs.thinger.io/arduino/

  /* Optional functionalities of EspMQTTClient */
  // MqttClient.enableDebuggingMessages();
  MqttClient.enableLastWillMessage(TOPIC(_PREFIX, _LastWill), LastWillMsg, true); // You can activate the retain flag by setting the third parameter to true
}

void loop()
{
  static unsigned long led_millis = millis();

  thing.handle();
  MqttClient.loop();

  if (millis() - led_millis > 1000)
  {
    led_millis = millis();
    digitalWrite(LED_BUILTIN, !digitalRead(LED_BUILTIN));
    MqttClient.publish(TOPIC(_PREFIX, _MOD_HEARTBEAT), String(led_millis), true);
  }
}

void onConnectionEstablished()
{
  MqttClient.publish(TOPIC(_PREFIX, _MOD_STATUS), "Online", true);
}

Hi @iotregenesis, I think the OTA can fail as it currently does not “stop” the normal device loop. Using other software, reading sensors, etc., may affect the OTA process and finally get transfer timeouts. Would work tomorrow on it, as I have a pending update regarding ESP8266 and ESP32 Webconfig.

3 Likes

hi @alvarolb ,

Thank you for responding.
I wish I were more fluent with this internal Arduino and ESP stuff, so I could be more helpful.

Cheers,

Hi,

I have uploaded a new library version that can be used for testing. It includes a better support for Webconfig both on ESP8266 and ESP32, and improves OTA management. It is not released on the master branch yet, so, for testing it is necessary to configure platformio.ini in the following way (to use the webconfig branch).

lib_deps =  
    https://github.com/thinger-io/Arduino-Library.git#webconfig

I noticed that sometimes OTA fails due to memory limit on devices, i.e., when you add some other libraries and code. In this case it is possible to reduce the required memory consumption for the OTA process. For example, setting it to 1024 bytes inside the setup():

ota.set_block_size(1024);

I have lowered it for the ESP8266 to 2048 bytes by default (it was set to 8192), and I am investigating if it can be adjusted dynamically depending on the free memory. There is a trade-off between memory footprint and OTA speed. A smaller memory footprint will require more packets being processed and acknowledged, so it will be a bit slower.

This branch also includes some callbacks to know the OTA state by the program. I will update soon an example with an OLED screen.

1 Like

ESP8266 Webconfig Example + OLED Display + OTA

#define THINGER_SERIAL_DEBUG

// Requires from Library Manager or https://github.com/tzapu/WiFiManager
#include <ThingerESP8266WebConfig.h>
#include <ThingerESP8266OTA.h>

#include "SSD1306Wire.h" // legacy include: `#include "SSD1306.h"`

ThingerESP8266WebConfig thing;
ThingerESP8266OTA ota(thing);
SSD1306Wire display(0x3c, D1, D2);   // ADDRESS, SDA, SCL  -  SDA and SCL usually populate automatically based on your board's pins_arduino.h e.g. https://github.com/esp8266/Arduino/blob/master/variants/nodemcu/pins_arduino.h

void setup() {
  // open serial for debugging
  Serial.begin(115200);

  display.init();
  display.flipScreenVertically();
  display.setContrast(255);

  pinMode(LED_BUILTIN, OUTPUT);

  //thing.clean_credentials();

  // digital pin control example (i.e. turning on/off a light, a relay, configuring a parameter, etc)
  thing["led"] << digitalPin(LED_BUILTIN);

  // resource output example (i.e. reading a sensor value)
  thing["millis"] >> outputValue(millis());

  thing.set_on_captive_portal_run([](WiFiManager& manager){
    manager.setAPCallback([](WiFiManager *myWiFiManager) {
        display.clear();
        display.setFont(ArialMT_Plain_10);
        display.setTextAlignment(TEXT_ALIGN_CENTER_BOTH);
        display.drawString(display.getWidth() / 2, display.getHeight() / 2 - 10, 
        "Configure WiFi\n" +
        myWiFiManager->getConfigPortalSSID() + "\n" + 
        WiFi.softAPIP().toString());
        display.display();
      });
  });

  thing.set_on_config_callback([](pson &){
    display.clear();
    display.setFont(ArialMT_Plain_10);
    display.setTextAlignment(TEXT_ALIGN_CENTER_BOTH);
    display.drawString(display.getWidth() / 2, display.getHeight() / 2 - 10, "Device Configured");
    display.display();
    delay(1000);
  });

  thing.set_on_wifi_config([](bool wifi){
    display.clear();
    display.setFont(ArialMT_Plain_10);
    display.setTextAlignment(TEXT_ALIGN_CENTER_BOTH);
    display.drawString(display.getWidth() / 2, display.getHeight() / 2 - 10, "WiFi Configured");
    display.display();
    delay(1000);
  });

  ota.set_block_size(2048);

  ota.on_start([]() {
    display.clear();
    display.setFont(ArialMT_Plain_10);
    display.setTextAlignment(TEXT_ALIGN_CENTER_BOTH);
    display.drawString(display.getWidth() / 2, display.getHeight() / 2 - 10, "OTA Update " + String(ESP.getFreeHeap()));
    display.display();
  });

  ota.on_progress([](size_t progress, size_t total) {
    display.drawProgressBar(4, 32, 120, 8, progress / (total / 100) );
    display.display();
  });

  ota.on_end([]() {
    display.clear();
    display.setFont(ArialMT_Plain_10);
    display.setTextAlignment(TEXT_ALIGN_CENTER_BOTH);
    display.drawString(display.getWidth() / 2, display.getHeight() / 2, "Restart");
    display.display();
  });

 thing.set_state_listener([&](ThingerClient::THINGER_STATE state){
    String status;

    switch(state){
        case ThingerClient::NETWORK_CONNECTING:
            status = "Connecting Network...";
            break;
        case ThingerClient::NETWORK_CONNECTED:
            status = "Network Connected\n" + WiFi.SSID() + "\n" + WiFi.localIP().toString();
            break;
        case ThingerClient::NETWORK_CONNECT_ERROR:
            status = "Error while connecting...";
            break;
        case ThingerClient::SOCKET_CONNECTING:
            status = "Connecting\n" THINGER_SERVER;
            break;
        case ThingerClient::SOCKET_CONNECTED:
            status = "Connected\n" THINGER_SERVER;
            break;
        case ThingerClient::SOCKET_CONNECTION_ERROR:
            status = "Cannot Connect\n" THINGER_SERVER;
            break;
        case ThingerClient::SOCKET_DISCONNECTED:
            status = "Disconnected\n" THINGER_SERVER;
            break;
        case ThingerClient::SOCKET_ERROR:
            status = "Connection Error\n" THINGER_SERVER;
            break;
        case ThingerClient::SOCKET_TIMEOUT:
            status = "Connection Timeout\n" THINGER_SERVER;
            break;
        case ThingerClient::THINGER_AUTHENTICATING:
            status = "Authenticating\n" THINGER_SERVER;
            break;
        case ThingerClient::THINGER_AUTHENTICATED:
            status = "Authenticated\n" THINGER_SERVER "\nWaiting Updates";
            break;
        case ThingerClient::THINGER_AUTH_FAILED:
            status = "Auth Failed\n" THINGER_SERVER;
            break;
        case ThingerClient::THINGER_STOP_REQUEST:
            status = "Stopping\n" THINGER_SERVER;
            break;
    }

      // Align text vertical/horizontal center
      display.clear();
      display.setTextAlignment(TEXT_ALIGN_CENTER_BOTH);
      display.setFont(ArialMT_Plain_10);
      display.drawString(display.getWidth() / 2, display.getHeight() / 2, status);
      display.display();

      // add some delay to be able to see all states
      delay(1000);
  });

  /*
    Steps for getting the ESP8266 WebConfig working:
    1. Connect to Thinger-Device WiFi with your computer or phone, using thinger.io as WiFi password
    2. Wait for the configuration window, or navigate to http://192.168.4.1 if it does not appear
    3. Configure the wifi where the ESP8266 will be connected, and your thinger.io device credentials
    4. Your device should be now connected to the platform.
    More details at http://docs.thinger.io/arduino/
  */
}

void loop() {
  thing.handle();
}
1 Like

Hi, @iotregenesis and @inelint
If you can do some tests to confirm what I identified in the Arduino-Thinger library under development (link):

Summary of the BUGs I identified:

  1. Connection failure if the device cannot get the date and time on power on: [NTP_SYN] Cannot sync time!
    It is necessary to create a routine for ESP8266 to make a new request to obtain the NTP date and time. Otherwise, the device cannot connect to the Thinger Server.

  2. First connection failed right after registering the SSID and password of the modem/router.
    The cause would be that ThingerWebConfig.h doesn’t call setClock() to get the NTP date and time. If the device is reset (second attempt), then it is able to connect to the Thinger Server.

  3. Failed to resume connection when simulating turning off (2 min) and on the modem/router.
    The cause may be the same as previously reported.

  4. Connection resume failure when simulating modem/router internet connection drop (2min → timeout)
    Device fails to reconnect with Thinger Server when modem/router internet works again

About OTA with WebConfig:
It’s working fine with ThingerESP8266WebConfig.h . I haven’t tested ThingerESP32WebConfig.h.

About OTA (ESP8266 and 2G/GSM - SIM800C modem):
In two attempts, an error occurred in the middle of the download process: Error while writing to device: Error: Request failed with status code 404 debug VS Code (vscode-extension).

1 Like

Buen trabajo!

1 Like

Hi, @inelint
Did you manage to repeat the bugs described in my post?
Today, I will test with ThingerESP32WebConfig.h

We really need the library to work well. We are depending on the library to release some products. We’re two months behind on our schedule.

We hope this feedback will help @alvarolb and that the new version will be released as soon as possible.

Update: OTA via GSM (with ThingerTinyGSM.h) is working correctly. I had problems with the modem configuration. After adjusting the settings, the device has been working properly for many hours.

I have not tested “ThingerESP32WebConfig.h”. Has anyone performed tests?

1 Like