Get the last value from the bucket: Some questions

1

Hey guys

I did a lot of research in the Community to find out how to “get last value from the bucket” to apply it to various purposes, for example to visualize values ​​on the display.
I read these posts:

2018/01 - Acces bucket from device - General - The Internet of Thinger
2018/07 - Consulta lectura datos desde data bucket - Reading data from data bucket - General - The Internet of Thinger
2018/9 - Data bucket query, from a device - How-To - The Internet of Thinger
2020/12 - How to take data from Api ans watch it in monitor serial in Arduino IDE? - How-To - The Internet of Thinger

I was successful in my initial code, but I have some questions:

Almost 3 years after the first post, I have some doubts:

1 - Is there already any functionality available in the Arduino-Thinger library to obtain this information (last value of the bucket), as announced by @alvarolb in this post(Acces bucket from device - #2 by alvarolb?

2 - The Esp8266 ‘HTTPClient::begin(String)’ function is deprecated. The new function is ‘HTTPClient::begin(client, String);’. See this post('bool HTTPClient::begin(String)' is deprecated - #2 by maxgerhardt - PlatformIO Community).
How do I use the new function with the Arduino-Thinger library without causing problems with the ‘thing.handle();’ function or between device connection and Host Thinger.io?

3 - ESP8266’s HTTPClient library makes GET requests with HTTP (unencrypted), how can I make GET requests with HTTPS (encrypted) to Host Thinger’s bucket APIs? I saw these post(HTTPS requests to any URL using the ESP8266 - Maakbaas), but I was wondering if there is a better solution with Thinger-Arduino library or that doesn’t cause problems between device connection and Host Thinger.io.

4 - I tried to get the ‘ts’ variable from the JSON (Bucket API Example - link) using the ArduinoJSON library and ESP8266. The problem is that the value stored in ‘ts’ is too big (64btis = long long) and I’m getting changed values ​​when trying to transform from milesseconds to seconds. Looks like ArduinoJSON v6.18 does a random ‘rounding’. Example: The JSON value (example) “ts”:1631803256689.0 is transformed to ‘1631803015’ when trying to divide the variable ts by 1000 (ts/1000, to transform milesseconds into seconds). Even applying the solution proposed in this post (Problem with large numbers after moving from v5 to v6 (EXPONENTIATION_THRESHOLD) · Issue #954 · bblanchon/ArduinoJson · GitHub) it’s that (Arduino: How to covert epoch with milliseconds to epoch - Stack Overflow) I think it doesn’t work for the new version of ArduinoJSON v6.18.

Can I use the ‘pson.h’ library (from Thinger) to extract the JSON variables from the Bucket, in particular the variable “ts”? Or some other solution or JSON library for Arduino/ESp8266?
The solution I used was to transform the JSON into a String and remove the characters to the left and right of the value “ts”:1631803256689.0 to get a String “1631803256” and then convert it to type long. But it is not ideal, as it may break in the future with the change of the JSON structure.

5 - Some GET requests with the ESP8266 HTTPClient library return “Error code: -11”. Does anyone know what this error is?. Some people report the same problem. In this post(Http error code -11 (negativ 11) from a postrequest sent from a Arduino board wi - #4 by Gormd - Networking, Protocols, and Devices - Arduino Forum), a user reports that he handles HTTP error 411, but I don’t know if it’s true.
At other times, the GET request returns HTTP error “204”. Why does this error occur? Is it a Host Thinger requirement? How can I avoid this error on GET requests with Host Thinger?

Thanks for listening

##############################################
Update:

About question 4:

#include <Arduino.h>

#define ARDUINOJSON_USE_LONG_LONG 1
#define ARDUINOJSON_USE_DOUBLE 1

#include <ArduinoJson.h>
#include <StreamUtils.h>

// https://github.com/PaulStoffregen/Time
#include <TimeLib.h> 

#include <ThingerESP8266.h>

// #include <WiFiManager.h>
#include <ESP8266HTTPClient.h>
#include <ESP8266WiFi.h>
#include <ESP8266WebServer.h>
#include <DNSServer.h>
#include "FS.h"

#define USERNAME "XXXXXXXXXXXX"
#define DEVICE_ID "YYYYYYYYYYYYYYYYY"
#define DEVICE_CREDENTIAL "ZZZZZZZZZZZZZZZZZ"

#define SSID "LLLLLLLLLLLL"
#define SSID_PASSWORD "PPPPPPPPPPPPPP"

unsigned long next_request;

ThingerESP8266 thing(USERNAME, DEVICE_ID, DEVICE_CREDENTIAL);

void setup() {

  thing.add_wifi(SSID, SSID_PASSWORD);

  Serial.begin(115200);

}

void loop() {

if(millis() > next_request ){ 

  WiFiClient client;
  HTTPClient http;

  // Send request
  // https://arduinojson.org/v6/how-to/use-arduinojson-with-httpclient/
  http.useHTTP10(true);

  // Replace httpS with http in the request link !!!!!!!!!!
  http.begin(client, "http://api.thinger.io/v1/users/jt/buckets/SensePoint/data?items=1&max_ts=0&sort=desc&authorization=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJqdGkiOiJEYXNoYm9hcmRfREVNTyIsInVzciI6Imp0In0.y_cwNQqIak5oQQhElcmO0hEYKEodds3UB8SHW7Rlmko");

  http.GET();

  //Use the Json Assistant (ArduinoJSON library)  to find out the proper size of the JSON in memory
  // https://arduinojson.org/v6/assistant/

  DynamicJsonDocument doc(2048);
  
   ReadLoggingStream loggingStream(http.getStream(), Serial);

  deserializeJson(doc, http.getStream());
  long time_thinger = ((long long) doc[0]["ts"] / 1000); // microseconds to milliseconds

            if(time_thinger == 0) {

                  Serial.println("Request Error - Empty JSON");

            } else{

            Serial.print("Time milliseconds: ");
            Serial.println(time_thinger);
            
            time_t ts = time_thinger;
            int hour_ts = hour(ts);
            int minute_ts = minute(ts);
            int second_ts = second(ts);
            int day_ts = day(ts);
            int month_ts = month(ts);
            int year_ts = year(ts);

            String data_time = String(hour_ts) + ":" + String(minute_ts) + ":" + String(second_ts) + " - " + String(year_ts) + "/" + String(month_ts) + "/" + String(day_ts);
          
            Serial.println(data_time);
           
            }    

  

  // Disconnect

  http.end();

  next_request = millis() + 10000;

}

}

About question 5:
I think “Error -11” is related to HTTP request timeout. I deduced this by refining this snippet of library code Arduino/ESP8266HTTPClient.h at 612e7ffd7fb398fc2cdcf8bd1d177b75d997d46c · esp8266/Arduino · GitHub .
The reason for HTTP 204 remains to be found

2

Hi @George_Santiago,

I’ll try to be brief as it seems you have done your research and thoroughly tested the possibilities.

  1. I believe there is no functionality at the moment in the Arduino Thinger.io library to read from a data bucket.

  2. You can create a new standalone client for this request and as per the documentation from Thinger.io good coding practices use a non locking structure:

void loop() {
  thing.handle();

  // Execute every 20 seconds
  if (millis() %20000==0) {
    WiFiClient client;
    HTTPClient http;

    if (http.begin(client, url)) {  // HTTP
      Serial.println("[HTTP] GET...");
      int httpCode = http.GET();

      // httpCode will be negative on error
      if (httpCode > 0) {
        // HTTP header has been send and Server response header has been handled
        Serial.printf("[HTTP] GET... code: %d\n", httpCode);
        // file found at server?
        if (httpCode == HTTP_CODE_OK) {
          String payload = http.getString();
          Serial.println(String("[HTTP] Received payload: ") + payload);
        }
      } else {
        Serial.printf("[HTTP] GET... failed, error: %s\n\r", http.errorToString(httpCode).c_str());
      }

      http.end();
    } else {
      Serial.printf("[HTTP] Unable to connect\n\r");
    }
  }
}
  1. Sadly is does seem possible nor I was successful while testing without breaking connection with Thinger.io. It seems that the ESP8266 is not able to handle two secure clients (which is what Thinger.io library uses if SSL is not disabled) simultaneously due to RAM requirements, as the official documentation states. I’ll leave a couple of additional references: link1 and link2. Let me know if you are willing to live with the breaking connection against Thinger.io, if the request is not to often, and I’ll post the code.

Sorry I can not help you with points 4 and 5, but let me give you a couple of extra ideas in case you haven’t yet thought of them, based on writing the data to the device instead of the device retrieving it.

If you have access to either the plugins in the platform or are willing to self-host a Node-RED instance, since last published version you are able to read the last item from a data bucket and write the message to a device. I believe this could solve both your issues regarding the secure connection and the JSON rounding error.

Another option would be to have an additional server that could periodically read the last item of the bucket and send it to the device through the API, similarly as to how you are retrieving the data from the bucket.

Hope it helps

3

Hi,

I could extract the values from the received JSON in the device handling it as String and looking for the positions, this I guess could solve the need instead to look for another library, in the quoted posts there is one mine where is a function to do it, let me know any doubt about it.

On other hand, regarding the value storing in bucket, what I think could work using the thinger native protoocol is to store it as a property too, so anytime the device will be able to call and consult the last value as there is already a native communication to the device’s properties, of course this will work for the device consulting its own last values, not crossed devices consulting, dont know if this works for your purpose.

Hope this helps.

4

Thanks for this explanation.

Thanks for that suggestion. I will try this the solution you proposed.

Thanks for the @ega suggestion. After a lot of research, I found the right solution with the ArduinoJSON library. The solution was to put: #define ARDUINOJSON_USE_LONG_LONG 1 and ARDUINOJSON_USE_DOUBLE 1)

I updated the post above and left the solution. I believe many people look for this solution too.

1 Like