Best way for streaming data?

Hello everyone, i’m working with a sketch for a device and recording some data… let’s say Temp and Hum for example…

This is my loop:

void loop() {
  if (WiFi.status() != 6) digitalWrite(ledPin, LOW);   //  (invertito)  
  else digitalWrite(ledPin, HIGH);                     // 

  thing.handle();

  sensors.requestTemperatures();                    //  DS18B20
  
  thing.stream(thing["LetturaDati"]);
  
  //if (millis() > timer + 60000) {                      //call each 1 minutes
     
     //thing.write_bucket("Bucket_ESP2_Colonna00", "LetturaDati");   //call thinger.io endpoint function and attacht the pson (ID Bucket, thing)
     //timer = millis();                                 //actualize time counter
  //};
  //ESP.deepSleep(SLEEP_MS*1000, WAKE_RF_DEFAULT);  
}

Is this ok if I stream data many times, but I show in the dashboard or register in buckets in a specific time? like every 1 hour… or I better insert the write call from the device?? there is a way to stream in a specific time? like every second with a command in the sketch?

Both of them, works perfectly maybe the logic is not the right one…

Thanks to everyone

Hi, what I think is you should add the stream instruction into a cycle that runs every second (as the commented “call each 1 minute” for example), this should execute the instruction once every second.

Let me know how it goes.

I tryied like this:

void loop() {
  if (WiFi.status() != 6) digitalWrite(ledPin, LOW);     
  else digitalWrite(ledPin, HIGH);                     

  thing.handle();

  if (millis() > timer + 2000) {                      //call each 2 sec
     sensors.requestTemperatures();                   DS18B20
     thing.stream(thing["LetturaDati"]);    
     
     //thing.write_bucket("Bucket_ESP2_Colonna00", "LetturaDati");   //call thinger.io endpoint function and attacht the pson (ID Bucket, thing)
     
     timer = millis();                                 //actualize time counter
  };

}

But I can’t see the API… i think something is in conflict… idk!!

Hi, paste the whole code, or at least the “setup()” function too, you should see the “LetturaDati” thing at the device’s API, let’s start solving that.

Hello, sorry for the late reply… I solved it!! i mean… i was trying the device without the sensors plugged in… an i thought at least to see a 0 value on the api, but after connecting it the api finally appears!! my bad…

BUT… i’m still not getting which is the best way to get data from the device… i mean… i need data at least every 1 hour… so what i’m doing now is to stream the device with the function “stream” without a timing… so i get so many data each second… and then i set the bucket to write from device every 1 hour… by doing this i understand that i’m overloading the device and the server, even if i set the recording on bucket every 1 hour.

The reason i want to stream the device so frequently is not for the collection of data but because sometimes i need to see in real time what is happening without waiting 1 hour!
That’s why i thought I can set the sketch as you can see in the loop above: I repeat it here

thing.handle();
  if (millis() > timer + 2000) {                      //call each 2 sec
     sensors.requestTemperatures();                   //DS18B20
     thing.stream(thing["LetturaDati"]);    
     //thing.write_bucket("Bucket_ESP2_Colonna00", "LetturaDati");   //call thinger.io endpoint function and attacht the pson (ID Bucket, thing)
     
     timer = millis();                                 //actualize time counter
  };

I thought by stream the device every 1/2/10 sec I don’t overload the server and the device.

Is my reasoning technically correct?? and, the sketch on loop, (apart the fact that I works…) is correct?? the way of setting the time is the “if millis…” the right way? I think I’m missing something… even if it’s properly working! but maybe I’m wrong.

Thank you
Bests

But you can set the dashboard to ask the device the last measured value at certain frequency, does this work for you?
This is better because if the dashboard is not open, the device doesnt execute the stream loop.

ohhh yes! thank you! this is what i was missing!! the famous “Sampling Interval” on refresh mode!

Now I got it! so I don’t Stress the device sensing infinite data, but I see them when I open the dashboard!! Right?

Every version works! BUT… i tried to do a test to delete the stream on the sketch and see if I get the data… it works perfectly on dashboard but not on the bucket… As you can see on the image, the bucket is set to log every 10 min,(rec on min 54,56,57) and it works but when I open the dashboard (that I set to see data every X sec… but this is another thing and it works properly ) the bucket collects data, not respecting anymore the 10min (rec on min 54,56,57) !
Should I use the device write call to write the bucket or there is something else I’m missing. I don’t want the bucket to be interrupted by opening dashboard, and to maintain a regular recording.

Thank you very much for everything. it’s a pleasure to solve problems all together!

The values han nothing to do… i was just looking at the time it recorded

hahaha yes it is exciting when you get the issues solved and understanding how things are done.

Ok there are two ways to record data in the bucket, one is by writing call from the device, and the other is that the bucket periodically ask for the value, which one you want to use? The second one would be like the dashboard, no code required just the bucket will ask the device when it is online every timelapse established.

The advantage of the 1st way is that you can even set another way to write data, besides the timelapse, for example when the value has a peak, you may code to write by the call before this event.

The methods are not compatible, the bucket could be configured just to store data by asking to a device or when a device writes on it, not both ways in the same bucket.

Let me know how you want to do things to help you setting it up.

Regards,

Yes I got this… I understood both ways… I was thinking to use the record on bucket by asking the device… and this Is what I did! In the sketch there is nothing on loop… indeed in my image of the bucket you can see it worked properly recording every 10 min (my sampling period set on bucket) but you can also see that I get record of irregular timing sometimes… to be precise when I open the dashboard… so it interrupts the recording of every 10 min…why??
I hope I explained myself properly

Thanks
Bests

That could be an abnormal behaviour, delete from the sketch both, the stream and the bucket write instruction, those are not needed if you use the dashboard and the bucket to ask directly to the device for data to show and to save, and lets see if it works properly.

I already dit it, i don’t know!! the pic I showed you before, it was already with the sketch without anything on loop!! and I saw when I enter on dashboard it gets registered on bucket data, not respecting the timing!

Can you show me the setup? to see how is defined the “thing” because is so weird, I have not had this kind of behaviour it is so weird this.

#define THINGER_SERVER "server"   
#define _DEBUG_                       
#define _DISABLE_TLS_                 // 
#include <ESP8266WiFi.h>              //
#include <ThingerESP8266.h>           // req'd by thinger
#include <OneWire.h>                  // 
#include <DallasTemperature.h>        // 
#include <HX711.h>                    // 


#define USERNAME ""       // 
#define DEVICE_ID ""              //
#define DEVICE_CREDENTIAL ""   // 
ThingerESP8266 thing(USERNAME, DEVICE_ID, DEVICE_CREDENTIAL);

#define SSID ""           // 
#define SSID_PASSWORD ""    // 

// SET A TIME PERIOD TO UPDATE TEMP AND HUMID READINGS
//#define SLEEP_MS 120000                // 60,000 milliseconds FOR DEEPSLEEP MODE

const int wire = 4;                      // Unico filo - D2 = GPIO4 
//const int wire = 0;                    // Unico filo - D3 = GPIO0
const int ledPin = 2;                    // Led scheda - D4 = GPIO2

const int GND1 = 15;                     // Primo HX711 (Sezione 1) - GND -   D8 = GPIO15
const int sezione1_SCK_PIN = 13;         // Primo HX711 (Sezione 1) - Clock - D7 = GPIO13
const int sezione1_DOUT_PIN = 12;        // Primo HX711 (Sezione 1) - Data -  D6 = GPIO12
const int alim1 = 14;                    // Primo HX711 (Sezione 1) - Alim -  D5 = GPIO14

float temp = 0;                          //
int hx1ass = 0;                          // 
int hx1zero = 0;                         // costante di 0 
int hx1sens = 1;                         // 
int carico1 = 0;                         // inizializzazione di carico1
int OffSet_Temp = 0;



int timer = 0; // timer per lettura dati temporizzata

OneWire oneWire(wire);                   // Sensore Temperatura DS18B20 - D3=GPIO0
DallasTemperature sensors(&oneWire);     // Pass our oneWire reference to Dallas Temperature

HX711 sezione1;                          // 


void setup() {
  pinMode(ledPin, OUTPUT);               // 
  pinMode(GND1, OUTPUT);                 // per creare un GND
  pinMode(alim1, OUTPUT);                // per creare una Alim +3.3V
  digitalWrite(GND1, LOW);               // configura un GND
  digitalWrite(alim1, HIGH);             // configura una Alim +3.3V

  sensors.begin();                       // per la lettura della temperatura
  
  sezione1.begin(sezione1_DOUT_PIN, sezione1_SCK_PIN);  // 
   
//// WIFI STUFF  
  Serial.begin(115200);
  delay(100);
  Serial.println("Booting");
//// WIFI STUFF END  


  thing.add_wifi(SSID, SSID_PASSWORD);
  
  thing["LetturaDati"] >> [](pson& out){ 
    out["link"] = WiFi.RSSI();
    out["temp"] = sensors.getTempCByIndex(0);
    out["hx1ass"] = sezione1.read_average(10);
    out["carico1"] = (sezione1.read_average(10) - hx1zero) / hx1sens;
    out["tempcorr2"] = sensors.getTempCByIndex(0) + OffSet_Temp;
  }; 

  thing["hx1zero"] << [](pson& in){
    if(in.is_empty()){
        in = hx1zero;
    }
    else{
        hx1zero = in;
    }
  };
  
  thing["hx1sens"] << [](pson& in){
    if(in.is_empty()){
        in = hx1sens;
    }
    else{
        hx1sens = in;
    }
  };

 thing["OffSet_Temp"] << [](pson& in){
    if(in.is_empty()){
        in = OffSet_Temp;
    }
    else{
        OffSet_Temp = in;
    }
  };
}
  
void loop() {
  if (WiFi.status() != 6) digitalWrite(ledPin, LOW);     
  else digitalWrite(ledPin, HIGH);                     

  thing.handle();

  sensors.requestTemperatures();                   //DS18B20
     
  }

Certainly there is no routine to write at the bucket.

What I am thinking is that the device maybe the device is reconnecting every 1 minute, so when it connects the platform recognize it and ask for the value, and restart the timer to ask again in 10 minutes, but if it reconnects in one minute will do again, the data requesting feature is tied to the time when the device connects.

If you want to avoid this, necesarily need that the device handle this writing frequency, and need to use the bucket writing routine.

How can it connect every min? is this a Thinger stuff or you suppose? because it stays connected… it happens just only when i open the dashboard…

I’m supposing… But it is so weird that behaviour, the dashboard how often refresh? one minute or any timelapse… Does it show something in the serial console? I’m thinking that maybe the uC it is getting rebooted by something else, does the uptime in the cloud counter resetted or it goes according the time you have the uC since last scheduled reboot?

I repeat, this is so weird what are you reporting, just planting my thoughts about the situations that I think may deliver the result you are showing us.