Best practice for storing large sweep (IV curve) data via MQTT in Thinger.io Data Buckets

1

Hi Thinger team :wave:,

I am working with Thinger.io Data Buckets using “From MQTT Topic” and I need guidance on the correct / recommended way to store and visualize large, event-based sweep data.

:small_blue_diamond: Setup

  • Device: ESP32-S3
  • Transport: MQTT
  • Broker: backend.thinger.io
  • Bucket source: From MQTT Topic
  • Topic example: solar/mppt/02/sweep
  • Payload format: JSON
  • Data is successfully stored and exportable from the bucket

:small_blue_diamond: Data Types I Send

:one: Periodic sensor data (WORKING PERFECTLY)

Example payload:

{
  "bpw": 1,
  "lux": 234.5,
  "aht_temp": 32.1,
  "aht_hum": 61.3,
  "roll": 1.2,
  "pitch": -0.5,
  "ntc_array": "26.1,26.3,26.0,..."
}

2️⃣ Sweep / IV curve data (PROBLEM AREA)

Sweep is event-based, triggered manually or by timer.

Because the sweep string is long, I currently send it in multiple MQTT messages:

{"event":"init","data":""}
{"event":"sweep_start"}
{"event":"sweep_data","data":"49.38,0.05,2.30;49.21,0.38,18.07;..."}
{"event":"sweep_data","data":"48.77,1.42,66.14;..."}
{"event":"sweep_end"}


Each publish creates a separate row in the bucket (expected behavior).

🔹 Observed Behavior

✔ Data is stored in the bucket

✔ Exported JSON contains all sweep chunks

❌ Dashboards cannot reconstruct or plot the sweep

❌ Array/string-based sweep data is not usable for visualization

I also have a Python dashboard consuming live MQTT, and it works perfectly because it reconstructs the sweep manually.

❓ Questions

1️⃣ Is it recommended or supported to store large, event-based sweep data (IV curves) in Thinger.io Data Buckets?

2️⃣ Is it better to send the entire sweep in a single JSON payload, for example:

{
  "V": [49.38,49.21,49.04],
  "I": [0.05,0.38,0.82],
  "P": [2.30,18.07,38.39],
  "Voc": 49.42,
  "Vmp": 40.81,
  "Imp": 11.92,
  "Pmp": 509.5
}


Are there payload size limits?

Is this dashboard-friendly?

3️⃣ Is the intended best practice to:

Store only summary values (Voc, Vmp, Imp, Pmp) in Thinger buckets

And handle full IV curve visualization outside Thinger dashboards?

4️⃣ Are Thinger dashboards mainly designed for scalar time-series data, not large array or event-based datasets?

🎯 Goal

I want to:

Use Thinger.io for monitoring, KPIs, alarms, and history

Use external tools (Python) for full IV curve visualization

I’d like to confirm if this split architecture is the intended and recommended approach.

Thanks in advance for any guidance or best-practice confirmation 🙏
2

Hi,

Current widgets are limited to time-series data. Since the X-axis is restricted to timestamps, I would try using an HTML widget to create an XY plot using two different variables. I haven’t implemented it yet, but I believe it is feasible.

Which plan do you have? Recently, I read a topic mentioning that changes were made to the community server regarding data retention. I’m not sure if the total data volume was also impacted. Private instances, however, don’t have these limitations (or at least I haven’t reached them).

I recommend testing with the worst-case scenario (the highest number of variables) to verify if it could be stored. In any case, the recommendation is to keep your datasets consistent within the bucket, I mean all entries should contain the same variables (and the same size).

BR