Importing Data from CSV into Bucket

How-To
1

Hello,
I am trying to test how to manually import CSV data into Thinger.
After perusing the following documentation I gather that the source .csv file is subject to a few restrictions: https://docs.thinger.io/features/buckets
Namely:

  1. “;” used as separator.
  2. A “ts” column in milliseconds.

Just for testing’s sake let us assume the following file:
prueba_manual.csv

ts; nombre
11210.524279; evento1
19917.620121; evento2

After importing the file it says “Your bucket data has been imported!”, yet the preview window of the bucket is completely empty.

Perhaps the format of this test .csv file is not correct. Yet Thinger claims the data has been correctly imported to the bucket (even when providing a csv file with no ts column), while nothing is shown.

Might you please provide a short valid test .csv file or perhapps indicate what other issue might be at fault?

Thank you.

2

Hi,

I see some things that can be causing the issue,

The timestamp is an integer, it should not have a ., and taking the integer part both ts are from 1970 (Thursday, 1 January 1970 5:31:57 the last one), note that the bucket shows data from selected period (last 24h by default), you would need to set from 1969 forward to check if it has the data stored.

The name of the column should be between ", for your example should be → "nombre", I think the strings evento1 and evento2, should be between " too.

Hope this helps.

3

Thank you ega for your kind reply.

I got that epoch value from dmesg, and silly me forgot that it doesnt add the current unix date seconds by default.

The test file currently looks like this:

ts; "nombre"
1686241640; "evento1"
1690230359; "evento2"

Both dates are more current now, and I believe having the values between " helped as there is now some sort of tab-completion of similar values in the editor.

I import the data again, yet still cant see anything displayed.

4

Wierd… although I still dont see anything in the preview window, I have managed to export the bucket to a .csv file.

In seconds:

"ts";"ANO";"D01";"D02";"D03";"D04";"D05";"D06";"D07";"D08";"D09";"D10";"D11";"D12";"D13";"D14";"D15";"D16";"D17";"D18";"D19";"D20";"D21";"D22";"D23";"D24";"D25";"D26";"D27";"D28";"D29";"D30";"D31";"ESTACION";"MAGNITUD";"MES";"MUNICIPIO";"PROVINCIA";"PUNTO_MUESTREO";"V01";"V02";"V03";"V04";"V05";"V06";"V07";"V08";"V09";"V10";"V11";"V12";"V13";"V14";"V15";"V16";"V17";"V18";"V19";"V20";"V21";"V22";"V23";"V24";"V25";"V26";"V27";"V28";"V29";"V30";"V31"
0.000000;2022;0;0;0.7;0.4;10.1;0.2;0;17;7;0.1;12.4;10.1;32.9;17.2;2.4;1.6;0.1;0.1;0;8;0.1;0;0.4;0;0;0;0;0;0;0;0;59;89;12;79;28;28079059;"V";"V";"V";"V";"V";"V";"V";"V";"V";"V";"V";"V";"V";"V";"V";"V";"V";"V";"V";"V";"V";"V";"V";"V";"V";"V";"V";"V";"V";"V";"V"
11.210000;" evento1"
1686241.640000;" evento1"
1690230.359000;" evento2"
11210524.279000;"evento1"

in milliseconds:

"ts";"ANO";"D01";"D02";"D03";"D04";"D05";"D06";"D07";"D08";"D09";"D10";"D11";"D12";"D13";"D14";"D15";"D16";"D17";"D18";"D19";"D20";"D21";"D22";"D23";"D24";"D25";"D26";"D27";"D28";"D29";"D30";"D31";"ESTACION";"MAGNITUD";"MES";"MUNICIPIO";"PROVINCIA";"PUNTO_MUESTREO";"V01";"V02";"V03";"V04";"V05";"V06";"V07";"V08";"V09";"V10";"V11";"V12";"V13";"V14";"V15";"V16";"V17";"V18";"V19";"V20";"V21";"V22";"V23";"V24";"V25";"V26";"V27";"V28";"V29";"V30";"V31"
0;2022;0;0;0.7;0.4;10.1;0.2;0;17;7;0.1;12.4;10.1;32.9;17.2;2.4;1.6;0.1;0.1;0;8;0.1;0;0.4;0;0;0;0;0;0;0;0;59;89;12;79;28;28079059;"V";"V";"V";"V";"V";"V";"V";"V";"V";"V";"V";"V";"V";"V";"V";"V";"V";"V";"V";"V";"V";"V";"V";"V";"V";"V";"V";"V";"V";"V";"V"
11210;" evento1"
1686241640;" evento1"
1690230359;" evento2"
11210524279;"evento1"

There’s some other ‘garbage’ at the begging of the archive from when I tried to import meteorological data from a file.

It seems Thinger was expecting unix epoch in milliseconds as apposed to seconds.
So the dates in my previous amendment were not in fact current.

Current file looks like this now:

ts; "nombre"
1686241640000; "evento1"
1690230359000; "evento2"
1690231764854; "evento3"

Now I can finally see some data displayed in the preview window. Oddly enough “evento1” is not shown.

I’ll probably start again with a new bucket and try and play/debug this further.

Thanks again for the help ega.

5

Hi @idatart ,

I agree with you, for sure I would recommend you to start from scratch with an empty bucket, note that is a bad practice to have unconsistent data sets, all the fields or variables should be filled with each dataset stored-imported.

Hope this helps.