HTTP post rejected from SIM900 GSM module

tormod · June 27, 2020, 10:31am

So I have code that pushes data (GPS coordinates) to a bucket, and until recently this code worked perfectly. Thinger.io apparently made some changes at the start of May (new platform release), and the code has since not worked.

Now, this is usually a good indication where to start, changes are made and stuff stops working! I first thought that the API was changed or something, but when trying the request through the postman service using the exact same URL, authorisation token and input data, the data is updated successfully, so apparently the API still works, and I’m left thinking it might be something with my syntax or the way I build the HTTP post request since the error I receive is this: “{“error”:{“message”:“Content-Type must be application/json”}}”?

I’m using an Arduino Leonardo with a SIM900 development board.

The output from the application is as follows (it also outputs the commands pushed to thinger API):

AT

OK
AT+CMGF=1

OK
AT+CNMI=1,2,0,0,0

OK
AT+SAPBR=3,1,"CONTYPE","GPRS"

OK
AT+SAPBR=3,1,"APN","TeleXXX"

OK
AT+SAPBR=1,1

OK
AT+SAPBR=2,1

+SAPBR: 1,1,"46.156.55.XXX"

OK
AT+HTTPINIT

OK
AT+HTTPPARA="CID",1

OK
AT+HTTPPARA="URL","http://backend.thinger.io/v3/users/tom/devices/CT/callback/data?authorization=eyJhbGKLJSDFKLJSsdfsdfsdf.eyJqdGkiOiJEZXZpY2VDYWxsYmFja19DVCIsInVzciI6InRvcm1vZCJ9.AR1gWvIZB9KmtI-5Z12YXKuchPTGn58AkwBJSZQIoxQ"

OK
AT+HTTPPARA="CONTENT","application/json"

OK
AT+HTTPDATA=51,10000

DOWNLOAD
{ "longitude" : 5.336664 , "latitude" : 50.365433 }

OK
AT+HTTPACTION=1

OK

+HTTPACTION:1,400,61
AT+HTTPREAD

+HTTPREAD:61
{"error":{"message":"Content-Type must be application/json"}}

OK
AT+HTTPTERM

OK

JorgeTrincado · June 29, 2020, 9:18am

Hy @tormod,

I think you have any mistake in the token, can you copy here your access token configuration?

best

tormod · June 29, 2020, 9:50pm

Hi @JorgeTrincado,

And thanks for the reply. What mistake are you referring to? I should probably mentioned that I have purposely replaced parts of the access token with random characters, so not to reveal the token publicly. I have attached a screenshot from my device setup callback page.!

I have also replaced the SSL part of the URL (using http and not https) due to the SIM900 module not supporting SSL. Now, I’ve tested the non-SSL URL with postman and it works, so I don’t think that would be a problem either, otherwise I would expect an “authorization failure” reply from the request.

21|689x364

JorgeTrincado · July 8, 2020, 7:00pm

Hy @tormod

always you have problems with authorizations, try creating an ad-hoc access token instead of using the original one that can maybe be replaced by the oauth system.

best

tormod · July 15, 2020, 8:46pm

@JorgeTrincado I’ve created a new access token, but it did not improve anything. Taking into account that they both update the bucket through the API using postman (200 - OK returned) I am assuming that the access tokens actually work.

I have seen similar posts regarding the updated API as well, so it looks like I’m not the only one. I tried to change “v3” to “v2” to use the old one, but that returned a “not authorised” reply.

tormod · September 27, 2020, 4:43pm

After picking up this where I left it a while ago, I’m still not getting anywhere! I can’t seem to get my head around this. I’ve attached screenshots from postman where you can clearly see that it works (return 200 - OK) with the same authorisation token, so the token must be ok. Even if the token was not ok, I would expect “not authorised” 403 or similar.

I have tried both “device token” and access token directly to data bucket. Both works with postman, but not when using AT commands to compile a HTTP POST request, it does not.

I’m also attaching the Arduino code with AT commands in case this is of any help.

There must surely be some other syntax errors that can cause this? After all, it started after the API change in May.

SIM900.println( F(“AT+HTTPINIT”) );
delay(1000);
updateSIM900();

SIM900.println( F(“AT+HTTPPARA="CID",1”) );
delay(1000);
updateSIM900();

SIM900.println(“AT+HTTPPARA="URL","http://backend.thinger.io/v3/users/tormod/devices/CT/callback/data?authorization=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJqdGkiOiJEZXZpY2VDYWxsYmFja19DVCIsInVzciI6Gn58AkwBJSZQIoxQ"”);
delay(2000);
updateSIM900();

SIM900.println( F(“AT+HTTPPARA="CONTENT","application/json"”) );
delay(1000);
updateSIM900();

SIM900.println(httpDataLen);
delay(1000);
updateSIM900();

SIM900.println(content);
delay(1000);
updateSIM900();

SIM900.println( F(“AT+HTTPACTION=1”) );
delay(10000);
updateSIM900();

SIM900.println(“AT+HTTPREAD”);
delay(2000);
updateSIM900();

SIM900.println( F(“AT+HTTPTERM”) );
delay(1000);
updateSIM900();

alvarolb · September 27, 2020, 6:59pm

Hi, I think that the problem is related either with the Content-Type parameter (but it seems to be ok on your code) or the payload being set to the request, that is not in JSON format (I cannot see the code where you set the payload)

I suggest you to debug your HTTP request by using an online service like this: https://requestcatcher.com/, in order to debug the actual request you are sending to the server. Just change the request URL by the URL you generate in this service. Then let us know

tormod · September 27, 2020, 8:08pm

Hi @alvarolb and thanks for your quick response Requestcatcher was indeed a useful service, and it did reveal something.

Here is the output from postman, which obviously work (but i put in the body/json data for reference):

POST /test HTTP/1.1
Host: ct.requestcatcher.com
Accept: /
Accept-Encoding: gzip, deflate, br
Connection: keep-alive
Content-Length: 45
Content-Type: application/json
Postman-Token: 5bc407cf-d81b-4ee4-ac88-225f498df734
User-Agent: PostmanRuntime/7.26.5

{“latitude”: 39.416775,“longitude”: 89.70379}

Now, here is the output from my Arduino code:

POST / HTTP/1.1
Host: ct.requestcatcher.com
Accept: /
Connection: Keep-Alive
Content-Length: 40
User-Agent: SIMCOM_MODULE

{“longitude”:40.0015,“latitude”:90.5032}

Indeed, the “content-type” is actually missing, so that probably explains why the request is rejected.

I don’t see why this line of code in the Arduino project should not work anymore though. As far as I can see, the content-type is set correctly:

SIM900.println( F(“AT+HTTPPARA="CONTENT","application/json"”) );

Anyway, since this might not be a Thinger related issue then, I might have to bring this to another forum… unless you have any ideas of course

alvarolb · September 27, 2020, 8:22pm

Nice! Now we have the root cause of the problem

Not sure why the “CONTENT” AT is not working as expected. Maybe your module require a firmware update?

In think we can force to set the Content-Type with the “USERDATA”, but I have not tested it, as it is something I read quickly from the Internet:

AT+HTTPPARA="USERDATA","Content-Type: application/json"

In theory, this method works for setting any header, including the access token

AT+HTTPPARA="USERDATA","Authorization: Bearer xyz..."

Hope you can progress something more. Please, let us know!

tormod · September 27, 2020, 8:51pm

Thanks @alvarolb for the tips. I tried the “USERDATA” command too, but it did not work (POST header did not change). But at least I know where the problem is now, and thats a big step forward… I’ll post an update when I figure it out…

alvarolb · September 27, 2020, 9:04pm

In theory it should work, it is even possible to set various headers, as someone pointed here:

Will wait for your update

tormod · September 28, 2020, 5:47pm

You sir, are a lifesaver Indeed, the “CONTENT” parameter was not supported in the older firmware. I had, for some reason, the oldest firmware you could possibly have on the chip, so I updated firmware, and it worked straight away!

The POST header now includes the content-type, and when I put back the thinger address, the data bucket is updated too! Embarrassed that I did not think of checking this in the first place, should be one of the first things to check, that everything is in its latest version. Got to say though, the update procedure was nothing less than a real pain…

So here it is:

POST / HTTP/1.1
Host: ct.requestcatcher.com
Accept: /
Connection: Keep-Alive
Content-Length: 42
Content-Type: application/json
User-Agent: SIMCOM_MODULE

{“longitude”: 40.0015,“latitude”: 90.5032}

And I’ve learned a lot in the process too… as http communication is not my strong suit.

For later reference, in case anyone else hits the same problem, the first firmware to support the CONTENT parameter is B10 (1137B10SIM900M64_ST) for the SIM900 chip.

alvarolb · September 29, 2020, 8:59am

Nice you got it working!

rascal · January 15, 2021, 12:47am

Hi - I have the same issue. Trying to use my SIM800L setup after shelving it for a months - and it doesn’t work - but: I can use the SIM800L together with a terminal (Realterm) and then it works with the following commands:

AT+SAPBR=3,1,"CONTYPE","GPRS"

AT+SAPBR=3,1,“APN”,“internet”
AT+SAPBR=1,1
AT+SAPBR=2,1

AT+HTTPINIT
AT+HTTPPARA=“CID”,1
AT+HTTPPARA=“URL”,“http://backend.thinger.io/v3/users/rascalsailor/devices/SIM800L/callback/data?authorization=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJqdGkiOiJEZXZpY2VDYWxsYmFja19TSU04MDBMIiwidXNyIjoicmFzY2Fsc2FpbG9yIn0.GYiTnCNkA4R3EJBqVRDkGbFJK1x-***************”
AT+HTTPPARA=“CONTENT”,“application/json”
AT+HTTPDATA=30,10000 //Content-Length: 30
{“Temperature”: 21}
AT+HTTPACTION = 1
AT+HTTPREAD
AT+HTTPTERM

But using the similar code to yours, I can’t get it to send to the server.
Regards
Russell