[HTTP-Requests] cookies - SPTrans Olho Vivo

eduzal

Hello

I'm building a visualization tool to get the immediate position of the buses available. There's this API that delivers these posions using a GET request. The return is a JSON.

I'm authenticating without problems, but when i request anything it blocks me any access. it's stinky documentation doesn't says that they need cookies to return the request.

The final product will be a mapped projection with this map, transported via syphon to another Resolume Arena. i need help in getting this cookie within this and processing it to get my requests.

import http.requests.*;

String baseURL="http://"+"api.olhovivo.sptrans.com.br/v0";
String token="2bc28344ead24b3b5fe273ec7389d2e4d14abd2bad45abe6d8d774026193f894";
String route="7272";

void setup() {
  size(1200, 1800);

  //API Olho Vivo
  authentication();
  getLinha(route);
}

void authentication() {
  String reqURL = "/Login/Autenticar?token=";
  PostRequest post = new PostRequest(baseURL+reqURL+token);
  post.send();
  println("Reponse Content: " + post.getContent());
  println("Reponse Content-Length Header: " + post.getHeader("Content-Length"));
}

void getLinha(String busLIne) {
  String reqURL = "/Posicao?codigoLinha=";
  GetRequest get = new GetRequest(baseURL+reqURL+busLine); 
  get.send();
  println("Reponse Content: " + get.getContent());
  println("Reponse Content-Length Header: " + get.getHeader("Content-Length"));
}

Responses:
Reponse Content: true
Reponse Content-Length Header: 4
May 15, 2017 12:05:34 AM org.apache.http.impl.client.DefaultRequestDirector handleResponse
WARNING: Authentication error: Unable to respond to any of these challenges: {}
Reponse Content: {"Message":"Authorization has been denied for this request."}
Reponse Content-Length Header: 61

API documentation: http://www.sptrans.com.br/desenvolvedores/APIOlhoVivo/Documentacao.aspx?1

koogs

wget -S --method POST "http:// api.olhovivo.sptrans.com.br/v0/Login/Autenticar?token=2bc28344ead24b3b5fe273ec7389d2e4d14abd2bad45abe6d8d774026193f894"

--2017-05-15 07:32:51--  http:// api.olhovivo.sptrans.com.br/v0/Login/Autenticar?token=2bc28344ead24b3b5fe273ec7389d2e4d14abd2bad45abe6d8d774026193f894
Resolving api.olhovivo.sptrans.com.br (api.olhovivo.sptrans.com.br)... 200.189.189.84
Connecting to api.olhovivo.sptrans.com.br (api.olhovivo.sptrans.com.br)|200.189.189.84|:80... connected.
HTTP request sent, awaiting response... 
  HTTP/1.1 200 OK
  Cache-Control: no-cache
  Pragma: no-cache
  Content-Type: application/json; charset=utf-8
  Expires: -1
  Server: Microsoft-IIS/7.5
  X-AspNet-Version: 4.0.30319
  Set-Cookie: apiCredentials=2B205020F9B70EABD2B449FCF6AB97D546C6C6C69AD3E2AA0F1F9A388BE17DB0CDC858B1610994758BBCE8FA339EE04047DFFC162DE32A8EED31CAEF138FBD808D6CA71A90C544167F1601BF3081E40227F078AF9C171BF3A48D61E4B39285DCAA6EC4716E751B871F01723FF73144A2E4AC69874D713CF0D998425DAA6D4A5337E3FC0437146AF06A6CC976534D56BB5F65FD82B98EE7E15BBCEFCB62AF67F0C21EEC0805BA68C5194A18966A148F31A9C586717772F1852F2FDF7662BB852CED365E46; path=/; HttpOnly
  X-Powered-By: ASP.NET
  Date: Mon, 15 May 2017 06:32:50 GMT
  Content-Length: 4
Length: 4 [application/json]
Saving to: ‘Autenticar?token=2bc28344ead24b3b5fe273ec7389d2e4d14abd2bad45abe6d8d774026193f894.1’

2017-05-15 07:32:53 (182 KB/s) - ‘Autenticar?token=2bc28344ead24b3b5fe273ec7389d2e4d14abd2bad45abe6d8d774026193f894.1’ saved [4/4]

there's a cookie in the response, try capturing and sending that in later requests.

kfrajer

try capturing and sending that in later requests

@koogs How can one do that? I did think about checking if the POST had the cookie attached (based on this post: http://stackoverflow.com/questions/17769011/how-does-cookie-based-authentication-work) but I was not able to get it with Processing code.

Kf

koogs

Good question.

this example suggests how to access them in the response

https://github.com/runemadsen/HTTP-Requests-for-Processing/blob/master/examples/get/get.pde

and they are easy enough to add, being just headers

koogs

https://en.wikipedia.org/wiki/HTTP_cookie#Setting_a_cookie

eduzal

@koogs i'm realy clueless about this. i believe i would need a walkthrough this time. what should i store?

this?

Set-Cookie: apiCredentials=2B205020F9B70EABD2B449FCF6AB97D546C6C6C69AD3E2AA0F1F9A388BE17DB0CDC858B1610994758BBCE8FA339EE04047DFFC162DE32A8EED31CAEF138FBD808D6CA71A90C544167F1601BF3081E40227F078AF9C171BF3A48D61E4B39285DCAA6EC4716E751B871F01723FF73144A2E4AC69874D713CF0D998425DAA6D4A5337E3FC0437146AF06A6CC976534D56BB5F65FD82B98EE7E15BBCEFCB62AF67F0C21EEC0805BA68C5194A18966A148F31A9C586717772F1852F2FDF7662BB852CED365E46; path=/; HttpOnly

or this?

Saving to: ‘Autenticar?token=2bc28344ead24b3b5fe273ec7389d2e4d14abd2bad45abe6d8d774026193f894.1’

capturing you mean storing it in a variable like?

String credentials = post.getHeader("apiCredentials")

and then sending it back

get.addHeader("apiCredentials", "credentials");

tks

GoToLoop

https://forum.Processing.org/two/discussions/tagged?Tag=addheader()

koogs

this line

Saving to: ‘Autenticar?token=2bc28344ead24b3b5fe273ec7389d2e4d14abd2bad45abe6d8d774026193f894.1’

is just output from wget telling me where it was saving the output to. ignore it.

i think you want the contents of your varaiable rather than the literal "credentials"

get.addHeader("apiCredentials", credentials);

i'm not sure whether the bit on the end of the credentials line - "path=/; HttpOnly" - is a separate header or part of the credentials. and i'm not currently in a position to try it.

eduzal

i'm still not getting it right.

i've added this line credentials = post.getHeader("Set-Cookie");

i'm receiving apiCredentials=F3F0B82C63A62E5AECEF949B44E72F73A528B311C85CA64D702F7D1DAAE3750568862C9B5636D5800BDDA6B21C3134321ED95643309A9A81976A3C75568C425C43F01D3AB9CF8E3FE096B0757A457BEEEEA43925452D3D1DE01C910AE735F46E683A89E799FDA50A38751A1502C6109B4E7FFD2468C6347E18CECB5CB06269E8F99C1816A6B3B576A6A5CEA75717C94F195B3470E152E434F7977DD1A1188E07C1C66B97472A8F493C5371F727E10AA96090AB76838834BA2BA25DF5670199A793CABC21; path=/; HttpOnly

when i call get.addHeader("Set-Cookie",credentials); i still receive authorizationdenied

... people are still using cookies....

koogs

works for me (when i updated the http request library to one that didn't NPE when you added headers)

// eduzal / koogy
import http.requests.*;

String baseURL="http://"+"api.olhovivo.sptrans.com.br/v0";
String token="2bc28344ead24b3b5fe273ec7389d2e4d14abd2bad45abe6d8d774026193f894";
String route="7272";
String cookie;

void setup() {
  size(1000, 800);

  //API Olho Vivo
  authentication();
  getLinha(route);
}

void authentication() {
  println("POST ==================");
  String reqURL = "/Login/Autenticar?token=";
  PostRequest post = new PostRequest(baseURL + reqURL + token);
  post.send();
  println("Reponse Content: " + post.getContent());
  println("Reponse Content-Length Header: " + post.getHeader("Content-Length"));
  cookie = post.getHeader("set-cookie");
  println("Cookie: " + cookie);
}

void getLinha(String busLIne) {
  println("GET ===================");
  String reqURL = "/Posicao?codigoLinha=";
  GetRequest get = new GetRequest(baseURL + reqURL + route); 
  get.addHeader("Cookie", cookie);
  get.send();
  println("Reponse Content: " + get.getContent());
  println("Reponse Content-Length Header: " + get.getHeader("Content-Length"));
}

which gives me

Reponse Content: {"hr":"16:52","vs":[]}

no idea what this means though.

eduzal

@koogs awesome! thank you!

koogs

get.addHeader("Set-Cookie",credentials);

ha, you just got the name of the header wrong - it's just Cookie in the second request, see the wikipedia link.

PauloBenson

i'm having a issues when i tried to request a sequence of information from the API. I don't know if the problem is with my sketch or with the API, but it works for some time. But after a while it give me a error:

java.lang.RuntimeException: A JSONObject text must begin with '{' at processing.data.JSONObject.(JSONObject.java:242) at processing.data.JSONObject.parse(JSONObject.java:401) at mapa_onibus_05.runPosicao(mapa_onibus_05.java:121) at mapa_onibus_05.draw(mapa_onibus_05.java:88) at processing.core.PApplet.handleDraw(PApplet.java:2439) at processing.opengl.PSurfaceJOGL$DrawListener.display(PSurfaceJOGL.java:849) at jogamp.opengl.GLDrawableHelper.displayImpl(GLDrawableHelper.java:692) at jogamp.opengl.GLDrawableHelper.display(GLDrawableHelper.java:674) at jogamp.opengl.GLAutoDrawableBase$2.run(GLAutoDrawableBase.java:443) at jogamp.opengl.GLDrawableHelper.invokeGLImpl(GLDrawableHelper.java:1293) at jogamp.opengl.GLDrawableHelper.invokeGL(GLDrawableHelper.java:1147) at com.jogamp.newt.opengl.GLWindow.display(GLWindow.java:759) at com.jogamp.opengl.util.AWTAnimatorImpl.display(AWTAnimatorImpl.java:81) at com.jogamp.opengl.util.AnimatorBase.display(AnimatorBase.java:452) at com.jogamp.opengl.util.FPSAnimator$MainTask.run(FPSAnimator.java:178) at java.util.TimerThread.mainLoop(Timer.java:555) at java.util.TimerThread.run(Timer.java:505)

What i'm trying to do is access the API and request to the buses location from time to time, like 15sec. But after a while it gives me that error. It's possible to enlarge the time of response, so the Processing sketch don't rush, and throw me that error?

here is my code:

import http.requests.*;

String baseURL="http://"+"api.olhovivo.sptrans.com.br/v2.1";
String token="4af5e3112da870ac5708c48b7a237b30206806f296e1d302e4cb611660e2e03f";
String cookie;
int codigoLinhaCount = 0;
// location quadrado;

JSONArray values;
JSONArray posicaoOnibus;
JSONObject busao;
int countBus = 0;
int frameCount=0;
int numbPrint=0;

float minY= -23.340850;
float maxY= -24.024043;
float maxX= -46.349509;
float minX = -46.864493;

float diffY = 683.193;
float diffX = 514.984;

int telaX = int(diffX*100);
int telaY = int(diffY*100);

float latMax = -23.343223;
float latMin = -24.01542;
float longMax = -46.326279;
float longMin = -46.869986;

float mapX1,mapX2;
float mapY1,mapY2;



void setup() {
  size(1545, 2050);
  frameRate(1);
  background(0,0,0);
  smooth();

  mapX1 = 0;
  mapX2 = width;
  mapY1 = 0;
  mapY2 = height;



}


void draw(){

  frameCount++;

  if(frameCount % 15 == 0){
    numbPrint++;
    runPosicao();
    String frame = nfp(numbPrint, 6);
    String hora = timer();
    text(hora,5,10);
    saveFrame("data/"+"onibus"+frame+".png");
    println(numbPrint);
  }

}

String timer() {
  int d = day(); 
  int m = minute();
  int h = hour();

  String dia = String.valueOf(d);
  String minuto = String.valueOf(m);
  String hora = String.valueOf(h);

  String timer = "d"+dia+":h"+hora+":m"+minuto;
  return timer;

}




void runPosicao() { 
  fill(0,0,0);
  rect(0,0,width,height);
  authentication();

  JSONObject sistema = JSONObject.parse(posicaoOnibus()); // Destino que terá a linha passada a branco
  values = sistema.getJSONArray("l");
  saveJSONArray(values, "data/new.json");
  //passar por todas as linhas
  for (int i = 0 ; i < values.size(); i++){
   JSONObject linha = values.getJSONObject(i);
   //int codigoLinha = linha.getInt("cl");
   //String nomeLinha = linha.getString("c");
   JSONArray onibusLinha = linha.getJSONArray("vs");


   //passar por cada linha especifica
   for(int l = 0; l < onibusLinha.size(); l++){
     JSONObject onibus = onibusLinha.getJSONObject(l);
     float py = onibus.getFloat("py");
     float px = onibus.getFloat("px");

     float xx = TX(px);
     float yy = TY(py); 
     fill(#F7E511);
     noStroke();
     ellipseMode(CENTER);
     ellipse(xx,yy,1.5,1.5);


    }    
   } 
   println("true");
  } 




void authentication() {
  println("POST ==================");
  String reqURL = "/Login/Autenticar?token=";
  PostRequest post = new PostRequest(baseURL + reqURL + token);
  post.send();
  println("Reponse Content: " + post.getContent());
  println("Reponse Content-Length Header: " + post.getHeader("Content-Length"));
  cookie = post.getHeader("set-cookie");
}

String getLinhas(String busLIne) {
  println("GET ===================");
  String reqURL = "/Linha/Buscar?termosBusca=";
  GetRequest get = new GetRequest(baseURL + reqURL + busLIne); 
  get.addHeader("Cookie", cookie);
  get.send();
  return get.getContent();  
}

String posicaoOnibus() {
  println("GET ===================");
  String reqURL = "/Posicao";
  GetRequest get = new GetRequest(baseURL + reqURL); 
  get.addHeader("Cookie", cookie);
  get.send();

  println("Reponse Content: " + get.getContent());
  return get.getContent();  
}



float TX(float x) {
  return map(x,longMin,longMax,mapX1,mapX2);
  }
float TY(float y) {
    return map(y,latMin,latMax,mapY2,mapY1);
  }

koogs

loading all that data every frame is madness, even with framerate set to 1. far too much work. especially as the data isn't being updated that often anyway.

i can also see sptrans getting annoyed by all you people DOSing their systems like this and they may throttle requests, returning errors if you're hitting the url too frequently. (and you are)

koogs

you can probably add debug to your GET requests that'll show you this.

eduzal

@PauloBenson You need to do a conditional to check if the information you get is valid. sometimes the api returns an empty request and makes you crash.

PauloBenson

@koogs, i'm making a call every 15 or 20 sec. But don't depend of that. I change for 1 call every minute, it crash eventually. i Did the debug, and they send me a null

koogs

Check the status as well, not just the content.

PauloBenson

@eduzal, i did the condition to skip if i received a null. but it's not the effect that a wanna.... But Thanks guys!

PauloBenson

The issue is probable because they are getting annoyed, and take me down.

eduzal

@PauloBenson also, you don't need to add frameCount. processing already does that.

eduzal

@PauloBenson sometimes they send an empty container. don't worry, they do that error, it's not personal. if you use null as the conditional value it wont work cause they send you an empty JSON, but with a header, you have to count the header.

you don't need to get the data every 20 seconds. there's no big change on that scale.

also notice that the authentication lasts only 30 mins

PauloBenson

Yep i did my condition with header count of 4. And i was asking for authentication every time a ask for informations.... Maybe thats a problem?

eduzal

@PauloBenson

get.addHeader("Content-Type", "application/json");
get.addHeader("Cookie", cookie);
get.send();
int contentSize=int(get.getHeader("Content-Length"));
if (contentSize > 4) {
 JSONObject JSONPosition = parseJSONObject(get.getContent());
} else {
getPosition(route);
}

PauloBenson

@eduzal The think is that i was planning to do a map of all the buses moving around with that gap on time. good point, about the authentication! thanks!

PauloBenson

I change the Post to happen only in 20 min.....and seams to work better!!

PauloBenson

@eduzal i make the contentSize condition outside of the get information, so if a got a null response, i would try at the next frame.... in this case the boolean "foi" it's used for printing or not.

something like this:

import http.requests.*;

String baseURL="http://"+"api.olhovivo.sptrans.com.br/v2.1";
String token="4af5e3112da870ac5708c48b7a237b30206806f296e1d302e4cb611660e2e03f";
String cookie;
int codigoLinhaCount = 0;
// location quadrado;

JSONArray values;
JSONArray posicaoOnibus;
JSONObject busao;
int countBus = 0;
int frameCount=0;
int numbPrint=0;

float minY= -23.340850;
float maxY= -24.024043;
float maxX= -46.349509;
float minX = -46.864493;

float diffY = 683.193;
float diffX = 514.984;

int telaX = int(diffX*100);
int telaY = int(diffY*100);

float latMax = -23.343223;
float latMin = -24.01542;
float longMax = -46.326279;
float longMin = -46.869986;

float mapX1,mapX2;
float mapY1,mapY2;

String parseSite;
JSONObject sistema;
boolean foi;

int checkCon = 10;
int checkConIdeal = 10; 



void setup() {
  size(1545, 2050);
  frameRate(1);
  background(0,0,0);
  smooth();
  authentication();

  mapX1 = 0;
  mapX2 = width;
  mapY1 = 0;
  mapY2 = height;



}


void draw(){


  frameCount++;

  if(frameCount % (60*25) == 0){
  authentication();
  }

  if(frameCount % checkCon == 0){

    runPosicao();

    if (foi == true){
    numbPrint++;  
    String frame = nfp(numbPrint, 6);
    String hora = timer();
    text(hora,5,10);
    saveFrame("data/"+"onibus"+frame+".png");
    println(numbPrint);
    println(hora);
    }
  }

}

String timer() {
  int d = day(); 
  int m = minute();
  int h = hour();

  String dia = String.valueOf(d);
  String minuto = String.valueOf(m);
  String hora = String.valueOf(h);

  String timer = "d"+dia+":h"+hora+":m"+minuto;
  return timer;

}




boolean runPosicao() { 
  fill(0,0,0);
  rect(0,0,width,height);

  parseSite = posicaoOnibus();
  if (parseSite.length() == 4){
    println("Reponse Content: " + parseSite);
    println("error");
    foi = false;
    checkCon = 1;

  } else {
  println("ok");
  display(parseSite);
  foi = true;
  checkCon = checkConIdeal;
    }
    return foi;
  } 

void display(String parse){

  sistema = JSONObject.parse(parse); // Destino que terá a linha passada a branco

  values = sistema.getJSONArray("l");
  saveJSONArray(values, "data/new.json");
  //passar por todas as linhas
  for (int i = 0 ; i < values.size(); i++){
   JSONObject linha = values.getJSONObject(i);
   //int codigoLinha = linha.getInt("cl");
   //String nomeLinha = linha.getString("c");
   JSONArray onibusLinha = linha.getJSONArray("vs");


   //passar por cada linha especifica
   for(int l = 0; l < onibusLinha.size(); l++){
     JSONObject onibus = onibusLinha.getJSONObject(l);
     float py = onibus.getFloat("py");
     float px = onibus.getFloat("px");
     JSONObject busao = new JSONObject();
     busao.setFloat("py",py);
     busao.setFloat("px",px);
     float xx = TX(px);
     float yy = TY(py); 
     fill(#F7E511);
     noStroke();
     ellipseMode(CENTER);
     ellipse(xx,yy,1.5,1.5);


    }    
   }


}


void authentication() {
  println("POST ==================");
  String reqURL = "/Login/Autenticar?token=";
  PostRequest post = new PostRequest(baseURL + reqURL + token);
  post.send();
  println("Reponse Content: " + post.getContent());
  println("Reponse Content-Length Header: " + post.getHeader("Content-Length"));
  cookie = post.getHeader("set-cookie");
}

String getLinhas(String busLIne) {
  println("GET ===================");
  String reqURL = "/Linha/Buscar?termosBusca=";
  GetRequest get = new GetRequest(baseURL + reqURL + busLIne); 
  get.addHeader("Cookie", cookie);
  get.send();
  return get.getContent();  
}

String posicaoOnibus() {
  println("GET ===================");
  String reqURL = "/Posicao";
  GetRequest get = new GetRequest(baseURL + reqURL); 
  get.addHeader("Cookie", cookie);
  get.send();

  //println("Reponse Content: " + get.getContent());
  return get.getContent();  
}



float TX(float x) {
  return map(x,longMin,longMax,mapX1,mapX2);
  }
float TY(float y) {
    return map(y,latMin,latMax,mapY2,mapY1);
  }

PauloBenson

I hope that the authentication after 25min works.....

koogs

i'm making a call every 15 or 20 sec.

You changed your post after my comment, originally you had it every frame, every second.

The new code is much more sensible, apart from the frameCount thing. Framecount already exists and is incrementing automatically so use that instead.

Also your code has an initial delay in it because of the checks in lines 66 and 70. Change the 0 to 1 to get rid of this.

rallyestyle

Have been reading up on this post as i have an identical error and am looking to understand how it exists and whats the best way to solve it. Id like to perhaps use try catch to handle the ' A JSONObject text must begin with '{' ' error, if the JSON becomes invalid all of a sudden i could restart the draw function. Does this sound like a sensible way to do things do you think?

rallyestyle

As the sketch is looped try catch seems to catch the error plus some NPE's and move on quite nicely, works perfectly for what i need to do.

Id love to spend some time figuring out the NPE's but for now the sketch starts over and runs fine again so im just happy thats the case. Onwards and upwards!

eduzal

@rallyestyle some tips after using the sketch for a while.

1. the cookie lasts 30 minutes

2. the system updates the positions each minute, you don't need to update it in less time

3. sometimes they send you an empty answer and it will make you crash, it's an all or nothing situation. you have to do something to deal with it. try/catch might be good, some examples with the twitter api have a try/catch and it might work just fine. me, i used recursion because i'm not that good with try/catch creation.

koogs

rallyestyle, it's not particularly helpful to revive 8 month old threads, especially if you've a thread for your problem already. better continue it there.

https://forum.processing.org/two/discussion/25686/http-header-with-api-key-to-access-json

rallyestyle

Noted, I thought this thread would be more applicable to the particular part of my sketch in question but for now i will keep it all under that one thread.

Thanks eduzal for the tips, i have try catch working nicely now but as always with a big project when one problem is solved another 10 appear! doh! :)