[HTTP-Requests] cookies - SPTrans Olho Vivo

edited May 2017 in Library Questions

Hello

I'm building a visualization tool to get the immediate position of the buses available. There's this API that delivers these posions using a GET request. The return is a JSON.

I'm authenticating without problems, but when i request anything it blocks me any access. it's stinky documentation doesn't says that they need cookies to return the request.

The final product will be a mapped projection with this map, transported via syphon to another Resolume Arena. i need help in getting this cookie within this and processing it to get my requests.

import http.requests.*;

String baseURL="http://"+"api.olhovivo.sptrans.com.br/v0";
String token="2bc28344ead24b3b5fe273ec7389d2e4d14abd2bad45abe6d8d774026193f894";
String route="7272";

void setup() {
  size(1200, 1800);

  //API Olho Vivo
  authentication();
  getLinha(route);
}

void authentication() {
  String reqURL = "/Login/Autenticar?token=";
  PostRequest post = new PostRequest(baseURL+reqURL+token);
  post.send();
  println("Reponse Content: " + post.getContent());
  println("Reponse Content-Length Header: " + post.getHeader("Content-Length"));
}

void getLinha(String busLIne) {
  String reqURL = "/Posicao?codigoLinha=";
  GetRequest get = new GetRequest(baseURL+reqURL+busLine); 
  get.send();
  println("Reponse Content: " + get.getContent());
  println("Reponse Content-Length Header: " + get.getHeader("Content-Length"));
}

Responses:
Reponse Content: true
Reponse Content-Length Header: 4
May 15, 2017 12:05:34 AM org.apache.http.impl.client.DefaultRequestDirector handleResponse
WARNING: Authentication error: Unable to respond to any of these challenges: {}
Reponse Content: {"Message":"Authorization has been denied for this request."}
Reponse Content-Length Header: 61

API documentation: http://www.sptrans.com.br/desenvolvedores/APIOlhoVivo/Documentacao.aspx?1

Answers

  • edited May 2017
    wget -S --method POST "http:// api.olhovivo.sptrans.com.br/v0/Login/Autenticar?token=2bc28344ead24b3b5fe273ec7389d2e4d14abd2bad45abe6d8d774026193f894"
    
    --2017-05-15 07:32:51--  http:// api.olhovivo.sptrans.com.br/v0/Login/Autenticar?token=2bc28344ead24b3b5fe273ec7389d2e4d14abd2bad45abe6d8d774026193f894
    Resolving api.olhovivo.sptrans.com.br (api.olhovivo.sptrans.com.br)... 200.189.189.84
    Connecting to api.olhovivo.sptrans.com.br (api.olhovivo.sptrans.com.br)|200.189.189.84|:80... connected.
    HTTP request sent, awaiting response... 
      HTTP/1.1 200 OK
      Cache-Control: no-cache
      Pragma: no-cache
      Content-Type: application/json; charset=utf-8
      Expires: -1
      Server: Microsoft-IIS/7.5
      X-AspNet-Version: 4.0.30319
      Set-Cookie: apiCredentials=2B205020F9B70EABD2B449FCF6AB97D546C6C6C69AD3E2AA0F1F9A388BE17DB0CDC858B1610994758BBCE8FA339EE04047DFFC162DE32A8EED31CAEF138FBD808D6CA71A90C544167F1601BF3081E40227F078AF9C171BF3A48D61E4B39285DCAA6EC4716E751B871F01723FF73144A2E4AC69874D713CF0D998425DAA6D4A5337E3FC0437146AF06A6CC976534D56BB5F65FD82B98EE7E15BBCEFCB62AF67F0C21EEC0805BA68C5194A18966A148F31A9C586717772F1852F2FDF7662BB852CED365E46; path=/; HttpOnly
      X-Powered-By: ASP.NET
      Date: Mon, 15 May 2017 06:32:50 GMT
      Content-Length: 4
    Length: 4 [application/json]
    Saving to: ‘Autenticar?token=2bc28344ead24b3b5fe273ec7389d2e4d14abd2bad45abe6d8d774026193f894.1’
    
    2017-05-15 07:32:53 (182 KB/s) - ‘Autenticar?token=2bc28344ead24b3b5fe273ec7389d2e4d14abd2bad45abe6d8d774026193f894.1’ saved [4/4]
    

    there's a cookie in the response, try capturing and sending that in later requests.

  • try capturing and sending that in later requests

    @koogs How can one do that? I did think about checking if the POST had the cookie attached (based on this post: http://stackoverflow.com/questions/17769011/how-does-cookie-based-authentication-work) but I was not able to get it with Processing code.

    Kf

  • Good question.

    this example suggests how to access them in the response

    https://github.com/runemadsen/HTTP-Requests-for-Processing/blob/master/examples/get/get.pde

    and they are easy enough to add, being just headers

  • edited May 2017

    @koogs i'm realy clueless about this. i believe i would need a walkthrough this time. what should i store?

    this?

    Set-Cookie: apiCredentials=2B205020F9B70EABD2B449FCF6AB97D546C6C6C69AD3E2AA0F1F9A388BE17DB0CDC858B1610994758BBCE8FA339EE04047DFFC162DE32A8EED31CAEF138FBD808D6CA71A90C544167F1601BF3081E40227F078AF9C171BF3A48D61E4B39285DCAA6EC4716E751B871F01723FF73144A2E4AC69874D713CF0D998425DAA6D4A5337E3FC0437146AF06A6CC976534D56BB5F65FD82B98EE7E15BBCEFCB62AF67F0C21EEC0805BA68C5194A18966A148F31A9C586717772F1852F2FDF7662BB852CED365E46; path=/; HttpOnly

    or this?

    Saving to: ‘Autenticar?token=2bc28344ead24b3b5fe273ec7389d2e4d14abd2bad45abe6d8d774026193f894.1’

    capturing you mean storing it in a variable like?

    String credentials = post.getHeader("apiCredentials")

    and then sending it back

    get.addHeader("apiCredentials", "credentials");

    tks

  • this line

    Saving to: ‘Autenticar?token=2bc28344ead24b3b5fe273ec7389d2e4d14abd2bad45abe6d8d774026193f894.1’
    

    is just output from wget telling me where it was saving the output to. ignore it.

    i think you want the contents of your varaiable rather than the literal "credentials"

    get.addHeader("apiCredentials", credentials);
    

    i'm not sure whether the bit on the end of the credentials line - "path=/; HttpOnly" - is a separate header or part of the credentials. and i'm not currently in a position to try it.

  • i'm still not getting it right.

    i've added this line credentials = post.getHeader("Set-Cookie");

    i'm receiving apiCredentials=F3F0B82C63A62E5AECEF949B44E72F73A528B311C85CA64D702F7D1DAAE3750568862C9B5636D5800BDDA6B21C3134321ED95643309A9A81976A3C75568C425C43F01D3AB9CF8E3FE096B0757A457BEEEEA43925452D3D1DE01C910AE735F46E683A89E799FDA50A38751A1502C6109B4E7FFD2468C6347E18CECB5CB06269E8F99C1816A6B3B576A6A5CEA75717C94F195B3470E152E434F7977DD1A1188E07C1C66B97472A8F493C5371F727E10AA96090AB76838834BA2BA25DF5670199A793CABC21; path=/; HttpOnly

    when i call get.addHeader("Set-Cookie",credentials); i still receive authorizationdenied

    ... people are still using cookies....

  • edited May 2017 Answer ✓

    works for me (when i updated the http request library to one that didn't NPE when you added headers)

    // eduzal / koogy
    import http.requests.*;
    
    String baseURL="http://"+"api.olhovivo.sptrans.com.br/v0";
    String token="2bc28344ead24b3b5fe273ec7389d2e4d14abd2bad45abe6d8d774026193f894";
    String route="7272";
    String cookie;
    
    void setup() {
      size(1000, 800);
    
      //API Olho Vivo
      authentication();
      getLinha(route);
    }
    
    void authentication() {
      println("POST ==================");
      String reqURL = "/Login/Autenticar?token=";
      PostRequest post = new PostRequest(baseURL + reqURL + token);
      post.send();
      println("Reponse Content: " + post.getContent());
      println("Reponse Content-Length Header: " + post.getHeader("Content-Length"));
      cookie = post.getHeader("set-cookie");
      println("Cookie: " + cookie);
    }
    
    void getLinha(String busLIne) {
      println("GET ===================");
      String reqURL = "/Posicao?codigoLinha=";
      GetRequest get = new GetRequest(baseURL + reqURL + route); 
      get.addHeader("Cookie", cookie);
      get.send();
      println("Reponse Content: " + get.getContent());
      println("Reponse Content-Length Header: " + get.getHeader("Content-Length"));
    }
    

    which gives me

    Reponse Content: {"hr":"16:52","vs":[]}
    

    no idea what this means though.

  • @koogs awesome! thank you!

  • Answer ✓

    get.addHeader("Set-Cookie",credentials);

    ha, you just got the name of the header wrong - it's just Cookie in the second request, see the wikipedia link.

  • edited May 2017

    i'm having a issues when i tried to request a sequence of information from the API. I don't know if the problem is with my sketch or with the API, but it works for some time. But after a while it give me a error:

    java.lang.RuntimeException: A JSONObject text must begin with '{' at processing.data.JSONObject.(JSONObject.java:242) at processing.data.JSONObject.parse(JSONObject.java:401) at mapa_onibus_05.runPosicao(mapa_onibus_05.java:121) at mapa_onibus_05.draw(mapa_onibus_05.java:88) at processing.core.PApplet.handleDraw(PApplet.java:2439) at processing.opengl.PSurfaceJOGL$DrawListener.display(PSurfaceJOGL.java:849) at jogamp.opengl.GLDrawableHelper.displayImpl(GLDrawableHelper.java:692) at jogamp.opengl.GLDrawableHelper.display(GLDrawableHelper.java:674) at jogamp.opengl.GLAutoDrawableBase$2.run(GLAutoDrawableBase.java:443) at jogamp.opengl.GLDrawableHelper.invokeGLImpl(GLDrawableHelper.java:1293) at jogamp.opengl.GLDrawableHelper.invokeGL(GLDrawableHelper.java:1147) at com.jogamp.newt.opengl.GLWindow.display(GLWindow.java:759) at com.jogamp.opengl.util.AWTAnimatorImpl.display(AWTAnimatorImpl.java:81) at com.jogamp.opengl.util.AnimatorBase.display(AnimatorBase.java:452) at com.jogamp.opengl.util.FPSAnimator$MainTask.run(FPSAnimator.java:178) at java.util.TimerThread.mainLoop(Timer.java:555) at java.util.TimerThread.run(Timer.java:505)

    What i'm trying to do is access the API and request to the buses location from time to time, like 15sec. But after a while it gives me that error. It's possible to enlarge the time of response, so the Processing sketch don't rush, and throw me that error?

    here is my code:

    import http.requests.*;
    
    String baseURL="http://"+"api.olhovivo.sptrans.com.br/v2.1";
    String token="4af5e3112da870ac5708c48b7a237b30206806f296e1d302e4cb611660e2e03f";
    String cookie;
    int codigoLinhaCount = 0;
    // location quadrado;
    
    JSONArray values;
    JSONArray posicaoOnibus;
    JSONObject busao;
    int countBus = 0;
    int frameCount=0;
    int numbPrint=0;
    
    float minY= -23.340850;
    float maxY= -24.024043;
    float maxX= -46.349509;
    float minX = -46.864493;
    
    float diffY = 683.193;
    float diffX = 514.984;
    
    int telaX = int(diffX*100);
    int telaY = int(diffY*100);
    
    float latMax = -23.343223;
    float latMin = -24.01542;
    float longMax = -46.326279;
    float longMin = -46.869986;
    
    float mapX1,mapX2;
    float mapY1,mapY2;
    
    
    
    void setup() {
      size(1545, 2050);
      frameRate(1);
      background(0,0,0);
      smooth();
    
      mapX1 = 0;
      mapX2 = width;
      mapY1 = 0;
      mapY2 = height;
    
    
    
    }
    
    
    void draw(){
    
      frameCount++;
    
      if(frameCount % 15 == 0){
        numbPrint++;
        runPosicao();
        String frame = nfp(numbPrint, 6);
        String hora = timer();
        text(hora,5,10);
        saveFrame("data/"+"onibus"+frame+".png");
        println(numbPrint);
      }
    
    }
    
    String timer() {
      int d = day(); 
      int m = minute();
      int h = hour();
    
      String dia = String.valueOf(d);
      String minuto = String.valueOf(m);
      String hora = String.valueOf(h);
    
      String timer = "d"+dia+":h"+hora+":m"+minuto;
      return timer;
    
    }
    
    
    
    
    void runPosicao() { 
      fill(0,0,0);
      rect(0,0,width,height);
      authentication();
    
      JSONObject sistema = JSONObject.parse(posicaoOnibus()); // Destino que terá a linha passada a branco
      values = sistema.getJSONArray("l");
      saveJSONArray(values, "data/new.json");
      //passar por todas as linhas
      for (int i = 0 ; i < values.size(); i++){
       JSONObject linha = values.getJSONObject(i);
       //int codigoLinha = linha.getInt("cl");
       //String nomeLinha = linha.getString("c");
       JSONArray onibusLinha = linha.getJSONArray("vs");
    
    
       //passar por cada linha especifica
       for(int l = 0; l < onibusLinha.size(); l++){
         JSONObject onibus = onibusLinha.getJSONObject(l);
         float py = onibus.getFloat("py");
         float px = onibus.getFloat("px");
    
         float xx = TX(px);
         float yy = TY(py); 
         fill(#F7E511);
         noStroke();
         ellipseMode(CENTER);
         ellipse(xx,yy,1.5,1.5);
    
    
        }    
       } 
       println("true");
      } 
    
    
    
    
    void authentication() {
      println("POST ==================");
      String reqURL = "/Login/Autenticar?token=";
      PostRequest post = new PostRequest(baseURL + reqURL + token);
      post.send();
      println("Reponse Content: " + post.getContent());
      println("Reponse Content-Length Header: " + post.getHeader("Content-Length"));
      cookie = post.getHeader("set-cookie");
    }
    
    String getLinhas(String busLIne) {
      println("GET ===================");
      String reqURL = "/Linha/Buscar?termosBusca=";
      GetRequest get = new GetRequest(baseURL + reqURL + busLIne); 
      get.addHeader("Cookie", cookie);
      get.send();
      return get.getContent();  
    }
    
    String posicaoOnibus() {
      println("GET ===================");
      String reqURL = "/Posicao";
      GetRequest get = new GetRequest(baseURL + reqURL); 
      get.addHeader("Cookie", cookie);
      get.send();
    
      println("Reponse Content: " + get.getContent());
      return get.getContent();  
    }
    
    
    
    float TX(float x) {
      return map(x,longMin,longMax,mapX1,mapX2);
      }
    float TY(float y) {
        return map(y,latMin,latMax,mapY2,mapY1);
      }
    
  • Answer ✓

    loading all that data every frame is madness, even with framerate set to 1. far too much work. especially as the data isn't being updated that often anyway.

    i can also see sptrans getting annoyed by all you people DOSing their systems like this and they may throttle requests, returning errors if you're hitting the url too frequently. (and you are)

  • Answer ✓

    you can probably add debug to your GET requests that'll show you this.

  • @PauloBenson You need to do a conditional to check if the information you get is valid. sometimes the api returns an empty request and makes you crash.

  • @koogs, i'm making a call every 15 or 20 sec. But don't depend of that. I change for 1 call every minute, it crash eventually. i Did the debug, and they send me a null

  • Check the status as well, not just the content.

  • @eduzal, i did the condition to skip if i received a null. but it's not the effect that a wanna.... But Thanks guys!

  • The issue is probable because they are getting annoyed, and take me down.

  • @PauloBenson also, you don't need to add frameCount. processing already does that.

  • @PauloBenson sometimes they send an empty container. don't worry, they do that error, it's not personal. if you use null as the conditional value it wont work cause they send you an empty JSON, but with a header, you have to count the header.

    you don't need to get the data every 20 seconds. there's no big change on that scale.

    also notice that the authentication lasts only 30 mins

  • Yep i did my condition with header count of 4. And i was asking for authentication every time a ask for informations.... Maybe thats a problem?

  • edited May 2017

    @PauloBenson

    get.addHeader("Content-Type", "application/json");
    get.addHeader("Cookie", cookie);
    get.send();
    int contentSize=int(get.getHeader("Content-Length"));
    if (contentSize > 4) {
     JSONObject JSONPosition = parseJSONObject(get.getContent());
    } else {
    getPosition(route);
    }
    
  • @eduzal The think is that i was planning to do a map of all the buses moving around with that gap on time. good point, about the authentication! thanks!

  • I change the Post to happen only in 20 min.....and seams to work better!!

  • edited May 2017

    @eduzal i make the contentSize condition outside of the get information, so if a got a null response, i would try at the next frame.... in this case the boolean "foi" it's used for printing or not.

    something like this:

    import http.requests.*;
    
    String baseURL="http://"+"api.olhovivo.sptrans.com.br/v2.1";
    String token="4af5e3112da870ac5708c48b7a237b30206806f296e1d302e4cb611660e2e03f";
    String cookie;
    int codigoLinhaCount = 0;
    // location quadrado;
    
    JSONArray values;
    JSONArray posicaoOnibus;
    JSONObject busao;
    int countBus = 0;
    int frameCount=0;
    int numbPrint=0;
    
    float minY= -23.340850;
    float maxY= -24.024043;
    float maxX= -46.349509;
    float minX = -46.864493;
    
    float diffY = 683.193;
    float diffX = 514.984;
    
    int telaX = int(diffX*100);
    int telaY = int(diffY*100);
    
    float latMax = -23.343223;
    float latMin = -24.01542;
    float longMax = -46.326279;
    float longMin = -46.869986;
    
    float mapX1,mapX2;
    float mapY1,mapY2;
    
    String parseSite;
    JSONObject sistema;
    boolean foi;
    
    int checkCon = 10;
    int checkConIdeal = 10; 
    
    
    
    void setup() {
      size(1545, 2050);
      frameRate(1);
      background(0,0,0);
      smooth();
      authentication();
    
      mapX1 = 0;
      mapX2 = width;
      mapY1 = 0;
      mapY2 = height;
    
    
    
    }
    
    
    void draw(){
    
    
      frameCount++;
    
      if(frameCount % (60*25) == 0){
      authentication();
      }
    
      if(frameCount % checkCon == 0){
    
        runPosicao();
    
        if (foi == true){
        numbPrint++;  
        String frame = nfp(numbPrint, 6);
        String hora = timer();
        text(hora,5,10);
        saveFrame("data/"+"onibus"+frame+".png");
        println(numbPrint);
        println(hora);
        }
      }
    
    }
    
    String timer() {
      int d = day(); 
      int m = minute();
      int h = hour();
    
      String dia = String.valueOf(d);
      String minuto = String.valueOf(m);
      String hora = String.valueOf(h);
    
      String timer = "d"+dia+":h"+hora+":m"+minuto;
      return timer;
    
    }
    
    
    
    
    boolean runPosicao() { 
      fill(0,0,0);
      rect(0,0,width,height);
    
      parseSite = posicaoOnibus();
      if (parseSite.length() == 4){
        println("Reponse Content: " + parseSite);
        println("error");
        foi = false;
        checkCon = 1;
    
      } else {
      println("ok");
      display(parseSite);
      foi = true;
      checkCon = checkConIdeal;
        }
        return foi;
      } 
    
    void display(String parse){
    
      sistema = JSONObject.parse(parse); // Destino que terá a linha passada a branco
    
      values = sistema.getJSONArray("l");
      saveJSONArray(values, "data/new.json");
      //passar por todas as linhas
      for (int i = 0 ; i < values.size(); i++){
       JSONObject linha = values.getJSONObject(i);
       //int codigoLinha = linha.getInt("cl");
       //String nomeLinha = linha.getString("c");
       JSONArray onibusLinha = linha.getJSONArray("vs");
    
    
       //passar por cada linha especifica
       for(int l = 0; l < onibusLinha.size(); l++){
         JSONObject onibus = onibusLinha.getJSONObject(l);
         float py = onibus.getFloat("py");
         float px = onibus.getFloat("px");
         JSONObject busao = new JSONObject();
         busao.setFloat("py",py);
         busao.setFloat("px",px);
         float xx = TX(px);
         float yy = TY(py); 
         fill(#F7E511);
         noStroke();
         ellipseMode(CENTER);
         ellipse(xx,yy,1.5,1.5);
    
    
        }    
       }
    
    
    }
    
    
    void authentication() {
      println("POST ==================");
      String reqURL = "/Login/Autenticar?token=";
      PostRequest post = new PostRequest(baseURL + reqURL + token);
      post.send();
      println("Reponse Content: " + post.getContent());
      println("Reponse Content-Length Header: " + post.getHeader("Content-Length"));
      cookie = post.getHeader("set-cookie");
    }
    
    String getLinhas(String busLIne) {
      println("GET ===================");
      String reqURL = "/Linha/Buscar?termosBusca=";
      GetRequest get = new GetRequest(baseURL + reqURL + busLIne); 
      get.addHeader("Cookie", cookie);
      get.send();
      return get.getContent();  
    }
    
    String posicaoOnibus() {
      println("GET ===================");
      String reqURL = "/Posicao";
      GetRequest get = new GetRequest(baseURL + reqURL); 
      get.addHeader("Cookie", cookie);
      get.send();
    
      //println("Reponse Content: " + get.getContent());
      return get.getContent();  
    }
    
    
    
    float TX(float x) {
      return map(x,longMin,longMax,mapX1,mapX2);
      }
    float TY(float y) {
        return map(y,latMin,latMax,mapY2,mapY1);
      }
    
  • I hope that the authentication after 25min works.....

  • edited May 2017

    i'm making a call every 15 or 20 sec.

    You changed your post after my comment, originally you had it every frame, every second.

    The new code is much more sensible, apart from the frameCount thing. Framecount already exists and is incrementing automatically so use that instead.

    Also your code has an initial delay in it because of the checks in lines 66 and 70. Change the 0 to 1 to get rid of this.

  • edited January 9

    Have been reading up on this post as i have an identical error and am looking to understand how it exists and whats the best way to solve it. Id like to perhaps use try catch to handle the ' A JSONObject text must begin with '{' ' error, if the JSON becomes invalid all of a sudden i could restart the draw function. Does this sound like a sensible way to do things do you think?

  • As the sketch is looped try catch seems to catch the error plus some NPE's and move on quite nicely, works perfectly for what i need to do.

    Id love to spend some time figuring out the NPE's but for now the sketch starts over and runs fine again so im just happy thats the case. Onwards and upwards!

  • edited January 9

    @rallyestyle some tips after using the sketch for a while.

    1. the cookie lasts 30 minutes

    2. the system updates the positions each minute, you don't need to update it in less time

    3. sometimes they send you an empty answer and it will make you crash, it's an all or nothing situation. you have to do something to deal with it. try/catch might be good, some examples with the twitter api have a try/catch and it might work just fine. me, i used recursion because i'm not that good with try/catch creation.

  • rallyestyle, it's not particularly helpful to revive 8 month old threads, especially if you've a thread for your problem already. better continue it there.

    https://forum.processing.org/two/discussion/25686/http-header-with-api-key-to-access-json

  • Noted, I thought this thread would be more applicable to the particular part of my sketch in question but for now i will keep it all under that one thread.

    Thanks eduzal for the tips, i have try catch working nicely now but as always with a big project when one problem is solved another 10 appear! doh! :)

Sign In or Register to comment.