log-in to a web-page via processing // Post-Request for login

General Discussion Other

der_muk

log-in to a web-page via processing // Post-Request for login

in General Discussion • Other • 21 days ago

Hello fellow coders,

I want to parse an html to create some custom statistics. Parsing is not the problem - accessing the html directly via loadStrings() is.

Problem: I need to login before I can view the html I want to parse. I figured out, that I may need to make some sort of a post-request in order to send my login-data to the service. How i this feasible with processing?

Thank you for helping out!

Replies(5)

akiersky

Re: log-in to a web-page via processing // Post-Request for login

21 days ago

hey der_muk,

I created this class to handle just such issues:

class HTTPSconnect {
HttpHost targetHost;
DefaultHttpClient httpclient;
AuthScope authScope;
UsernamePasswordCredentials upc;
AuthCache authCache;
BasicScheme basicAuth;
BasicHttpContext localcontext;
HttpGet httpget;
HTTPSconnect () {
targetHost = new HttpHost("<<Put your base url here eg: google.com>>", 80, "http");
httpclient = new DefaultHttpClient();
authScope = new AuthScope(targetHost.getHostName(), targetHost.getPort(), AuthScope.ANY_REALM, "basic");
upc = new UsernamePasswordCredentials("<<put your username here>>", "<<put your password here>>");
httpclient.getCredentialsProvider().setCredentials(authScope, upc);
authCache = new BasicAuthCache();
basicAuth = new BasicScheme();
authCache.put(targetHost, basicAuth);
localcontext = new BasicHttpContext();
localcontext.setAttribute(ClientContext.AUTH_CACHE, authCache);
}
JSONObject getStoriesJson (String url) {
JSONObject jo = null;
try {
httpget = new HttpGet(url);
org.apache.http.HttpResponse response = httpclient.execute(targetHost, httpget, localcontext);
jo = new JSONObject(EntityUtils.toString(response.getEntity()));
}
catch(Exception e) {
println("this is some error : "+e.getStackTrace());
}
return jo;
}
PImage getImage(String url) {
PImage img = null;
try {
httpget = new HttpGet(url);
org.apache.http.HttpResponse response = httpclient.execute(targetHost, httpget, localcontext);
byte[] bytes = EntityUtils.toByteArray(response.getEntity());
ByteArrayInputStream bis = new ByteArrayInputStream(bytes);
BufferedImage jimg = ImageIO.read(bis);
img=new PImage(jimg.getWidth(), jimg.getHeight(), PConstants.ARGB);
jimg.getRGB(0, 0, img.width, img.height, img.pixels, 0, img.width);
img.updatePixels();
}
catch(Exception e) {
println("this is some error : "+e.getStackTrace());
}
return img;
}
}

So you'll want to put in your base url, without the http:// or https:// and add your username and password. Then create an HTTTPSconnect object. Then you can call getStoriesJson (you could change this method name, I was getting json files) with the rest of the url to the file you are trying to load. For example, just use everything after the .com/.edu/.whatever. the method will return a JSONObject, but again you may want to change that to just give a String.

As a bonus, it also works with images behind the password wall. these get pretty crazy because you end up getting a byte array back rather than an actual image. so this will parse the byte array and return the PImage.

you'll have to add some imports to the file as well, I have all these, but some may not be used (I haven't cleaned them up):

import org.apache.http.*;
import org.apache.http.client.*;
import org.apache.http.auth.*;
import org.apache.http.protocol.*;
import org.apache.http.util.*;
import org.apache.http.impl.client.*;
import org.apache.http.impl.auth.*;
import org.apache.http.client.protocol.*;
import org.apache.http.client.methods.*;

hope this helps!

akiersky

Re: log-in to a web-page via processing // Post-Request for login

21 days ago

Also, I'm not sure what your html parsing process is, but I found jsoup to be crazy useful in getting data out of html.

-ak

der_muk

Re: Re: log-in to a web-page via processing // Post-Request for login

19 days ago

*DANG*

That was a fast reply! Many thanks for helping, akiersky!
I tried your code, but the code gives me pain - by prompting this message (from your try / catch - thingy):

this is some error : [Ljava.lang.StackTraceElement;@1ee2433b

I am also not sure, if i used the correct .jar-files from the apache library. It took me some fiddling around until i managed to get the library from HERE. I put the .jar-files into the "code"-folder. Processing is now able to load the libraries. I read about some replacements at apache (http-client vs. http-components). May this cause the error?

I am using the lib/.jar-files from: HttpClient 4.3 (GA)
And Processing 2.0.3

Thanks a bunch!

dimkir

Re: log-in to a web-page via processing // Post-Request for login

20 days ago

jsoup is amazing. Remember to set up your user-agent (othewise you may get wrong version of the site) and increase timeout, as default timeout seems to be too short when working with servers under load.

akiersky

Re: log-in to a web-page via processing // Post-Request for login

17 days ago

the error may be from trying to convert the string to a json file. Is the content you are loading json? if not, it may be better to just keep it a string.

As for the libraries, they are for java, so they do take some re-arranging to get them to work in processing. Generally though, if it compiles, then it should work. the try catch error would be from actual loading, so it sounds like you have the lib's set up correctly.

try removing this line
jo = new JSONObject(EntityUtils.toString(response.getEntity()));

and see if you still get the catch error. It will throw an error about not returning a JSONObject then. to fix that, just change all the JSONObject stuff to String and that should take care of it.

hope this helps!

Top Reply