How to access html data from a page that requires logging in?

Programming Questions

patakk

How to access html data from a page that requires logging in?

in Programming Questions • 1 year ago

Hello everybody! I apologise for my bad English in advance..
So here's the problem, I'm using proHTML library for fetching some html data from a certain page, but this page requires logging in. I can login manualy, but it doesn't solve the problem, the code I have just stops at a login page.

For example, this code below doesn't do anything special, I used it just to simply demonstrate my problem, so, it should print contets of a "a" element and "html" attribute from this url "http://www.fer2.net/misc.php?do=whoposted&t=51670", but from results I can see that it just got to the login page.

import prohtml.*;
HtmlElementFinder htmlElementFinder;
void setup(){
//enter your url here
htmlElementFinder = new HtmlElementFinder("http://www.fer2.net/misc.php?do=whoposted&t=51670","a");
List links = htmlElementFinder.getElements();
for (int i = 0;i<links.size();i++){
println(((StandAloneElement)links.get(i)).getAttribute("href"));
}
}

So, I wonder if there is a simple way I could provide login data to my code in order to access the data I want?

THANKS!!

Replies(2)

PhiLho

Re: How to access html data from a page that requires logging in?

1 year ago

Fact: it isn't useful to log in manually, as your credential is save in a cookie in the browser you use. If you navigate to this page with another browser, you have to log in again. Java is seen as "another browser"... So it would have to serve the same cookie as your.

I fear I don't have experience in accessing such page. Moreover, it depends on the kind of required authentication. Some pages require an HTTP-level authentication (dialog box asking your name and password), others, more classically, just show a login page. For automated authentication, you need to send (POST) the form data, get back the cookie, and send it with each request.
Perhaps I gave enough keywords for you to search the Net for a Java solution... Don't hesitate to share if you are successful.

patakk

Re: How to access html data from a page that requires logging in?

1 year ago

Thank you very much for your time, some things seem so much clearer.
I just have to mention that I've managed to skip that step (connecting to internet using Processing) just by c/p-ing the text I needed to a sepparate txt file which I then parsed and used as I intended to, but I will certainly get back to that step (logging in) and, if I find the solution, share it with you.

Here's basically the result I wanted. There is a certain thread on that forum (student forum) from which I c/p-ed the "who posted" file, matched each user with it's number of posts and visualised the data.
Bigger number of posts => bigger/more saturated/centered nickname.

Telefunken     36
Tsubasa     19
Patak     14
g0g0     13
Jonathan Richman     12
MILe     12
Summoning     9
Stealth     9
...

Top Reply