use xpath with a xml

General Discussion Other

zotzot

in General Discussion • Other • 2 years ago

i'd like to extract data from an html document using xPath.

I've seen there are different libraries for java to do this, but i did'n found any example in processing.

Can somebody suggest me the easist way to do it?

thanks

Replies(4)

PhiLho

Re: use xpath with a xml

2 years ago

I suggest to use an HTML to Dom or XML library like Jsoup or TagSoup. Jsoup has its own query language, somehow simpler that Xpath.

zotzot

Re: use xpath with a xml

2 years ago

ok, i've manage to do it with Jsoup, thanks phi.lho.

i publish the code as i didn't found an example using it with processing.

The code get the search page from yahoo and print the search results:

import org.jsoup.Jsoup;
import org.jsoup.helper.Validate;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
import java.io.IOException;
void setup() {
//the page i want load
String url = "http://search.yahoo.com/search?p=processing.org";
println("Fetching %s..." + url);
Document doc;
//there's an unhandled exception in the library, so we must use try-catch
try {
doc = Jsoup.connect(url).get();
}
catch (Exception e) {
println(e);
doc = null;
}
//get only <a> tag under <h3> (the search results in yahoo)
Elements test = doc.select("h3 > a");
//print it out
println("-------------");
for (Element t : test) {
println(t.text()+" | "+t.attr("abs:href"));
}
}

any suggestion to improve it is welcome.

Anyway, i have a problem using it with google.

if you run the same code with the url "http://www.google.com/search?q=processing.org", it returns this error:

java.io.IOException: 403 error loading URL http://www.google.com/search?q=processing.org

Exception in thread "Animation Thread" java.lang.NullPointerException

at jsoup_test.setup(jsoup_test.java:52)

at processing.core.PApplet.handleDraw(PApplet.java:1608)

at processing.core.PApplet.run(PApplet.java:1530)

at java.lang.Thread.run(Thread.java:680)

any suggestion?

PhiLho

Re: use xpath with a xml

2 years ago

If you search the forum, you will find the problem is quite common with Google: it doesn't like automated queries. There are workarounds although I don't recall them right now.

zotzot

Re: use xpath with a xml

2 years ago

yes you're right. Google doesn't allow automated queries. You must set the correct parameters to the URLConnection object.

I wrote a simple class to handle google queries, here is the code (i haven't found other examples in the forum, sorry if i'm double-posting it).

import org.jsoup.Jsoup;
import org.jsoup.select.Elements;
import java.net.URI;
class GoogleSearch {
String query;
int resultNum, startFrom;
Elements results;
GoogleSearch(String query, int resultNum, int startFrom) {
this.query = query;
this.resultNum = resultNum;
this.startFrom = startFrom;
}
Elements execute() {
Elements results;
try {
String visiturl = "http://www.google.com/search?q="+URLEncoder.encode(this.query).replace("+","%20")+"&num="+this.resultNum+"&start="+this.startFrom;
URL url = new URL(visiturl);
String html="";
URLConnection urlc = url.openConnection();
urlc.setRequestProperty("User-Agent", "");
InputStream is = urlc.getInputStream();
int c;
while ( (c = is.read ()) != -1) {
html += (char) c;
}
Document doc = Jsoup.parse(html);
results = doc.select("h3 > a");
}
catch(Exception e) {
results = null;
println(e);
}
this.results = results;
return results;
}
}

Top Reply