Wikipedia search, problem with re-searching the first results

Programming Questions

joshuakoo..

Wikipedia search, problem with re-searching the first results

in Programming Questions • 2 months ago

Dear all readers,

I'm currently working on a tool that looks up related topics with the Wikipedia API.

The idea:

Search for one topic to get back related topics. Then these related topics should also be used in a new search. These words should also send back their related topics.

The next code does only the first part (getting all the results for one keyword):

XML myXML;
XML[] links;
String keyword = "anders";
String query;
String search1;
String search2;
XML myXML2;
XML[] links2;
void setup() {
}
void draw() {
query = "http://nl.wikipedia.org/w/api.php?titles="+keyword+"&format=xml&action=query&prop=links&pllimit=500";
try {
myXML = loadXML(query);
links = myXML.getChildren("query/pages/page/links/pl");
for (int i = 0; i < links.length; i++) {
String title = links[i].getString("title");
search1 = search1+","+title;
}
}
catch (Exception exception) {
println(exception);
}
println(search1);
noLoop();
}

This code should get the result of the first word, and then uses the results to search again, so every result of the first search should be a keyword to get results from.

XML myXML;
XML[] links;
String keyword = "anders";
String query;
String search1;
String search2;
XML myXML2;
XML[] links2;
void setup() {
}
void draw() {
query = "http://nl.wikipedia.org/w/api.php?titles="+keyword+"&format=xml&action=query&prop=links&pllimit=500";
try {
myXML = loadXML(query);
links = myXML.getChildren("query/pages/page/links/pl");
for (int i = 0; i < links.length; i++) {
String title = links[i].getString("title");
search1 = search1+","+title;
}
}
catch (Exception exception) {
println(exception);
}
String[] search1_results = split(search1, ',');
for (int i = 0; i < search1_results.length; i++) {
query = "http://nl.wikipedia.org/w/api.php?titles="+search1_results[i]+"&format=xml&action=query&prop=links&pllimit=500";
myXML2 = loadXML(query);
links = myXML.getChildren("query/pages/page/links/pl");
for (int i2 = 0; i2 < links.length; i2++) {
String title = links[i2].getString("title");
search2 = search1+","+title;
}
}
println(search1);
}

The code above, however, is not working. I can't see what I'm doing wrong at this point. Hope you can help me out with this code.

Best,

Joshua

Replies(7)

Chrisir

Re: Wikipedia search, problem with re-searching the first results

2 months ago

here is a working version

it still throws a lot of excep

I catched them with

checking search1_results[i] for ""
and myXML2 for null

// This code should get the result of the first word, and then uses the results to search again, so every result of the first search should be a keyword to get results from.
XML myXML;
XML[] links;
String keyword = "anders";
String query;
String search1;
String search2;
XML myXML2;
XML[] links2 = new XML[0];
void setup() {
noLoop();
}
void draw() {
query = "http://nl.wikipedia.org/w/api.php?titles="+keyword+"&format=xml&action=query&prop=links&pllimit=500";
try {
myXML = loadXML(query);
links = myXML.getChildren("query/pages/page/links/pl");
for (int i = 0; i < links.length; i++) {
String title = links[i].getString("title");
search1 = search1+","+title;
}
}
catch (Exception exception) {
println(exception);
}
String[] search1_results = split(search1, ',');
// second loop
search2 = "______________________________________\n";
for (int i = 0; i < search1_results.length; i++) {
// println ( search1_results[i] );
search2 = search2 + "For "+search1_results[i]+":\n";
if (!search1_results[i].equals("")) {
query = "http://nl.wikipedia.org/w/api.php?titles="+search1_results[i]+"&format=xml&action=query&prop=links&pllimit=500";
myXML2 = loadXML(query);
if (myXML2!=null) {
links = myXML2.getChildren("query/pages/page/links/pl");
for (int i2 = 0; i2 < links.length; i2++) {
String title = links[i2].getString("title");
search2 = search2+","+title;
} // for
search2 = search2+"\n";
} // if
else println ("xml was null");
} // if
else println ("search1_results[i] was empty");
} // for
println(search2);
}

Greetings, Chrisir

If you need an answer, please send me a personal message since this forum doesn't notify.

joshuakoo..

Re: Wikipedia search, problem with re-searching the first results

2 months ago

thank you!

I also cleared out the exceps:

// This code should get the result of the first word, and then uses the results to search again, so every result of the first search should be a keyword to get results from.
XML myXML;
XML[] links;
String keyword = "anders";
String query;
String search1;
String search2;
XML myXML2;
XML[] links2 = new XML[0];
void setup() {
noLoop();
}
void draw() {
query = "http://nl.wikipedia.org/w/api.php?titles="+keyword+"&format=xml&action=query&prop=links&pllimit=500";
try {
myXML = loadXML(query);
links = myXML.getChildren("query/pages/page/links/pl");
for (int i = 0; i < links.length; i++) {
String title = links[i].getString("title");
search1 = search1+","+title;
}
}
catch (Exception exception) {
println(exception);
}
String[] search1_results = split(search1, ',');
// second loop
search2 = "______________________________________\n";
for (int i = 0; i < search1_results.length; i++) {
// println ( search1_results[i] );
search2 = search2 + "For "+search1_results[i]+":\n";
if (!search1_results[i].equals("")) {
query = "http://nl.wikipedia.org/w/api.php?titles="+search1_results[i]+"&format=xml&action=query&prop=links&pllimit=500";
query = query.replaceAll(" ","&20");
myXML2 = loadXML(query);
if (myXML2!=null) {
links = myXML2.getChildren("query/pages/page/links/pl");
for (int i2 = 0; i2 < links.length; i2++) {
String title = links[i2].getString("title");
title = title.replaceAll("Wikipedia:Doorverwijspagina","");
search2 = search2+","+title;
} // for
search2 = search2+"\n";
} // if
else println ("xml was null");
} // if
else println ("search1_results[i] was empty");
} // for
println(search2);
}

This was the trick:

query = query.replaceAll(" ","&20");

The wikisearch doesn't support spaces. They need to be properly replaced by &20.

Thank you so much!

Chrisir

Re: Wikipedia search, problem with re-searching the first results

2 months ago

nice!

I also have an entry with your program
For W?adys?aw Anders:

Maybe you need to check how the ? comes in and replace it equally?
like query = query.replaceAll(" ","&20");

could be & or å or so.....

ah, I looked it up, it's Władysław
so it's the ł here

but it's wrong in the xml already, so no use

query = query.replaceAll(" ","&20");

no idea...

Greetings, Chrisir

joshuakoo..

Re: Wikipedia search, problem with re-searching the first results

2 months ago

I think it's a encoding error. There's a fix for this. Had the same problem a year ago. Need to look up how.

Still one error though:

First query is NULL, so it looks for everything with NULL in wiki. Weird.

PhiLho

Re: Wikipedia search, problem with re-searching the first results

2 months ago

The &20 sounds suspicious. In HTML encoding, it would be &20; In URL encoding (more probably for a query), it would be %20

The page is probably encoded in UTF-8, so the special characters are decomposed in several 8-bit chars, and recomposed to UTF-16 Java chars thereafter. But if the font used to display them doesn't support this code page, you won't see them properly.

About the null query, I don't know where you see that. Can you show the current state of your code, if different from above? Is the whole query variable null, or only the keyword? Since it is initialized at declaration, I don't see why it would be null.

Chrisir

Re: Wikipedia search, problem with re-searching the first results

2 months ago

he is searching for null (as a string: the keyword has the String entry "null")
so maybe the api is giving "null" at 1st entry?
It's a String, not a var that is not defined imho

______________________________________
For null:
,/dev/null,ACID,Aggregaatfunctie,Alter (SQL),Axioma's van Armstrong,CRUD,C (programmeertaal),Commit,Create (SQL),Cursor (databank),Data Control Language,Data Definition Language,Data Manipulation Language,Databankmodel,Database,Databasemanagementsysteem,Databasenormalisatie,Datatype,Delete (SQL),Drop (SQL),From (SQL),Index (databank),Informatica,Insert (SQL),Join (SQL),Kandidaatsleutel,Lijst van relationele databases,Merge (SQL),Opgeslagen procedure,Partitie (databank),Pointer (programmeerconcept),Primaire sleutel,Programmeertaal,Referentiële integriteit,Relatie (databank),Relationeel model,Relationele algebra,Relationele database,Rollback (dataopslag),SQL,Select (SQL),Tabel (databank),Transactie (dataopslag),Trigger (databank),Truncate (SQL),Union (SQL),Unix,Update (SQL),Variabele (informatica),View (databank),Vreemde sleutel,Where (SQL)
For Anders (voornaam):
,Anders (voornaam),Anders Hechtel-Eksel,Andersen,Anderson,Andrea Anders,Andrea Anders (James Bond),Brouwerij Anders!,David Anders,Hans Anders,Hendrik Anders,Samuel Anders,Thomas Anders,Wiktionary,William Anders,W?adys?aw Anders,
For Anders Hechtel-Eksel:

as if null would be a search result:
see line 34

// This code should get the result of the first word, and then uses the results to search again,
// so every result of the first search should be a keyword to get results from.
XML myXML;
XML[] links;
String keyword = "anders";
String query;
String search1;
String search2;
XML myXML2;
XML[] links2 = new XML[0];
void setup() {
noLoop();
}
void draw() {
query = "http://nl.wikipedia.org/w/api.php?titles="+keyword+"&format=xml&action=query&prop=links&pllimit=500";
try {
myXML = loadXML(query);
links = myXML.getChildren("query/pages/page/links/pl");
//println(links );
for (int i = 0; i < links.length; i++) {
String title = links[i].getString("title");
//println(title);
search1 = search1+","+title;
}
}
catch (Exception exception) {
println(exception);
}
String[] search1_results = split(search1, ',');
// second loop
search2 = "______________________________________\n";
for (int i = 0; i < search1_results.length; i++) {
// println ( search1_results[i] );
search2 = search2 + "For "+search1_results[i]+":\n";
if (!search1_results[i].equals("")) {
query = "http://nl.wikipedia.org/w/api.php?titles="+search1_results[i]+"&format=xml&action=query&prop=links&pllimit=500";
query = query.replaceAll(" ", "&20");
myXML2 = loadXML(query);
if (myXML2!=null) {
links = myXML2.getChildren("query/pages/page/links/pl");
for (int i2 = 0; i2 < links.length; i2++) {
String title = links[i2].getString("title");
title = title.replaceAll("Wikipedia:Doorverwijspagina", "");
search2 = search2+","+title;
} // for
search2 = search2+"\n";
} // if
else println ("xml was null");
} // if
else println ("search1_results[i] was empty");
} // for
println(search2);
}

Greetings, Chrisir

PhiLho

Re: Wikipedia search, problem with re-searching the first results

2 months ago

Ah, he wrote "first query", so the wording was ambiguous.

The null is there because search1 is not initialized, so it is null when entering the loop, in line 23.

I suggest to make an array of size links.length , to fill it with the titles, then to use join() to build search1.

Uh, no, I don't see the point to build a string from an array to split it thereafter to get an array!

Just use the resulting array as search1_results.

Top Reply