Building a spider

General Discussion Other

asimes

in General Discussion • Other • 1 year ago

Is it possible to use Processing to visit a website and get the source code? I wanted to try to see if I could build something that visits a website and finds keywords I choose.

Replies(7)

amnon.owed

Re: Building a spider

1 year ago

Sure. If by source code you mean HTML.

Code Example

String[] s = loadStrings("http://www.processing.org/");
println(s);
exit();

asimes

Re: Building a spider

1 year ago

Didn't realize it was that simple, ha ha. Thanks amnon.owned

asimes

Re: Building a spider

1 year ago

EDIT: I removed this post because it is no longer relevant and it makes the thread too long

PhiLho

Re: Building a spider

1 year ago

The only reliable way to process arbitrary HTML data is to use a library made for that, because the syntax of the language is complex. Something like JSoup can be useful.

asimes

Re: Building a spider

1 year ago

I know, I'm not trying to make a general processor, I just wanted to make sure the way I'm handling Strings makes sense.

EDIT: I removed this section of the post because it is no longer relevant

asimes

Ok, I changed the code to find anchor tags instead of removing sections of text. There are two conditions I can think of that would be problematic and if either one happens I don't want to return anything. If the bad conditions don't happen I return a String array with all the anchor tags.

Right now in the code I just return a dummy String array when the bad conditions happen, how should I rewrite the function so that instead of returning a dummy String array it lets me know the task failed? I highlighted the bad conditions that return the dummy array in red:

String[] source = loadStrings("http://www.processing.org/");
String singleLine = join(source, "");
void setup() {
size(200, 200);
String[] links = getTags(singleLine, "<a", "</a>");
println(links);
}
void draw() {
}
// Always make sure start and end are lowercase
String[] getTags(String in, String start, String end) {
ArrayList startIndexes = new ArrayList();
ArrayList endIndexes = new ArrayList();
String lowerIn = in.toLowerCase();
boolean noMatch = false;
int position = 0;
// Get all the indexes of start and end keywords
while (true) {
int index = lowerIn.indexOf(start, position);
if (index != -1) {
startIndexes.add(index);
position = index+1;
int indexStop = lowerIn.indexOf(end, position);
if (indexStop != -1) endIndexes.add(indexStop+end.length());
else {
noMatch = true;
break;
}
}
else break;
}
if (startIndexes.size() == 0) {
println(start+" not found");
String[] thatsBad = new String[1];
return thatsBad; // I don't actually want to return this but is causes an error if I don't
}
else if (noMatch) {
println(end+" not found for every case of "+start);
String[] thatsBad = new String[1];
return thatsBad; // I don't actually want to return this but is causes an error if I don't
}
else {
String[] subs = new String[startIndexes.size()];
for (int i = 0; i < subs.length; i++) {
subs[i] = in.substring((Integer)startIndexes.get(i), (Integer)endIndexes.get(i));
}
return subs;
}
}

PhiLho

Re: Building a spider

1 year ago

It isn't bad practice, but in general we return an empty array: new String[0]. Which is generally better than null (which was a common error tag as return value), because you can still process it like a normal array (instead of making a special test for null values), just with no work (if you do a for loop on an empty array, you just don't enter the loop). And still allow to check for errors ( if (returnedArray.length == 0) {}).