using proHTML scraping text

dockhands

in Contributed Library Questions • 6 months ago

Hi,

I'm a new programmer, so please excuse my poor use of language.

I'm trying to scrape specific HTML text off an URL.

My strategy is (was) to use the proHTML library, from which I am able to create a 'List" of all the page elements and text (language?). I then want to convert that List into a String, so I can split the List and retrieve my specific text that I'm after (I'm assuming I need to convert List to String - I just don't know how to do this, or if there is another way around this).

Thanks for your help/suggestions.

dan

Code:

import prohtml.*;

HtmlList htmlList;

void setup(){

size(100,100);

//enter your url here

htmlList = new HtmlList("http://www.lyricfind.com/services/lyrics-search/try-our-search/?q=ape+punch+run");

for (int i = 0;i<htmlList.pageList.size();i++){

// println(htmlList.pageList.get(i));



//toString(htmlList.pageList);



//String htmlJunk = htmlList.pageList;

//String [] list1 = split(htmlJunk,"<h2>");

println(htmlList.pageList);

}

}

Replies(6)

Chrisir

Re: using proHTML scraping text

6 months ago

here..................

I noticed that the small arrow before every song is this image:
http://www.lyricfind.com/wp-content/themes/lyricfind/common-assets/PlayButton.png

so I just look for that image name and started to collect the String for that song in an array myResult.
When the image occurs again, I know, now comes the next song and I go to the next entry in the array myResult

//
// http://www.lyricfind.com/services/lyrics-search/try-our-search/?q=ape+punch+run
// http://creativecomputing.cc/p5libs/prohtml/index.htm
import prohtml.*;
//
HtmlList htmlList;
String [] myResult = new String [100];
int indexmyResult=-1;
boolean firstFinding = false;
//
void setup() {
size(1300, 800);
//enter your url here
htmlList = new HtmlList("http://www.lyricfind.com/services/lyrics-search/try-our-search/?q=ape+punch+run");
//
String a1="";
//
for (int i = 0;i<htmlList.pageList.size();i++) {
a1=(htmlList.pageList.get( i ).toString());
//
if ( a1.equals ( "<img src=\"http://www.lyricfind.com/wp-content/themes/lyricfind/common-assets/playbutton.png\">" ) )
{
// println ("equals");
firstFinding = true;
indexmyResult++;
myResult[indexmyResult] = "";
}
else
{
if (firstFinding)
myResult[indexmyResult] = myResult[indexmyResult] + a1 + " ";
}
}
println("end setup()");
}
//
void draw() {
background(111);
for (int i = 0;i<indexmyResult;i++) {
// replace etc.
myResult[i] = myResult[i].replace("", ": ");
myResult[i] = myResult[i].replace("", " -- ");
myResult[i] = myResult[i].replace("<h2>", "");
//
text(myResult[i], 30, 17*i + 29);
println (myResult[i] );
}
}
// -------------------------------------------------------------
//

dockhands

Re: using proHTML scraping text

6 months ago

Chrisir, thanks! That's so awesome. That looks great. I just spent the last two hours doing it the old dirt nasty way, using loadString (as suggested on a previous post)

I think I prefer yours!

I'd like to search different keywords in the URL.

For instance I have my baseURL like:

URLbase = "http://www.lyricfind.com/services/lyrics-search/try-our-search/?q="

And I want to generate keywords to search.. something like:

URLbase + keyword1+ keyword2+ keyword3

to give: "http://www.lyricfind.com/services/lyrics-search/try-our-search/?q=keyword1+keyword2+keyword3"

Can you do this inside a URL?

my shitty code that I wont be using anymore

size(600,200);
String lines[] = loadStrings("http://www.lyricfind.com/services/lyrics-search/try-our-search/?q=rock+try+bill+book");
println("there are " + lines.length + " lines");
for (int i = 0 ; i < lines.length; i++) {
}
String lyric = lines[252];
println(lyric);
//String htmlJunk = htmlList.pageList;
String [] list1 = split(lyric,">");
String [] list2 = split(list1[1],"<em");
println(list1[1]);
println(list1[2]);
String [] emPart = split(list1[2],"</em");
//println(emPart[0]);
//println(list2[0]);
//println(list2[0] + emPart[0]);
String toExclude = "" ;
String lyric0 = lyric.replaceAll(toExclude, "");
String toExclude2 = "" ;
String lyric1 = lyric0.replaceAll(toExclude2, "");
String toExclude3 = "" ;
String lyric2 = lyric1.replaceAll(toExclude3, "");
String toExclude4 = "" ;
String lyric3 = lyric2.replaceAll(toExclude4, "");
// I couldn't figure out how to exclude more than one text at a time
//println(lyric3);
String [] stanzas = split(lyric3,"/");
print(stanzas[0]);
print(stanzas[1]);
print(stanzas[2]);
color(0);
text(stanzas[0], 10,10);
text(stanzas[1], 10, 50);
text(stanzas[2], 10, 100);

Chrisir

Re: using proHTML scraping text

6 months ago

sure. it's like

url...." + keyword1 + "+" + keyword2 + "+"

etc.

dockhands

Re: using proHTML scraping text

6 months ago

Sorry, it was a stupid question. I figured it out; I think I was having a hard time using the " " or something.

I'm working on a image+lyric generator... it's slowly coming along.

Thanks for your help (and proHTML?).

My next step is to try rearrange the images to match the keywords/stanzas

And then open it up to include any film.

Here's what I have so far.... (draft version)....

//========================================================
// INTRO
//========================================================
//========================================================
// GLOBAL VARIABLES
//========================================================
import prohtml.*;
PImage webImg1, webImg2, webImg3;
HtmlImageFinder htmlImageFinder;
PFont font;
font = loadFont("MyriadPro-Bold-24.vlw");
int imgW = 512; // widthy of the image stills
int imgH = 238; // height of the image stills
int rand15 = int(random(1,15));
int rand30 = int(random(15,30));
int rand45 = int(random(30,45));
//insert your url here
htmlImageFinder = new HtmlImageFinder("http://film-grab.com/2010/07/06/2001-a-space-odyssey/");
PImage[] images = new PImage[htmlImageFinder.getNumbOfImages()];
println(htmlImageFinder.getNumbOfImages());
size(imgW,imgH*3);
try{
for(int i = 0;i<50;i++){
images[i] = loadImage(htmlImageFinder.getImageLink(i));
images[i].resize(imgW,imgH);
// println(htmlImageFinder.getImageLink(i));
}//end LOOP
} // end try
catch (NullPointerException e) {
}
//========================================================
// GET IMAGES AND KEYWORDS
//========================================================
// extract keyword from image
String kW1a = htmlImageFinder.getImageLink(rand15);
String[] kW1b = split(kW1a,".png");
String[] kW1c = split (kW1b[0], "-");
String [] kW1final= split (kW1c[1], "1");
String kW1F = kW1final[0];
println(kW1F);
String kW2a = htmlImageFinder.getImageLink(rand30);
String[] kW2b = split(kW2a,".png");
String[] kW2c = split (kW2b[0], "-");
String[] kW2final= split (kW2c[1], "1");
println(kW2final[0]);
String kW3a = htmlImageFinder.getImageLink(rand45);
String[] kW3b = split(kW3a,".png");
String[] kW3c = split (kW3b[0], "-");
String[] kW3final= split (kW3c[1], "1");
println(kW3final[0]);
// make images better resolution, remove ?w=150& from URL
String betterPic1 = htmlImageFinder.getImageLink(rand15);
String[] picture1 = split(betterPic1,"?w=150&");
String betterPic2 = htmlImageFinder.getImageLink(rand30);
String[] picture2 = split(betterPic2,"?w=150&");
String betterPic3 = htmlImageFinder.getImageLink(rand45);
String[] picture3 = split(betterPic3,"?w=150&");
println(picture1[0]);
//PImage newImage = new PImage (picture1[0]);
//image(newImage,0,0);
int randStanza = int(random(0,11));
String url1 = picture1[0];
String url2 = picture2[0];
String url3 = picture3[0];
// Load image from a web server
webImg1 = loadImage(url1, "gif");
webImg1.resize(imgW,imgH);
webImg2 = loadImage(url2, "gif");
webImg2.resize(imgW,imgH);
webImg3 = loadImage(url3, "gif");
webImg3.resize(imgW,imgH);
image(webImg1, 0, 0);
image(webImg2, 0, imgH);
image(webImg3, 0, imgH*2);
//========================================================
// GET KEYWORDS AND SEARCH
//========================================================
String keyword1temp= join( kW1final, "+");
String keyword1 = keyword1temp;
String keyword2temp= join( kW2final, "+");
String keyword2 = keyword2temp;
String keyword3temp= join( kW3final, "+");
String keyword3 = keyword3temp;
println (keyword1);
println (keyword2);
println (keyword3);
// search
String baseURL = "http://www.lyricfind.com/services/lyrics-search/try-our-search/?q=";
String request = baseURL + keyword1 +keyword2 +keyword3;
println(request);
String lines[] = loadStrings(request);
println("there are " + lines.length + " lines");
for (int i = 0 ; i < lines.length; i++) {
}
String lyric = lines[252];
println(lyric);
//String htmlJunk = htmlList.pageList;
String [] list1 = split(lyric,">");
String [] list2 = split(list1[1],"<em");
println(list1[1]);
println(list1[2]);
String [] emPart = split(list1[2],"</em");
//println(emPart[0]);
//println(list2[0]);
//println(list2[0] + emPart[0]);
String toExclude = "" ;
String lyric0 = lyric.replaceAll(toExclude, "");
String toExclude2 = "" ;
String lyric1 = lyric0.replaceAll(toExclude2, "");
String toExclude3 = "" ;
String lyric2 = lyric1.replaceAll(toExclude3, "");
String toExclude4 = "" ;
String lyric3 = lyric2.replaceAll(toExclude4, "");
//println(lyric3);
String [] stanzas = split(lyric3,"/");
print(stanzas[0]);
print(stanzas[1]);
print(stanzas[2]);
textSize(21); // text size
textFont(font, 24); //font
fill(250); // white text
textAlign(CENTER); // center Text
text(stanzas[0], 10, (imgH)-55, width-10, (imgH));
text(stanzas[1],10, (imgH*2)-55, width-10, (imgH*2));
text(stanzas[2], 10, (imgH*3)-55, width-10, (imgH*3));

Chrisir

Re: using proHTML scraping text

6 months ago

Sounds great!

dockhands

Re: using proHTML scraping text

6 months ago

Thanks.

Hmm, but I'm running into problems with the array limits when I load other movies.

Your code is more robust/error free. But the one thing I liked about my code was that I was able to separate the one set of lyrics into 3 sentences, using "/" to split the sentences.

I suppose the HTML you parsed didn't contain the "/" as delimiters. A pity. Because I don't know how I would parse your results ??? I could split them based on Capital Letters, but that's not as accurate.

Any ideas?

Your result returns:

Silent Drive The Punch : died I want to run but my feet get in the way of the getaway that day You claimed if I could know well I tried to know But defeat grabs for me and its

I want to convert this into:

1) died

2) I want to run but my feet get in the way of the getaway that day

3) You claimed if I could know well I tried to know

I can easily remove the Artist and Song name using ":" as a split token, but I can't think of a way to break up the lyrics into separate sentences using any characters -- besides capitalization??

On solution would be to split into 3 parts based on character length... less "poetic", but maybe the only solution...

Thanks

Top Reply