Load Image From Web

General Discussion Other

captainco..

in General Discussion • Other • 1 year ago

Hello!

I'm trying to retrieve an image from the internet without explicitly knowing its URL. For instance, an image off of a webpage like http://apod.nasa.gov/apod/astropix.html where the image changes daily. I think XMLEelement is the direction I should be heading but the reference example is confusing to me and I haven't been able to find a tutorial that meets my needs. Any help or direction to a relevant tutorial would be greatly appreciated!

Replies(9)

PhiLho

Re: Load Image From Web

1 year ago

" For instance" or precisely?
By nature, Web pages are complex and varied. Either you make a generic page crawler that will find and download all the images it can find one a give page (some criteria can be used), or you write a program targeting a specific page, to download a specific image (here, the only one).

XMLElement can't help in the page you point to, because the page isn't using XHTML (looks like HTML5, looking strongly like good old HTML1 to 4...).
For a generic page crawler, I would advise to use an HTML page parser, like JSoup. It offers facilities to quickly get a given element.

If the page is consistent over time (always using the same pattern / template), you can write some specific code looking precisely for the URL of the image.
When I look at the source code of the page I get today, I see:

<a href="image/1207/AR1520_071112friedman.jpg">
<IMG SRC="image/1207/AR1520_071112friedman900.jpg"
alt="See Explanation. Clicking on the picture will download  the highest resolution version available."></a>

So, first, it depends if you want the displayed image or the high resolution one.

PhiLho

Re: Load Image From Web

1 year ago

My solution:

String URL_BASE = "http://apod.nasa.gov/apod/";
String URL_PAGE = "astropix.html";

PImage astroImage;

// Optional spaces, the IMG tag and its attribute, capture of the URL, anything after it
Pattern pat = Pattern.compile("\\s*<IMG SRC=\"(image/.*?)\".*");

void setup()
{
  size(1024, 768);
  frameRate(30);
  
  DateFormat df = new SimpleDateFormat("yyyy-MM-dd");
  String imageName = "Nasa-" + df.format(new Date()) + ".jpg";
  
  String url = findImageURL(URL_BASE + URL_PAGE);
  if (url != null) // Found
  {
    astroImage = loadCachedImage(imageName, URL_BASE + url);
  }
}

void draw()
{
  image(astroImage, 0, 0);
}

String findImageURL(String pageURL)
{
  String url = null;
  String[] lines = loadStrings(pageURL);
  for (String line : lines)
  {
    Matcher m = pat.matcher(line);
    if (m.matches())
    {
      url = m.group(1);
      break;
    }
  }
  return url;
}

PImage loadCachedImage(String fileName, String url)
{
  PImage img = loadImage(fileName);
  if (img == null) // Not downloaded yet
  {
    img = loadImage(url);
    if (img != null)
    {
      img.save(fileName); // Cache of the file
    }
    else
    {
      println("Unable to load the image from " + url);
      exit();
    }
  }
  return img;
}

I made a cache, so you get the image on your hard disk the first run to the day, and if you run several times the same day, it just load it from the disk, reducing the stress on the site and showing it much faster.

Regular expressions are not good to parse HTML in general, but are OK if we target a specific page / element...

captainco..

Re: Load Image From Web

1 year ago

Thanks PhiLho!

I guess it was both precisely and in general as I'm just trying to figure out how it works on something simple and then go from there. I am fairly new to Processing/writing code so I'll have to pick this apart and look at the reference section a lot more to understand exactly how it works but I appreciate seeing a working model of what I would like to do!

PhiLho

Re: Load Image From Web

1 year ago

I used something that isn't in the Processing Reference: Pattern and Matcher are pure Java libraries. Processing offers some regular expression support, but I used these out of habit.

joshuakoo..

Re: Load Image From Web

1 year ago

I've been trying to scrape images from a Dutch news website (www.nrc.nl).

If I take one page for example:

http://www.nrc.nl/nieuws/2012/08/26/oostenrijkse-bondskanselier-wil-griekenland-wel-meer-tijd-geven/

Then this is the first part of the HTML:

<!DOCTYPE HTML>
<html lang="nl-NL">
	<head>
		<meta charset="utf-8">
		<script src="../../../../../js/nmt.scripts.js?v=201208032037"></script>
		<noscript><img src="http://b.scorecardresearch.com/p?c1=2&amp;c2=11546962&amp;c4=nrc.nl" height="1" width="1" alt=""></noscript>

		<title>Oostenrijkse bondskanselier wil Griekenland wel meer tijd geven :: nrc.nl</title>

		<link rel="stylesheet" href="../../../../../css/minified.css?v=1345715119">
		
		<link rel="canonical" href="http://www.nrc.nl/nieuws/2012/08/26/oostenrijkse-bondskanselier-wil-griekenland-wel-meer-tijd-geven/" />
		<link rel="alternate" media="handheld"  href="http://m.nrc.nl/nieuws/2012/08/26/oostenrijkse-bondskanselier-wil-griekenland-wel-meer-tijd-geven/" />

		<meta name="keywords" content="Nieuws, Buitenland, Economie, Angela Merkel, Antonis Samaras, Duitsland, ecb, EU, eurozone, François Hollande, Frankrijk, Griekenland, IMF, Oostenrijk, schuldencrises, Werner Faymann"/>
		<meta property="og:title" content="Oostenrijkse bondskanselier wil Griekenland wel meer tijd geven" />
		<meta property="og:url" content="http://www.nrc.nl/nieuws/2012/08/26/oostenrijkse-bondskanselier-wil-griekenland-wel-meer-tijd-geven/" />
		<meta property="og:image" content="http://www.nrc.nl/wp-content/uploads/2012/08/2012-08-14T100951Z_01_NIL08_RTRMDNP_3_AUSTRIA-300x197.jpg" />
		<meta property="og:description" content="De Oostenrijkse bondskanselier Werner Faymann heeft zich vandaag uitgesproken ten gunste van meer tijd voor Griekenland om de schulden terug te betalen. Er is bovendien een grote kans dat dat gebeurt, zei hij." />
		<meta property="og:site_name" content="nrc.nl" />
		<meta property="og:type" content="article" />

What I need is the <meta property="og:image url. What would be the most logic way to isolate the image? I think the best way to start is to search for <meta property="og:image and start to get the text till />.

I only don't know how to isolate this part from the rest of the text and convert it to a string. Does anybody know what's the best way to deal with this?

joshuakoo..

Re: Load Image From Web

1 year ago

(my last question is marked a spam. This was not my intention)

PhiLho

Re: Re: Load Image From Web

1 year ago

You used the sentence "best way to deal" and Zoho didn't like it...

akiersky

Re: Load Image From Web

1 year ago

Check out the jsoup lib ( http://jsoup.org/), a Java lib, but it has some very nice methods to break an html document down to elements and extract attributes and content out.

Here's an example of how I am able to extract images out of an html doc and return an array of the images:

String[][] extractImages(String s) {
String[] imgUrls = {"none"};
Document jDoc = Jsoup.parse(s);
Elements img = jDoc.select("img[src]");
String[] storyImgs = new String[img.size()];
int count = 0;
for(Iterator<Element> i = img.iterator(); i.hasNext();) {
Element e = i.next();
storyImgs[count++] = e.attr("src");
e.remove();
}
Elements paragraphs = jDoc.getElementsByTag("p");
Elements unwanted = paragraphs.select("iframe, i, b, strong, hr, blockquote, em, span, a, br, small");
String story;
for(Iterator<Element> i = unwanted.iterator(); i.hasNext();) {
Element e = i.next();
e.unwrap(); //removes the elemnt, but keeps the content
}
println("_______________________________________________________________________________________________________");
println("after formatting : " +paragraphs.html());
String[] formattedText = {paragraphs.html()};
String[][] formatted = {formattedText, storyImgs};
return formatted;
}

But check out the docs ( http://jsoup.org/apidocs/) for all it's power!

good luck!

PhiLho

Re: Load Image From Web

1 year ago

akiersky is right. I answered with a regular expression to address a very specific need in a specific page, supposing it is quite stable. But if it is quick to write, it is still a quite brittle solution that might break someday, if they change the page layout, etc. Well, that's the case for most Web parsing anyway.
But for more generic tasks, robuster solutions, picking unknown elements ("all images of a page") and such, using a real HTML parser like Jsoup, understanding the HTML syntax, its quirks and its escape rules, is a much better solution.

Top Reply