Loading Strings, JSON, CSV, etc from the web (if server blocked Java crawler)

tnagel

It seems some web services prevent crawlers to access their services or web pages. As Processing does not allow to set the User-Agent header to change the default "Java x.x.x", I needed to write an own loadSomething method. Maybe this is interesting for others, too.

JSONArray loadJSONArrayWithHeader(String urlString) {
  JSONArray jsonArray = null;
  try {
    URL url = new URL(urlString);
    URLConnection urlConnection = url.openConnection();
    urlConnection.setRequestProperty("User-Agent", "Mozilla (Student project");
    BufferedReader reader = new BufferedReader(new InputStreamReader(urlConnection.getInputStream()));
    jsonArray = new JSONArray(reader);
    reader.close();
  } 
  catch (Exception e) {
    println("Data not loaded correctly. " + e.toString());
  }
  return jsonArray;
}

You simply need to change the JSONArray above to the appropriate Processing object / method to load your stuff.

PhiLho

Moved to Share your Work, since it is not a Programming Question.
Thanks for sharing.