HTTP GET requests - strange results

Core Library Questions

theboypip

HTTP GET requests - strange results

in Core Library Questions • 2 years ago

I'm trying the Networking example and I'm getting strange results. Ultimately I simply want to return a twitter search query which there searchAPI says you can get with a simple HTTP GET on say http://search.twitter.com/search.atom?q=Nottingham and indeed you get a nice string of XML in a browser with this URL.

However, I'm getting a strange return on www.processing.org. Running:

import processing.net.*;
Client c;
String data;
void setup() {
size(200, 200);
background(50);
fill(200);
c = new Client(this, "www.processing.org", 80); // Connect to server on port 80
c.write("GET / HTTP/1.1\r\n"); // Use the HTTP "GET" command to ask for a Web page
c.write("Host: 82.69.234.94\n\n"); // Be polite and say who we are
}
void draw() {
if (c.available() > 0) { // If there's incoming data from the client...
data = c.readString(); // ...then grab it and print it
println(data);
}
}

Returns

HTTP/1.1 200 OK

Date: Fri, 29 Apr 2011 11:10:00 GMT

Server: Apache/2.2.14 (Ubuntu)

Set-Cookie: cookiecookiecookie=82.69.234.94.1304075400820406; path=/; expires=Tue, 22-Apr-36 11:10:00 GMT; domain=.processing.org

Last-Modified: Sat, 07 Aug 2010 18:10:10 GMT

ETag: "830c90d-1cd-48d3fb1a9e480"

Accept-Ranges: bytes

Content-Length: 461

Vary: Accept-Encoding

Content-Type: text/html

<html>



<head> <title>

android.processing.org

</title> </head>

<body>

<center> <table width="500"> <tr> <td>

<font face="verdana, sans-serif" size=1>

<h3> Processing for Android </h3>

This page has moved <a href="http://wiki.processing.org/w/Android">to the wiki</a>.

</td> </tr> </table>

</body>

</html>

WHich is not what I was expecting. And when I run

search.twitter.com/search.atom?q=Nottingham

I get

java.net.UnknownHostException: search.twitter.com/search.atom?q=Nottingham

And a whole load more error lines.

I'm behind a domestic router which has the external IP 82.69.234.94 and my laptop is on 192.168.2.17 behind that router. I've tried on processing 1.2.1 and dev 0194, same results. Running Ubuntu with java 6...

Replies(7)

theboypip

Re: HTTP GET requests - strange results

2 years ago

Just seen there's now Processing 1.5! Installed, still same result.

Addition - putting

c = new Client(this, "search.twitter.com", 80); // Connect to server on port 80

Gets

HTTP/1.1 301 Moved Permanently

Date: Fri, 29 Apr 2011 12:01:25 GMT

Server: Apache

Location: http://search.twitter.com/

Cache-Control: max-age=300

Expires: Fri, 29 Apr 2011 12:06:25 GMT

Vary: Accept-Encoding

Content-Length: 234

Connection: close

Content-Type: text/html; charset=iso-8859-1

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">

<html><head>

<title>301 Moved Permanently</title>

</head><body>

<h1>Moved Permanently</h1>

<p>The document has moved <a href="http://search.twitter.com/">here</a>.</p>

</body></html>

Which is bizarre, as it seems to be saying that there's no web page at search.twitter.com as its moved to search.twitter.com!!! And in my browser http://search.twitter.com/ works fine...

I looked up the documentation on the Client object - no insights.

And yes, me and MatthewTrivett are working together on the same problem just from different angles... I was hoping http get was going to be easy...!

PhiLho

Re: HTTP GET requests - strange results

2 years ago

I don't know where the Client example to get HTTP pages comes from, but there are several things wrong with the code you show.
First, Host indicates " the naming authority of the origin server or gateway given by the original URL", not " who we are", which is the task of the User-Agent field. See http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.23
Second, a request always ends with a pair of \r\n, in addition to the \r\n pair coming with each header field.
Last, if you don't specify who you are (impersonating a Web browser) and what you want (a Web page), the server can fall back on defaults, like serving an Android device...
Here is a request that works:

void setup() {
size(200, 200);
background(50);
fill(200);
c = new Client(this, "processing.org", 80); // Connect to server on port 80
c.write("GET / HTTP/1.1\r\n"); // Can replace / with, eg., /reference/ or similar path
c.write("Host: www.processing.org\r\n"); // Which server is asked
c.write("User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.0.7) Gecko/2009021906 Firefox/3.0.7\r\n");
c.write("Accept: text/html,application/xhtml+xml,application/xml\r\n");
c.write("Accept-Language: en-us,en;q=0.5\r\n");
c.write("Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\n");
c.write("\r\n");
}

theboypip

Re: HTTP GET requests - strange results

2 years ago

That's great many many thanks.

The example is from processing 1.5 under Examples > Libraries > Network > HTTPClient

So I have

import processing.net.*;
Client c;
String data;
String domain = "search.twitter.com";
String addr = "/search.atom?q=Nottingham";
void setup() {
size(200, 200);
background(50);
fill(200);
c = new Client(this, domain, 80); // Connect to server on port 80
c.write("GET "+addr+" HTTP/1.1\r\n"); // Can replace / with, eg., /reference/ or similar path
c.write("Host: "+domain+"\r\n"); // Which server is asked
c.write("User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US) AppleWebKit/534.16 (KHTML, like Gecko) Ubuntu/10.04 Chromium/10.0.648.205 Chrome/10.0.648.205 Safari/534.16\r\n");
c.write("Accept: application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5\r\n");
c.write("Accept-Language: en-us,en;q=0.5\r\n");
c.write("Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\n");
c.write("\r\n");
}
void draw() {
if (c.available() > 0) { // If there's incoming data from the client...
data = c.readString(); // ...then grab it and print it
println(data);
}
}

Works a treat, ta.

PhiLho

Re: HTTP GET requests - strange results

2 years ago

Made Issue 655

jainsapna

Re: HTTP GET requests - strange results

2 years ago

Hello,

When I tried to run the above code I got this exception:

java.net.ConnectException: Connection refused: connect
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
    at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
    at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
    at java.net.Socket.connect(Socket.java:529)
    at java.net.Socket.connect(Socket.java:478)
    at java.net.Socket.<init>(Socket.java:375)
    at java.net.Socket.<init>(Socket.java:189)
    at processing.net.Client.<init>(Unknown Source)
    at sketch_may02a.setup(sketch_may02a.java:34)
    at processing.core.PApplet.handleDraw(Unknown Source)
    at processing.core.PApplet.run(Unknown Source)
    at java.lang.Thread.run(Thread.java:619)
java.lang.NullPointerException
    at processing.net.Client.write(Unknown Source)
    at processing.net.Client.write(Unknown Source)
    at sketch_may02a.setup(sketch_may02a.java:35)
    at processing.core.PApplet.handleDraw(Unknown Source)
    at processing.core.PApplet.run(Unknown Source)
    at java.lang.Thread.run(Thread.java:619)
java.lang.NullPointerException
    at processing.net.Client.write(Unknown Source)
    at processing.net.Client.write(Unknown Source)
    at sketch_may02a.setup(sketch_may02a.java:36)
    at processing.core.PApplet.handleDraw(Unknown Source)
    at processing.core.PApplet.run(Unknown Source)
    at java.lang.Thread.run(Thread.java:619)
java.lang.NullPointerException
    at processing.net.Client.write(Unknown Source)
    at processing.net.Client.write(Unknown Source)
    at sketch_may02a.setup(sketch_may02a.java:37)
    at processing.core.PApplet.handleDraw(Unknown Source)
    at processing.core.PApplet.run(Unknown Source)
    at java.lang.Thread.run(Thread.java:619)
java.lang.NullPointerException
    at processing.net.Client.write(Unknown Source)
    at processing.net.Client.write(Unknown Source)
    at sketch_may02a.setup(sketch_may02a.java:38)
    at processing.core.PApplet.handleDraw(Unknown Source)
    at processing.core.PApplet.run(Unknown Source)
    at java.lang.Thread.run(Thread.java:619)
java.lang.NullPointerException
    at processing.net.Client.write(Unknown Source)
    at processing.net.Client.write(Unknown Source)
    at sketch_may02a.setup(sketch_may02a.java:39)
    at processing.core.PApplet.handleDraw(Unknown Source)
    at processing.core.PApplet.run(Unknown Source)
    at java.lang.Thread.run(Thread.java:619)
java.lang.NullPointerException
    at processing.net.Client.write(Unknown Source)
    at processing.net.Client.write(Unknown Source)
    at sketch_may02a.setup(sketch_may02a.java:40)
    at processing.core.PApplet.handleDraw(Unknown Source)
    at processing.core.PApplet.run(Unknown Source)
    at java.lang.Thread.run(Thread.java:619)
java.lang.NullPointerException
    at processing.net.Client.write(Unknown Source)
    at processing.net.Client.write(Unknown Source)
    at sketch_may02a.setup(sketch_may02a.java:41)
    at processing.core.PApplet.handleDraw(Unknown Source)
    at processing.core.PApplet.run(Unknown Source)
    at java.lang.Thread.run(Thread.java:619)

theboypip

Re: HTTP GET requests - strange results

2 years ago

A quick google suggests that its either a firewall problem or some other kind if internet connection problem. Check your not behind a proxy, strict firewall, or something else that would stop processing connecting to the internet.

http://www.coderanch.com/t/207717/sockets/java/resolve-java-net-ConnectException-Connection

Otherwise the code in the post above yours, copied and pasted verbatim into a new sketch, works for me.

ben.fry

Re: HTTP GET requests - strange results

2 years ago

fwiw, it's also rarely necessary that you need to set up a special client just for an HTTP GET. You can use:

String[] lines = loadStrings("http://processing.org");
println(lines);

to do the same thing as the original example.

Top Reply