xml retrieval

Core Library Questions

Dimitar B..

in Core Library Questions • 2 months ago

Hello processOrs.

I'm new to processing but I already have figured out most of the basic concepts.

I'm currently trying to make an interface to visualize the data in that xml

http://www.teamliquid.net/video/streams/?xml=1&filter=live

My problem is that I can't even get the data

I have tried with getXML - not working

void setup() {
XML xml = loadXML("http://www.teamliquid.net/video/streams/?xml=1&filter=live");
println(xml);
}

java.io.IOException: Server returned HTTP response code: 406 for URL: http://www.teamliquid.net/video/streams/?xml=1&filter=live

at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1436)

at java.net.URL.openStream(URL.java:1010)

at processing.core.PApplet.createInputRaw(PApplet.java:7001)

at processing.core.PApplet.createInput(PApplet.java:6969)

at processing.core.PApplet.createReader(PApplet.java:6763)

at processing.core.PApplet.loadXML(PApplet.java:6105)

at processing.core.PApplet.loadXML(PApplet.java:6095)

at HTTPClient.setup(HTTPClient.java:18)

at processing.core.PApplet.handleDraw(PApplet.java:2280)

at processing.core.PGraphicsJava2D.requestDraw(PGraphicsJava2D.java:243)

at processing.core.PApplet.run(PApplet.java:2176)

at java.lang.Thread.run(Thread.java:662)

http://www.teamliquid.net/video/streams/?xml=1&filter=live does not exist or could not be read

java.net.MalformedURLException

at java.net.URL.<init>(URL.java:601)

at java.net.URL.<init>(URL.java:464)

at java.net.URL.<init>(URL.java:413)

at com.sun.org.apache.xerces.internal.impl.XMLEntityManager.setupCurrentEntity(XMLEntityManager.java:648)

at com.sun.org.apache.xerces.internal.impl.XMLVersionDetector.determineDocVersion(XMLVersionDetector.java:186)

at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:772)

at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:737)

at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:119)

at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:232)

at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:284)

at processing.data.XML.<init>(XML.java:170)

at processing.core.PApplet.loadXML(PApplet.java:6105)

at processing.core.PApplet.loadXML(PApplet.java:6095)

at HTTPClient.setup(HTTPClient.java:18)

null

at processing.core.PApplet.handleDraw(PApplet.java:2280)

at processing.core.PGraphicsJava2D.requestDraw(PGraphicsJava2D.java:243)

at processing.core.PApplet.run(PApplet.java:2176)

at java.lang.Thread.run(Thread.java:662)

I tried to use the example HTTPclient - not working

import processing.net.*;
Client c;
String data;
void setup() {
size(200, 200);
background(50);
fill(200);
c = new Client(this, "http://www.teamliquid.net/video/streams/?xml=1&filter=live", 80); // Connect to server on port 80
c.write("GET / HTTP/1.0\r\n"); // Use the HTTP "GET" command to ask for a Web page
c.write("\r\n");
}
void draw() {
if (c.available() > 0) { // If there's incoming data from the client...
data = c.readString(); // ...then grab it and print it
println(data);
}
}

java.net.UnknownHostException: http://www.teamliquid.net/video/streams/?xml=1&filter=live

at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:195)

at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)

at java.net.Socket.connect(Socket.java:529)

at java.net.Socket.connect(Socket.java:478)

at java.net.Socket.<init>(Socket.java:375)

at java.net.Socket.<init>(Socket.java:189)

at processing.net.Client.<init>(Unknown Source)

at HTTPClient.setup(HTTPClient.java:42)

at processing.core.PApplet.handleDraw(PApplet.java:2280)

at processing.core.PGraphicsJava2D.requestDraw(PGraphicsJava2D.java:243)

at processing.core.PApplet.run(PApplet.java:2176)

at java.lang.Thread.run(Thread.java:662)

java.lang.NullPointerException

at processing.net.Client.write(Unknown Source)

at processing.net.Client.write(Unknown Source)

at HTTPClient.setup(HTTPClient.java:43)

at processing.core.PApplet.handleDraw(PApplet.java:2280)

at processing.core.PGraphicsJava2D.requestDraw(PGraphicsJava2D.java:243)

at processing.core.PApplet.run(PApplet.java:2176)

at java.lang.Thread.run(Thread.java:662)

java.lang.NullPointerException

at processing.net.Client.write(Unknown Source)

at processing.net.Client.write(Unknown Source)

at HTTPClient.setup(HTTPClient.java:44)

at processing.core.PApplet.handleDraw(PApplet.java:2280)

at processing.core.PGraphicsJava2D.requestDraw(PGraphicsJava2D.java:243)

at processing.core.PApplet.run(PApplet.java:2176)

at java.lang.Thread.run(Thread.java:662)

I also tried to use saveStrings(); - still not working.

Can someone help me with that please.

Thank you!

Replies(2)

PhiLho

At http://bazaar.launchpad.net/~philho/+junk/Processing/view/head:/_QuickExperiments/_Net/HTTP_GET/HTTP_GET.pde I give an example to get data from an URL needing a user agent: some servers prefer to avoid automated connections, or rely on user agent to personalize answers or something.

Notice how the Client constructor takes only the top-level URL, the parameters must be given in the GET request.

Showing the lines I changed for testing your URL:

void draw()
{
if (step > 0)
{
println("------ " + step);
client = new Client(this, "www.teamliquid.net", 80);
}
switch (step)
{
case 1:
client.write("GET /video/streams/?xml=1&filter=live HTTP/1.1\r\n");
client.write("Host: www.teamliquid.net\r\n");

The fetched data is gzipped, so it shows binary data.

PhiLho

Re: xml retrieval

2 months ago

OK, after some reverse engineering, trial and error, and sweat, I got it working:

import processing.net.*;
Client client;
int step = 0;
void setup()
{
size(200, 200);
frameRate(5);
step = 1;
}
void draw()
{
if (step > 0)
{
println("------ " + step);
}
switch (step)
{
case 1:
client = new Client(this, "www.teamliquid.net", 80);
client.write("GET /video/streams/?xml=1&filter=live HTTP/1.1\r\n");
client.write("Host: www.teamliquid.net\r\n");
// client.write("User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.0.7) Gecko/2009021906 Firefox/3.0.7\r\n");
client.write("Accept: text/html,application/xhtml+xml,application/xml\r\n");
client.write("Accept-Language: en-us,en;q=0.5\r\n");
client.write("Accept-Encoding: gzip\r\n");
client.write("Accept-Charset: utf-8;q=0.7,*;q=0.7\r\n");
// client.write("Keep-Alive: 300\r\n");
// client.write("Connection: keep-alive\r\n\r\n");
client.write("\r\n");
break;
case 2:
exit();
}
if (step > 0)
{
step = -++step; // Next step, and negative to stop treatment until an answer is received
}
if (step < 0 && client.available() > 0)
{
byte[] dataIn = client.readBytes();
String textData = new String(dataIn);
if (textData.contains("text/plain"))
{
println(textData); // Likely to be an error message (eg. if omitting to accept gzip data)
}
else
{
byte[] gzipData = null;
for (int i = 0; i < dataIn.length - 3; i++)
{
// print(hex(dataIn[i]) + " ");
if (dataIn[i] == 0x0D && dataIn[i + 1] == 0x0A && dataIn[i + 2] == 0x0D && dataIn[i + 3] == 0x0A) // Start of POST data
{
println("Found");
int len = 0;
i += 4; // No check on bounds, don't do that in production code! :-)
while (dataIn[i] != 0x0D)
{
// println(i + " " + hex(dataIn[i]));
len = len * 16 + (dataIn[i] > 57 ? dataIn[i] - 87 : dataIn[i] - 48);
// println("-> " + hex(len));
i++;
}
println("Length: " + len + " (" + hex(len) + ")");
i += 2;
gzipData = new byte[len];
arrayCopy(dataIn, i, gzipData, 0, len);
break;
}
}
//*
if (gzipData != null)
{
try
{
XML xml = new XML(new java.util.zip.GZIPInputStream(new java.io.ByteArrayInputStream(gzipData)));
println(xml);
}
catch (Exception e)
{
e.printStackTrace();
}
}
else
{
println("Gzip data not found!");
println(textData);
}
//*/
}
//println(dataIn);
step = -step;
}
}

User-agent isn't needed, but the server wants the client to tell it accepts gzipped data (it won't deliver it unzipped, unless the API has a special parameter for that).

The main problem was that if the request is OK, the answer is a standard HTTP one, with binary data inside.

I guessed that the start of the data (after a double CRLF sequence) is the length of the Gzipped data encoded as a hexadecimal number. After that, I just had to copy this data in a buffer and unzip it to feed the XML parser.

Top Reply

xml retrieval

Replies(2)

Re: xml retrieval

Re: xml retrieval

Statistics

Tags

Actions

Related Posts