We closed this forum 18 June 2010. It has served us well since 2005 as the ALPHA forum did before it from 2002 to 2005. New discussions are ongoing at the new URL http://forum.processing.org. You'll need to sign up and get a new user account. We're sorry about that inconvenience, but we think it's better in the long run. The content on this forum will remain online.
IndexProgramming Questions & HelpIntegration › Reading Python Script Output
Page Index Toggle Pages: 1
Reading Python Script Output (Read 3394 times)
Reading Python Script Output
Feb 10th, 2010, 7:11am
 
I've got a python script that reads data from some speciality hardware and massages the data. I'd like to use processing to display that data, and assumed there would be some way to having processing call the python script (or any bash/windows cmd.exe command) and read the lines as they're output.

I've pieced this together from forum postings and it seems to do more or less what I'd like:

Code:

import java.io.*;

void setup() {
 size(640,480);
 try {
   String line;
   OutputStream stdin = null;
   
   Process p = Runtime.getRuntime().exec("ping google.com -n 100");
   
   BufferedReader input = new BufferedReader(new InputStreamReader(p.getInputStream()));
 
   while ((line = input.readLine()) != null) {
      println(line);
   }
 } catch (Exception err) {
   err.printStackTrace();
 }
}


As this is written, if I add a draw() routine it isn't called until the ping command has exited (as expected, but not as desired). What I'd like is for the lines of input to be read in to a buffer, which is then checked and acted upon during each call of draw(). Could someone post a quick example of this? Perhaps drawing one background color when there isn't a new line in the buffer, and another when there is?
Re: Reading Python Script Output
Reply #1 - Feb 10th, 2010, 8:43am
 
You need to run the code in a separate thread. Plenty of examples on this forum.
Re: Reading Python Script Output
Reply #2 - Feb 11th, 2010, 8:34am
 
Thanks for the tip, I suspected as much.

I've implemented a version with threads (mostly by poking around the shiffman tutorial and java docs) and it works very well except for one (potentially deal-breaking) issue. The values make it python okay, but with a very consistent half-second or so lag. Here is the code in question...

Code:

import java.io.*;

DataThread ping_thread;
float time = 0;
int min = 160;
int max = 170;
int index = 0;
float[] numbers;

void setup() {
size(640,480);
frameRate(33);
numbers = new float[100];
ping_thread = new DataThread();
ping_thread.start();
background(255);
fill(0,40);
stroke(0,40);
smooth();
}

void draw() {
background(255);
time += 0.1/20;
stroke(0,30);
line(0,height/2-height/8,width,height/2-height/8);
line(0,height/2+height/8,width,height/2+height/8);
line(0,height/2+height-height/6,width,height/2+height-height/6);
numbers[index % numbers.length] = ping_thread.getLatest(0);
//println(ping_thread.getLatest(0));
beginShape();
for (int i = 0; i < numbers.length + 1; i++) {
float value = numbers[(index+i+1)%numbers.length];
float spacing = (float) width / (float) numbers.length;
value = map(value,5000,pow(2,16),height/2+height/8,height/2-height/8);
noFill();
curveVertex(spacing * i, value);
//ellipse(spacing * i, value, 20, 20);
stroke(0,180);
}
endShape();
index = (index + 1) % numbers.length;
}

/* Classes */

public class DataThread extends Thread {
private boolean running; // Is the thread running? Yes or no?
private boolean fresh; // Is there fresh data to be polled?
int[] labjack_fields = new int[16];

// Constructor, create the thread
// It is not running by default
public DataThread() {
running = false;
}

public void start()
{
// Set running equal to true
running = true;
fresh = false;
// Print messages
System.out.println("Starting data thread...");
// Do whatever start does in Thread, don't forget this!
super.start();
}

public void run () {
try {
String line;
OutputStream stdin = null;

Process p = Runtime.getRuntime().exec("python C:/code/grabber.py");

BufferedReader input = new BufferedReader(new InputStreamReader(p.getInputStream()));

while ((line = input.readLine()) != null) {
String[] m = split(line, " ");
labjack_fields[int(m[0])] = int(m[1]);
Thread.currentThread().sleep(1); # Mysteriously Important Sleep Line
}
} catch (Exception err) {
err.printStackTrace();
}
}

public int getLatest(int index) {
return labjack_fields[index];
}

}


The python script's output is along the lines of:

Code:

0 193411
1 56782
2 343434
3 893
0 342
1 1111
2


With on the order of a few hundred lines being output per second. The line commented "mysteriously important sleep line" was added on a spooky hunch after initial tests showed that the strip chart was only being updated every half-second or so with the last sample received from the python script... it cleared that issue up (now every sample is received beautifully) but a delay occurs somewhere between the python script and the screen.

What can I do to get rid of the delay? Is the buffer simply too large? Can I change the size?

Thanks for your help.
Re: Reading Python Script Output
Reply #3 - Feb 12th, 2010, 4:28am
 
Changing the buffer size didn't seem to have an effect. I'm really lost as to how to approach this.
Re: Reading Python Script Output
Reply #4 - Feb 15th, 2010, 11:36pm
 
Would creating am array to act as buffer in the data thread potentially help? Perhaps some sort of locking?

I'm sorry to bump this thread again, but it's holding me up on a very important project. Are there other (Java?) communities I could check out for help, or documentation I should read?
Re: Reading Python Script Output
Reply #5 - Feb 16th, 2010, 1:45am
 
I don't have any experience reading from streams in Java, so the discussion below is wild speculation...

If you sleep(1) all the time, on average you're going to have a half-second delay (sometimes almost no delay, sometimes about 1 second delay, mostly somewhere between).

I'm not entirely sure what you mean by "initial tests showed that the strip chart was only being updated every half-second or so with the last sample received from the python script".

Did you try displaying what values the sketch was reading?

Code:
while ((line = input.readLine()) != null) {
 println("line: [" + line + "]"); // #
 String[] m = split(line, " ");
 println(m);


Edit: # left off a closing parenthesis.

My sneaking suspicion is that perhaps the lines are incomplete, but perhaps BufferedReader.readLine() waits for a complete line to become available.

If a half-second delay is too long, have you considered a smaller delay How responsive does it need to be

-spxl
Re: Reading Python Script Output
Reply #6 - Feb 16th, 2010, 6:37am
 
subpixel: Thanks for the response.

Before I address anything else I should point out that the delay is actually around 1.5-2 seconds - I am just terrible at estimating.

The sleep() delay in the data thread's while loop is not 1 second, but 1 millisecond. If I remove that line the strip chart (and gauges, and printed values) are updated around once every 2 seconds.

To make this more concrete, let me give some examples. If the sleep statement is included as above and the python script is outputting a sine wave at 0.01Hz, it is drawn on the stripchart beautifully but with a constant ~2 second delay. If the sleep statement is removed the strip chart's plot is stepped (e.g. http://www.solarray.com/Images/ImagesTGuides/Stepped%20Wave.png) with the jumps occurring every ~2 seconds.

I have discovered that I can use the "nanosecond" argument to sleep() and get a slight reduction in the delay, but if I set it too low the stripcharts/gauges stop updating entirely.

Here are the docs for the sleep function, amongst others.
http://java.sun.com/j2se/1.4.2/docs/api/java/lang/Thread.html

It seems to me that the problem is the sleep statement in one way or another, and that it shouldn't be necessary. Why then does the input stutter without it
Re: Reading Python Script Output
Reply #7 - Feb 16th, 2010, 8:44am
 
Oops. Bad guess by me that the sleep() parameter was in seconds.

Nevertheless, on with other possibilities!

I didn't look at your code before, so didn't know what it was doing. I've copied it into the PDE now to play with it. I don't have your python sctipt / hardware / input, so manufacture my own data in DataThread.run().

I now see there are at least two issues here:

1) is the data being read correctly and in a timely manner by the DataThread.run() method?

How are you measuring this ~2 second delay? How do you know when the Python script is outputting the values? Are you generating a signal (sine wave or otherwise) from some hardware after a period of "no signal" and not seeing a response in your sketch for 2 seconds? Otherwise, how do you know what the phase should be (are you looking at an oscilloscope or something?)

Possibility: the Python output is buffered (in/by Python) before it gets to your readline(). Look for a flush() method or similar.

If you take a dump of the Python output (not called from your sketch), say two seconds worth, does the data appear as expected (matching your input in time, "smoothly" changing values)?

If you stick println("line: [" + line + "]"); into DataThread.run(), does it spit out something like the raw output from the Python program?

2) what is the data (supposing it is being emitted at the expected time, and is being received at the expected time) supposed to mean?

Your sample data is very short, and without much explanation. Why output values into array positions 0, 1, 2, 3, 0, 1, 2...? Is this reporting a single reading, or is it 4 separate inputs, of which only the first you are interested in plotting? (The current program only plots values obtained via ping_thread.getLatest(0)).
Re: Reading Python Script Output
Reply #8 - Feb 16th, 2010, 9:13am
 
YESSSSSSSSSSSSSSSSS!!!

Thank you so much subpixel! python was buffering after all! How obvious is that in retrospect?

I see from your site you're in australia, so I'm unlikely to be there in the near future, but if you're ever in Washington DC, I'll buy you a drink.
Re: Reading Python Script Output
Reply #9 - Feb 16th, 2010, 12:20pm
 
Tip: always test your input before you worry about your output. Garbage in, garbage out!

List me in the "thanks to" section on your project?... (Unless it is a school thing and you're supposed to pretend you did it all yourself)
Re: Reading Python Script Output
Reply #10 - Feb 19th, 2010, 8:10pm
 
I'd love to, but unfortunately it's a project I'm working on for a very small company. Approximately half a dozen people will ever use it.
Page Index Toggle Pages: 1