Processing Forum

simple question about loadStrings() and URLs

[5 Replies]
- 25-May-2013 04:12 PM
- Forum: Programming Questions
Hello, I'd like a widget to log my Xively (previously Cosm, Pachube) data to file. The data can be grabbed using their API. You enter a URL in your browser and you get CSV results. Eg: http://api.xively.com/v2/feeds/76854.csv gives me the six latest readings from the weather station I've got sending data to Xively in the CSV structure <datastream name>, <timestamp>, <data>.

When I use loadStrings() to try and grab this data I get a 401. Any other URL I enter gives me the text of the html. Eg.

http://processing.org/about/index.html

https://xively.com/dev/docs/api/data/read/

Can anyone tell me why this is the case?

Thanks,

NC

EDIT

Here's some code you can comment/uncomment to test the different URLs. The first two are for my data and don't work. The other two are just web pages and work. One of them is from the Xively site itself so there's no problem connecting to that server in particular.
1. //String[] lines = loadStrings("http://api.xively.com/v2/feeds/76854/datastreams/BMPp.csv");
  
  //String[] lines = loadStrings("http://api.xively.com/v2/feeds/76854.xml");
  
  String[] lines = loadStrings("https://xively.com/dev/docs/api/data/read/");
  
  //String[] lines = loadStrings("http://processing.org/about/index.html");
  
  println(lines);
  
  exit();
If you copy any of these URLs into your browser they all work as desired, just not in Processing.

how to parse byte arrays to 64bit floats?

[2 Replies]
- 13-Feb-2013 01:55 AM
- Forum: General Discussion
Apparently Processing doesn't support doubles. Is there any other way to work with base64 encoded 64bit floats?

converting Strings to base64Binary

[5 Replies]
- 23-Jan-2013 05:45 PM
- Forum: Contributed Library Questions
Hello all, I'm afraid I'm back with another dose of naivety and confusion. I am writing some software to handle large data sets generated by my mass spectrometer. In order to view the results in the open source viewer called TOPPView I need to be able to convert my results to xs:base64Binary to be incorporated into an XML file with the .mzML file extension. It seems that there are ways to do this in Java, using the Apache commons codec library. I have imported this into Processing but I am still struggling to make sense of it. There is example Java code at the end of this document describing a closely related open MS data format. I deleted a few things that Processing objected to and it looks like it runs now but I can't figure out how to feed it strings to convert.
2. import java.util.List;
3. import java.util.Vector;
6. import org.apache.commons.codec.DecoderException;
7. import org.apache.commons.codec.EncoderException;
8. import org.apache.commons.codec.binary.Base64;
10. /**
11. *
12. * @author Kai Runte
13. */
15. public class Base64Util {
17. public String floatListToBase64String(List floatList, boolean bigEndian) throws EncoderException {
18. byte[] raw = floatListToByteArray(floatList, bigEndian);
19. Base64 base64 = new Base64();
20. byte[] encoded = base64.encode(raw);
21. return new String(encoded);
22. }
24. /**
25. * Returns a byte array representing the float values in the IEEE
26. * 754 floating-point "single format" bit layout.
27. * @param floatList a list of float values
28. * @param bigEndian
29. * @return a byte array representing the float values in the
30. * IEEE 754 floating-point "single format" bit layout.
31. */
32. public byte[] floatListToByteArray(List floatList,
33. boolean bigEndian) {
34. int floatListSize = floatList.size();
35. byte[] raw = new byte[floatListSize * 4];
36. int jjj = 0;
37. if (bigEndian) {
38. for (int iii = 0; iii < floatListSize; iii++) {
39. Float aFloat = (Float) floatList.get(iii);
40. int ieee754 = Float.floatToIntBits(aFloat.floatValue());
41. raw[jjj] = (byte) ((ieee754 >> 24) & 0xff);
42. raw[jjj + 1] = (byte) ((ieee754 >> 16) & 0xff);
43. raw[jjj + 2] = (byte) ((ieee754 >> 8) & 0xff);
44. raw[jjj + 3] = (byte) ((ieee754) & 0xff);
45. jjj += 4;
46. }
47. } else {
48. for (int iii = 0; iii < floatListSize; iii++) {
49. Float aFloat = (Float) floatList.get(iii);
50. int ieee754 = Float.floatToIntBits(aFloat.floatValue());
51. raw[jjj] = (byte) ((ieee754) & 0xff);
52. raw[jjj + 1] = (byte) ((ieee754 >> 8) & 0xff);
53. raw[jjj + 2] = (byte) ((ieee754 >> 16) & 0xff);
54. raw[jjj + 3] = (byte) ((ieee754 >> 24) & 0xff);
55. jjj += 4;
56. }
57. }
58. return raw;
59. }
61. public List base64StringToFloatList(String base64String, boolean bigEndian) throws DecoderException {
62. Base64 base64 = new Base64();
63. byte[] encoded = base64String.getBytes();
64. byte[] raw = base64.decode(encoded);
65. List floatList = byteArrayToFloatList(raw, bigEndian);
66. return floatList;
67. }
69. public List byteArrayToFloatList(byte[] raw,
70. boolean bigEndian) {
71. Vector floatList = new Vector();
72. if (bigEndian) {
73. for (int iii = 0; iii < raw.length; iii += 4) {
74. int ieee754 = 0;
75. ieee754 |= (((int) raw[iii]) & 0xff);
76. ieee754 <<= 8;
77. ieee754 |= (((int) raw[iii + 1]) & 0xff);
78. ieee754 <<= 8;
79. ieee754 |= (((int) raw[iii + 2]) & 0xff);
80. ieee754 <<= 8;
81. ieee754 |= (((int) raw[iii + 3]) & 0xff);
82. float aFloat = Float.intBitsToFloat(ieee754);
83. floatList.add(new Float(aFloat));
84. }
85. } else {
86. for (int iii = 0; iii < raw.length; iii += 4) {
87. int ieee754 = 0;
88. ieee754 |= (((int) raw[iii + 3]) & 0xff);
89. ieee754 <<= 8;
90. ieee754 |= (((int) raw[iii + 2]) & 0xff);
91. ieee754 <<= 8;
92. ieee754 |= (((int) raw[iii + 1]) & 0xff);
93. ieee754 <<= 8;
94. ieee754 |= (((int) raw[iii]) & 0xff);
95. float aFloat = Float.intBitsToFloat(ieee754);
96. floatList.add(new Float(aFloat));
97. }
98. }
99. return floatList;
100. }
101. }
This code is obviously for Java and I don't know how to make it work in Processing- or even if it can.
Help plz?

NC

what are the limits of Processing WRT array sizes?

[5 Replies]
- 21-Jan-2013 08:03 PM
- Forum: General Discussion
I have files of CSV values. They contain 10,755,000 lines of 3 CSVs. I've written a sketch to take two of these files and subtract the third value from each line in one file from the third value in the next and rewrite the results to a new file. My problem is that I have to use an int to address lines in the initial file and therefore can't address lines beyond the 32,768th line. I cannot address an array using a long. Can someone confirm that I am not missing something elementary (ha!) and advise me of a way around this issue?

NB1 - When I load the files using loadStrings() it loads the first 2,412,536 lines. Why is this and is there a way around it beyond reading the code in line-by line?

NB2 - I know I can create an arrayList of 10,000,000 elements but I'm curious as to the limits to array().

Cheers,

NC

nesting two selectInput() functions

[8 Replies]
- 16-Jan-2013 12:53 PM
- Forum: Programming Questions
Hello all, hopefully this should be an easy one: I'd like to use the above function to select two different text files containing CSV data and then perform a mathematical operation on the data. Currently its throwing an error and I don't understand why. Have I structured this correctly?
2. /*
3. This code selects two files and performs a mathematical operation on the signal data in each row.
4. */
5. PrintWriter output;
6. String folderPath; // folder path where logfiles will be saved
7. String blankFilePath;
8. String testFilePath;
9. String blankFileName;
10. String[] blankLines;
11. String testFileName;
12. String testFileFolderPath;
13. String outputFile;
14. String[] testLines;
15. long index = 0;
16. String[] blankData;
18. // ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~
20. void setup() {
22. selectInput("Select a blank file:", "blankSelected");
23. }
26. // ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~
28. void blankSelected(File selection) {
29. if (selection == null) {
30. println("Window was closed or the user hit cancel.");
31. exit();
32. }
33. else {
34. blankFilePath = selection.getAbsolutePath();
35. blankFileName = selection.getName();
36. println("blank file selected: " + blankFilePath);
37. blankLines = loadStrings(selection);
38. println(blankLines[0]);
39. selectInput("Select a file to subtract:", "fileSelected");
40. }
42. output = createWriter(outputFile);
44. for(String i : testLines) { // this is the main loop of the sketch - contains just test code for now
46. blankData = split(i, ',');
47. println(blankData[0]);
48. println(index);
52. index++;
53. }
54. exit();
55. }
58. // ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~
60. void fileSelected(File selection) {
61. if (selection == null) {
62. println("Window was closed or the user hit cancel.");
63. exit();
64. }
65. else {
66. testFilePath = selection.getAbsolutePath();
67. testFileFolderPath = selection.getParent();
68. testFileName = selection.getName();
69. String[] tFN = split(testFileName, '-');
70. println("test file selected: " + testFilePath);
71. outputFile = testFileFolderPath + "\\" + "LCMS mathematical operator output - " + tFN[1]; // text file to save output to
72. println("output file: " + outputFile);
73. testLines = loadStrings(selection);
75. println(testLines[0]);
76. }
78. }
The output window produces the following (note that I have prefaced each row of correct output with "$$$$" to highlight where the code has done thigns correctly):
$$$$ blank file selected: C:\Google Drive\Chris Pook AUT\research\lipids LC-MS\2ul injections 12-12-13\ultimate txt writer\LCMS ultimate .txt writer output - 2ul blank crop.txt

$$$$ 590.0,500.0,0

java.lang.reflect.InvocationTargetException

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

at java.lang.reflect.Method.invoke(Method.java:597)

at processing.core.PApplet.selectCallback(PApplet.java:6049)

at processing.core.PApplet.access$000(PApplet.java:153)

at processing.core.PApplet$5.run(PApplet.java:5975)

at java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:209)

at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:646)

at java.awt.EventQueue.access$000(EventQueue.java:84)

at java.awt.EventQueue$1.run(EventQueue.java:607)

at java.awt.EventQueue$1.run(EventQueue.java:605)

at java.security.AccessController.doPrivileged(Native Method)

at java.security.AccessControlContext$1.doIntersectionPrivilege(AccessControlContext.java:87)

at java.awt.EventQueue.dispatchEvent(EventQueue.java:616)

at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:269)

at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:184)

at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:174)

at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:169)

at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:161)

at java.awt.EventDispatchThread.run(EventDispatchThread.java:122)

Caused by: java.lang.NullPointerException

at java.io.File.<init>(File.java:222)

at processing.core.PApplet.saveFile(PApplet.java:7024)

at processing.core.PApplet.createWriter(PApplet.java:6168)

at LCMS_mathematical_operator.blankSelected(LCMS_mathematical_operator.java:61)

... 21 more

$$$$ test file selected: C:\Google Drive\Chris Pook AUT\research\lipids LC-MS\2ul injections 12-12-13\ultimate txt writer\LCMS ultimate .txt writer output - 2ul 500mg.l tristearin crop.txt

$$$$ output file: C:\Google Drive\Chris Pook AUT\research\lipids LC-MS\2ul injections 12-12-13\ultimate txt writer\LCMS mathematical operator output - 2ul 500mg.l tristearin crop.txt

java.lang.reflect.InvocationTargetException

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

at java.lang.reflect.Method.invoke(Method.java:597)

at processing.core.PApplet.selectCallback(PApplet.java:6049)

at processing.core.PApplet.access$000(PApplet.java:153)

at processing.core.PApplet$5.run(PApplet.java:5975)

at java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:209)

at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:646)

at java.awt.EventQueue.access$000(EventQueue.java:84)

at java.awt.EventQueue$1.run(EventQueue.java:607)

at java.awt.EventQueue$1.run(EventQueue.java:605)

at java.security.AccessController.doPrivileged(Native Method)

at java.security.AccessControlContext$1.doIntersectionPrivilege(AccessControlContext.java:87)

at java.awt.EventQueue.dispatchEvent(EventQueue.java:616)

at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:269)

at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:184)

at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:174)

at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:169)

at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:161)

at java.awt.EventDispatchThread.run(EventDispatchThread.java:122)

Caused by: java.lang.OutOfMemoryError: Java heap space

at java.util.Arrays.copyOfRange(Arrays.java:3209)

at java.lang.String.<init>(String.java:215)

at java.io.BufferedReader.readLine(BufferedReader.java:331)

at java.io.BufferedReader.readLine(BufferedReader.java:362)

at processing.core.PApplet.loadStrings(PApplet.java:6646)

at processing.core.PApplet.loadStrings(PApplet.java:6633)

at processing.core.PApplet.loadStrings(PApplet.java:6567)

at LCMS_mathematical_operator.fileSelected(LCMS_mathematical_operator.java:92)

... 21 more

Many thanks,

NC

why does writing to arrays massively slow my sketch down?

[5 Replies]
- 10-Jan-2013 06:55 PM
- Forum: Programming Questions
Hello Processing types, I have a query about my sketch. I am reading data from a text file and creating from it a new text file of comma separated values. The input files are about 10MB and the output files about 50MB. The extra size comes from filling gaps in the data with zero values. Ultimately I want to do maths operations with two files, subtracting one from another to determine differences. For this purpose I adapted my sketch to put the data into arrays so instead of a text file containing columns of three CSV values I would have three arrays of similar length. However, the code below is now 100x slower to run than without the array writing code. Can someone suggest why this is and hopefully a way of speeding the code up? Instead of taking 20 seconds to run its now been about 20 minutes and its barely 20% through.
Apologies for the crudeness of my code. I am very much an enthusiastic amateur at this game.
2. /*
3. This code takes a text file of LCMS data produced by MSConvertGUI and produces another text
4. file in the same folder containing three colummns of CSV data which hold time (seconds), m/z
5. and signal of the LCMS TIC chromatogram.
6. */
8. import java.io.File;
9. PrintWriter output;
10. String folderPath; // folder path where logfiles will be saved
11. int count;
12. String time;
13. String mz;
14. String signal;
15. int index = 0;
16. int present = 1;
17. Float floatTime;
18. Float roundedTime;
20. Float[] timeArray = new Float[1];
21. Float[] mzArray = new Float[1];
22. String[] signalArray = new String[1];
26. // ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~
28. void setup() {
30. selectInput("Select a file to process:", "fileSelected");
31. }
34. // ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~
36. void fileSelected(File selection) {
37. if (selection == null) {
38. println("Window was closed or the user hit cancel.");
39. exit();
40. }
41. else {
42. println("User selected " + selection.getAbsolutePath());
43. }
45. folderPath = selection.getParent(); // folder path to save to
46. String name = selection.getName();
47. // println(folderPath);
48. String file = folderPath + "\\" + "LCMS ultimate .txt writer output - " + name; // text file to save to
50. String lines[] = loadStrings(selection);
52. output = createWriter(file);
53. println(file);
56. for(String i : lines) { // this is the main loop of the sketch
58. int nums = i.indexOf("["); // identifies which lines hold m/z or signal data
59. int secs = i.indexOf("cvParam: scan start time,"); // identifies which lines hold the time signature in seconds
61. if(nums > 0 || secs > 0) { // what to do with those lines
63. if(nums > 0){
64. String[] Trim = split(i, ']'); // cuts out the text guff
65. Trim = trim(Trim); // trims off whitespace to leave a number
66. if(index == 2) { signal = Trim[1];} // saves result in one of the ints
67. else {mz = Trim[1];}
68. index++;
69. }
71. else {
72. String[] Trim = split(i, ','); // does same as above for time result
73. time = trim(Trim[1]);
74. floatTime = float(time) * 1000; // rounds to 3 sig figs
75. roundedTime = round(floatTime) / 1000.0;
76. index++;
77. }
79. }
81. if (index == 3) { // every three lines write the data as a line of CSV values
82. String[] mzData = split(mz," ");
83. String[] signalData = split(signal," ");
84. count = mzData.length;
86. for(int u = 5000; u < 9501; u++) {
87. output.print(roundedTime);
88. output.print(", ");
89. timeArray = (Float[]) append(timeArray,roundedTime); // this is new array writing code
90. Float thisMZ = u / 10.0;
91. output.print(thisMZ);
92. output.print(", ");
93. mzArray = (Float[]) append(mzArray, thisMZ); // this is new array writing code
95. String mzString = str(thisMZ);
96. present = 0;
97. for(int e = 0; e < count; e++) {
99. if(mzString.equals(mzData[e]) == true) {
100. output.println(signalData[e]);
101. signalArray = append(signalArray, signalData[e]); // this is new array writing code
102. present = 1;
103. }
104. }
106. if(present < 1) {
107. output.println("0");
108. signalArray = append(signalArray, "0"); // this is new array writing code
109. }
110. present = 1;
111. }
113. println(roundedTime);
115. output.flush();
117. }
119. if(index > 2) {index = 0;}
120. }
123. output.flush();
125. output.println(" "); // this is all new array writing code
126. output.println(timeArray);
127. output.flush();
128. output.println(mzArray);
129. output.flush();
130. output.println(signalArray);
131. output.flush();
133. output.close();
136. exit();
138. }
In hindsight I can probably see a way of doing the mathematical operations using the text files using loadStrings() to put the CSV rows into an array and doing the calculations a row at a time. The zero-filling is designed to make the layout of the data identical and enables this. Maybe I will just proceed along these lines but I am still very interested to know why writing the data to arrays slows it down so much.

Many thanks in advance for your generously dispensed wisdom.

converting CSV data in a text file into an array - solved I think :)

[0 Replies]
- 26-Oct-2012 02:59 AM
- Forum: General Discussion
Hi, I've got a sketch logging CSV data from an Arduino, sent via the serial port, to a text file and also plotting it.The data indicates the state of a 3-way solenoid valve switching between two streams of milk flowing through a bioreactor. A pair of thermocouples measure the inflow and outflow temperature. The data is seconds run for, temp A, temp B and valve state. The sketch runs for days and data is sent at 0.2Hz so there's a fair amount of data there. What I'd like to do is adapt this sketch to logging data from HPLC detectors. These transmit analogue data as 0-1V, which I intend to process with an 18-bit ADC breakout for my Arduino.

My problem is that I like logging CSV to text files with a new line for every data point but I want to exploit the subset() function for array data, inconjunction with click-and drag, to zoom in on areas of the plot. This data will only be logged for an hour or so but at 10Hz so still tens of thousands of data. I have worked out how to import CSV data into an array with loadStrings() and then parse it into individual arrays for each column of values plot it but I want to know if there's an easier way of doing this than reading in and processing the array data every SerialEvent()?

~ <BRAINWAVE!> ~

The solution is obvious, now I write it down! Instead of working from a single repository, in the text file, and wasting processor time reading it in and out I just maintain parallel data streams, logging new data to a new line in the text file and appending it to an array in memory. I can maintain a second subset array of data to be plotted whose contents will be dictated by the region of the plot selected for display- zoomed or otherwise.

Simples!

Now I've written it I'll leave this here in case anyone else faces the same issue.

accumulating data in arrays

[3 Replies]
- 06-Jun-2012 02:44 PM
- Forum: Programming Questions
Hi, I'm a massive noob to programming and I'm trying to bash together some code to plot data from the serial cable sent by my Arduino. The data comes in as individual lines of comma separated values from a few temperature and light sensors. Using code from this thread I can split the CSV values and convert them to floats but I'm struggling to work out how then to add them to an array for real time plotting. I can't find the blog that I got the plotting code from but it needs the data as arrays.Can someone help me out?

Many thanks,
2. import processing.serial.*; // serial library
4. Serial myPort; // The serial port
5. String inBuffer = null; // one line of data from serial port
8. void setup () {
10. // List all the available serial ports
11. println(Serial.list());
13. myPort = new Serial(this, Serial.list()[2], 9600);
15. }
18. void draw () {
19. // everything happens in the serialEvent()
20. }
22. void serialEvent (Serial myPort) {
23. // get the ASCII string:
24. String inString = myPort.readStringUntil('\n');
26. if (inString != null) {
27. // trim off any whitespace:
28. inString = trim(inString);
30. float[] dat; // array of numbers read in on one line from serial port
32. println("SERIAL:" + inString); // show the line of serial input
33. dat = float(split(inString, ',')); // parse comma-separated number string into numbers
34. println("DAT_IN:" + dat[0] + "," + dat[1] + "," + dat[2]);
35. println();
36. float[] arrayA = dat[0];
37. float[] arrayB = dat[1];
38. float[] arrayC = dat[2];
41. println(arrayA);
43. println(arrayB);
45. println(arrayC);
47. }
49. }

Activity Trend