converting Strings to base64Binary

NinjaChris

converting Strings to base64Binary

in Contributed Library Questions • 8 months ago

Hello all, I'm afraid I'm back with another dose of naivety and confusion. I am writing some software to handle large data sets generated by my mass spectrometer. In order to view the results in the open source viewer called TOPPView I need to be able to convert my results to xs:base64Binary to be incorporated into an XML file with the .mzML file extension. It seems that there are ways to do this in Java, using the Apache commons codec library. I have imported this into Processing but I am still struggling to make sense of it. There is example Java code at the end of this document describing a closely related open MS data format. I deleted a few things that Processing objected to and it looks like it runs now but I can't figure out how to feed it strings to convert.

import java.util.List;
import java.util.Vector;
import org.apache.commons.codec.DecoderException;
import org.apache.commons.codec.EncoderException;
import org.apache.commons.codec.binary.Base64;
/**
*
* @author Kai Runte
*/
public class Base64Util {
public String floatListToBase64String(List floatList, boolean bigEndian) throws EncoderException {
byte[] raw = floatListToByteArray(floatList, bigEndian);
Base64 base64 = new Base64();
byte[] encoded = base64.encode(raw);
return new String(encoded);
}
/**
* Returns a byte array representing the float values in the IEEE
* 754 floating-point "single format" bit layout.
* @param floatList a list of float values
* @param bigEndian
* @return a byte array representing the float values in the
* IEEE 754 floating-point "single format" bit layout.
*/
public byte[] floatListToByteArray(List floatList,
boolean bigEndian) {
int floatListSize = floatList.size();
byte[] raw = new byte[floatListSize * 4];
int jjj = 0;
if (bigEndian) {
for (int iii = 0; iii < floatListSize; iii++) {
Float aFloat = (Float) floatList.get(iii);
int ieee754 = Float.floatToIntBits(aFloat.floatValue());
raw[jjj] = (byte) ((ieee754 >> 24) & 0xff);
raw[jjj + 1] = (byte) ((ieee754 >> 16) & 0xff);
raw[jjj + 2] = (byte) ((ieee754 >> 8) & 0xff);
raw[jjj + 3] = (byte) ((ieee754) & 0xff);
jjj += 4;
}
} else {
for (int iii = 0; iii < floatListSize; iii++) {
Float aFloat = (Float) floatList.get(iii);
int ieee754 = Float.floatToIntBits(aFloat.floatValue());
raw[jjj] = (byte) ((ieee754) & 0xff);
raw[jjj + 1] = (byte) ((ieee754 >> 8) & 0xff);
raw[jjj + 2] = (byte) ((ieee754 >> 16) & 0xff);
raw[jjj + 3] = (byte) ((ieee754 >> 24) & 0xff);
jjj += 4;
}
}
return raw;
}
public List base64StringToFloatList(String base64String, boolean bigEndian) throws DecoderException {
Base64 base64 = new Base64();
byte[] encoded = base64String.getBytes();
byte[] raw = base64.decode(encoded);
List floatList = byteArrayToFloatList(raw, bigEndian);
return floatList;
}
public List byteArrayToFloatList(byte[] raw,
boolean bigEndian) {
Vector floatList = new Vector();
if (bigEndian) {
for (int iii = 0; iii < raw.length; iii += 4) {
int ieee754 = 0;
ieee754 |= (((int) raw[iii]) & 0xff);
ieee754 <<= 8;
ieee754 |= (((int) raw[iii + 1]) & 0xff);
ieee754 <<= 8;
ieee754 |= (((int) raw[iii + 2]) & 0xff);
ieee754 <<= 8;
ieee754 |= (((int) raw[iii + 3]) & 0xff);
float aFloat = Float.intBitsToFloat(ieee754);
floatList.add(new Float(aFloat));
}
} else {
for (int iii = 0; iii < raw.length; iii += 4) {
int ieee754 = 0;
ieee754 |= (((int) raw[iii + 3]) & 0xff);
ieee754 <<= 8;
ieee754 |= (((int) raw[iii + 2]) & 0xff);
ieee754 <<= 8;
ieee754 |= (((int) raw[iii + 1]) & 0xff);
ieee754 <<= 8;
ieee754 |= (((int) raw[iii]) & 0xff);
float aFloat = Float.intBitsToFloat(ieee754);
floatList.add(new Float(aFloat));
}
}
return floatList;
}
}

This code is obviously for Java and I don't know how to make it work in Processing- or even if it can.

Help plz?

Replies(5)

PhiLho

Re: converting Strings to base64Binary

8 months ago

Side answer: no need for special library, now...
First hit of the search of Base64 in the forum: https://forum.processing.org/topic/base64-encode-string

NinjaChris

Re: converting Strings to base64Binary

8 months ago

Hi PhiLho, sorry, I should have mentioned that I tried that code on some data I copied from a results file and it produced gibberish.

Here's my adaptation:

import javax.xml.bind.DatatypeConverter;
void setup()
{
// String b64 = DatatypeConverter.printBase64Binary("Java has well hidden resources!".getBytes());
// println(b64);
String input = "zcwSRDPTEkSa2RJEAOASRJq5F0QAwBdEZsYXRM3MF0Qz0xdEmtkXRADgF0Rm5hdEzewXRAAAGERmBhhEzQwYRDMTGESaGRhEACAYRADAIERmxiBEzcwgRDPTIESa2SBEAOAgRGbmIETN7CBEM/MgRJr5IEQAACFEZgYhRM0MIUQzEyFEmhkhRAAgIURmJiFEzSwhRGamJUTNrCVEM7MlRJq5JUQAwCVEZsYlRM3MJUQz0yVEmtklRADgJURm5iVEzewlRDPzJUSa+SVEAAAmRGYGJkTNDCZEMxMmRJoZJkQAICZEZiYmRM0sJkQzMyZEmjkmRABAJkRmRiZEzUwmRDNTJkSaWSZEAGAmRGZmJkTNbCZEM3MmRJp5JkQAgCZEZoYmRM2MJkQzkyZEmpkmRACgJkRmpiZEzawmRDOzJkTNzCZEM9MmRJrZJkQA4CZEmrknRADAJ0RmxidEzcwnRDPTJ0Sa2SdEAOAnRGbmJ0TN7CdEM/MnRJr5J0QAAChEZgYoRM0MKEQzEyhEmhkoRAAgKERmJihEzSwoRDMzKESaOShEAEAoRM1MKEQzUyhEmlkoRABgKERmZihEzWwoRAAAKURmBilEzQwpRDMTKUSaGSlEACApRGYmKUTNLClEMzMpRJo5KUTNTClEM1MpRJpZKUQAYClEZmYpRM1sKUSa2SlEzQwqRDMTKkSaGSpEACAqRGYmKkTNLCpEzUwqRDNTKkSaWSpEAGAqRGZmKkTNLCtEMzMrRJo5K0QAQCtEZkYrRM1MK0QzUytEmlkrRABgK0RmZitEzWwrRDNzK0SaeStEAIArRGaGK0TNjCtEM5MrRJqZK0QAoCtEZqYrRM2sK0QzsytEZsYrRM3MK0Qz0ytEmtkrRADgK0Rm5itEzewrRDPzK0RmRi1EzUwtRDNTLUSaWS1EAGAtRGZmLUTNbC1EM3MtRGaGLUTNjC1EM5MtRJqZLUQAoC1EAMAuRGbGLkTNzC5EM9MuRJrZLkQA4C5EZuYuRM3sLkQz8y5EmvkuRAAAL0RmBi9EzQwvRDMTL0SaGS9EACAvRGYmL0TNLC9EMzMvRJo5L0QAQC9EZkYvRM1ML0QzUy9EmlkvRABgL0RmZi9EzWwvRDNzL0SaeS9EZoYvRM2ML0Qzky9EmpkvRACgL0Rmpi9EmtkvRADgL0Rm5i9EmvkvRAAAMERmBjBEzQwwRDMTMESaGTBEACAwRGYmMETNLDBEMzMwRJo5MEQAQDBEZkYwRM1MMEQzUzBEmlkwRABgMERmZjBEzWwwRDNzMESaeTBEM5MwRJqZMEQAoDBEZqYwRM2sMERmBjFEzQwxRDMTMUSaGTFEACAxRGYmMUTNLDFEAEAxRGZGMUTNTDFEM1MxRJpZMUQAYDFEZmYxRM1sMUQzczFEAEA0RGZGNETNTDREM1M0RJpZNEQAYDREZmY0RM1sNEQzczREmnk0RACANERmhjREzYw0RDOTNESamTREAKA0RGamNETNrDREM7M0RJq5NEQAwDREZsY0RM3MNEQz0zREmtk0RADgNERm5jREzew0RDPzNEQzEzVEmhk1RAAgNURmJjVEZsY2RM3MNkQz0zZEmtk2RADgNkRm5jZEzew2RAAAN0RmBjdEzQw3RDMTN0SaGTdEACA3RGYmN0TNLDdEMzM3RJo5N0QAQDdEZkY3RM1MN0QzUzdEmlk3RABgN0RmZjdEzWw3RDNzN0SamTdEAKA3RDMTOESaGThEmlk4RABgOERmZjhEM3M7RJp5O0QAgDtEZoY7RM2MO0QzkztEmpk7RACgO0RmpjtEzaw7RDOzO0SauTtEAMA7RGbGO0TNzDtEM9M7RJrZO0QA4DtEZuY7RM3sO0Qz8ztEmvk7RAAAPERmBjxEzQw8RDMTPESaGTxEACA8RGYmPETNLDxEMzM8RDNTPESaWTxEAGA8RJoZP0QAID9EzQxNRDMTTUSaGU1EACBNRGYmTUTNLE1EMzNNRJpZTUQAYE1EZmZNRM1sTUSaWVNEAGBTRJo5V0QAQFdEZkZXRM1MV0QzU1dEmllXRABgV0RmZldEzWxXRDNzV0SaeVdEAIBXRGaGV0TNjFdEM5NXRJqZV0QAoFdEZqZXRM2sV0Qzs1dEmrlXRDPTV0Sa2VdEAOBXRGbmV0TN7FdEMzNbRJo5W0QAQFtEZkZbRM1MW0QzU1tEmllbRABgW0RmZltEzWxbRDNzW0SaeVtEAIBbRGaGW0TNjFtEM5NbRJqZW0QAoFtEZqZbRM2sW0Qzs1tEmrlbRADAW0RmxltEzcxbRDPTW0Sa2VtEAOBbRGbmW0TN7FtEM/NbRJrZZEQA4GREAMBoRGbGaETNzGhEM9NoRJrZaEQA4GhEZuZoRM3saEQz82hEmvloRAAAaURmBmlEzQxpRDMTaUSaGWlEACBpRGYmaUTNLGlEMzNpRJo5aUTNTGlEM1NpRJpZaUQAYGlEZmZpRM1saURmxmxEzcxsRDPTbESa2WxEAOBsRGbmbETN7GxEM/NsRJr5bETNDG1EMxNtRJoZbUQAIG1EZiZtRM0sbUQ=";
byte[] origB = DatatypeConverter.parseBase64Binary(input);
String orig = new String(origB);
println(orig);
exit();
}

The input String is the line of data I copied from my .mzData results file. A copy of the file can be found here. The data is encoded between the first pair of <mzArrayBinary> tags.

I've just tried to decode the first line of data in this .mzML version of the same file with similarly gibberish results.

Your code is much shorter than the example code I pasted in my original post, PhiLho. Can you see how it could be simplified?

EDIT

The descriptor data for the mz array in the .mzML file is as follows:

<binaryDataArray encodedLength="448">

<cvParam cvRef="MS" accession="MS:1000514" name="m/z array" unitAccession="MS:1000040" unitName="m/z" unitCvRef="MS" />

<cvParam cvRef="MS" accession="MS:1000523" name=" 64-bit float" />

<cvParam cvRef="MS" accession="MS:1000576" name="no compression" />

This same information for the corresponding intensity array is:

<binaryDataArray encodedLength="224">

<cvParam cvRef="MS" accession="MS:1000515" name="intensity array" unitAccession="MS:1000131" unitName="number of counts" unitCvRef="MS"/>

<cvParam cvRef="MS" accession="MS:1000521" name="32-bit float" />

<cvParam cvRef="MS" accession="MS:1000576" name="no compression" />

So the MZ values are 64 bit floats and the intensity values are 32 bit floats. Does this make a difference?

Similarly, in the .mzData file, the description of the MZ data is:

<data precision="32" endian="little" length="479">

and that for the MZ data:

<data precision="32" endian="little" length="479">

Here the data precisions are the same but the little Endian parameter is 'little', whereas the example code in my original post frequently mentions 'bigEndian'.

I am so confused by this it is driving me nuts.

NinjaChris

Re: converting Strings to base64Binary

8 months ago

The encoding is not base64. There are plenty of online converters which cannot parse the data into anything legible. I've also downloaded Notepad++ and installed the MIME toolkit with its base64 text decoder and that produces nothing legible from the data either. Here's an example data string:

cpxSR2bnPUihaNxIFFA8Sc+8gUnWuJxJi8ywSW5ju0npWrFJHW2MSZAeMklcZ71IjrBySF2ci0gdecRI0vD/SM4ZF0m1ZSJJBY8bSZJl/0intaxIlQJBSPdY3EdBLM9Hp4wLSOK8Nki8L1pIoglxSAvNbkjHg0VIDlv7R6B2Y0elCqVGFWuVRgagAkdxHTdHELxRR9i2VkdyEUlHU84kRzTS2kalKFxG

This should convert into the following string:

53916.4453125 194461.59375 451397.03125 771329.25 1062809.875 1283866.75 1448337.375 1535085.75 1452893.125 1150371.625 729577 387898.875 248514.21875 285922.90625 402376.90625 524166.5625 618908.875 665179.3125 637168.3125 523052.5625 353709.21875 197642.328125 112817.929688 106072.507813 142898.609375 187123.53125 223422.9375 246822.53125 244532.171875 202255.109375 128694.109375 58230.625 21125.3222656 19125.5410156 33440.0234375 46877.4414063 53692.0625 54966.84375 51473.4453125 42190.3242188 28009.1015625 14090.1611328

Please do try it for yourself.

NinjaChris

Re: converting Strings to base64Binary

8 months ago

Okay, David Schwartz at StackOverflow has pointed out to me that the floats I am expecting to see when I decode the base64 are encoded in 32 bit binary and not plaintext. Here is the source of my confusion, frustration and general ennui.

Splendid!

NinjaChris

Re: converting Strings to base64Binary

8 months ago

In case anyone other raw n00bs encounter similar challenges and confusion as I have I will post three short sketches here which do useful things in relation to base64 encoded data. I will describe what I now understand this encoding to mean: base64 encrypts a string of bytes into a series of ascii characters. Each three bytes becomes four characters.

32bit floats are represented by 4 bytes (4 x 8bits = 32bits). A string of three floats, represented by 12 bytes will be encoded in base64 to 16 characters.

The following sketches are not my own work but merely adaptations of existing code that Google threw up.

This one converts an array of floats to an array of bytes ready for base64 encoding.

import java.nio.*;
float[] input = { 14554.261, 18106.002, 17326.701 }; // example data
byte byteArray[] = new byte[input.length*4];
// wrap the byte array to the byte buffer
ByteBuffer byteBuf = ByteBuffer.wrap(byteArray);
// create a view of the byte buffer as a float buffer
FloatBuffer floatBuf = byteBuf.asFloatBuffer();
// now put the float array to the float buffer,
// it is actually stored to the byte array
floatBuf.put (input);
println(byteArray);
exit();

This one (actually one of PhiLho's examples) converts a text string of characters to base64.

import javax.xml.bind.DatatypeConverter;
String input = "Java has well hidden resources!";
byte[] byteInput;
void setup()
{
byteInput = input.getBytes();
String b64 = DatatypeConverter.printBase64Binary(byteInput);
println(b64);
byte[] origB = DatatypeConverter.parseBase64Binary(b64);
String orig = new String(origB);
println(orig);
exit();
}

This one converts a string of characters encoding little endian 32bit floats in base64 to an array of floats.

import javax.xml.bind.DatatypeConverter;
import java.nio.*;
float[] floaty = new float[0];
String[] DATA = new String[1];
DATA[0] = "C2ljRgF0jUZnXYdG"; // three little endian 32bit floats encoded in base 64
byte[] input = DatatypeConverter.parseBase64Binary(DATA[0]);
println(input);
int inputLength = input.length;
println(inputLength);
for (int i = 0; i < inputLength; i = i + 4) {
byte[] data = new byte[4];
data[0] = input[i];
data[1] = input[i + 1];
data[2] = input[i + 2];
data[3] = input[i + 3];
ByteBuffer buff;
buff = ByteBuffer.wrap(data);
buff = buff.order(ByteOrder.LITTLE_ENDIAN);
floaty = append(floaty, buff.getFloat());
}
println(floaty);
exit();