Reliably send images over net

TorbjornLunde

Hello,

I’m trying to transfer images over net between Processing-sketches on different computers. I’ve been making making some small test sketches to test this out. I’m using built-in java library to convert PImage to JPEG-encoded byte-arrays (to keep size down) which I in turn send over the net.

However, I’ve never really done much networking programming before and I’m having trouble getting this to work reliably. Generally my problem is that very often I just get a part of the image, but not the whole thing.

I am not really looking for code help per se, just what approach I should take. Do I need to send the image in parts and then stitch it together (or the corresponding bytes together)? Will I then need some way to make sure the server know it received a whole part? Or should I just read from the buffer in a different way? Are there any ready-made solutions like libraries that can deal with this?

And generally my goal here is simply to transfer a image from one computer to another, am I going about this in the easiest and smartest way? Would be easier to just bounce this stuff off a web server somewhere?

Any pointers here would be hugely appreciated.

Here is my server code:

import processing.net.*;

Server server;
JPGEncoder jpg;
PImage img;

void setup() {
    size(1400, 800);
    jpg = new JPGEncoder();
    // Start a server at port 5204
    server = new Server(this, 5204);

    img = createImage(100, 100, RGB);
    println("Starting server");
}

void draw() {
    checkForIncomingImage();
    image(img, 0, 0);
}


void checkForIncomingImage() {
    Client nextClient = server.available();
    if (nextClient != null) {
        println("Client is available, reading bytes");

        byte[] byteBuffer = nextClient.readBytes();

        if (byteBuffer.length > 0) {
            println("Received data. Trying to decode.");
            try {
                img = jpg.decode(byteBuffer);
            } catch (IOException e) {
                println("IOException");
            } catch (NullPointerException e) {
                println("Probs incomplete image");
            } catch (ArrayIndexOutOfBoundsException e) {
                println("Probs also incomplete image (Out of Bounds)");
            }
        } else {
            println("Byte amount not above 0");
        }
    }
}

And here is my client code:

import processing.net.*;
import processing.video.*;

Client client;
Capture cam;
JPGEncoder jpg;

void setup() {
    jpg = new JPGEncoder();

    cam = new Capture(this, Capture.list()[1]);
    cam.start();

    // String server = "192.168.1.15";
    String server = "127.0.0.1";
    client = new Client(this, server, 5204);

    background(255);
    println("Starting client");
}

void draw() {

}

void keyTyped() {
    if (cam.available()) {
        println("Cam available. Going to read");
        cam.read();
        try {
            println("Getting image to memory");
            PImage img = cam.get();
            img.resize(500, 0);

            println("Encoding");
            byte[] encoded = jpg.encode(img, 0.1F);

            println("Writing to server");
            client.write(encoded);
        } catch (IOException e) {
            // Ignore failure to encode
            println("IOException");
        }


    }
}

You can also view the code at GitHub here: https://github.com/torbjornlunde/processing-experiments

Example of what an incomplete transfer of image looks like:

koogs

I would start by printing the number of bytes sent and received, see if there's a discrepancy

Lord_of_the_Galaxy

I guess the data input stream is buffered and hence not the entire set of bytes is available.

leopessanha

Have you found a better solution?

I stumbled upon the same problem. In order to work I have downgraded the framerate and the resolution of the sent image.

I would like to be able to send larger images in a faster pace.

If you improved your solution, let me know!

pbp

Hello everyone,

@TorbjornLunde you almost had it. Let me point out the problems:

1. I have not tested it, but I think the problem with the code you posted was that when you read with the server, the image may not have arrived entirely (there are still bytes to come).

2. I saw your code on github (https://github.com/torbjornlunde/processing-experiments) and spoted the only error you had: first you transmit the length of the array of bytes that comprises the image; then you transmit the array of bytes. But the length of the byte array is an int! An int has 32 bits in Java. So, an int can be decomposed into 4 bytes, but not 2! The only way to decompose it in two bytes is to make sure that the length of the byte array is less than 2^16. For that reason, you could send complete images only if they were very compressed.

3. The solution can be found, together with a finite-state machine that manages the server and the client (I do not know if this is better or worse for efficiency) in:

https://github.com/PBernalPolo/sendPImage

However, the key is:

• from int to bytes:

int l = jpgBytes.length;
byte[] lengthBytes = new byte[]{ (byte)( l & 0xFF ) , (byte)( ( l >> 8 ) & 0xFF ) , (byte)( ( l >> 16 ) & 0xFF ) , (byte)( ( l >> 24 ) & 0xFF ) };

• from bytes to int:

imageLength = ( ( (lengthBytes[3] & 0xFF) << 24 ) | ( (lengthBytes[2] & 0xFF) << 16 ) | ( (lengthBytes[1] & 0xFF) << 8 ) | (lengthBytes[0] & 0xFF) );

Finally, I will mention @leopessanha because he seemed interested, and I do not know if he would receive a notification otherwise.

P.D.: does anyone knows how to format properly the code after "from bytes to int"? I can not use "< pre lang="processing">" and "< /pre>" without something disappearing...

GoToLoop

Rather than using <pre>, just highlight your code and hit CTRL+O. ;;)
If you'd still prefer <pre>, replace < w/ <. :\">

pbp

Thanks @GoToLoop. I did not like that the line was so long with CTRL+O...

GoToLoop

You can remove most of those parentheses to make the whole statement shorter: :ar!
imageLength = lengthBytes[3] << 030 | (lengthBytes[2] & 0xff) << 020 | (lengthBytes[1] & 0xff) << 010 | lengthBytes[0] & 0xff;

And even place them in separate lines: :bz

imageLength =
  lengthBytes[3] << 030 |
  (lengthBytes[2] & 0xff) << 020 |
  (lengthBytes[1] & 0xff) << 010 |
  lengthBytes[0] & 0xff;

And if we assume that each indexed value of lengthBytes[] is never greater than 255 (0xff), we can remove all & 0xff and parentheses too: \m/

imageLength =
  lengthBytes[3] << 030 |
  lengthBytes[2] << 020 |
  lengthBytes[1] << 010 |
  lengthBytes[0];

TorbjornLunde

Thank you for answering! I was meaning to get back to this (and post) after I'd finish my class on OS and networks but I never got around to it. Thank you anyway!

pbp

@TorbjornLunde, thank you for your JPGEncoder class. You did all the hard work.

@GoToLoop, I'm afraid that part of your code is incorrect:

1. The use you made of octatal literals (010, 020, 030, ...) is correct, but I think it could cause confusion. I'm not familiar with these expressions, and I do think many others either. However, it is correct.

2. It is correct to know the operators precedence and save some parentheses with it, but I often doubt the exact order, so I prefer to use parentheses to clarify the expression. Even so, your first and second expressions are correct.

3. Your third expression is not correct. The variable lengthBytes is a byte array. It contains bytes. When you apply the left shift operator (<<), the result is still a byte. Then, after applying the bitwise inclusive ORs (|) you get a byte. The & 0xFF mask is necessary to transform the byte into an int, so that the left shift can be done correctly.

Please, for the next time consider testing the code before posting. It could have saved the time I inverted writing this, or in case I had not responded, the time that others could have lost using that code, then debug...

GoToLoop

Your third expression is not correct. The variable lengthBytes is a byte array. It contains bytes. When you apply the left shift operator (<<), the result is still a byte.

• Your statement above is incorrect! b-(
• In Java, and even in C, any datatype below int is auto-converted to int when used as operands. @-)
• So at lengthBytes[3] << 030, the value stored in lengthBytes[3] is converted to int before being used as the 1st operand of the operator <<. :-B
• Still doubt it is as I say? Well, just attempt to compile the sample code below if you can: :-\"

byte b = 10;
b = b << 0; // cannot convert from int to byte

println(b);
exit();

pbp

You are right. That statement was incorrect. Still, check the following code:

for(int l=-1000; l<1000; l++){
  // conversion from int to byte array
  byte[] lengthBytes = new byte[]{ (byte)( l & 0xFF ) , (byte)( ( l >> 8 ) & 0xFF ) , (byte)( ( l >> 16 ) & 0xFF ) , (byte)( ( l >> 24 ) & 0xFF ) };
  // conversion from byte array to int
  int r0 = ( ( (lengthBytes[3] & 0xFF) << 24 ) | ( (lengthBytes[2] & 0xFF) << 16 ) | ( (lengthBytes[1] & 0xFF) << 8 ) | (lengthBytes[0] & 0xFF) );
  // conversion from byte array to int using your first expression
  int r1 = lengthBytes[3] << 030 | (lengthBytes[2] & 0xff) << 020 | (lengthBytes[1] & 0xff) << 010 | lengthBytes[0] & 0xff;
  // conversion from byte array to int using your second expression
  int r2 = 
           lengthBytes[3] << 030 |
          (lengthBytes[2] & 0xff) << 020 |
          (lengthBytes[1] & 0xff) << 010 |
           lengthBytes[0] & 0xff;
  // conversion from byte array to int using your third expression
  int r3 = 
           lengthBytes[3] << 030 |
           lengthBytes[2] << 020 |
           lengthBytes[1] << 010 |
           lengthBytes[0];
  // we print the conversion if it is different from the initial value
  if( l != r0 ) System.out.println( "l: " + l + "\tr0: " + r0 );
  if( l != r1 ) System.out.println( "l: " + l + "\tr1: " + r1 );
  if( l != r2 ) System.out.println( "l: " + l + "\tr2: " + r2 );
  if( l != r3 ) System.out.println( "l: " + l + "\tr3: " + r3 );
}

I already had to test your code twice ...

GoToLoop

Hello again @pbp! B-)

As a condition to this shortcut:

imageLength =
  lengthBytes[3] << 030 |
  lengthBytes[2] << 020 |
  lengthBytes[1] << 010 |
  lengthBytes[0];

I've stated: X_X

And if we assume that each indexed value of lengthBytes[] is never greater than 255 (0xff), we can remove all & 0xff and parentheses too.

Actually I shoulda stated instead: within the range from 0 to 255. :-\"

I confess I haven't really paid much attention that your lengthBytes[] was of datatype byte[].
I was assuming it'd much likely be of int[] instead. 3:-O

Java's datatype byte is very problematic, b/c its Byte.MAX_VALUE is 127, not 255: :-O
println(Byte.MAX_VALUE); // 127

Therefore, (byte) 0x80 isn't 128, but -128: println((byte) 0x80); // -128
And (byte) 0xff isn't 255, but -1: println((byte) 0xff); // -1

Funnily, applying & 0xff to a byte value, removes its negative form: :ar!
println((byte) 0x80 & 0xff); // 128

If you change byte[] to short[], char[] or int[], the negative glitch goes away! O:-)

Here's another recent post w/ that problematic byte conversion. This time to String: =;
https://Discourse.Processing.org/t/udp-problem-sending-byte-127/3545

pbp

@GoToLoop, certainly, "Java's datatype byte is problematic", but it is the only 8-bit datatype that Java has. Communication methods need to use that datatype. I have had some problems trying to send data between sketches, and also trying to send data between Arduino and Processing. You came to help, I'm sorry if I was a bit harsh in my response. I probably should have pointed out your error in a private message, but I guess we both learned something from this. Again, thanks for your help.

GoToLoop

I'm sorry if I was a bit harsh in my response.

Never mind that at all, @pbp. Always ready for a tech debate! :-j
I wanna learn as much as I wanna teach! O:-)