Missing some data from a call to readBytes()

Mag748

Hello,

I'm writing a client/server application and am having some issues with the network library. When a client requests a large amout of data (like an image file) from the server, during the call to readBytes(), it only gets part of the message.

This is an excerpt of the client code for your reference:

if (client.available() > 0)
{
  byte[] chunk = new byte[1024];
  chunk = client.readBytes();
  println("DEBUG: received data length: " + chunk.length);
  String new_string = new String(chunk);
  println("DEBUG: stringified data: " + new_string);
  JSONObject server_response = parseJSONObject(new_string);
}

And here's some of the server code:

File file = new File(filepath);
String encoded_photo = "";
JSONObject server_response = new JSONObject();
byte[] data;
try {
    data = server_response.toString().getBytes("UTF-8");
    println("DEBUG: Size of JSONObject bytes: " + server_response.size());
} catch (java.io.UnsupportedEncodingException e) {
  throw new RuntimeException(e);
}
println("DEBUG: written data length: " + data.length);
thisClient.write(data);

The debug output from the server application would indicate the correct amout of data is being sent:

DEBUG: received data length: 60
DEBUG: received request for image 4459648.png
DEBUG: Loading photo...
DEBUG: Size of encoded image: 2457044
DEBUG: written data length: 2457119

But the debug output from the client seems to indicate that it's not receiving all the data before it starts trying to process it.

DEBUG: received data length: 8806
DEBUG: stringified data: {
  "meetup_chat_id": "4459648",
  "data": "iVBORw0KGgoAAAANSUh...(continues but never is terminated)

When the line String new_string = new String(chunk); runs, about half the time I'll get the full image file saved, but the other half I'll get an "Unterminated String" exception, and the data received size is not the same as the data sent.

How would one go about solving this type of problem? Any suggestions welcome.

Thanks, Marcus

GoToLoop

https://Processing.org/reference/libraries/net/Client_readBytesUntil_.html
https://Processing.org/reference/libraries/net/clientEvent_.html

kfrajer

You can build an MCVE as that will allow to tackle the problem directly. For the time being, this next link provides some guidance:

https://stackoverflow.com/questions/8370268/missing-bytes-in-tcp-file-transfer

Kf

Mag748

Dear GoToLoop, kfrajer,

Thanks for the response and recategorizing my question.

I've thought about the situation from a TCP stream point of view in the sense kfrajer points out and rewrote everything.

The networking part of the program will continue to build the stream as it arrives until the "\n" character is received. Then it finalizes the packet and passes it onto the program. This is working well.

Is the "\n" character the best to delimit packets with? I've run into a problem when sending JSONObjects using the object.toString() function, as it automatically adds "\n" characters after each item. For now I've solved this simply by replacing all char(10) with char(32) before sending, and it's all good. But what is the industry standard in that regard?

Thanks again for the good suggestions.

Sincerely, Marcus

kfrajer

I am not an expert on this, and I am not sure if there is a standard, but there is another approach if you cannot define a stopping char. The second approach is to send a token first with the size of bytes you will be transferring and then you send that many bytes. Additionally, you can add a check sum at the end of the stream to check for the integrity of the received package. This concept works when you are managing your byte transfer yourself. Another concept will be transferring files using FTP which is a protocol designed exactly for this purpose of file transfer.

Kf