reading UTF-8 strings over a socket?

Programming Questions

fearless_..

reading UTF-8 strings over a socket?

in Programming Questions • 1 year ago

Do I have to do anything special to condition a Client to receive UTF-8 strings?

When I create and display a UFT encoded string from within the app, the characters display as expected:

String s1 = "Swifty.\u221e";
PFont font;
void setup() {
size(800,800);
background(127);
font = loadFont("Helvetica-32.vlw");
textFont(font);
textAlign(CENTER);
text(s1, width/2, height/2);
}

..but when I read a similar string via a client connection:

Client client = new Client(_parent, host, port);
if (client.available() > 0) {
inString = _client.readStringUntil('\n');
...
}

... the strings are displayed with odd characters. (Disclaimer: I haven't tried too hard to trace this down -- my problem may lay elsewhere. But Client seems like the most likely culprit...)

Replies(2)

fearless_..

Re: reading UTF-8 strings over a socket?

1 year ago

More information. Here's a complete program that displays a "local_string" and a "network_string" read from a file. First, the results:

The file "data/utf_test.txt" contains the same string as the local_string, i.e. "Swifty.<infinity sign>" and is fed to the processing app via netcat:

cat data/utf_test.txt | nc -l 5204

Here's the program in its entirety:

// Testing reading UTF-8 characters over a net connection.

// Start server first "cat data/utf_test.txt | nc -l 5204"
// before launching this app.
import processing.net.*;
void setup() {
textAlign(CENTER);
fill(0);
String local_string = "Swifty.∞";
text(local_string, width/2, height * 0.333);
Client client = new Client(this, "localhost", 5204);
String network_string = client.readStringUntil(' ');
text(network_string, width/2, height * 0.666);
}

As you can see the "local_string" displays correctly, the string read over the network connection does not.

Any suggestions on where/how to fix this?

PhiLho

Re: reading UTF-8 strings over a socket?

1 year ago

Ouch, you are mixing up lot of things!

I see nowhere in the code you show you converted data to UTF-8.
Java internally encodes strings in UTF-16, which is quite different.
The notation you used is neither, it is just the UCN notation for non-Ascii characters.
This notation is automatically converted by the compiler to their corresponding UTF-16 characters.
But data you read from a file isn't processed by the compiler! And this notation isn't handled by the file loading API: you have to save your file in UTF-8 or UTF-16 format and read it by specifying the same encoding. I think that by default it uses the same encoding that your system, so for me it must be Windows-1252 or similar.

I suppose you are on a Unix system (or a Mac one, which is still Unix). It is probably set up to use UTF-8 encoding, so all you have to do is to paste the ∞ sign in text with a text editor set up to write UTF-8, and perhaps your cat command will send the right sequence of characters.

Note; ∞ looks like âˆž when seen in ISO-8859-1 encoding...

Top Reply