Processing 1.0 - Processing Discourse - problem - unicode string in P2D and P3D mode

We closed this forum 18 June 2010. It has served us well since 2005 as the ALPHA forum did before it from 2002 to 2005. New discussions are ongoing at the new URL http://forum.processing.org. You'll need to sign up and get a new user account. We're sorry about that inconvenience, but we think it's better in the long run. The content on this forum will remain online.

Index › Programming Questions & Help › Syntax Questions › problem - unicode string in P2D and P3D mode

‹ Previous Topic | Next Topic ›

Pages: 1

problem - unicode string in P2D and P3D mode (Read 3028 times)

trebari

problem - unicode string in P2D and P3D mode
Feb 25^th, 2009, 2:47am

The Korean part of a string is not shown when using P2D or P3D mode instead of JAVA2D.
Is this a limitation of P2D and P3D?

I also found that Korean characters does not work in PGraphics either.

Code:

PFont   myFont = createFont("AppleGothic", 32);
String  myStr = "some korean letters here";

size(400, 400, P3D);

background(0);
fill(255);
textFont(myFont);
text(myStr, 30, 200);

* when I put real Korean letters in the code above, this BBS converts in into UTF-8 codes, so I replaced it with "some korean letters here" for this post.

PhiLho

Re: problem - unicode string in P2D and P3D mode
Reply #1 - Feb 25^th, 2009, 12:35pm

createFont() reference says:
Quote:

When possible, the text() function will use a native font rather than the bitmapped version created behind the scenes with createFont(). For instance, when using the default renderer setting (JAVA2D), the actual native version of the font will be employed by the sketch, improving drawing quality and performance. With the P2D, P3D, and OPENGL renderer settings, the bitmapped version will be used.

I haven't checked, but I bet by default createFont limits itself to Ascii char set (including high Ascii as it seems): it would be a bit wasteful and pointless to create thousands of characters each time.
The charset parameter is precisely there to specify which characters you need.

Example of use:
Code:

char[] charset = { '<', '>', 'o', '\u2600', '\u2605' };
PFont   myFont = createFont("Arial Unicode MS", 32, true, charset);
String  myStr = "o>\u2600\u2605<o";
 
size(400, 400, P3D);
 
background(0);
fill(255);
textFont(myFont);
text(myStr, 30, 200);

Of course, you can make a loop to fill charset:
Code:

char[] charset = new char[16];
StringBuilder strB = new StringBuilder();
for (int i = 0; i < 16; i++) 
{
  charset[i] = Character.toChars(0x2600 + i)[0];
  strB.append(charset[i]);
}
PFont   myFont = createFont("Arial Unicode MS", 48, true, charset);
 
size(800, 200, P2D);
 
background(0);
fill(255);
textFont(myFont);
text(strB.toString(), 10, 100);

trebari

Re: problem - unicode string in P2D and P3D mode
Reply #2 - Feb 26^th, 2009, 11:46am

Quote:

when using the default renderer setting (JAVA2D), the actual native version of the font will be employed by the sketch, improving drawing quality and performance. With the P2D, P3D, and OPENGL renderer settings, the bitmapped version will be used.

Oops, I missed this part.
So, this is why createFont() seemed working for Korean characters when using JAVA2D.

The reason I do not use Create Font menu is that it causes memory error even though the memory allocation is set to the maximum like 4GB. ("java.lang.OutOfMemoryError: Java heap space")

I don't know what to do next...

PhiLho

Re: problem - unicode string in P2D and P3D mode
Reply #3 - Feb 26^th, 2009, 12:53pm

trebari wrote on Feb 26^th, 2009, 11:46am:

I don't know what to do next...

Well, as I wrote, trying to generate the full Unicode range can be demanding... As I shown, use createFont, but indicate it to include Korean characters. These are Hangul chars, right

Searching... Right.
The interesting page Korean Unicode Fonts shows lot of fonts supporting this set. It also points to the useful CJK FAQ. Looks like we need Jamo too (I don't know Korean language, alas). Or not, Hangul is perhaps enough.

Anyway, let's do an example: using the good BabelMap utility, I see that Hangul Syllables start at 0xAC00 and ends... at 0xD7A3. Wow, lot of characters!

Anyway, reusing my previous sketch, I did:
Code:

size(800, 200, P2D); // Must be first!
 
char[] charset = new char[0xD7A3 - 0xAC00 + 1];
StringBuilder strB = new StringBuilder();
for (int i = 0, code = 0xAC00; code <= 0xD7A3; i++, code++)  
{
  charset[i] = Character.toChars(code)[0];
  if (i % 256 == 0)
  {
    strB.append(charset[i]);
  }
}
println("Creating font");
PFont myFont = createFont("Arial Unicode MS", 24, true, charset);
 
background(0);
fill(255);
textFont(myFont);
text(strB.toString(), 10, 100);

My old computer creates the font subset without problem although that's a bit slow.
Maybe a possible improvement to the create font dialog would be to indicate subsets. Or Processing should allow saving the font after calling createFont, would be even more flexible (although rarely used, I fear).

trebari

Re: problem - unicode string in P2D and P3D mode
Reply #4 - Feb 26^th, 2009, 6:40pm

Thanks a lot, PhiLho!

Your code works like a charm.
I'm sorry that I did not carefully see your code in your first reply.

Anyway...
I had to include basic Latin characters and Hangul 'jamo' to have complete set for using Korean fonts.

Here is a class called "Hangul" for managing all of this.
There may be something I missed in this code, but it works perfectly so far for me.

I'll post upgraded version of this when available for other Korean guys.
Thanks again!

P.S. I still don't understand why "Create Font" menu does not work for Korean fonts. It always causes Out Of Memory error, even though I set enough memory in the Preferences.

Code:

/*
  Hangul

    version 0.1
    written by Jaeho Chang (trebari@gmail.com)
    special thanks to Philippe Lhoste
    
  IMPORTANT
  
    You must call "size()" before creating any instance of this class.
    
    If you encounter a memory error,
    adjust the memory setting in the Preferences.
    (http://processing.org/reference/troubleshooting/index.html#memory)
 
  EXAMPLE
 
    Hangul gothic;
    Hangul myungjo;

    void setup() {
	size(400, 400, P2D);
	myungjo = new Hangul("AppleMyungjo", 24);
	gothic = new Hangul("AppleGothic", 24);
    }

    void draw() {
	background(0);
	fill(255);

	gothic.use();
	text("any Hangul words here", 10, 100);
 
	myungjo.use();
	text("any Hangul words here", 10, 150);
    }
*/

class Hangul {
  char[]   mCharset;
  PFont    mFont;

  Hangul(String fontName, int fontSize) {
    int index = 0, count = 0;

    // calculate the number of characters
    count += (0x007F - 0x0000 + 1);    // basic Latin
    count += (0x11FF - 0x1100 + 1);    // Hangul jamo
    count += (0xD7AF - 0xAC00 + 1);    // Hangul syllables

    // allocate memory
    mCharset = new char[count];

    // loading basic Latin characters
    for (int code = 0x0000; code <= 0x007F; code++) {
	mCharset[index] = Character.toChars(code)[0];
	index++;
    }

    // loading Hangul jamo
    for (int code = 0x1100; code <= 0x11FF; code++) {
	mCharset[index] = Character.toChars(code)[0];
	index++;
    }

    // loading Hangul syllables
    for (int code = 0xAC00; code <= 0xD7AF; code++) {
	mCharset[index] = Character.toChars(code)[0];
	index++;
    }

    // creating font
    print("Creating font " + fontName + "... please wait... ");
    mFont = createFont(fontName, fontSize, true, mCharset);
    println("done!");
  }

  void use() {
    textFont(mFont);
  }
}

PhiLho

Re: problem - unicode string in P2D and P3D mode
Reply #5 - Feb 27^th, 2009, 8:27am

trebari wrote on Feb 26^th, 2009, 6:40pm:

I still don't understand why "Create Font" menu does not work for Korean fonts. It always causes Out Of Memory error, even though I set enough memory in the Preferences.

I tried to hint above...
In short, when you check "All Characters" in the Create Font dialog, Processing attempts to generate 65536 characters (the whole base Unicode!).
Fortunately, it is smart enough to generate only the characters exiting in the font: lot of amateur-made fonts include only base Ascii, most good fonts also include Latin-1 chars (high Ascici/accented chars) and some fonts like Arial Unicode or Bitstream Cyberbit try to cover as much as possible of defined Unicode chars.
So the resulting file can be more or less big. If you try to generate your Korean chars from Arial Unicode, chances are you are indeed exhausting memory, even more if you chose a big font size, Processing generating a PImage per character!

I wrote: Quote:

Processing should allow saving the font after calling createFont

To answer the first part, I took a look at the source, and found something handy not documented in the official Reference: PFont has a save() function!

The following isn't tested but inspired from P5's source:
Code:

PFont font = createFont(fontName, fontSize, true, mCharset); // End of your code
File fontFile = new File( "where you need to save", "KoreanFont.vlw" ); // Of course, adjust name and path!
font.save(new FileOutputStream(fontFile)); // Probably need a try/catch on IOException

You get the best of two worlds: a very flexible font creation tool.

trebari

Re: problem - unicode string in P2D and P3D mode
Reply #6 - Feb 27^th, 2009, 3:25pm

Wow, you did a great job.
Based on your code, I made a handy sketch for creating Hangul font files.

One more thing is that I always get "ATSFontGetPostScriptName failed to return valid ASCII PostScript name" error message. This error does not seem harmful, but I'm investigating what this is.

Thanks a lot PhiLho!

Code:

///////////////////////////////////////////////// Settings

String   fontName = "AppleGothic";
int	fontSize = 24;
String   fontFilePath = "/Users/jaeho/Desktop/AppleGothic.vlw";

///////////////////////////////////////////////// Create and Test

PFont theFont;

void setup() {
  size(400, 200);
  createHangulFont(fontName, 24, fontFilePath);
  theFont = loadFont(fontFilePath);
  textFont(theFont);
}

void draw() {
  background(0);
  fill(255);
  text("some hangul words here", 10, 100);
}

///////////////////////////////////////////////// Core Function

void createHangulFont(String fontName, int fontSize, String fontFilePath) {
  char[]   charset;
  int	index = 0, count = 0;

  // calculate the number of characters
  count += (0x007F - 0x0000 + 1);    // basic Latin
  count += (0x11FF - 0x1100 + 1);    // Hangul jamo
  count += (0xD7AF - 0xAC00 + 1);    // Hangul syllables

  // allocate memory
  charset = new char[count];

  // loading basic Latin characters
  for (int code = 0x0000; code <= 0x007F; code++) {
    charset[index] = Character.toChars(code)[0];
    index++;
  }

  // loading Hangul jamo
  for (int code = 0x1100; code <= 0x11FF; code++) {
    charset[index] = Character.toChars(code)[0];
    index++;
  }

  // loading Hangul syllables
  for (int code = 0xAC00; code <= 0xD7AF; code++) {
    charset[index] = Character.toChars(code)[0];
    index++;
  }

  // creating font
  PFont    font;
  print("Creating font " + fontName + "...");
  font = createFont(fontName, fontSize, true, charset);
  println("done!");

  // creating file
  try {
    print("Creating font file... ");
    File fontFile = new File(fontFilePath);
    font.save(new FileOutputStream(fontFile));
    println("done!");
  }
  catch(IOException e) {
    println("failed to create font file");
  }
}

Pages: 1

‹ Previous Topic | Next Topic ›