How can I make my Processing.js applet activate the virtual keyboard in Safari/iOS?

otherthin..

How can I make my Processing.js applet activate the virtual keyboard in Safari/iOS?

in Processing with Other Languages • 1 year ago

I'm working on an applet that requires keyboard input, and using Processing.js so it will work in the widest variety of contexts. Here's an earlier one I published using the same methods, just for an example:

http://otherthings.com/apps/judder1/

I'm able to get this to run in Safari in iOS (e.g. on my iPhone) just fine. But I can't get keyboard input to work, because clicking on the applet doesn't bring up the virtual keyboard.

The example above could be redesigned to not require keyboard input (e.g. I could have mapped the parameters to mouse X and Y.) But the one I'm writing now really does require keyboard input, because it's all about text.

Does anyone know how to make this happen?

Replies(8)

therealpo..

Re: How can I make my Processing.js applet activate the virtual keyboard in Safari/iOS?

1 year ago

rely on the web page instead. Define a function called "getUserInput()" in your javascript that does:

1) create a see-through text element (css color: transparent and background-color: transparent) and moves focus onto it. This prompts the browser to fire up the virtual keyboard.

2) add an event handler to the element so that on every keytyped event, that character is passed to your sketch for "doing something with".

3) make the event handling also remove the text element on a return/enter/newline so that it disappears when the user have confirmed their text.

This will look something like this:

function startGettingUserInput() {
var e = document.createElement("input");
e.type = "text";
e.style.color = "transparent";
e.style.backgroundColor = "transparent";
e.style.border = "none";
e.onkeyup = function(evt) {
var event = evt | window.event;
sketch.userTypedLetter(event.keyCode);
if(event.keyCode == 10 || event.keyCode==13) {
document.body.removeChild(e);
}
}
document.body.appendChild(e);
e.focus();
}

You also want a string buffer in your sketch that gets appended to when you require input, something like this:

String buffer;
// we call this method in the sketch
void startGettingUserInput() {
buffer = "";
if(javascript) {
javascript.startGettingUserInput(this);
}
}
// this method gets called by javascript every time a letter is typed
void userTypedLetter(int keyCode) {
if(keyCode == 10 || keyCode == 13) {
// newline / return
finishUserInput();
}
else { buffer += String.valueOf((char) keyCode); }
}
// we can call this when we see that the user is done
void finishUserInput() {
// do something with the contents of buffer
}

The "javascript" variable is one that you created by tying your sketch and javascript together, along the lines of http://processingjs.org/articles/PomaxGuide.html#interface

- Pomax

otherthin..

Re: How can I make my Processing.js applet activate the virtual keyboard in Safari/iOS?

1 year ago

Fantastic! This is so helpful!

My app only needs to process one keystroke at a time, but I think I can modify your example code to make it do that... time to start tinkering...

Thanks a million!

otherthin..

Re: How can I make my Processing.js applet activate the virtual keyboard in Safari/iOS?

1 year ago

Alright, I've tried to cobble together the pieces you've showed me, but there are some things I clearly still don't understand. Here's the html of my web page:

<html>
<head>
<title>Processing.js keyboard input test</title>

<script src="processing-1.3.6.min.js"></script>
<script type="text/javascript">
    function startGettingUserInput() {
      var e = document.createElement("input");
      e.type = "text";
      e.style.color = "transparent";
      e.style.backgroundColor = "transparent";
      e.style.border = "none";
      e.onkeyup = function(evt) {
          var event = evt | window.event;
            sketch.userTypedLetterJS(event.keyCode);
            if(event.keyCode == 10 || event.keyCode==13) {
             document.body.removeChild(e);
            }
          }
      document.body.appendChild(e);
      e.focus();
    }
</script>

</head>

<body marginwidth=36 leftmargin=72 rightmargin=72>

<center>
<table width=500><tr><td>
<br>
<br>
<center>
<h1>Processing.js keyboard input test</h1>
<p><i><a href="http://otherthings.com">Cassidy Curtis</a>
<br>March 17, 2012</a></i></p>
<p>Type some letters and they should show up below.</p>

<canvas id="sketch" data-processing-sources="keyboard_js_test.pde"></canvas>
<br><br>
<i><font size=-1>Processing source code:
<a href="keyboard_js_test.pde">keyboard_js_test.pde</a></font></i>
</center>

</td></tr></table></center>

</body></html>

And here's what's in "keyboard_js_test.pde":

PFont font;
String lastTyped;

interface JavaScript {
void startGettingUserInput();
}

void bindJavascript(JavaScript js) {
javascript = js;
}

JavaScript javascript;

void setup() {
size(400, 400);
background(0);
smooth();
noStroke();
font = createFont("Averia-Bold-48.vlw", 36);
textFont(font);
textAlign(CENTER);
lastTyped = "";
startGettingUserInput();
}

void draw() {
background(0);
text(lastTyped, width/2, height/2);
}

void startGettingUserInput() {
if(javascript != null) {
    javascript.startGettingUserInput();
    lastTyped = "we have javascript!";
} else {
    lastTyped = "no javascript.";
}
}

void keyPressed() {
if(key == CODED) {
    userTypedCode(keyCode);
} else {
    userTypedLetter(key);
}
}

void userTypedCode(int typed) {
lastTyped = "keyCode = " + str(typed);
}

void userTypedLetter(char typed) {
lastTyped = str(typed);
}

void userTypedCodeJS(int typed) {
lastTyped = "with love from JS: " + typed;
}

What am I doing wrong here? (Probably many things, but am I at least on the right track?)

therealpo..

Re: How can I make my Processing.js applet activate the virtual keyboard in Safari/iOS?

1 year ago

let's turn that into a jsfiddle.net thing instead. With a few notes:

- <center> is not valid HTML, it's a tag from an HTML version gone by

- <br> for spacing is also no longer considered correct. We have CSS for these things =)

- "vlw" fonts are not fonts (they're a bmp image with some non-standardised header information, and Processing.js does not support them). Use the real font instead (.ttf or .otf)

With that said: http://jsfiddle.net/Vd5U3/6 (I inlined your code so that jsfiddle can run it). Note how the sketch is set up, and javascript independently tries to call your sketch's bindJavascript (you didn't call it, so the sketch never knew about the browser's javascript environment). You now have to click the sketch to start getting user input, which makes javascript insert that text element. when you type something, the letter gets sent back to your sketch and the element's killed off again.

Trying this on an ipad 2 almost works, it pops up the keyboard, but then closes it again so it's possible the element needs some more things so that the ipad knows it's "real". I haven't looked into that (I'll leave that as an exercise for you =D)

- Pomax

otherthin..

Re: How can I make my Processing.js applet activate the virtual keyboard in Safari/iOS?

1 year ago

Pomax, you're blowing my mind here! That jsFiddle thing is brilliant!

I cut and pasted all the code into a flat html file, just to convince myself I could get it to run, and it works!

I was also able to substitute in a version of my real sketch, with mixed success. It doesn't seem to handle text input the way I'd expect: for example, both "a" and "A" show up as "A", and "?" appears as "¿" for some reason.

Now I just need to figure out why JavaScript's idea of a keystroke is so utterly different from Processing's, and how to make them compatible. In the sketch I actually want to publish this way, I'm making use of both "key" and "keyCode" to distinguish visible characters like letters, numbers and punctuation from irrelevant events like Shift and Ctrl. It was working great within Processing, but now, not so much. I'll dig into the JavaScript documentation and see if maybe there's a flavor of keystroke that's a better match for what I need.

therealpo..

Re: How can I make my Processing.js applet activate the virtual keyboard in Safari/iOS?

1 year ago

yeah it's using charCode at the moment, which only catches the "absolute" ascii number. You can try using keyCode, but some browsers are quirky with it (that should, however, give you the actual letter value).

For why javascript's so different from Processing (more accurately in this case, Java), have a look at http://unixpapa.com/js/key.html (try to have something calming available within reaching distance. it's a bucket of wtf =) Short version: key handling is hard. Long version: key handling is stupidly hard.

If you're up for it you can have a look inside Processing.js to see how it deals with turning JavaScript key events into Processing-conformant key events. The coded character concept is mostly "when a special key is pressed, record its "ispressed" value, and keep it set to that value until keyup shows that particular special character was let go of again".

Also, you probably want to change the CSS for the input element by assigning it a width:0px, height: 0px and position:absolute -- now it's properly invisible =)

- Pomax

otherthin..

Re: How can I make my Processing.js applet activate the virtual keyboard in Safari/iOS?

1 year ago

Haha, wow, you're not kidding about the craziness of keyboard inputs. That stuff is nutty!

I'm also discovering that Processing.js and Processing have very different ideas about what a "char" is. JS seems to always treat it as a number, whereas Processing treats it like a character. My sketch is full of code like:

String s = "";
void addLetter(char c) {
s = s + c;
}

In Processing, if I say "addLetter(key)", it does what I expect: appends the key I pressed to the end of the string, so typing "abc" results in a string "abc". In JavaScript, bizarrely, the same code converts the char into a number before appending it to the string, so typing "abc" gives you "979899"! Likewise, tests like this work fine in Processing, but fail in Processing.js:

char c = key;
if (c >= 'a' && c <= 'z') {
addLetter(c);
} else {
// it's not a letter
}

I'm starting to think it might be less work for me to just bite the bullet and develop this as an iOS app... ;-)

therealpo..

Re: How can I make my Processing.js applet activate the virtual keyboard in Safari/iOS?

1 year ago

That's a problem inherent to Javascript (html5 iOS apps suffer from the same problem =). JavaScript only knows "number" and "string", so the "it's both" char datatype is a big problem for JavaScript. We try to make Processing.js do the right thing most of the time, so for comparison purposes for instance it works just fine (stuff like if(key=='a') { ... } will work) but for other things it won't, like string concatenation.

The code you pasted works fine in Processing.js 1.3.6 from what I can tell, though?

char a = 'c';
println( "is " + (a >= 'a' && a <= 'z' ? "" : "not ") + "letter");
char b = (char) 72;
println( "is " + (b >= 'a' && b <= 'z' ? "" : "not ") + "letter");
char c = (char) 102;
println( "is " + (c >= 'a' && c <= 'z' ? "" : "not ") + "letter");

This prints "is letter", "is not letter", and "is letter" in both Processing and Processing.js

- Pomax

Top Reply