you have to import javax.media.opengl.*; and you have to prefix all constants with the class they're in, e.g. GLUT.GLUT_DOUBLE.

HOWEVER, as far as I can tell jogl (which Processing uses for OpenGL( doesn't include the full set of glut instructions, and glutInitDisplayMode isn't one of them.
Jogl handles display options with GLCapabilities instead.

There's yet another problem, in that unless you're doing somethign really fancy, you'll have trouble setting up a new display, as processing handles all that itself when you do size(640,480,OPENGL);