Sensorium Project - 3D - 360 Degree - HID Sensor

Share your Work

robotmake..

Sensorium Project - 3D - 360 Degree - HID Sensor

in Share your Work • 2 years ago

Sensorium Project - 3D-360-degree Multi-touch(less) HID sensor

We've now successfully interfaced ROBOTmaker's solid-state 360 degree HID multi-touch(less) sensor to various 3D Java Processing Sketches - as a proof of concept -> ( see videos ).

...and finallly also got a USB interface working (via Teensyduino) - so now it can also emulate a mouse, keyboard, joystick e.g. for massive online multiplay online gaming (MMOG), control of Google Earth or 3D CAD application such as 123D or Inventor from Autodesk.

We are also taking this much further and developing gesture sensing applications - but there is still a long way to go to.

We therefore have opened up discussions on processing forums for some further ideas and programming support.

Some possible development directions could also be...:

3D gesture sensing, for example using the TUIO interface
3D midi interface (see 3D drum machine example)
3D interface to kinetic art projects
3D Touchless input to mobile device
Control of multi axis CNC machines and Robots

moving a hand over the sensor controls PC applications

Background to Sensorium Project

see further details of the project -> here:

We used Arduino UNO & Teensy to control the serial communications between 360 degree HID sensor and the Arduino Uno , Teensyduino and Java / Processing sketches on the PC/Mac. The sensor controls 3D graphics (Java sketches), or can emulate Mouse, Joystick or Keyboard on the PC (or possible would also work on Apple Mac, Andriod, Ubuntu, etc but have not test this yet.) to control any 3D applications such as Google Sketchup, Google maps, AutoDesk Inventor - CAD type application.

Develop your own Arduino Sketch or Processing.org sketches

We are using Processing.org , Arduino, Teensy as a development platform as it's real easy to develop cool java graphics programs and Processing GUI is free and really a great way to learn about programming and interfacing. .

3D Mouse / Keyboard / Joystick Control

Plug the IRCF360 sensor into a Teensy2 to enable 3D mouse, joystick, keyboard interface to your Arduino projects. The Teensy2 can be configured as a USB device such as mouse, Joystick, keyboard which allows the sensor to control your PC or Mac ....adding new dimensions to your projects.

So how does this work?...By moving your hand over the sensor the direction and distance of your hand / fingers are sensed and translated into values within the Teensy2.....

Gaming control

Keyboard emulation interface

Sensing directions are translated into keyboard keys. This is programmed into the Teensyduinon - sketch using the usb keyboard commands:

Keyboard.set_modifier(0);

Keyboard.set_key1(KEY_W);

Keyboard.send_now();

e.g. configurate of the keys WASD to control online or Massive Multiple online (MMO) games - spaceship speed control is achieved by mapping mouse or keyboard keys to the distance readings from sensor.

Mouse emulation interface

Sensing directions are translated to mouse directions. This is programmed into the Teensyduinon - sketch using the Mouse.move (x, y) USB mouse command ; e.g. configurate of the keys WASD to control online or Massive Multiple online (MMO) games - engine / speed control is achieved by mapping mouse or keyboard keys to the distance readings from sensor.

Joystick emulation interface

Sensing directions are translated to Joystick directions or commands. This is programmed into the Teensyduinon - sketch using USB Joystick command to add Z directional control to your project too.

The joystick usb interface has

32: Buttons
6: Axis (X, Y, Z, Rotate Z, Left Slider, Right Slider)
1: Hat Switch

Joystick.X(value);

// "value" is from 0 to 1023 Joystick.Y(value);

// 512 is resting position Joystick.Z(value);

Joystick.Zrotate(value);

Joystick.sliderLeft(value);

Joystick.sliderRight(value);

Adding 3D Control to your Java and Processing Sketches

The example below is the multi-blob tracking demo, where the two large blobs are tracking the closest and 2nd closest object. The brown 'spider-web' is a representation of all other directions. For more advanced Java programs & sketches the pattern could be used to interpret different hand gestures to add more advanced controls.