Kinect to Processing to OSC to Blender

dhall5150

Hello everyone, I've started a project to create a sketch that produces compatible OSC output to use NIMate's official Blender plugin unmodified in blender.

So far my results are fantastic - except for 1 joint rotation I can not figure out. When reaching up to cover the models eyes, for example, there is a point where the elbow does an unnatural rotation. It seems like when the hand joint is higher than the elbow joint the hand twists around, in an impossible way, showing the palm. Continuing to the eyes, the rotation eventually comes back to a natural position.

The sketch, and the version of NIMate's Big Buck Blender Model I'm using can be found here 1drv.ms/1BstWXN

The sketch is using simpleopenni, and a few other libraries that I've included in the link.

Surprisingly the math up to this point has been easy to understand.
Right now the bulk of the math sits in the "OSC" file inside the "sendJointMsg" function. I am rotating the majority of joints based on its relation to the body's CoM.

The only 2 that I'm not doing that with is the shoulder and the elbow. When I use CoM the positions of the arms aren't matching me. Instead, I found using a new vector of (1,0,0) satisfies those joints, but the limbs seem more rigid compared to leg motions, head motions, leaning forward/backward, etc.

Finally, one more puzzle is that using NIMates plugin in blender, you can auto-generate shapes at joint points. Using a blank blend, and enabling this option, you see a human shape and the joints seem to be rotating like you would expect. Its hard to know that for sure since there is no bone between joints though.

Any help on getting the arms to rotate naturally would be appreciated thanks

dhall5150

For anyone that is interested, I figured out my problem and have the sketch and sample blend file here. http://1drv.ms/1LArCUh

The sketch produces compatible OSC output to use NIMates Blender Plugin unmodified in blender. You'll need SimpleOpenNI, toxiclibs, oscP5/netP5 libraries to use it.

My needs require a separate webcam feed since the Kinect actor will be in a different room. You can safely remove "Capture webcam" from the sketch without impacting functionality.

On an unrelated note, if someone finds it helpful, I also created for my project a blender game engine lip sync addon that plays animations based on sound input. It can be found here http://1drv.ms/1zsRN8S