Leap Motion Tremor Recognition?

Grad_student

Hey Community,

I'm a fairly new developer that has been working on stuff using Leap Motion for Processing https://github.com/voidplus/leap-motion-processing. I'm enjoying Processing thus far.

I came across this demo on YouTube

I need to build something like this but I have no idea where to start?! I cant even tell what language this app is built in and I cant find any documentation for it online....

If anybody could provide some pointers in the right direction it would be great! i'm going to continue to lurk the internet for more information....

colouredmirrorball

Looks doable. Just curious, what is the application?

How far did you get with the code? Did you manage to display the finger positions on the screen? I'm using a different package and I don't have access to the Leap I used for my own experiments so I'm not sure how much of a help I can be.

Looks like it just tracks the orientation of your hand and does some signal processing on that data. Getting the data and visualizing it shouldn't be a problem, but I have no clue how to do the FFT in Processing and find the highest amplitude...

There are some methods in the library to get the hand orientation so you could just grab that and store it in an array. Do you want the orientation of the hand itself, or the orientation of the fingers? If you want the latter, you'll need to do some linear fit as well.

Grad_student

@colouredmirrorball

I want to use this to potentially track hand tremor for people with a physical disability. I haven't written any code as of yet but getting finger position with the Leap Processing is a pretty simple task: finger_position = finger.getPosition();

They also have all of these:

 PVector hand_position    = hand.getPosition();
 PVector hand_stabilized  = hand.getStabilizedPosition();
 PVector hand_direction   = hand.getDirection();
 PVector hand_dynamics    = hand.getDynamics();
 float   hand_roll        = hand.getRoll();
 float   hand_pitch       = hand.getPitch();
 float   hand_yaw         = hand.getYaw();

Which I'm sure will be useful I just have to figure out how each of these functions work by looking into the Leap API https://developer.leapmotion.com/docs.

I am completely clue-less when it comes to signal processing and FFT though. My math skills are somewhat lacking as well. Can you explain why these would be necessary?

colouredmirrorball

What the program is doing in your vid is storing the orientation of the hand/fingers in an array or list of some sort and then using the FFT to determine the frequency at which the hand tremors. It also determines the amplitude (I guess mean value?), i.e. the angle over which the hand oscillates.

What information do you want to retrieve? I guess frequency and amplitude are the most obvious ones, I don't see anything else.

Grad_student

@colouredmirrorball

I see. I need to refresh myself on these topics before I even begin to look at code. I think frequency and amplitude is all that will be needed.

colouredmirrorball

One thing that is vitally important is that you need to keep track of the time. The easiest way is to limit the framerate to a fixed level, so you know the interval time between two adjacent draw() calls. Downside is that if you ever get lag for whatever reason (processor can't handle it, computer is old, another program acts up, ...) the measurement gets screwed up. So the most sensible way is to use a twodimensional arraylist, in which you store the interval time in the first column and the actual angle in the second column.

I had another look at your vid, and it looks like it stores three values at each measurement: the hand angle as determined by the leap, the angle of your fingers, and some kind of mean between the two. If you want to do this as well, you'll need four columns (one for each of these data plus time).

Grad_student

@colouredmirrorball

Ok... I am following you somewhat... that part doesnt sound too hard to do. But then what about this FFT/signal processing part? Can that be done within Processing? An explanation in English would be great lol

colouredmirrorball

Honestly I have no clue... so I'm hoping somebody else can tell us this :P Java probably has some libraries or what are they called that can do this, but I have no experience with that.

colouredmirrorball

Maybe these libraries are useful...

http://www.gicentre.org/utils/ https://github.com/SableRaf/signalfilter https://github.com/jagracar/grafica http://adilapapaya.com/papayastatistics.html

I didn't find a FFT capable one though.

Grad_student

@colouredmirrorball

You keep mentioning FFT. Why do you think this is necessary?

colouredmirrorball

Because they use it in your vid. How else would you determine the frequency?

Grad_student

@colouredmirrorball

Lol, I had no idea. I'm clearly starting from scratch on this