Creating LeapMotion Gestures

Now that I have my LeapMotion controller and I have successfully managed to model a simplified pinball machine, I am keen to link the two together; and be able to play my pinball machine using the LeapMotion controller.

The first step along the road to achieving this is to define a way of interacting with the pinball machine - by defining gestures for using the Plunger and the Flippers.

The Plunger's Gesture

Source: gametrailers.com

The plunger is used to propel the pinball in to the playfield with a varying degree of force - depending on how far back the spring-loaded rod is pulled, before it is released. 

With this in mind, I set out to create a gesture which would track when the user grabbed the plunger and how far, along the Z axis, they pulled back the plunger before releasing it.

Source: leapmotion.com

Due to the limitations of the LeapMotion controller, and for simplicity's sake, I modelled the action of grabbing the plunger as a horizontal two-fingered pinch. Once the Leap Motion controller identifies two fingers as being brought within a certain distance of each other (along the X axis) they are considered as being pinched together - the plunger has been grabbed. The code then keeps track of how far back (away from the LeapMotion controller/towards the user) the fingers are drawn (along the Z axis) before the pinch is released (the two fingers move a certain distance apart from each other on the X axis). The distance in which the pinch gesture has travelled along the Z axis can then be translated in to an amount of force to be applied to the pinball in order to launch it in to the playfield. 

Based on the documentation for setting up a Java project and the Sample Java class - which provides a nice introduction to the API for getting hold of the Frames and the information contained within each of the Frames (Hands, Fingers, Gestures etc.) (Sample.java can be found as part of the SDK's files) - I created a new project of my own.

In order for your application to get hold of the event information dispatched by the Controller, the API contains a Listener Class; a subclass of which can override a number of callback functions in order to get hold of the dispatched event. The most useful of these callback functions is the onFrame method which is called every time the Controller has a new Frame of information - which could be anything between every 2 ms to 33 ms (i.e. 30 to 500 fps). My PinchGestureListener class, therefore, extends the Listener class and overrides the onFrame method:

There are a few notable things to mention about what the above code does: 

1. First of all it makes sure that we only have one hand in view; that, that hand only has two fingers visible and it works on the assumption that it is the right hand; I chose to impose these constraints to make it easier to make the gesture robust. Production-ready code would require more complexity and a lot more testing. 
2. Any state outside of the constraints will effectively reset/end the gesture in order to prevent the code from ending up in an erroneous state e.g if a pinch gesture is recognised and then hand disapears/is taken out of view of the controller.
3. Thresholds are defined for how close the fingers need to be together to constitute a pinch and how far they need to be away from each other for the pinch to subsequently be released (measured along the X axis). Since the Controller loses track of one of the fingers when they touch, I found it necessary to not require the fingers to touch for the gesture to be complete and to handle this boundary case.

The result looks like this:

The Flippers' Gesture

Source: wikimedia.org

Flippers are levers used to redirect the pinball up the playfield. Depending on the timing on which the button is pressed, to active the flipper, they can be used to control the direction and velocity of the pinball. 

Keeping with the idea of simplicity, I next went on to create a gesture for the right-hand flipper - activated when only the right hand is in view of the Controller. The reason that I chose to impose this constraint is that from experimenting with the Diagnostic Visualizer I found that when both of my hands were in view, the Controller tended to get confused more often; losing track of fingers or have them jutting out at what would be agonisingly painful angles. Using this tool was, however, a great help and saved me many hours of frustration. I had originally set out with the image of a user creating a flipper gesture by rotating a palm-opened hand that was facing towards them around the wrist in a flicking motion. After some experimentation with the Diagnostic Visualizer, I found that extending just one finger, pointing it horizontally flat; facing across the body to the left-hand side and wiggling it back and forwards was the best mechanic for robustly identifying a gesture for activating a flipper. 

Similarly to my PinchGestureListener class, the RightHandFlipGestureListener class is a subclass of Listener and overrides the onFrame method:

Whilst simpler than the PinchGestureListener class, there are still a few notable things to mention about what the above code does: 

1. For simplicity, we the code ignores Frames that have more than one hand in them or where the hand doesn't have any visible fingers.
2. It doesn't strictly enforce only that only one finger can be in view, but attempts to isolate the finger that is furthest over to the left (smallest value on the X axis) as the one to follow - in a hope that the code is resilient to those few occasions/Frames when random fingers are detected by the Controller.
3. Thresholds are defined for how far above and below the Z axis the finger needs to be, to be considered as either completely up (flipper activated) or completely down (flipper deactivated).
4. Instead of making the flipper completely binary (i.e. either on or off), I wanted to introduce the idea of the flipper being in a "falling" state (changing from being activated, but having not come to rest yet), so that if it is activated during this state that it applies less force to the pinball. This models a situation where the flipper is press repeatedly in quick succession. 

The result looks like this: 

Prev (Handling User Input and Simple Joints in JBox2D)