PlayFeat! - Hand tracking game for iPhone and iPad

The brief was to build with a community near us. We picked PUSPADI, an NGO in Bali that makes prosthetics, orthotics and active wheelchairs for people across Bali and eastern Indonesia. We expected to hear about the equipment. What we heard about was where it could not go. Public parks, mostly. The kerbs, the gates, the paths that stop halfway. Going outside to move for an hour was not about wanting to. It was about whether the place would let you in. The problem was not in anyone's body. It was in the paths. So we asked a smaller question: if the park is shut to you, what does moving for fun look like at home?

PlayFeat! is a cooking game you play with your hands in the air. Vision tracks your hand joints through the front camera. You pinch to grab floating ingredient bubbles, drop them on a plate, then carry the dish to a bell at the side of the screen. Each dish has a timer and you get three lives. Nothing to hold, nothing to stand up for. I was the lead coder: the mechanics, the hand tracking, and every threshold in it.

My Approach

Three layers. Camera at the back, a transparent SpriteKit scene in the middle, SwiftUI on top. HandPoseManager owns the camera and the Vision work. GameStateManager owns the rules.

The split was for testing. Anything with tricky timing became a small struct that runs without a camera: PinchDetector for grabbing, HoverDetector for the bin. I tested timing in seconds instead of waving at an iPad.

Vision and Innovation

I wanted the food to feel like it was really there. You reach out, close your fingers, and the bubble comes with you. Break that link once and the illusion goes.

So my work went into attaching the hand to the player, not the camera. I tested a fist against a pinch and picked pinch. A fist hides the fingers at the moment you need them, and you can do a pinch with your elbow resting on a chair arm.

The Challenges

Vision reads one frame at a time and has no memory. Blur your hand, or let your thumb pass behind a finger, and for one frame it reports nothing. Handle that naively and the hand vanishes and drops what it held. It reads as a broken game, not a struggling camera. Vision also returns hands in no fixed order, and will track someone walking past behind you.

How I Solved Them

PinchDetector decides from a run of frames instead of one. Three agreeing frames commit a change, a tenth of a second at 30Hz. A gap between the close and open thresholds stops a hand on the boundary chattering.

The real fix was smaller. The first version treated a missing reading as an open hand, so the grab let go. "I cannot see it" is not "it is open". Now a missing measurement is ignored and the hand keeps its state. On top I added handGracePeriod: a hand Vision stops reporting stays drawn at its last position until that runs out. Before, the cursor blinked out and back as you played. Now you never see it.

HumanBodyPoseManager settles whose hand. It takes the nearest person, the widest shoulders on screen, and only hands near their wrists.

Player-Centric Design

One hand mode. You pick left or right in settings and the game tracks only that hand. People play one handed for all sorts of reasons, and a two handed game locks them out. I made it a toggle, not a separate mode.

Seat calibration runs before play, sitting you where both hands and your torso fit in frame. Otherwise the game fails mid round and leaves you guessing why.

What I Built

• PinchDetector: reads a pinch from a run of frames, so one bad frame cannot drop what you hold.

• Hand grace period: holds a hand at its last position for 0.35 seconds after Vision loses it.

• Hand identity tracking: matches hands frame to frame so two never swap what they hold.

• HumanBodyPoseManager: picks the nearest person and ignores hands behind them.

• One hand mode: limits tracking to a chosen left or right hand.

• HoverDetector: fires once when you hold a hand over the bin to dump the plate.

• Dish clock and lives: times each dish and takes a life when it runs out.

Architecture

Files are grouped by job. Managers holds the brains and no UI, Scenes the SpriteKit layer, Views the SwiftUI. GameScene is one class split across five files, so input, layout, spawning and serving each have their own. Anything fiddly lives in a standalone struct, which is why the coordinate mapping and gesture timing have real tests.

What I Took From It

The lesson was about missing data. My first version treated "no reading" and "a reading of zero" as the same thing, and that assumption caused most of the bugs I chased. Absence is not a value. I owned the core systems while my teammate worked on the details around them, which held only because they could change a screen without touching how hands are tracked.