poseHook

This is an app that continuously detects the body parts of a single person in the frames seen by your device's camera. It transmits the detected positions to a host in form of OSC messages. OSC is a UDP based protocol understood by many music or visual art software, e.g. TouchDesigner, Max for Live, Resolume Arena, etc.

You can download Ableton Live sets demonstrating OSC conversion to MIDI, and much more:

https://bitbucket.org/constanze/posehook/

Detected are 33 body points: nose, mouth left/right corner eyes, ears, etc. It is possible to configure the OSC address names, and IP / port of the receiver. It doesn't support detection of multiple persons. poseHook displays the detected points on top of the camera picture, and plots them.

You can replay the last sent 30 OSC messages in a loop by pushing the play button on the plot.

poseHook uses MediaPipe BlazeNet for the detection. The OSC messages are sent as fast as they are detected. A detection rate of > 10fps is possible with a S10-like phone.

On host side, you can receive OSC messages with various kinds of musical or creative coding software. Note that the data is quite noisy, so you have to take that into account when creating an effect.

Questions how to use poseHook? Or would you like to share your idea for improvement? Get in contact with us on twitter: @holllyhook (trippple L).

Detection works to 100% offline, the data is neither saved nor shared, besides of the intended OSC usage.

poseHook is open source, the code is available at https://bitbucket.org/constanze/posehook/. For a limited time, you can view the poses detected by poseHook v2 online. Send the OSC data to 18.192.229.240, and open http://ec2-18-192-229-240.eu-central-1.compute.amazonaws.com:8081 in the browser. Source code for the demo is available, ready to be used as a starting points for p5.js: https://bitbucket.org/constanze/posehook-demo/