Motion Capture in the Age of AI
New foundations for the capture and analysis of human movement
21 July 2024 2 minute read
Motion capture, or mocap, is broken and has been for a long time. This post describes what's wrong and how to fix it using a new paradigm driven by data and machine learning.
The use of cameras to capture and analyze human movement dates back to the work of Muybridge at the dawn of photography. Today the so-called "gold standard" approach is based on using reflective markers attached to the body and a set of cameras around the subject that detect the makers and compute their 3D coordinates. Given the 3D marker locations, one then "solves" for the 3D skeleton that plausibly gave rise to the markers. The motion of this skeleton can then be retargetted to new graphics characters, analyzed to detect disease, or used to evaluate athletic performance.
The technology has been around for 40 years and mocap labs are quite common. Commercial systems can estimate the 3D locations of markers with sub-millemeter precision. So why do I say mocap is broken and needs to be fixed?
body model
from skeleton to surface
Nuance.
Soft tissue is not a "problem" -- it's part of us
still normative but you have a way of analyzing bias and a way to fix it
errors
what is a gold standard. is marker accuracy relevant
full body v2v error
model soft tissue
Markers
every pixel is a marker
From the outside in with data
MRI and CT to predict the outside from the inside
Goal
from pixels to dynamics, muscle activation, and diagnosis
Full realism.
Capturing a surfer
The Perceiving Systems Department is a leading Computer Vision group in Germany.
We are part of the Max Planck Institute for Intelligent Systems in Tübingen — the heart of Cyber Valley.
We use Machine Learning to train computers to recover human behavior in fine detail, including face and hand movement. We also recover the 3D structure of the world, its motion, and the objects in it to understand how humans interact with 3D scenes.
By capturing human motion, and modeling behavior, we contibute realistic avatars to Computer Graphics.
To have an impact beyond academia we develop applications in medicine and psychology, spin off companies, and license technology. We make most of our code and data available to the research community.