4D Processing Pipeline

4D performance capture is a great source of data on where each millimeter of the actor’s skin is moving at each moment in time.

Different capture methods use depth cameras, infrared or color cameras to achieve that. Some methods use a single camera (monocular capture), others use multiple synchronized cameras capturing images of an actor from different views.

In this documentation we only cover the method of videogrammetry where an actor’s performance is captured using multiple synchronized cameras producing a set of image sequences for each camera. Images for each frame are then used to reconstruct a 3D scan of the actor’s face using traditional photogrammetry reconstruction software. The resulting sequences of 3D scans contain a lot of useful data but it can’t be used directly in production because each scan has a unique topology and UV.

The next step should be to convert the sequence of 3D scans into a consistent topology that can be used in production. This is where our 4D processing pipeline comes in handy. The pipeline is built around 4 standalone applications:

R3DS Wrap4D

Wrap4D is an extended version of Wrap. It contains 10 special nodes designed specifically for the processing of 4D sequences. Wrap4D takes a sequence of textured 3D scans as an input and produces a sequence of meshes with consistent topology as the output. It’s important that each frame is computed independently, which means we can process them in parallel. In order to produce good results Wrap4D requires a sparse set of markers (dots) to be painted on an actor’s face. The marker positions are used as initialization for the wrapping process and greatly increase robustness when processing extreme facial expressions. Wrap4D also uses information about eyelid and lip contours that are detected in R3DS Track.

Wrap4D includes the following nodes for 4D processing:

R3DS Track

R3DS Track is a 2D tracking software that fulfills two purposes:

  • Tracking markers on the actor’s face.

  • Detecting lip and eyelid contours using a personalized facial detector.

R3DS Node

R3DS Node is a headless version of R3DS Wrap4D with a command-line interface. It can be used in combination with R3DS Rush to run computations over a network.

R3DS Rush

R3DS Rush is a simple to use tool that allows running R3DS Wrap4D project computations on multiple machines over a network. R3DS Rush automatically distributes and manages the workload over multiple R3DS Node instances.

4D Processing Video Tutorials

Getting Started

Tutorial 01. Preparing a Neutral Mesh

Tutorial 02. Labeling Markers

Tutorial 03. Rendering Scans

Tutorial 04. Tracking Markers

Tutorial 05. Lip and Eyelid Detection

Tutorial 06. Facial Annotation

Tutorial 07. Facial Wrapping

Tutorial 08. Introduction into Rush