Match moving
This article has multiple issues. Please help improve it or discuss these issues on the talk page. (Learn how and when to remove these template messages)
|
In
Match moving is primarily used to track the movement of a camera through a shot so that an identical virtual camera move can be reproduced in a
As it is mostly software-based, match moving has become increasingly affordable as the cost of computer power has declined; it is now an established visual-effects tool and is even used in live television broadcasts as part of providing effects such as the yellow virtual down-line in American football.
Principle
The process of match moving can be broken down into two steps.
Tracking
The first step is identifying and
Calibration
The second step involves solving for 3D motion. This process attempts to derive the motion of the camera by solving the inverse-projection of the 2-D paths for the position of the camera. This process is referred to as calibration.
When a point on the surface of a three-dimensional object is photographed, its position in the 2-D frame can be calculated by a
The projection function P takes as its input a camera vector (denoted camera) and another vector the position of a 3-D point in space (denoted xyz) and returns a 2D point that has been projected onto a plane in front of the camera (denoted XY). We can express this:
- XY = P(camera, xyz)
The projection function transforms the 3-D point and strips away the component of depth. Without knowing the depth of the component an inverse projection function can only return a set of possible 3D points, that form a line emanating from the
- xyz ∈ P'(camera, XY)
or
- {xyz :P(camera, xyz) = XY}
Let's say we are in a situation where the features we are tracking are on the surface of a rigid object such as a building. Since we know that the real point xyz will remain in the same place in real space from one frame of the image to the next we can make the point a constant even though we do not know where it is. So:
- xyzi = xyzj
where the subscripts i and j refer to arbitrary frames in the shot we are analyzing. Since this is always true then we know that:
- P'(camerai, XYi) ∩ P'(cameraj, XYj) ≠ {}
Because the value of XYi has been determined for all frames that the feature is tracked through by the tracking program, we can solve the reverse projection function between any two frames as long as P'(camerai, XYi) ∩ P'(cameraj, XYj) is a small set. Set of possible camera vectors that solve the equation at i and j (denoted Cij).
- Cij = {(camerai,cameraj):P'(camerai, XYi) ∩ P'(cameraj, XYj) ≠ {})
So there is a set of camera vector pairs Cij for which the intersection of the inverse projections of two points XYi and XYj is a non-empty, hopefully small, set centering on a theoretical stationary point xyz .
In other words, imagine a black point floating in a white void and a camera. For any position in space that we place the camera, there is a set of corresponding parameters (orientation, focal length, etc.) that will photograph that black point exactly the same way. Since C has an infinite number of members, one point is never enough to determine the actual camera position.
As we start adding tracking points, we can narrow the possible camera positions. For example, if we have a set of points {xyzi,0,...,xyzi,n} and {xyzj,0,...,xyzj,n} where i and j still refer to frames and n is an index to one of many tracking points we are following. We can derive a set of camera vector pair sets {Ci,j,0,...,Ci,j,n}.
In this way multiple tracks allow us to narrow the possible camera parameters. The set of possible camera parameters that fit, F, is the intersection of all sets:
- F = Ci,j,0 ∩ ... ∩ Ci,j,n
The fewer elements are in this set the closer we can come to extracting the actual parameters of the camera. In reality errors introduced to the tracking process require a more statistical approach to determining a good camera vector for each frame,
In all, the 3D solving process is the process of narrowing down the possible solutions to the motion of the camera until we reach one that suits the needs of the composite we are trying to create.
Point-cloud projection
Once the camera position has been determined for every frame it is then possible to estimate the position of each feature in real space by inverse projection. The resulting set of points is often referred to as a point cloud because of its raw appearance like a nebula. Since point clouds often reveal some of the shape of the 3-D scene they can be used as a reference for placing synthetic objects or by a reconstruction program to create a 3-D version of the actual scene.
Ground-plane determination
The camera and point cloud need to be oriented in some kind of space. Therefore, once calibration is complete, it is necessary to define a ground plane. Normally, this is a unit plane that determines the scale, orientation and origin of the projected space. Some programs attempt to do this automatically, though more often the user defines this plane. Since shifting ground planes does a simple transformation of all of the points, the actual position of the plane is really a matter of convenience.
Reconstruction
3-D reconstruction is the interactive process of recreating a photographed object using tracking data. This technique is related to photogrammetry. In this particular case we are referring to using match moving software to reconstruct a scene from incidental footage.
A reconstruction program can create three-dimensional objects that mimic the real objects from the photographed scene. Using data from the point cloud and the user's estimation, the program can create a virtual object and then extract a texture from the footage that can be projected onto the virtual object as a surface texture.
2-D vs. 3-D
This article's use of external links may not follow Wikipedia's policies or guidelines. (November 2020) |
Match moving has two forms. Some compositing programs, such as
Three-dimensional match moving tools make it possible to extrapolate three-dimensional information from two-dimensional photography. These tools allow users to derive camera movement and other relative motion from arbitrary footage. The tracking information can be transferred to
- 3DEqualizer from Science.D.Visions (which won an Academy Award for Technical Achievement)[1]
- Blender (open source; uses libmv)
- Voodoo
- ACTS an automatic camera tracking with dense depth recovery system for handling image/video sequences
- LS-ACTS a robust and efficient structure-from-motion system which can handle large image/video sequence datasets in near real-time and robustly work in challenging cases (e.g. loopback sequences and multiple sequences)
- VISCODA VooCAT
- Icarus (University of Manchester research project, now discontinued but still popular)
- Maya MatchMover
- The Pixel Farm PFTrack, PFMatchit, PFHoe (based on PFTrack algorithms)
- KeenTools GeoTracker, PinTool
- SynthEyes by Andersson Technologies
- Boujou (which won an Emmy awardin 2002)
- NukeX from The Foundry
- CameraTracker (a plug-in for Adobe After Effects) from The Foundry.
- VideoTrace from Punchcard (software for generating 3D models from video and images)
- IXIR 2D Track Editor It is capable of 2D tracks and Mask files of software like 3D Equalizer, PFTrack, Boujou, SynthEyes, Matchmover, Movimento, Nuke, Shake, Fusion, After Effects, Combustion, Mocha, Silhouette
- mocha Pro from Imagineer Systems, Planar Tracker-based utility for post production
- fayIN a plug-in for Adobe After Effects from fayteq (acquired by Facebook in 2017)
- Meshroom from Alicevision, a free and open-source photogrammetry application that also allows users to export an animated camera along with a 3D reconstruction of a scene
Automatic vs. interactive tracking
There are two methods by which motion information can be extracted from an image. Interactive tracking, sometimes referred to as "supervised tracking", relies on the user to follow features through a scene. Automatic tracking relies on computer
The advantage of automatic tracking is that the computer can create many points faster than a human can. A large number of points can be analyzed with statistics to determine the most reliable data. The disadvantage of automatic tracking is that, depending on the algorithm, the computer can be easily confused as it tracks objects through the scene. Automatic tracking methods are particularly ineffective in shots involving fast camera motion such as that seen with hand-held camera work and in shots with repetitive subject matter like small tiles or any sort of regular pattern where one area is not very distinct. This tracking method also suffers when a shot contains a large amount of motion blur, making the small details it needs harder to distinguish.
The advantage of interactive tracking is that a human user can follow features through an entire scene and will not be confused by features that are not rigid. A human user can also determine where features are in a shot that suffers from motion blur; it is extremely difficult for an automatic tracker to correctly find features with high amounts of motion blur. The disadvantage of interactive tracking is that the user will inevitably introduce small errors as they follow objects through the scene, which can lead to what is called "drift".
Professional-level motion tracking is usually achieved using a combination of interactive and automatic techniques. An artist can remove points that are clearly anomalous and use "tracking mattes" to block confusing information out of the automatic tracking process. Tracking mattes are also employed to cover areas of the shot which contain moving elements such as an actor or a spinning ceiling fan.
Tracking mattes
A tracking matte is similar in concept to a
Refining
Since there are often multiple possible solutions to the calibration process and a significant amount of error can accumulate, the final step to match moving often involves refining the solution by hand. This could mean altering the camera motion itself or giving hints to the calibration mechanism. This interactive calibration is referred to as "refining".
Most match moving applications are based on similar algorithms for tracking and calibration. Often, the initial results obtained are similar. However, each program has different refining capabilities.
Real time
On-set, real-time camera tracking is becoming more widely used in feature film production to allow elements that will be inserted in post-production be visualised live on-set. This has the benefit of helping the director and actors improve performances by actually seeing set extensions or CGI characters whilst (or shortly after) they do a take. No longer do they need to perform to green/blue screens and have no feedback of the result. Eye-line references, actor positioning, and CGI interaction can now be done live on-set giving everyone confidence that the shot is correct and going to work in the final composite.
To achieve this, a number of components from hardware to software need to be combined. Software collects all of the 360 degrees of freedom movement of the camera as well as metadata such as zoom, focus, iris and shutter elements from many different types of hardware devices, ranging from motion capture systems such as active LED marker based system from PhaseSpace, passive systems such as Motion Analysis or Vicon, to rotary encoders fitted to camera cranes and dollies such as Technocranes and Fisher Dollies, or inertia & gyroscopic sensors mounted directly to the camera. There are also laser based tracking systems that can be attached to anything, including Steadicams, to track cameras outside in the rain at distances of up to 30 meters.
Motion control cameras can also be used as a source or destination for 3D camera data. Camera moves can be pre-visualised in advance and then converted into motion control data that drives a camera crane along precisely the same path as the 3-D camera. Encoders on the crane can also be used in real time on-set to reverse this process to generate live 3D cameras. The data can be sent to any number of different 3D applications, allowing 3D artists to modify their CGI elements live on set as well. The main advantage being that set design issues that would be time-consuming and costly issues later down the line can be sorted out during the shooting process, ensuring that the actors "fit" within each environment for each shot whilst they do their performances.
Real time motion capture systems can also be mixed within camera data stream allowing virtual characters to be inserted into live shots on-set. This dramatically improves the interaction between real and non-real MoCap driven characters as both plate and CGI performances can be choreographed together.
See also
References
- Matchmoving: The Invisible Art of Camera Tracking, by Tim Dobbert, Sybex, Feb 2005, ISBN 0-7821-4403-9
- 3-D Estimation and Applications to Match Move - An early paper on match moving, which gets in depth about mathematics.
- Comparison of matchmoving and tracking applications
- Tracking and 3D Matchmoving Tutorials*Dead Link*
External links
- Matchmoving explained at FLIP Animation blog Retrieved May 2013