Video cutout in real-time with commodity hardware
ADSC has created and demonstrated a video object cutout technique that automatically segments a foreground layer from a live VGA or HD video stream in real time, using only a single webcam and a commodity desktop or laptop CPU [Lu11a]. A video of the cutout system in action can be seen here.
Grounded on a solid inference basis, the cutout algorithm probabilistically fuses different cues together into a unified optimization framework. The cutout approach includes many novel and robust design choices, which are key to achieving the high accuracy and good performance shown in the prototype implementation of the cutout algorithm. Further, the overall solution framework and some of its key components are applicable for real-time video object cutout with stereo cameras or depth cameras, another key area of interest at ADSC [LMPD11, MLD11a, MLD11b].
ADSC’s new approach to video object cutout advances the state of the art for this classical computer vision problem in three major ways.
- Reliable segmentation with a single webcam. ADSC’s approach works reliably with a single webcam, yet produces accurate results whose quality is comparable to those obtained with more costly or complex setups, such as stereo cameras, depth sensors, and flash and no-flash image pairs. ADSC’s approach is robust against intensity and color variations in the subject’s appearance and a variety of background changes present in real-life applications, all of which previous methods find very challenging to handle: background and foreground with similar colors, dramatic movements in the foreground, a non-stationary background with moving objects, illumination changes, and shaky cameras.
- Real-time performance. ADSC’s cutout approach extracts foreground objects at unprecedented speeds on an ordinary PC, e.g., processing 18 frames per second at a 1280×960 resolution on a mainstream laptop CPU, with a CPU load of less than 40%. This contrasts with previous methods, which process even 160×120 or 320×240 low-resolution videos at only 7-15 frames per second, or entirely offline.
- Ease of use. ADSC’s cutout approach requires simple and minimal user participation during system initialization, and is completely automated thereafter. These advantages make ADSC’s cutout approach appealing and practical for wide application in video conferencing, e-learning, and virtual reality.