Video chat with low cost, high quality

figure 1

Researchers at the Advanced Digital Sciences Center (ADSC) have achieved a major advance inthe state of the art for real-time, robust video object cutout, and demonstrated this breakthrough in CuteChat, a lightweight video chat system that uses an ordinary laptop, webcam, and the public internet. CuteChat offers a variety of features to provide the illusion that users are somewhere other than where they really are. Overall, CuteChat is a major advance over previous teleimmersive systems, which are typically bulky, require high Internet bandwidth, are too computationally intensive to run on an ordinary PC, and are not always easy to use.

To provide the illusion of being somewhere else, CuteChat must remove the background from the user’s camera feed, and replace it by another background or video feed. Although it is easy for humans to ignore a visual backdrop, no previous algorithm can do a decent job of cutting out a user out from her backdrop in real-time, using a conventional camera and processor -– not even with hardware assistance in the form of a GPU or FPGA –- in spite of intense interest in both the industrial and research communities. CuteChat provides bilateral cutout at VGA resolution for two or more users, while CuteChat’s cutout algorithm on its own can process 18 frames per second at 1280×960 resolution on an ordinary laptop, with a CPU load of less than 40%, and is robust against challenges such as movement and similar colors in the foreground and background. CuteChat uses object-based video coding to reduce bandwidth requirements and speed up the video coding process, both by a factor of 3 to 4 compared to standard approaches.

CuteChat has a quick and easy user initialization process, and is entirely automated thereafter. CuteChat’s fun features include the ability to merge users onto a shared background, beautify their skin, outline objects in black, or render objects in a limited range of colors. With a depth camera, participants can interact with each other or with the background content using CuteChat’s gesture recognition, such as paging through presentation slides or photo albums with a wave of the arm.