Visual Modeling and Analytics of Dynamic Environments for the Mass is a research project conducted at Advanced Digital Sciences Center (ADSC), as part of the Data Analytics sub-program. This project aims to address emerging challenges to visual data analytic by developing fundamental technical tools needed to bring structure into ever-growing visual data captured from various sources.

Due to rapid advances in consumer imaging and mobile computing, mass users can now easily capture and process live images anywhere and anytime. How to interpret, manage, and infer information from this ever-growing, big visual data, which records our living environments from different perspectives at different times, is of great significance. The key challenge for making sense of such big visual data is that it often comes in unstructured form, in which images were captured from separate locations, times, and conditions.

Recently, there have been many efforts by industry and governments to build large-scale image databases and 3D models, such as Google Street View and 3D virtual Singapore. The construction of these visual databases requires specialized equipment, controlled conditions, and significant investment. As a result, they cannot be updated often and mainly focus on static and common environments.

We propose to develop a number of key technical tools that enable bringing structure into an arbitrary sequence of images captured by commodity mobile cameras, and through visual correspondence linking these images to established image databases and 3D models on the cloud. Specifically, we propose to develop innovative algorithms that are robust, efficient, and scalable for the following fundamental tasks:

  1. Localization – recovering the geometric locations of the user, the camera viewpoint, and the objects in the environment around the user;
  2. Registration – aligning and modeling dynamically captured images and measurements of the scene over different time and viewpoints;
  3. Inference – estimating and analyzing the semantic information of the scene from the registered visual information and recovered geometric information.

These key computational tools will expand visual modeling and analytics to the masses with commodity mobile devices, and allow mass users to leverage the increasingly rich visual databases on the cloud. Consequently, these tools will lead to numerous applications that have overwhelming practical significance, including visual analytics of dynamic environments with opportunistic sampling, the transferring of annotation from the cloud to consumer visual mobile devices, and augmented reality. In particular, we will focus on three driving applications: 1) holistic indoor modeling and semantic annotation on mobile devices, 2) dense landmark recognition and urban monitoring, and 3) personalized large-scale environment exploration. By collaborating with other research groups in ASDC and I2R, as well as external companies and universities, we will also optimize and customize the developed core technology for other compelling real-life applications.