Detecting Shadows from Outdoor Videos under Moving Viewpoints for Augmented Reality

To detect shadows from a live video captured with a free moving camera in augmented reality (AR), an online algorithm based on optical flow tracking is proposed in this paper. Firstly, the algorithm divides shadows in the current frame into the shadows tracked from previous frame with the optical flow tracking algorithm, and the new shadow caused by the camera movement. To alleviate the inevitably cumulative errors produced in tracking process, we refine tracking results under the framework of Graph Cut optimization. Observing that the tracked and new shadow of a frame should have the same illumination condition such as the same magnitude ratio between the sunlight and skylight, the shadow features from tracked shadows after optimization is extracted to detect new shadows. Finally, to eliminate the error of detecting isolated new shadows based on the thresholding method, the Graph Cut optimization is utilized again to refine new detected shadows. The experimental results demonstrate that the algorithm significantly reduces the cumulative error of tracking. Meanwhile, it is able to accurately detect the soft shadows frequently occurring in outdoor scenes.