Abstract: Real time integration of virtual objects with real world by interface of live video is an emerging topic of mixed reality.One difficult problem is how to detect the presence of moving obstacles,such as walking people in the live video so as to control the motion of virtual objects accordingly.We propose a solution based on the concept of computer vision.By analyzing the input video in real time,the motion states of objects in the real scene are recovered and then compose the motion constraints of the virtual characters to avoid unrealistic collisions among real and virtual objects.To illustrate the correct visibility of each virtual object,we extract the silhouette of each potential occluder in the video accurately and perform the hidden parts removal in image space.Examples demonstrate the efficacy of our proposed algorithm.