Abstract: In an augmented reality environment, users interact with virtual objects by hands.Making real-time collision detection between virtual and real objects is essential to enhance the immersion and authenticity of the environment.In the proposed approach, the key points of hands are marked with different colors, and are captured by a pair of common web cameras.The histograms of marked colors are then used to track the key points along time.The disparity of the key points in two consecutive images is computed based on the stereo vision principle.In this way, the 3D locations of the key points are calculated in real-time, and the accurate collision detection is quickly achieved by representing the virtual object models with the OBB structure.Experimental results show that our approach achieves accurate collision detection results when the distance between the objects and the camera is smaller than two meters.