Abstract: In the field of image-based 3D reconstruction, real-time reconstruction approaches based on silhouette or stereo suffered from lacking of details.On the other hand, reconstruction algorithms with high quality need pixel-level energy optimization, which leads to relatively low performance.Based on those problems, this paper presents a real-time 3D reconstruction algorithm, which based on visual hull refined by depth map from time of flight(TOF) sensor.First, the depth data from TOF sensor is refined by a simplified depth calibration method.A fusion approach using both foreground silhouette and depth data is applied to recover an initial coarse model.Finally the model is refined by a local stereo-matching search.Our experiments illustrate that this algorithm is more efficient than heterogeneous cameras-based algorithms in improving the reconstruction quality similar, and the concaves can be recovered effectively.