Abstract: We present a system framework for restoration of Han dynasty brick and stone relief(BSR) height from their rubbing images.In our approach we divide BSR into the base relief of low frequency component and the high frequency details.The base relief estimation involves detection of object regions on rubbing images,determination of transitive region for raised slop inside object regions with a histogram based edge sharpness estimation method,definition of raised slope with empirical functions and estimation of height variations inside object regions by a simple interpolation scheme.The estimation of high frequency details involves inverting pixel values of rubbing images and minimal interactive processing.The final height map for restored BSR is obtained by blending height maps of the low frequency component and high frequency details.We tested our method by successfully restoring some BSR.