Abstract: This paper proposes a robust method of tracking human head poses from a sequence of monocular images.First we estimate the head pose parameters in the first frame by an affine correspondence based method.Then linear brightness and depth constraint equations are derived from the small interframe rigid motion assumption.At the same time,we use a 3D head model to provide depth measurements and take advantage of geometry information of the features on the face surface to weight the brightness and depth constraints and refine the results.Finally,in order to remove the effects of gradual lighting changes and occlusions,we estimate the reliability of the features frame by frame and dynamically update the reliable feature set.Experiments show that the proposed method can robustly track the head poses even for heads moving with large depth changes.