Abstract: With rapid developments of virtual reality and human computer interaction,next-generation public games for home and office wide have drawn wide attentions.In this paper,a natural interaction technique for public games,i.e.hand gesture interaction technique,is well studied from the aspects both hardware and software.A low-cost hand-held interactive device is chosen and re-engineered for gesture interaction games.A two-pass method for moving target detection is proposed to improve the robustness of hand-held target detection,which takes into account the image luminance,color and shape features.A deterministic finite state automaton is developed for hand gesture interaction.Hand gestures are parameterized and a rule-based approach is proposed to classify them.As an experimental example,a hand-held LED light pen is adopted to drive the popular PC game Tetris,which replaces the traditional key controls.Experimental results show that the proposed hand gesture interactions are natural and effectively enhances user's experiences on the game.