Abstract: Two key techniques of surgical navigation system are studied. For the problem of motion tracking of a surgical instrument in cluttered background, a surgical instrument identification plate with circular texture pattern is designed, and it is tracked by continuously adaptive mean-shift (CamShift) algorithm based on hue, saturation and texture features. Texture is extracted by uniform local binary pattern (LBP) which is a way of point sample estimation. And then Hough-circle transform is introduced to overcome the problem of re-initializing search window when the object is lost in order to implement a fully automatic tracking. For visualization of probe in the three-dimensional model, the probe position captured by the camera is put into the coordinate system of three-dimensional focus model reconstructed preoperatively, according to the relationship that there is only a scale factor difference between the probe position under patient coordinate system and the probe position under model mark coordinate system. The simulation experiment result shows that the system is simple and stable, and can meet the requirements of surgical navigation precision.