Abstract: In order to solve the tracking problem caused by the deformation and irregular motion of gestures in the process of human-computer interaction, a nonparametric kernel density estimation algorithm based on the feature space segmentation modeling is proposed. Firstly, the AdaBoost classifier in the detection module is used to detect the presence of gestures in the image, and the gesture position information is sent to the tracking module, which accurately extracts the gesture target for color modeling. Then, the probability density image of motion target is obtained by estimating the posterior probability density of each frame image using the color model of the target, which is decomposed into the gesture motion area and the similar color interference region. Finally, the mixed Gauss model is used to weaken the interference of close color objects in the similar color region. The redetection module is started when the target is lost, and the gesture position can be detected by adopting the Bayesian classifier and the variance classifier. The experimental results show that the proposed method resolves the similar color interference and redetection problem in deformed gesture tracking by segmenting the feature space and cascading different classifiers. The proposed algorithm improves the tracking accuracy (>81.5%) and is suitable for complex scenes involving the irregular motion of nonrigid objects, and has a high tracking accuracy.