Abstract: To obtain high recognition accuracy of human actions in daily life under different illumination conditions and angles of view, a recognition method based on Kinect is proposed. Inspired by the fact that the human's attention focuses on the detailed partial action in most cases of action recognitions, the proposed recognition method processes the sensed joints data hierarchically. Roughly, actions are classified into the upper limb action and the trunk action by judging the change speed of the position of the trunk joint. Particularly, the hand joints trajectories in the upper limb action and the trunk joint trajectory in the trunk action are focused on. Then, key points are extracted from the two rough categories of trajectories by C-means clustering algorithm for each category separately. And the trajectory of the action is mapped to the corresponding key points. By this means, the sequence of key points in each category is obtained. And the sequences of key points are modeled based on the concept of the proposed temporal order histogram. Nextly, the action recognition is accomplished by comparing the similarities of the sequences of the key points among the trajectories. Finally, the recognition rate of 99% is realized on the collected datasets. The results show that the method is effective for human action recognition task.