A Multi-modal Interactive Physiotherapy Robot with Multiple Physiotherapy Operations and Acupoint Recognition

To address the shortcomings of physiotherapy robots in dynamic acupoint recognition and interaction performance, a multimodal interactive physiotherapy robot system is proposed, which combines multiple physiotherapy operations with dynamic acupoint precise recognition. Firstly, a multimodal human-machine interaction framework integrating vision, touch, force, position, and language is proposed by considering the individual user characteristics and the requirements for safety and compliance in the physiotherapy process, and a physiotherapy robot system is constructed. Additionally, a pneumatic end-effector capable of simulating various physician operations is integrated into the system. Secondly, a dynamic acupoint recognition paradigm for the human back is developed, including dataset construction, annotation and processing of datasets, as well as training and optimization of acupoint recognition models. A two-stage acupoint processing method is introduced, and a multiscale acupoint recognition model is established, named RTOP-AcuDet (real-time object and pose-based acupoint detection), to achieve recognition and tracking of back acupoints. Finally, the end-effector is controlled based on the characteristics of physiotherapy operations by using a hybrid force-position control method combined with task planning methods for efficient reproduction of physiotherapy operations. In experimental verification, the proposed high-precision dynamic recognition method of back acupoints achieves the model average error of about 5.52 mm, the average recall rate of about 91.43%, and the model detection efficiency of 31.5 frames/s, and can be combined with the multi-modal humanmachine interaction framework to form a multi-modal physiotherapy robot system suitable for a variety of physiotherapy service scenarios.