Learning and Transfer Methods for Active Rehabilitation Strategy of Upper-limb Rehabilitation Robot

GUO Shijie; SONG Yuanhao; WANG Xusheng; LIU Zuojun; LI Yang

doi:10.13973/j.cnki.robot.240014

Volume 46 Issue 5

Sep. 2024

Turn off MathJax

Article Contents

Abstract

References

ROBOT > 2024 > 46(5): 562-575. > DOI: 10.13973/j.cnki.robot.240014 CSTR: 32165.14.j.cnki.robot.240014

GUO Shijie, SONG Yuanhao, WANG Xusheng, LIU Zuojun, LI Yang. Learning and Transfer Methods for Active Rehabilitation Strategy of Upper-limb Rehabilitation Robot[J]. ROBOT, 2024, 46(5): 562-575. DOI: 10.13973/j.cnki.robot.240014

Citation:

PDF (1490 KB)

Learning and Transfer Methods for Active Rehabilitation Strategy of Upper-limb Rehabilitation Robot

GUO Shijie^{1, 2, 3},
SONG Yuanhao^{1, 2, 3},
WANG Xusheng⁴,
LIU Zuojun^{1, 2, 3},
LI Yang^{1, 2, 3, ,}

1.
Engineering Research Center of the Ministry of Education for Intelligent Rehabilitation Equipment and Detection Technologies, Hebei University of Technology, Tianjin 300130, China
2.
State Key Laboratory of Reliability and Intelligence of Electrical Equipment, Hebei University of Technology, Tianjin 300130, China
3.
Hebei Key Laboratory of Smart Sensing and Human-Robot Interaction, Tianjin 300130, China
4.
Shanghai Fushi Intelligent Technology Co., Ltd, Shanghai 201306, China

More Information

Received Date: January 17, 2024
Revised Date: April 10, 2024
Accepted Date: April 06, 2024

Graphical Abstract

Abstract

Abstract

To achieve active upper-limb rehabilitation for stroke patients, a feedback control strategy based on reinforcement learning is proposed, to transfer the control strategy to the physical environment after training. Firstly, human upper-limb movement is analyzed to establish an equivalent model, and a random trajectory planning method is proposed to mimic voluntary movement in joint space. Subsequently, a simulation environment for man-robot system is established based on the equivalent model. To train the control strategy, the complete states of the system are mapped to observable states through kinematic analysis on the human body and robot. Finally, a feedback control strategy is used to align the feature distribution between environments, in order to transfer the trained control strategy to the robot and avoid negative transfer. Simulation results demonstrate the effectiveness of the reinforcement learning-based control strategy in accomplishing active rehabilitation tasks for stroke patients. The control strategy trained in simulation is transferred to the real robot, and the experimental results verify the effectiveness of the learning and transfer method of rehabilitation strategy, which provides a new technical solution for the active rehabilitation of movement disorder patients.
- active rehabilitation,
- reinforcement learning,
- feedback control,
- man-machine system,
- control strategy transfer

FullText(HTML)

References (35)

References

[1]	HILL M, JÖRGENSEN S, ENGSTRÖM G, et al. Functional and structural impairments of the pulmonary system in middleaged people with cervical and upper thoracic spinal cord injuries[J]. The Journal of Spinal Cord Medicine, 2023, 46(5): 732-741. doi: 10.1080/10790268.2022.2031478
[2]	王陇德, 彭斌, 张鸿祺, 等. 《中国脑卒中防治报告2020》概要[J]. 中国脑血管病杂志, 2022, 19(2): 136-144. doi: 10.3969/j.issn.1672-5921.2022.02.011 WANG L D, PENG B, ZHANG H Q, et al. Brief report on stroke prevention and treatment in China, 2020[J]. Chinese Journal of Cerebrovascular Disease, 2022, 19(2): 136-144. doi: 10.3969/j.issn.1672-5921.2022.02.011
[3]	何畅, 熊蔡华, 陈文斌. 脑损伤上肢康复机器人及其临床应用研究[J]. 机械工程学报, 2023, 59(19): 65-80. doi: 10.3901/JME.2023.19.065 HE C, XIONG C H, CHEN W B. Review on upper-limb rehabilitation robots for patients with brain injury and clinical applications[J]. Journal of Mechanical Engineering, 2023, 59(19): 65-80. doi: 10.3901/JME.2023.19.065
[4]	罗胜利, 孟巧玲, 喻洪流. 我国康复机器人技术研究与应用概况[J]. 中国康复医学杂志, 2023, 38(12): 1762-1768. doi: 10.3969/j.issn.1001-1242.2023.12.023 LUO S L, MENG Q L, YU H L. Survey on research and application of rehabilitation robot technology in our country[J]. Chinese Journal of Rehabilitation Medicine, 2023, 38(12): 1762-1768. doi: 10.3969/j.issn.1001-1242.2023.12.023
[5]	PEREZ-IBARRA J C, SIQUEIRA A A G, SILVA-COUTO M A, et al. Adaptive impedance control applied to robot-aided neuro-rehabilitation of the ankle[J]. IEEE Robotics and Automation Letters, 2019, 4(2): 185-192. doi: 10.1109/LRA.2018.2885165
[6]	梁文渊, 毕胜. 主动康复训练机器人的感知交互与控制策略[J]. 科技导报, 2019, 37(22): 26-36. https://kns.cnki.net/kcms2/article/abstract?v=8WLnD7pOpNFKoroIUYhZckqqzGWu-te37P_7MbxSviNSmygxaK1uaI9PlyHqGOi5p0ToqqnAPEk4qkzK8OSIyE4JQ238FQ3BUPyAOPLVah4NG4c90kHUwVf6dWZwYEHsido1C7b1Hvg=&uniplatform=NZKPT&flag=copy LIANG W Y, BI S. Sensory interaction and control strategy in rehabilitation robot with active training[J]. Science and Technology Review, 2019, 37(22): 26-36. https://kns.cnki.net/kcms2/article/abstract?v=8WLnD7pOpNFKoroIUYhZckqqzGWu-te37P_7MbxSviNSmygxaK1uaI9PlyHqGOi5p0ToqqnAPEk4qkzK8OSIyE4JQ238FQ3BUPyAOPLVah4NG4c90kHUwVf6dWZwYEHsido1C7b1Hvg=&uniplatform=NZKPT&flag=copy
[7]	ZHANG J J, CHEAH C C. Passivity and stability of humanrobot interaction control for upper-limb rehabilitation robots[J]. IEEE Transactions on Robotics, 2015, 31(2): 233-245. doi: 10.1109/TRO.2015.2392451
[8]	PEHLIVAN A U, LOSEY D P, O'MALLEY M K. Minimal assist-as-needed controller for upper limb robotic rehabilitation[J]. IEEE Transactions on Robotics, 2016, 32(1): 113-124. doi: 10.1109/TRO.2015.2503726
[9]	李洋, 朱立爽, 刘今越, 等. 基于动力学模型辨识的全臂柔顺控制[J]. 机械工程学报, 2022, 58(3): 45-54. doi: 10.3901/JME.2022.03.045 LI Y, ZHU L S, LIU J Y, et al. Dynamic model identification for whole-arm compliance control[J]. Journal of Mechanical Engineering, 2022, 58(3): 45-54. doi: 10.3901/JME.2022.03.045
[10]	李洋, 冯适意, 朱德良, 等. 面向移乘护理的冗余双臂机器人行为安全控制[J]. 机械工程学报, 2023, 59(9): 76-89. doi: 10.3901/JME.2023.09.076 LI Y, FENG S Y, ZHU D L, et al. Safety control of a redundant dual-arm robot for transfer-care task[J]. Journal of Mechanical Engineering, 2023, 59(9): 76-89. doi: 10.3901/JME.2023.09.076
[11]	LIU P, GE X Q, LI Y, et al. Study on control strategy of active rehabilitation training for upper limb rehabilitation robots[J]. Journal of University of Chinese Academy of Sciences, 2019, 36(4): 570-576. doi: 10.7523/j.issn.2095-6134.2019.04.018
[12]	吴青聪, 王兴松, 吴洪涛, 等. 上肢康复外骨骼机器人的模糊滑模导纳控制[J]. 机器人, 2018, 40(4): 457-465. doi: 10.13973/j.cnki.robot.18093 WU Q C, WANG X S, WU H T, et al. Fuzzy sliding mode admittance control of the upper limb rehabilitation exoskeleton robot[J]. Robot, 2018, 40(4): 457-465. doi: 10.13973/j.cnki.robot.18093
[13]	程龙, 夏修泽. 上肢康复外骨骼智能控制综述[J]. 机器人, 2022, 44(6): 750-768. doi: 10.13973/j.cnki.robot.210450 CHENG L, XIA X Z. A survey of intelligent control of upper limb rehabilitation exoskeleton[J]. Robot, 2022, 44(06): 750-768. doi: 10.13973/j.cnki.robot.210450
[14]	梁旭, 王卫群, 苏婷婷, 等. 下肢康复机器人的主动柔顺自适应交互控制[J]. 机器人, 2021, 43(5): 547-556. doi: 10.13973/j.cnki.robot.210029 LIANG X, WANG W Q, SU T T, et al. Active compliant and adaptive interaction control for a lower limb rehabilitation robot[J]. Robot, 2021, 43(5): 547-556. doi: 10.13973/j.cnki.robot.210029
[15]	LEI X Y, ZHANG Z A, DONG P F. Dynamic path planning of unknown environment based on deep reinforcement learning[J]. Journal of Robotics, 2018. doi: 10.1155/2018/5781591
[16]	KAHN G, VILLAFLOR A, DING B, et al. Self-supervised deep reinforcement learning with generalized computation graphs for robot navigation[C]//IEEE International Conference on Robotics and Automation. Piscataway, USA: IEEE, 2018: 5129-5136. doi: 10.1109/ICRA.2018.8460655
[17]	TAI L, PAOLO G, LIU M. Virtual-to-real deep reinforcement learning: Continuous control of mobile robots for mapless navigation[C]//IEEE/RSJ International Conference on Intelligent Robots and Systems. Piscataway, USA: IEEE, 2017: 31-36. doi: 10.1109/IROS.2017.8202134
[18]	LAKE B M, ULLMAN T D, TENENBAUM J B, et al. Building machines that learn and think like people[J]. Behavioral and Brain Sciences, 2017, 40. doi: 10.1017/S0140525X16001837
[19]	Wu Y H, Yu Z C, Li C Y, et al. Reinforcement learning in dual-arm trajectory planning for a free-floating space robot[J]. Aerospace Science and Technology, 2020, 98. doi: 10.1016/j.ast.2019.105657
[20]	CHEN X, GHADIRZADEH A, FOLKESSON J, et al. Deep reinforcement learning to acquire navigation skills for wheellegged robots in complex environments[C]//IEEE/RSJ International Conference on Intelligent Robots and Systems. Piscataway, USA: IEEE, 2018: 3110-3116. doi: 10.1109/IROS.2018.8593702
[21]	MENG F C, DAI Y P. Reinforcement learning adaptive control for upper limb rehabilitation robot based on fuzzy neural network[C]//31st Chinese Control Conference. Piscataway, USA: IEEE, 2012: 5157-5161. https://ieeexplore.ieee.org/abstract/document/6390836
[22]	ZHANG Y, LI S, NOLAN K J, et al. Adaptive assist-asneeded control based on actor-critic reinforcement learning[C]//IEEE/RSJ International Conference on Intelligent Robots and Systems. Piscataway, USA: IEEE, 2019: 4066-4071. doi: 10.1109/IROS40897.2019.8968464
[23]	BOEING A, BRÄUNL T. Leveraging multiple simulators for crossing the reality gap[C]//International Conference on Control, Automation, Robotics and Vision. Piscataway, USA: IEEE, 2012: 1113-1119. doi: 10.1109/ICARCV.2012.6485313
[24]	WANG T, BAO X, CLAVERA I, et al. Benchmarking modelbased reinforcement learning[DB/OL]. (2019-07-03)[2023-12-11]. https://arxiv.org/abs/1907.02057.
[25]	GAO C, JIANG Y, CHEN F. Transferring hierarchical structures with dual meta imitation learning[C]//Proceedings of the 6th Conference on Robot Learning. New York, USA: ACM, 2023: 762-773. https://proceedings.mlr.press/v205/gao23b.html
[26]	MEHTA B, DIAZ M, GOLEMO F, et al. Active domain randomization[C]//Proceedings of the Conference on Robot Learning. New York, USA: ACM, 2020: 1162-1176. https://proceedings.mlr.press/v100/mehta20a.html
[27]	CHEN J, WU X X, DUAN L X, et al. Domain adversarial reinforcement learning for partial domain adaptation[J]. IEEE Transactions on Neural Networks and Learning Systems, 2022, 33(2): 539-553. doi: 10.1109/TNNLS.2020.3028078
[28]	SLAOUI R B, CLEMENTS W R, FOERSTER J N, et al. Robust domain randomization for reinforcement learning[DB/OL]. (2020-03-06)[2023-12-11]. https://arxiv.org/abs/1910.10537.
[29]	PENG X B, COUMANS E, ZHANG T, et al. Learning agile robotic locomotion skills by imitating animals[DB/OL]. (2020-07-21)[2023-12-11]. https://arxiv.org/abs/2004.00784.
[30]	严浩, 王洪波, 陈鹏, 等. 具有广义肩关节的上肢康复机器人优化设计[J]. 兵工学报, 2021, 42(11): 2491-2502. doi: 10.3969/j.issn.1000-1093.2021.11.022 YAN H, WANG H B, CHEN P, et al. Optimal design of an upper limb rehabilitation robot with generalized shoulder joint[J]. Acta Armamentarii, 2021, 42(11): 2491-2502. doi: 10.3969/j.issn.1000-1093.2021.11.022
[31]	HE D X, WANG H P, TIAN Y. Fractional-order ultra-local model-based optimal model-free control for 7-DOF iReHave upper-limb exoskeleton[J]. IEEE Transactions on Circuits and Systems Ⅱ: Express Briefs, 2022, 69(8): 3510-3514. doi: 10.1109/TCSII.2022.3161659
[32]	HUANG R, CHENG H, CHEN Q M, et al. Interactive learning for sensitivity factors of a human-powered augmentation lower exoskeleton[C]//IEEE/RSJ International Conference on Intelligent Robots and Systems. Piscataway, USA: IEEE, 2015: 6409-6415. doi: 10.1109/IROS.2015.7354293
[33]	CHIA E Y, CHEN Y L, CHIEN T C, et al. Velocity field based active-assistive control for upper limb rehabilitation exoskeleton robot[C]//IEEE International Conference on Robotics and Automation. Piscataway, USA: IEEE, 2020: 1742-1748. doi: 10.1109/ICRA40945.2020.9196766
[34]	ASL H J, YAMASHITA M, NARIKIYO T, et al. Field-based assist-as-needed control schemes for rehabilitation robots[J]. IEEE/ASME Transactions on Mechatronics, 2020, 25(4): 2100-2111. doi: 10.1109/TMECH.2020.2992090
[35]	陈文斌. 人体上肢运动学分析与类人肢体设计及运动规划[D]. 武汉: 华中科技大学, 2012. doi: 10.7666/d.D232844 CHEN W B. Human upper limb kinematics and anthropomorphic robot kinematic design and motion planning[D]. Wuhan: Huazhong University of Science and Technology, 2012. doi: 10.7666/d.D232844

Cited By

Get Citation

PDF

XML

Article views (189) PDF downloads (130)

Learning and Transfer Methods for Active Rehabilitation Strategy of Upper-limb Rehabilitation Robot

Abstract

References

Catalog

Related

Learning and Transfer Methods for Active Rehabilitation Strategy of Upper-limb Rehabilitation Robot

Abstract

References

Catalog

Related

Export File

Citation

Format

Content