A Self-supervised Learning Method of Target Pushing-Grasping Skills Based on Affordance Map
WU Peiliang1,2, LIU Ruijun1, MAO Bingyi1,2, SHI Haoyang1, CHEN Wenbai3, GAO Guowei3
1. School of Information Science and Engineering, Yanshan University, Qinhuangdao 066004, China; 2. The Key Laboratory for Computer Virtual Technology and System Integration of Hebei Province, Qinhuangdao 066004, China; 3. School of Automation, Beijing Information Science & Technology University, Beijing 100192, China
Abstract：A self-supervised learning method of target pushing-grasping skills based on affordance map is presented. Firstly, the self-supervised learning problem is described for robot to learn target pushing-grasping skills in cluttered environment. The decision process of robot pushing and grasping operation in workspace is defined as a new Markov decision process (MDP), in which the vision mechanism module and action mechanism module are trained separately. Secondly, the adaptive parameters and group split attention module are fused in the vision mechanism module to design the feature extraction network RGSA-Net, which can generate the affordance map from the original state image of the input network, and provide a good premise for the target pushing-grasping operation. Then, a deep reinforcement learning based self-supervised training framework DQAC based on actor-critic framework is built in the action mechanism module. After the robot performs the action according to the affordance map, the DQAC framework is used to evaluate the action, and thus better cooperation between pushing and grasping is realized. Finally, experimental comparison and analysis are carried out to verify the effectiveness of the proposed method.
[1] Morrison D, Corke P, Leitner J.Multi-view picking:Next-bestview reaching for improved grasping in clutter[C]//International Conference on Robotics and Automation.Piscataway, USA:IEEE, 2019:8762-8768. [2] Redmon J, Angelova A.Real-time grasp detection using convolutional neural networks[C]//IEEE International Conference on Robotics and Automation.Piscataway, USA:IEEE, 2015:1316-1322. [3] Levine S, Pastor P, Krizhevsky A, et al.Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection[J].International Journal of Robotics Research, 2018, 37(4-5):421-436. [4] Schmidt P, Vahrenkamp N, Wachter M, et al.Grasping of unknown objects using deep convolutional neural networks based on depth images[C]//IEEE International Conference on Robotics and Automation.Piscataway, USA:IEEE, 2018:6831-6838. [5] Mahler J, Liang J, Niyaz S, et al.Dex-Net 2.0:Deep learning to plan robust grasps with synthetic point clouds and analytic grasp metrics[C]//Robotics:Science and Systems.Cambridge, USA:MIT, 2017.DOI:10.15607/rss.2017.xiii.058. [6] Kumra S, Joshi S, Sahin F.Antipodal robotic grasping using generative residual convolutional neural network[C]//IEEE/RSJ International Conference on Intelligent Robots and Systems.Piscataway, USA:IEEE, 2020:9626-9633. [7] Lou X, Yang Y, Choi C.Learning to generate 6-DoF grasp poses with reachability awareness[C]//IEEE International Conference on Robotics and Automation.Piscataway, USA:IEEE, 2020:1532-1538. [8] Shao Q Q, Hu J, Wang W M, et al.Suction grasp region prediction using self-supervised learning for object picking in dense clutter[C]//IEEE 5th International Conference on Mechatronics System and Robots.Piscataway, USA:IEEE, 2019:7-12. [9] He K M, Zhang X Y, Ren S Q, et al.Deep residual learning for image recognition[C]//IEEE Conference on Computer Vision and Pattern Recognition.Piscataway, USA:IEEE, 2016:770-778. [10] Pinto L, Gupta A.Supersizing self-supervision:Learning to grasp from 50K tries and 700 robot hours[C]//IEEE International Conference on Robotics and Automation.Piscataway, USA:IEEE, 2016:3406-3413. [11] Shukla P, Kumar H, Nandi G C.Robotic grasp manipulation using evolutionary computing and deep reinforcement learning[J].Intelligent Service Robotics, 2021, 14(1):61-77. [12] Kalashnikov D, Irpan A, Pastor P, et al.Learning dexterous in-hand manipulation[J].International Journal of Robotics Research, 2018, 39(1):3-20. [13] Sarantopoulos I, Kiatos M, Doulgeri Z, et al.Total singulation with modular reinforcement learning[J].IEEE Robotics and Automation Letters, 2021, 6(2):4117-4124. [14] Quillen D, Jang E, Nachum O, et al.Deep reinforcement learning for vision-based robotic grasping:A simulated comparative evaluation of off-policy methods[C]//IEEE International Conference on Robotics and Automation.Piscataway, USA:IEEE, 2018:6284-6291. [15] Hou Y X, Li J, Fang Z H, et al.An initialization method of deep Q-network for learning acceleration of robotic grasp[C]//IEEE International Conference on Networking, Sensing and Control.Piscataway, USA:IEEE, 2020.DOI:10.1109/ICNSC48988.2020.9238061. [16] Deng Y H, Guo X F, Wei Y X, et al.Deep reinforcement learning for robotic pushing and picking in cluttered environment[C]//IEEE/RSJ International Conference on Intelligent Robots and Systems.Piscataway, USA:IEEE, 2019:619-626. [17] Xie X, Li C Y, Zhang C, et al.Learning virtual grasp with failed demonstrations via Bayesian inverse reinforcement learning[C]//IEEE/RSJ International Conference on Intelligent Robots and Systems.Piscataway, USA:IEEE, 2019:1812-1817. [18] Johannink T, Bahl S, Nair A, et al.Residual reinforcement learning for robot control[C]//IEEE International Conference on Robotics and Automation.Piscataway, USA:IEEE, 2019:6023-6029. [19] Mohammadi H B, Zamani M A, Kerzel M, et al.Mixed-reality deep reinforcement learning for a reach-to-grasp task[C]//28th International Conference on Artificial Neural Networks.Cham, Switzerland:Springer, 2019:611-623. [20] Ni P Y, Zhang W G, Zhang H R, et al.Learning efficient push and grasp policy in a totebox from simulation[J].Advanced Robotics, 2020, 34(13):873-887. [21] Gui B X, Qian K, Chen S H, et al.Knowledge induced deep Qnetwork for robot push and grasp manipulation skills learning[C]//Chinese Automation Congress.Piscataway, USA:IEEE, 2020:4078-4083. [22] Joshi S, Kumra S, Sahin F.Robotic grasping using deep reinforcement learning[C]//IEEE 16th International Conference on Automation Science and Engineering.Piscataway, USA:IEEE, 2020:1461-1466. [23] Zhang J H, Zhang W, Song R, et al.Grasp for stacking via deep reinforcement learning[C]//IEEE International Conference on Robotics and Automation.Piscataway, USA:IEEE, 2020:2543-2549. [24] Zeng A, Song S, Welker S, et al.Learning synergies between pushing and grasping with self-supervised deep reinforcement learning[C]//IEEE/RSJ International Conference on Intelligent Robots and Systems.Piscataway, USA:IEEE, 2018:4238-4245. [25] Yang Y, Liang H, Choi C.A deep learning approach to grasping the invisible[J].IEEE Robotics and Automation Letters, 2020, 5(2):2232-2239. [26] Gong L Y, He D, Li Z H, et al.Efficient training of BERT by progressively stacking[C]//36th International Conference on Machine Learning.2019:2337-2346. [27] Krizhevsky A, Sutskever I, Hinton G E.ImageNet classification with deep convolutional neural networks[J].Communications of the ACM, 2017, 60(6):84-90. [28] Badrinarayanan V, Kendall A, Cipolla R.SegNet:A deep convolutional encoder-decoder architecture for image segmentation[J].IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(12):2481-2495. [29] Li D Q, Hu X Q, Wang S Q, et al.Hyperspectral images ground object recognition based on split attention[C]//IEEE 2nd International Conference on Big Data, Artificial Intelligence and Internet of Things Engineering.Piscataway, USA:IEEE, 2021:324-330. [30] Barto A G, Sutton R S, Anderson C W.Neuronlike adaptive elements that can solve difficult learning control problems[J].IEEE Transactions on Systems, Man, and Cybernetics, 1983, 13(5):834-846. [31] Rohmer E, Singh S P N, Freese M.V-REP:A versatile and scalable robot simulation framework[C]//IEEE/RSJ International Conference on Intelligent Robots and Systems.Piscataway, USA:IEEE, 2013:1321-1326.
