Session: 07-17-03: Machine Learning and Artificial Intelligence in Dynamics, Vibrations and Control

Paper Number: 146151

146151 - Vision-Guided End Effector Trajectory Planning and Workspace Edge Drifting for Enhanced Real-Time Robot Teleoperation 

Introduction

Teleoperated robotic systems that have computer vision technology are expanding across different industries such as surgical, industrial, and space or ocean exploration applications. While the benefits of vision-guided robot control are evident, challenges arise due to limitations in traditional workspace Cartesian mapping methods, particularly in avoiding singularity points. These limitations can result in jerking movements or system shutdowns at the remote end, presenting significant hurdles to seamless operation.  This work integrates two algorithms in the robot teleoperation: 1) the edge drifting algorithm for the teleoperation workspace mapping, and 2) the deep learning algorithm for obstacle recognition and collision avoidance. By combining these two approaches in robot control, we aim to cover the robot's workspace during task operation with improved accuracy and efficiency, ensuring both safety and seamless motion planning for robot teleoperation.

Contribution of the Work

This work contributes to the advancement of real-time teleoperation in robotic systems utilizing artificial intelligent algorithm in vision-guided robot control. The integration of collision detection and object recognition via convolutional Neural Networks (CNNs) enables the system to dynamically adjust to changes in the workspace, ensuring robust performance even in challenging conditions. Additionally, the implementation of workspace edge drifting techniques further improves the system's adaptability, enabling seamless navigation within constrained spaces. Overall, these innovations enhance the capabilities of robotic systems, making them more versatile and effective for various teleoperation applications.

Methodology

This work begins with the investigation of control methods in MATLAB Simulink for operating a master-follower telerobotic system, where a robotic arm is controlled using a haptic phantom device operated by a human. Additionally, a camera observing the arm robot's end effector and environment aids in obstacle detection. The system also integrates a depth camera (Intel RealSense i435) with Convolutional Neural Networks (CNNs) for obstacle detection, where the CNN analyzes depth data to identify potential hazards, allowing the controller to adjust the robot's path for enhanced safety. A fuzzy PID motion control method is developed to stabilize the follower robot's movement while tracking the master robot's motion. For operation, bilateral communication between Simulink and the robot arm is established through ROS (Robot Operating System) on TCP/IP (Transmission Control Protocol/Internet Protocol). Initial simulation of the robot's control method helps determine optimal joint control to avoid workspace singularities and ensure smooth movement along a vision-guided path. The simulation of the robot arm is performed in MATLAB Simulink. To understand how the haptic robot coordinates with the robot arm, workspace mapping analysis based on minimum jerk principles is conducted. The workspace mapping method is a position-joint hybrid mapping method based on edge drifting. The edge drifting algorithm approach is helpful in avoiding singularity points in robot motion planning. Metrics for measuring the robot arm's performance encompass factors such as trajectory deviation, velocity profiles, and endpoint accuracy, providing a comprehensive evaluation of the control methods used.

Preliminary results and conclusions

We attach a depth camera on the follower robot end effector for object and collision detection. The CNN algorithm is used to analyze image data to identify object shape, depth and potential collision zones. Proactive safety measures are established by identifying potential collisions before they occur. Increased accuracy in tracking the end effector ensures it follows the intended path by the master robot, leading to more delicate and controlled procedures. This allows the control system to proactively adjust the robot's trajectory, preventing accidents and ensuring a safer motion planning. We have also constructed the MATLAB-ROS communication portal for bilateral teleoperation. Future work will focus on assessing the effectiveness of teleoperation when combining workspace edge drifting and the deep-learned vision system.

Presenting Author: Salman Saeed San Jose State University

Presenting Author Biography: Salman Saeed is a graduate research assistant in Mechanical Engineering at San Jose State University, specializing in Controls. His research interests lie in applying control theory principles to solve real-world engineering challenges. Salman brings valuable practical experience to his studies, having worked in the medical device field for over two years. Currently, he leverages his skills as a Manufacturing Engineer at Johnson & Johnson's surgical robotics division. In this role, he contributes to the development and production of cutting-edge robotic surgical systems.

Authors:

Salman Saeed San Jose State University
Katie Sun San Jose State University
Han Xuyen Duong San Jose State University
Lin Jiang San Jose State University