Session: 15-01-01: ASME International Undergraduate Research and Design Exposition

Paper Number: 100682

100682 - Toward Automatic Ground-Truth Image Labelling Using Industrial Robotics and Motion Capture 

Training and validation of a Deep Neural Network (DNN) for object tracking in images requires a series of large, ground-truth data sets. Previous work has identified the most cumbersome and error-prone part of the training/validation process is labeling of ground-truth. This work focuses on automated ground-truth labeling of large image data sets using industrial robotic manipulators, commercial motion capture (MoCap), camera calibration, and image projection. The preliminary application of this work is ground-truth labeling for DNN training associated with the last 15 feet of an aircraft’s approach during autonomous aerial refueling. In this work, we are using the North Atlantic Treaty Organization KC-130 high speed aerial refueling system. The goal of this work is to create a data set to supplement imagery collected during actual aerial refueling to later train and evaluate a DNN providing feedback for an aircraft’s refueling approach.

 

The system developed for this work consists of a rigidly fixed 7-DoF (degree of freedom) Yaskawa SIA20F robotic manipulator, a 6-DoF Universal Robots CB-Series UR10, a 12-camera OptiTrack Prime 41x MoCap, a KC-130 refueling drogue rigidly mounted Yaskawa end-effector, and a machine vision camera rigidly mounted to the end-effector of the UR10 manipulator. The UR10 manipulator is mounted to an industrial cart providing lateral movement relative to the Yaskawa manipulator. This movement extends the UR10 manipulator’s workspace to provide the 15-foot range required for our application. 

 

For DNN training, our system precisely defines a bounding box for images containing the KC-130 aerial refueling drogue. To do this, we need: (1) camera parameters; and (2) the relative pose of the KC-130 drogue frame relative to the camera frame. 

 

Noting that a “rigid body” is defined by fixing three or more reflective MoCap spheres to a rigid object, our system natively measures the following information:

MoCap Measurements:

H^w_bU – the UR10 base MoCap frame relative to the MoCap world

H^w_tU – the UR10 end-effector MoCap frame relative to the MoCap world

H^w_bY – the Yaskawa manipulator base MoCap frame relative to the MoCap world

H^w_tY – the UR10 end-effector MoCap frame relative to the MoCap frame

H^w_m – the checkerboard camera calibration fiducial MoCap frame

UR10 Measurements:

H^oU_eU – the UR10 end-effector frame relative to the UR10 base

Yaskawa Measurments:

H^oY_eY_eY_oY – the Yaskawa end-effector frame relative to the Yaskawa base

Camera Measurements

H^c_f – the checkerboard camera calibration fiducial frame relative to the camera

 

To establish a bounding box, Hdc must be defined for a given image. Using a homogeneous definition of rigid body transformations (i.e. H∈SE(3)), Eq. 1 provides the simplest definition of H^c_d. 

H^c_d=H^c_tU * H^tU_w * H^w_tY * H^tY_d  . (1)

Assuming that that H^t=tY_d can be established using a future computer aided design (CAD) model, the remaining unknown is H^c_tU

 

To establish H^c_tU, we collect n images of a checkerboard fiducial relative the camera along with the corresponding H^w_tU and H^w_m MoCap pose information. Using camera calibration, H^c_f is recovered for each image, and the resultant data is summarized as n transformation triplets {H^c_f, H^tU_w, H^w_m}_i for i∈{1,2,…n}. Noting that H^f_m and H^c_tU are fixed based on MoCap marker placement, H^c_tU and H^f_m can be established using the AX=XB solution presented by Park when the correspondence is defined {H^c_f, (H^w_tU)^-1, H^w_m}_i for i∈{1,2,…n}.

Preliminary results show an initial average extrinsic error of 1.23in and a reprojection error of 42.2 pixels in a 1280x960 image. Given the 30-inch diameter of the KC-130 aerial refueling drogue, these results suggest feasibility of our approach.

Presenting Author: Charles Doherty United States Naval Academy

Presenting Author Biography: United States Naval Academy senior majoring in Robotics and Control Engineering.

Authors:

Charles Doherty United States Naval Academy
Harrison Helmich United States Naval Academy
Michael Kutzer United States Naval Academy
Donald Costello United States Naval Academy