
Dataset Generation Method.
Transparent objects are common in daily life, while their optical properties pose challenges for RGB-D cameras to capture accurate depth information. This issue is further amplified when these objects are hand-held, as hand occlusions further complicate depth estimation. For assistant robots, however, accurately perceiving hand-held transparent objects is critical to effective human-robot interaction. This paper presents a Hand-Aware Depth Restoration (HADR) method based on creating an implicit neural representation function from a single RGB-D image. The proposed method utilizes hand posture as an important guidance to leverage semantic and geometric information of hand-object interaction. To train and evaluate the proposed method, we create a high-fidelity synthetic dataset named TransHand-14K with a real-to-sim data generation scheme. Experiments show that our method has better performance and generalization ability compared with existing methods. We further develop a real-world human-to-robot handover system based on HADR, demonstrating its potential in human-robot interaction applications.
We create a high fidelity synthetic dataset named TransHand-14K using Blender. This dataset is used for the percpetion research of hand-held transparent objects.
Dataset Generation Method.
Teaser of TransHand-14K.
We introduce a Hand-Aware Depth Restoration (HADR) method for hand-held tranparent objects. (a) Ray and voxel features are generated from the RGB and corrupted point cloud. (b) We introduce hand pose as an additional guidance for depth restoration. (c) The terminated probability and position of each ray-voxel pair are predicted. The final restored depth is obtained by ray-wise maxpooling.
Overview of HADR.
We develop a handover pipeline for transparent objects based on the proposed HADR method. The whole handover process is divided into three stages: (1) Wait & Observe, (2) Approach & React, (3) Grasp & Retrieve.
Overview of Proposed Handover Pipeline for Transparent Objects.
@article{yu2024depth,
title = {Depth Restoration of Hand-Held Transparent Objects for Human-to-Robot Handover},
author = {Yu, Ran and Yu, Haixin and Yan, Huang and Song, Ziwu and Li, Shoujie and Ding, Wenbo},
journal = {arXiv preprint arXiv:2408.14997},
year = {2024}
}