Vision for robotic manipulation of deformable objects

Deformable textile objects such as towels and garments can assume highly varied configurations, making robust grasp perception substantially harder than for rigid objects. Our research addresses this challenge directly from visual observations of cloth, covering both top-down grasping and bimanual unfolding in the air.

The two project lines share center-direction regression for robust grasp-point localization and differ in the pose representation and robotic task they address:

Robot cloth unfolding with 6-DoF grasp poses

Vision-guided bimanual unfolding of deformable cloth using a regrasping-in-the-air strategy: one arm suspends the cloth while the other selects a grasp that opens it. CeDiRNet-6DoF predicts an effective image location and the complete gripper orientation, while camera depth completes the 3D position needed to execute the grasp on the real robot setup.

Top-down cloth grasping with 3-DoF poses

Top-down visual perception for grasping deformable cloth resting on a support surface. CeDiRNet-3DoF combines center-direction regression with orientation estimation to predict an image-plane grasp location and in-plane angle. The method and the accompanying ViCoS Towel Dataset cover varied towel and garment configurations, lighting, backgrounds, and clutter for robust real-world evaluation.