Researchers
Vision-guided robot cloth unfolding
Second place at the ICRA 2024 Cloth Competition, outperforming nine of the ten other competing methods.
Overview
Robotic cloth manipulation is difficult not only because fabric can assume an enormous range of configurations, but also because every grasp changes its geometry: folds move, layers separate, new self-occlusions appear, and visually similar grasp locations can produce very different outcomes. Unlike rigid objects, cloth has no stable shape from which a grasp can be planned directly. Successful unfolding therefore requires estimating both where to grasp and how to orient the gripper from a partial visual observation.
This work follows the task and evaluation procedure introduced by the ICRA 2024 Cloth Manipulation Challenge, also referred to as the ICRA 2024 Cloth Competition. A cloth is first picked up from a random configuration. One robot arm then grasps its lowest point and suspends it in front of the camera, while the second arm must autonomously select and execute a new grasp. Finally, the two arms move apart to stretch the cloth open. The selected grasp must maximize the final unfolded area while remaining reachable and suitable for execution by the real robot system.
Our CeDiRNet-6DoF model predicts both an effective grasp point and the complete spatial orientation of the gripper from the observed hanging-cloth configuration. Center-direction regression localizes the grasp, dedicated rotation heads estimate yaw, pitch, and roll, and depth at the selected image location completes the 3D position. Joint cloth segmentation and pose prediction, randomized training backgrounds, and robot-aware cropping improve transfer from the training data to the real bimanual setup.
Results
At the ICRA 2024 Cloth Competition, Team Ljubljana’s CeDiRNet-6DoF placed second among 11 competing methods, outperforming nine other methods. Across the official 16-trial evaluation, it achieved 57% average cloth coverage, 63% grasp success, and 69% coverage when considering successful grasps only. Only the winning method obtained a higher average coverage, with 60%.
For the paper, we repeated the benchmark with the same competition cloth set and ten trials per item on our bimanual setup. CeDiRNet-6DoF achieved 61.3% average cloth coverage and 72.5% grasp success; considering successful grasps only, average coverage reached 72.6%. On an additional cloth-item set, it achieved 57.4% average coverage, 62.5% grasp success, and 70.4% coverage for successful grasps.
The ablation study shows that joint segmentation, randomized backgrounds, and robot-aware image cropping are important for reliable transfer to the real setup. Compared with the base 6-DoF extension, the complete model improved combined-set coverage from 43.7% to 60.9% and grasp success from 30.6% to 68.8%.
CeDiRNet-6DoF jointly localizes a cloth grasp point and estimates the complete grasp orientation.
Downloads
Code
Models
- Best-performing paper model (RGB, segmentation, background randomization, and robot-aware cropping)
- Model used for the ICRA 2024 Cloth Competition
Both checkpoints contain the learnable center estimator and do not require a separate center-model checkpoint.
Training datasets
- ViCoS Towel Dataset: public real-image base used for pretraining; the recovered training configuration selects 5,120 samples.
- Extended ViCoS Towel Dataset: 559 additional real-image samples used during pretraining.
- Synthetic MuJoCo towel dataset: 12,000 selected synthetic samples used during pretraining.
- ICRA 2024 Cloth Competition dataset: the original CC BY 4.0 dataset used for 6-DoF fine-tuning and validation.
- CeDiRNet-6DoF segmentation-mask sidecar: 503 cloth masks to extract over the original ICRA dataset.
- COCO train2017: external image pool used only for randomized background augmentation; COCO images are not redistributed here.
SHA-256 checksums for the downloadable CeDiRNet-6DoF artifacts