This work proposes DOFS, a pilot dataset of 3D deformable objects (DOs) (e.g., elasto-plastic objects) with full spatial information (i.e., top, side, and bottom information) using a novel and low-cost data collection platform with a transparent operating plane.
The dataset consists of active manipulation action, multi-view RGB-D images, well-registered point clouds, 3D deformed mesh, and 3D occupancy with semantics, using a pinching strategy with a two-parallel-finger gripper. In addition, we trained a neural network with the down-sampled 3d occu- pancy and action as input to model the dynamics of an elasto-plastic object. Our dataset and all CADs will be released soon.
The dynamics of DO (i.e., Plasticine) will perform dramatically differently due to the hollow bottom. For two pieces of plasticine of the same appearance and size, one is solid and another is hollow, the deformation results are completely different with the same action.
Hardware setup of our data collection platform. The blue part is a transparent acrylic board that serves as an operating plane. The orange parts are 4 cameras above the plane to collect top and side information. The green parts are 2 cameras below the plane to collect bottom-side information.
Six RGB-D images captured from RealSense D435i. Four images from the cameras installed above the operating plane show the top and side information of the plasticine. Two images from the cameras installed below the operating plane show the bottom side information of the plas- ticine.
Data visualization of one frame during manipulation. (a): Well-registered point cloud of pinched plasticine without background. (b): Reconstructed deformed mesh. (c): 3D occupancy of plasticine.
Visualization of Full spatial information collection. (a): We embed the cartoon model into the bottom of the plasticine and take it out. (b): The visualization of the point cloud of plasticine. (c): The deformed mesh of plasticine.
State transformation after pinching the plasticine. (a): Visualization of the initial state of the plasticine and two fingers, the yellow arrow indicates the pinch direction. (b): The ground- truth state after the pinch. (c): The inference state of the learned dynamics model.
@article{zhang2024dofs,
title={DOFS: A Real-world 3D Deformable Object Dataset with Full Spatial Information for Dynamics Model Learning},
author={Zhang, Zhen and Chu, Xiangyu and Tang, Yunxi and Au, KW},
journal={arXiv preprint arXiv:2410.21758},
year={2024}
}@inproceedings{
zhang2024dofs,
title={{DOFS}: A Real-world 3D Deformable Object Dataset with Full Spatial Information for Dynamics Model Learning},
author={Zhen Zhang and Xiangyu Chu and TANG Yunxi and K. W. Samuel Au},
booktitle={CoRL Workshop on Learning Robot Fine and Dexterous Manipulation: Perception and Control},
year={2024},
url={https://openreview.net/forum?id=QADznDlGM4}
}