This contact-rich data provides a clear demonstration of how touch can enhance manipulation and how combining touch with vision can bring robots closer to human-level dexterity. The system was tested on tasks like grasping fragile items, such as eggs and grapes, and performing in-hand manipulation, like reorienting a hex key between fingers or adjusting the grip on a spatula. The tactile and visual fusion significantly outperformed systems relying solely on visual inputs, which was especially useful when handling fragile items or doing tasks with limited visibility.

Thin, flexible tactile sensors transform robots from clunky tools into ones capable of precise, fluid manipulation 

The researchers developed a dense, flexible tactile sensor array integrated into a soft robotic gripper. The data from the sensors, combined with visual data, generate a 3D-point cloud, like a visual representation or a scene, that enables the robot to both “see” and “feel” its surroundings. The tactile feedback allows the robot to adjust its grip strength in real time, which is especially crucial when visual information is limited or occluded. 

Equipped with “fingers” capable of feeling the world around them, these robots can now handle fragile objects with care. Thin, flexible tactile sensors cover their hands, enabling them to perceive the slightest pressure and adjust their movements accordingly. This innovation has transformed the robots from clunky tools into ones capable of precise, fluid manipulation once thought impossible for machines.

“This breakthrough also enables robots to handle occluded objects more reliably and effectively,” said Binghao Huang, the project lead and a Columbia Engineering PhD student who works with Li. Occlusion occurs when an object is hidden from view, which is problematic for robots that rely on visual information to manipulate objects. “As the demand for humanoid robots to assist with household chores grows, our bimanual system equipped with tactile sensors presents a promising solution for achieving such tasks.”

What’s next?

With this leap forward in robotics, the line between human and machine skills begins to blur, opening the door to a future where robots cannot only see the world but feel it, too. 

The next step for the researchers is to further improve the system's scalability and to scale up data collection. They are also developing a tactile simulation and integrating it into the robot learning process. This will allow for larger-scale data collection and better generalization of the policy, enabling the system to perform well in new situations for which it was not explicitly trained.

About the Study

Conference: Conference on Robot Learning (CoRL) in Munich, Germany, November 6-9, 2024.

Title: 3D ViTac: Learning Fine-Grained Manipulation with Visuo-Tactile Sensing

Authors: Binghao Huang¹, Yixuan Wang¹, Xinyi Yang², Yiyue Luo³, Yunzhu Li¹, 

1. Columbia University
2. University of Illinois Urbana-Champaign
3. University of Washington

Funding: This work is partly funded by the Toyota Research Institute (TRI). This research solely reflects the opinions and conclusions of its authors and not TRI or any other Toyota entity.