This contact-rich data provides a clear demonstration of how touch can enhance manipulation and how combining touch with vision can bring robots closer to human-level dexterity. The system was tested on tasks like grasping fragile items, such as eggs and grapes, and performing in-hand manipulation, like reorienting a hex key between fingers or adjusting the grip on a spatula. The tactile and visual fusion significantly outperformed systems relying solely on visual inputs, which was especially useful when handling fragile items or doing tasks with limited visibility.

Thin, flexible tactile sensors transform robots from clunky tools into ones capable of precise, fluid manipulation 

The researchers developed a dense, flexible tactile sensor array integrated into a soft robotic gripper. The data from the sensors, combined with visual data, generate a 3D-point cloud, like a visual representation or a scene, that enables the robot to both “see” and “feel” its surroundings. The tactile feedback allows the robot to adjust its grip strength in real time, which is especially crucial when visual information is limited or occluded. 

Equipped with “fingers” capable of feeling the world around them, these robots can now handle fragile objects with care. Thin, flexible tactile sensors cover their hands, enabling them to perceive the slightest pressure and adjust their movements accordingly. This innovation has transformed the robots from clunky tools into ones capable of precise, fluid manipulation once thought impossible for machines.

“This breakthrough also enables robots to handle occluded objects more reliably and effectively,” said Binghao Huang, the project lead and a Columbia Engineering PhD student who works with Li. Occlusion occurs when an object is hidden from view, which is problematic for robots that rely on visual information to manipulate objects. “As the demand for humanoid robots to assist with household chores grows, our bimanual system equipped with tactile sensors presents a promising solution for achieving such tasks.”

What’s next?

With this leap forward in robotics, the line between human and machine skills begins to blur, opening the door to a future where robots cannot only see the world but feel it, too. 

The next step for the researchers is to further improve the system's scalability and to scale up data collection. They are also developing a tactile simulation and integrating it into the robot learning process. This will allow for larger-scale data collection and better generalization of the policy, enabling the system to perform well in new situations for which it was not explicitly trained.

About the Study

Conference: Conference on Robot Learning (CoRL) in Munich, Germany, November 6-9, 2024.

Title: 3D ViTac: Learning Fine-Grained Manipulation with Visuo-Tactile Sensing

Authors: Binghao Huang¹, Yixuan Wang¹, Xinyi Yang², Yiyue Luo³, Yunzhu Li¹, 

1. Columbia University
2. University of Illinois Urbana-Champaign
3. University of Washington

Funding: This work is partly funded by the Toyota Research Institute (TRI). This research solely reflects the opinions and conclusions of its authors and not TRI or any other Toyota entity.

New York, NY—August 23, 2021—A new way to control the motion of bubbles from researchers at Columbia Engineering might one day help separate useful metals from useless dirt using much less energy and water than is currently needed.

When mining for metals such as the copper used in most electronics and the lithium used in many batteries, only a small fraction of the material that is mined is useful metal, with the vast majority just useless dirt-like particles.

"We have to separate the useful metals from the useless particles, and we do this by blowing air bubbles up through them," said Chris Boyce, assistant professor of chemical engineering at Columbia Engineering. However, "this process utilizes a large amount of energy and water, causing climate change and water shortages, thus creating problems we are trying to prevent. We have this issue in part because we currently cannot control the motion of these bubbles."

Now Boyce and his colleagues reveal that if they vibrate these particles while blowing air up through them, the normally chaotic motion of these bubbles becomes orderly and controllable. The vibrations cause the particles to quickly shift between solid-like to fluid-like behavior, which in turn helps structure the bubbles into regularly spaced triangular arrays.

"I think the simple addition of vibration to go from chaos to order is beautiful," Boyce said. Their study appears August 23 in the journal Proceedings of the National Academy of Sciences.

Having a way to control the behavior of these bubbles can help scale up and optimize separation techniques. "We expect that the ability to create structure in flows can reduce energy and water use in mining as well as improve the efficiency of many clean energy processes," Boyce said.

The researchers now aim to apply this structured bubbling to sustainable mining separation techniques.

Image

Columbia Engineering

Columbia Engineering, based in New York City, is one of the top engineering schools in the U.S. and one of the oldest in the nation. Also known as The Fu Foundation School of Engineering and Applied Science, the School expands knowledge and advances technology through the pioneering research of its more than 220 faculty, while educating undergraduate and graduate students in a collaborative environment to become leaders informed by a firm foundation in engineering. The School’s faculty are at the center of the University’s cross-disciplinary research, contributing to the Data Science Institute, Earth Institute, Zuckerman Mind Brain Behavior Institute, Precision Medicine Initiative, and the Columbia Nano Initiative. Guided by its strategic vision, “Columbia Engineering for Humanity,” the School aims to translate ideas into innovations that foster a sustainable, healthy, secure, connected, and creative humanity.

About the Study

The study is titled "Dynamically structured bubbling in vibrated gas-fluidized granular materials."

The study appeared in the journal Proceedings of the National Academy of Sciences on August 23, 2021.

Authors are: Qiang Guo, Yuxuan Zhang, Azin Padash, Kenan Xi, Thomas M. Kovar, and Christopher M. Boyce.

Department of Chemical Engineering, Columbia Engineering.

The researchers received support from the China Scholarships Council and the Bakhmeteff Fellowship for Fluid Mechanics.

Image

Can you tell us a bit more about compound rain clusters and why is this happening?

An open question has been whether the climate will become “stickier” or more random as it warms - stickier meaning that the wind patterns just persist for longer periods of time, random meaning things get disorganized. It seems that there are modest but detectable changes -- at least over the United States -- where the incidence of persistent patterns in which waves of moisture are repeatedly directed to the same places is increasing. Each wave brings rain that starts to fill up dams and then, if a bigger rain event happens and the reservoir is vulnerable, there is the potential for failure.

Our study is the first analysis of rainfall sequences and events associated with recent

hydrologic failures of 552 dams in the U.S. We found that persistent atmospheric circulation patterns that lead to recurrent rainfall events, rather than just more moisture in the atmosphere, are a possible reason. The probability of these compound precipitation risks has increased across part of the country. With over 90,000 aging dams still in service, the increasing likelihood of intense rainfall sequences raises urgent concerns about future dam failures.

What are the near-term solutions?

We need to visit the portfolio of more than 90,000 dams in the U.S. and check not just their state of maintenance but also how they are being operated to decide which ones should be demolished, which ones repaired, and which ones given better strategies to hold on to water they already contain while we improve our predictions of floods or droughts. We need to improve the near-term prediction of persistent rainfall patterns. It’s urgent that we do a portfolio risk analysis that considers the climate, fragility, and operational risk factors with a mapping to potential impacts from dam failure so that we can better understand the collective risk of cascading failure of critical infrastructure that would be triggered by dam failure and its socio-economic impacts.

Is this happening across the globe, not just in the U.S.?

Yes, we see similar behavior in the data in many places around the world, especially in the higher latitudes (>30), where the storm tracks organize and then persist. But a more comprehensive study is needed.

What have been some of the successes out of this collaboration to date? 

Our work has generated the first air pollution data from highly populated cities such as Kinshasa, Democratic Republic of the Congo. We are leading the state of the science in low-cost sensor applications, building relationships with local decision makers and informing policy in India, and increasing capacity among thousands of local practitioners on air quality management and air pollution science. 

Dams in which areas of the U.S. are more prone to collapsing and why?

We have concerns with older dams that are not being maintained; these are primarily concentrated in the Northeast, Upper Midwest, and Southeast. And while Western dams tend to be newer and larger, in 2017 the tallest U.S. dam operated by the State of California -- Oroville -- nearly failed, and 200,000 people had to be evacuated. Persistently high, but not extraordinarily high, rainfall was implicated. The reservoir was full because the operators wanted to hang on to the water in case they lapsed back into drought, and a modest amount of rain forced them to use the overflow spillway (which was in poor repair), and it failed - the concrete just ripped off as the water went over it. They switched to the emergency spillway, which also failed.

Are hurricanes playing a major role in these scenarios? 

Hurricanes are really concerning since the amount of water that they can drop could easily overwhelm the capacity of local reservoir managers to cope with. In 2015, 52 dams failed in South Carolina in just one hurricane event–Hurricane Joaquin. With Hurricane Harvey, the Addicks and Barker Reservoirs in Houston were very vulnerable to failure, and a last-ditch effort to release water from them and flood downstream areas– in the middle of the night–prevented a more catastrophic failure. 

There has been a concern that hurricanes will be more frequent and stronger with global warming, and evidence has been presented in favor of and against this argument. Harvey persisted in place and kept on raining longer than any prior hurricane in the region, so the emerging question is whether we will see more of that. If we do, the vulnerability of dams may go way up since they may not be able to handle the resulting deluge.

What do you think about all this?

This seems to be a silent crisis so far. Since 2000, we’ve had a dam failure due to overtopping, on average every two weeks in the U.S. Luckily, most of these have been small dams, and the loss of life and immediate impact have not been catastrophic. But if a large dam were to fail above a major population center or a power plant or a super fund site or a bridge, the impacts could be devastating and long-lasting. The silence would be broken with a bang.

Lead Photo Credit: Kirk Fisher/Shutterstock