Researchers Take a Leaf From Humanity's Book to Make Robots Smarter, More Capable

Researchers working at Columbia University have joined forces with the Toyota Research Institute to bring household robotics a step closer to reality — by teaching them core concepts like object permanence.

"The progress that Carl Vondrick and Shuran Song have made with their research contributes directly to Toyota Research Institute's mission," explains Eric Krotkov, PhD, Toyota's advisor to the Columbia University research program. "TRI's research in robotics and beyond focuses on developing the capabilities and tools to address the socioeconomic challenges of an aging society, labor shortage, and sustainable production. Endowing robots with the capabilities to understand occluded objects and handle deformable objects will enable them to improve the quality of life for all.”

"Some of the hardest problems for artificial intelligence are the easiest for humans," explains Vondrick, associate professor of computer science, of the work he and his team have been doing. The problem: computer vision systems losing track of an object when it's blocked from view, a skill toddlers learn during games of peek-a-boo with their parents and carers.

Imbuing a robot with object permanence, though, proved tricky — and Vondrick and colleagues happened upon the trick of showing the neural network videos demonstrating the physical concepts the robots would need to understand, standing in for the sort of interactive play between children and carers that would teach the same concepts naturally. By converting camera data into three dimensions, plus time as the fourth, the network proved able to understand that objects still exist even when you can't see them.

"In our work, we are trying to investigate how humans intuitively do things," adds Song, assistant professor in computer science and the leader of the Columbia Artificial Intelligence and Robotics (CAIR) lab, though her team had a different focus: the soft-body problem, trying to anticipate what soft objects — like a length of rope — will do when manipulated. Borrowing a trick from how humans learn, Song's team allowed a robot to attempt to hit a cup with a rope through trial and error learning in an iterative residual policy (IRP) algorithm — with it succeeding in just seven tries, compared to the 100 to 1,000 tries it took a traditional machine learning algorithm.

The same research was then extended to a more useful task, of the sort a household robot might need to carry out daily: opening a bag. Using a self-supervised learning framework dubbed DextAIRity, the resulting robot figured out that it could open plastic bags and unfold cloths with a carefully-positioned puff of air. "One of the interesting strategies the system developed with the bag-opening task is to point the air a little above the plastic bag to keep the bag open," explains PhD student Zhenjia Xu. "We did not annotate or train it in any way; it learned it by itself."

More information on the object permanence project and the iterative residual policy algorithm are available on their respective websites, along with their papers and source code.

Main article image courtesy of Zhenjia Xu/Columbia University.