Humans know how to interact with each other using body language and communication, but robots developed to work alongside people don’t have this innate sensibility. 

Artificial intelligence (AI) is helping collaborative robots predict how a person likely will behave so it can respond accordingly.  That is one reason why researchers have been using movies, popular TVs shows, and sports to “train” the system on human behavior.

Researchers from Columbia Engineering developed a computer-vision technique for giving machines a more intuitive sense for patterns of human behavior by using higher-level associations between people, animals, and objects. They also leveraged an ancient form of geometry in their mathematical framework for how the AI functions.

The algorithm, the most accurate method to date for predicting video action events up to several minutes in the future, analyzed thousands of hours of movies, sporting events, and TV shows like “The Office.” In this way, it learned to predict hundreds of human interactions and activities, such as handshakes or fist bumps.

When the system can’t predict the specific action, it finds a higher-level concept that is related to the action, such as, in the case of handshakes or fist bumps, the word “greeting,” researchers said.

Related:Will Advanced Artificial Skin Raise Hope in Restoring a Sense of Touch?

“Our algorithm is a step toward machines being able to make better predictions about human behavior, and thus better coordinate their actions with ours,” said Carl Vondrick, assistant professor of computer science at Columbia, who directed the study. “Our results open several possibilities for human-robot collaboration, autonomous vehicles, and assistive technology.”

Evolution of Technology

The team is no stranger to predictive machine learning. However, in the past, its efforts, like those of other scientists, included predicting just one action at a time. In this scenario, the algorithms must decide how to classify an act—such as a hug, high five, or even someone’s lack of response, which would be classified as “ignore.”

However, when algorithms couldn’t with high certainty identify an action, most couldn’t find common threads between following potential options for action, researchers said.

To find a new way to develop long-range prediction models, Columbia Engineering Ph.D. students Didac Suris and Ruoshi Liu took a different approach based on the idea that “ not everything in the future is predictable,” Suris said in a press statement.

“When a person cannot foresee exactly what will happen, they play it safe and predict at a higher level of abstraction,” he said in the statement. “Our algorithm is the first to learn this capability to reason abstractly about future events.”

Related:Can New Materials Discovery get a Boost From Artificial Intelligence?

In other words, the new AI model can recognize when it can’t predict a future action with certainty and, like people do all the time, can make a guess as to what it will be by relating it to a concept.

To develop the model based on this idea, Suris and Liu used unusual geometries that aren’t commonly taught in high-school mathematics; instead, they date back to the time of ancient Greeks. These geometries have counter-intuitive properties in which the straight lines of typical geometry can bend, and triangles don’t have three even sides but instead bulge, researchers said.

Using principles of this unusual type of geometry, researchers constructed AI models that could organize high-level concepts to predict future human behavior based on how predictable events are in the future, they said.

For example, humans know that swimming and running are both types of exercise. The system can categorize such related activities on its while also being aware of uncertainty. The latter instance provides more specific actions when there is a certainty and more generic predictions when there is not, researchers said.

“Human behavior is often surprising,” Vondrick said in a press statement. “Our algorithms enable machines to anticipate better what they are going to do next.”

Improving Human-Robot Relations

Researchers wrote a paper on their work and presented the study at the International Conference on Computer Vision and Pattern Recognition on June 24.

While computers currently take action based on preprogramming, the model developed by the team can help give collaborative robots more spontaneity, just as humans do in their interactions, Liu said. This can help humans who in the future work closely with robots form a type of relationship, he said.

 “Trust comes from the feeling that the robot understands people,” Liu explained in a press statement. “If machines can understand and anticipate our behaviors, computers will be able to seamlessly assist people in daily activity.”

Researchers plan to continue their work by verifying how the algorithm works outside of the lab in diverse settings rather than merely testing it on benchmark tasks, they said. If it’s successful, the model can develop and eventually use robots that can work alongside humans to improve our safety, health, and security, researchers said.

The team also plans to continue to improve the algorithm’s performance with larger datasets and computers, and other forms of geometry.

Elizabeth Montalbano is a freelance writer who has written about technology and culture for more than 20 years. She has lived and worked as a professional journalist in Phoenix, San Francisco, and New York City. In her free time, Elizabeth enjoys surfing, traveling, music, yoga, and cooking. She currently resides in a village on the southwest coast of Portugal.