Lead Photo Caption: Sami Haddadin (left) and Dean Shih-Fu Chang (right).

Lead Photo Credit: Eileen Barroso/Columbia University

“Many people truly grasped AI’s rapid evolution when ChatGPT emerged, but AI research at Columbia had been thriving for years,” said Vishal Misra, vice dean for computing and AI at Columbia Engineering and professor of computer science. “This event was a chance to spotlight the work happening across the University and underscore our unique interdisciplinary approach to AI.”

Columbia leverages its broad academic strengths to explore AI beyond just technological development from medicine to law and business to journalism.

“AI is transforming every field,” said Richard Zemel, the Trianthe Dakolias Professor of Engineering and Applied Science in the Department of Computer Science, and event co-host. “Columbia’s approach ensures we are not just advancing AI technology, but also examining its broader implications and shaping policies for its responsible use.”

A showcase and network opportunity 

“I'm a researcher and I love to share what I've been doing,” said Yunzhu Li, assistant professor of computer science. Li and his students presented a robot equipped with advanced tactile sensors that significantly improved its ability to perform tasks requiring precise gripping. Li emphasized the importance of participating in these types of demo events, noting that they create valuable opportunities for collaboration. 

“These sessions open the door for partnerships with fellow professors and, of course, students who can bring fresh ideas and perspectives to the research,” he added.

The steady hum of conversation between participants and the crowd of attendees filled the space, and discussions ranged from technical breakthroughs to real-world applications of the projects. 

“We have only scratched the surface with our research and need more collaborators,” said Judah Goldfelder, a demo session participant and Engineering PhD student in the Creative Machines Lab. His project, developed in partnership with Google, focused on reducing buildings’ carbon emissions by five percent. Advocating for an open-source model, he convinced Google to make the project publicly accessible, enabling contributors to collaborate via GitHub.

This initiative produced a publicly available toolkit to help researchers develop AI-powered heating and cooling systems, reducing energy waste and pollution. The open-source benchmark aims to standardize research efforts in this area and foster collaboration across institutions. “Scaling this project requires more collaborators, and I got a lot of inquiries about the project at this event,” Goldfelder said.

The AI demo session featured 13 projects from across the School. Visit this project page for a list of the day’s demo descriptions and participants.

Lead Photo Credit: David Dini/Columbia Engineering

In the new study, the researchers instead developed a way for robots to autonomously model their own 3D shapes using a single regular 2D camera. This breakthrough was driven by three brain-mimicking AI systems known as deep neural networks. These inferred 3D motion from 2D video, enabling the robot to understand and adapt to its own movements. The new system could also identify alterations to the bodies of the robots, such as a bend in an arm, and help them adjust their motions to recover from this simulated damage.

Such adaptability might prove useful in a variety of real-world applications. For example, "imagine a robot vacuum or a personal assistant bot that notices its arm is bent after bumping into furniture," Hu says. "Instead of breaking down or needing repair, it watches itself, adjusts how it moves, and keeps working. This could make home robots more reliable—no constant reprogramming required."

Another scenario might involve a robot arm getting knocked out of alignment at a car factory. "Instead of halting production, it could watch itself, tweak its movements, and get back to welding—cutting downtime and costs," Hu says. "This adaptability could make manufacturing more resilient."

As we hand over more critical functions to robots, from manufacturing to medical care, we need these robots to be more resilient. “We humans cannot afford to constantly baby these robots, repair broken parts and adjust performance. Robots need to learn to take care of themselves, if they are going to become truly useful,” says Lipson. “That’s why self-modeling is so important.”

The ability demonstrated in this study is the latest in a series of projects that the Columbia team has released over the past two decades, where robots are learning to become better at self-modeling using cameras and other sensors. 

In 2006, the research team’s robots were able to use observations to only create simple stick-figure-like simulations of themselves. About a decade ago, robots began creating higher fidelity models using multiple cameras. In this study, the robot was able to create a comprehensive kinematic model of itself using just a short video clip from a single regular camera, akin to looking in the mirror. The researchers call this newfound ability “Kinematic Self-Awareness.” 

“We humans are intuitively aware of our body; we can imagine ourselves in the future and visualize the consequences of our actions well before we perform those actions in reality,” explains Lipson. “Ultimately, we would like to imbue robots with a similar ability to imagine themselves, because once you can imagine yourself in the future, there is no limit to what you can do.”

The researchers detailed their findings February 25 in the journal Nature Machine Intelligence.

Lead Photo Description: A robot observes its reflection in a mirror, learning its own morphology and kinematics for autonomous self-simulation. The process highlights the intersection of vision-based learning and robotics, where the robot refines its movements and predicts its spatial motion through self-observation. 

Credit: Jane Nisselson/Columbia Engineering 

About The Study

Journal: Nature Machine Intelligence

Title: Teaching Robots to Build Simulations of Themselves

Authors: Yuhang Hu 1, Jiong Lin 1, and Hod Lipson 1, 2

Affiliations:

1 Creative Machines Laboratory, Mechanical Engineering Department, Columbia University, New York, NY 10027, USA

2 Data Science Institute, Columbia University, New York, NY, 10027, USA

DOI: 10.1038/s42256-025-01006-w

Funding/Acknowledgements: This work was supported in part by the U.S. National Science Foundation (NSF) AI Institute for Dynamical Systems (DynamicsAI.org), grant 2112085.

All the authors declare that they have no competing interests.