A major theme of the conference was the application of engineering principles and data analysis techniques to a wide range of underaddressed problems in women’s health. 

“Our field can learn from other areas of engineering, such as automotive and aerospace, that already use physics- and data-driven digital twins and models to accelerate product design and solutions,” said co-organizer Kristin Myers, a professor of mechanical engineering at Columbia Engineering. “Women’s health challenges mirror a lot of the major outstanding challenges in health more generally, such as designing sensors and monitors that fit a patient’s need while conveying actionable information to a medical provider.”

The daylong event featured keynotes from Silvia Blemker, professor of biomedical engineering at the University of Virginia; Steven Levine, the senior director of virtual human modeling at Dassault Systèmes; and Sandra Brown, the dean of the College of Nursing and Allied Health, Southern University and A&M College. The expert panels covered gynecologic health, preterm/pregnancy health, pelvic floor health, commercialization and the regulatory path, and education and workforce.

Lead Photo Caption: Christine Hendon, associate professor of electrical engineering, who develops innovative imaging technologies to study human tissue and disease.

Lead Photo Credit: David Dini/Columbia Engineering

Dr. Masaki Suwa, the head of corporate research and development at Omron, and the president and CEO of OMRON SINIC X Corporation, said, “Our Automated Optical Inspection (AOI) solutions play a central role in ensuring the quality of printed circuit boards. Because Columbia’s MPS technology is robust to spurious reflections when inspecting mirror-like surfaces such as solder joints, dies, and chip surfaces, it has proved essential to reliable 3D inspection. As electronic components continue to miniaturize, a technology like MPS that can capture 3D shapes with high precision will become increasingly important to printed circuit board manufacturing.” 

It is rare for a technology developed in a university laboratory to achieve large-scale adoption in a fast-moving and highly demanding field such as factory automation. 

“The successful commercialization of Micro Phase Shifting underscores both the strength of Columbia’s creative fundamental research and the value of close collaboration between academia and industry to bring breakthrough innovations into real-world manufacturing environments,” said Ofra Weinberger, director of Columbia Technology Ventures at Columbia University.

”When we began this research project, we were motivated by a fundamental question: How do you recover accurate 3D information when light behaves in complex and non-ideal ways?” said Gupta. “We showed that by coding light smartly, one could separate the true 3D signal from the noise due to interreflections — a long-standing open problem in 3D imaging. Seeing that idea evolve into a method deployed at scale to help ensure the reliability of critical technologies has been a career highlight.” 

By creating an approach adopted by industry, the researchers demonstrated the value of academic research in bringing fresh ideas and rigorous thinking to business.

“Academic researchers explore a wide spectrum of problems, ranging from theoretical questions that seek to advance the knowledge base of the field to novel solutions to known practical problems,” Nayar said. “It is exciting to see one of our innovations solving a critical problem in the manufacturing of products we use on a daily basis.”

For more information about MPS technology, please visit the project page.

Lead Photo Caption: An artist’s rendering of micro phase shifting. 

Lead Photo Credit: Anna Collevecchio/Columbia Engineering

Robot watches itself talking 

Achieving realistic robot lip motion is challenging for two reasons: First, it requires specialized hardware containing a flexible facial skin actuated by numerous tiny motors that can work quickly and silently in concert. Second, the specific pattern of lip dynamics is a complex function dictated by sequences of vocal sounds and phonemes. 

Human faces are animated by dozens of muscles that lie just beneath a soft skin and sync naturally to vocal chords and lip motions. By contrast, humanoid faces are mostly rigid, operating with relatively few degrees of motion, and their lip movement is choreographed according to rigid, predefined rules. The resulting motion is stilted, unnatural, and uncanny.

In this study, the researchers overcame these hurdles by developing a richly actuated, flexible face and then allowing the robot to learn how to use its face directly by observing humans. First, they placed a robotic face equipped with 26 motors in front of a mirror so that the robot could learn how its own face moves in response to muscle activity. Like a child making faces in a mirror for the first time, the robot made thousands of random face expressions and lip gestures. Over time, it learned how to move its motors to achieve particular facial appearances, an approach called a “vision-to-action” language model (VLA).

Then, the researchers placed the robot in front of recorded videos of humans talking and singing, giving AI that drives the robot an opportunity to learn how exactly humans’ mouths moved in the context of various sounds they emitted. With these two models in hand, the robot’s AI could now translate audio directly into lip motor action.

The researchers tested this ability using a variety of sounds, languages, and contexts, as well as some songs. Without any specific knowledge of the audio clips' meaning, the robot was then able to move its lips in sync.

The researchers acknowledge that the lip motion is far from perfect. “We had particular difficulties with hard sounds like ‘B’ and with sounds involving lip puckering, such as ‘W’. But these abilities will likely improve with time and practice,” Lipson said. 

More importantly, however, is seeing lip sync as part of more holistic robot communication ability. 

“When the lip sync ability is combined with conversational AI such as ChatGPT or Gemini, the effect adds a whole new depth to the connection the robot forms with the human,” explained Yuhang Hu, who led the study for his PhD. “The more the robot watches humans conversing, the better it will get at imitating the nuanced facial gestures we can emotionally connect with.” 

“The longer the context window of the conversation, the more context-sensitive these gestures will become,” he added. 

The missing link of robotic ability

The researchers believe that facial affect is the ‘missing link’ of robotics. 

“Much of humanoid robotics today is focused on leg and hand motion, for activities like walking and grasping,” said Lipson. “But facial affection is equally important for any robotic application involving human interaction.”

Lipson and Hu predict that warm, lifelike faces will become increasingly important as humanoid robots find applications in areas such as entertainment, education, medicine, and even elder care. Some economists predict that over a billion humanoids will be manufactured in the next decade.

“There is no future where all these humanoid robots don’t have a face. And when they finally have a face, they will need to move their eyes and lips properly, or they will forever remain uncanny,” Lipson estimates.

“We humans are just wired that way, and we can’t help it. We are close to crossing the uncanny valley,” added Hu.

Risks and limits

This work is part of Lipson’s decade-long quest to find ways to make robots connect more effectively with humans, through mastering facial gestures such as smiling, gazing, and speaking. He insists that these abilities must be acquired by learning, rather than being programmed using stiff rules. 

“Something magical happens when a robot learns to smile or speak just by watching and listening to humans,” he said. “I’m a jaded roboticist, but I can’t help but smile back at a robot that spontaneously smiles at me.”

Hu explained that human faces are the ultimate interface for communication, and we are beginning to unlock their secrets.

“Robots with this ability will clearly have a much better ability to connect with humans because such a significant portion of our communication involves facial body language, and that entire channel is still untapped,” Hu said. 

The researchers are aware of the risks and controversies surrounding granting robots greater ability to connect with humans. 

“This will be a powerful technology. We have to go slowly and carefully, so we can reap the benefits while minimizing the risks,” Lipson said. 

Lead Photo Caption: Featured on the cover of Science Robotics, Hod Lipson and his team have created a robot that, for the first time, is able to learn facial lip motions for tasks such as speech and singing.

Lead Photo Credit: Jane Nisselson/ Columbia Engineering

About the Study

Title: “Learning Realistic Lip Motions for Humanoid Face Robots”

Authors: Yuhang Hu, Jiong Lin, Judah Allen Goldfeder, Philippe M. Wyder, Yifeng Cao, Steven Tian, Yunzhe Wang, Jingran Wang, Mengmeng Wang, Jie Zeng, Cameron Mehlman, Yingke Wang, Delin Zeng, Boyuan Chen and Hod Lipson

Funding: The study was supported by US National Science Foundation (NSF) AI Institute for Dynamical Systems (DynamicsAI.org) and a gift from Amazon to Columbia AI institute.