To date, robots have operated by having a human explicitly model the robot. “But if we want robots to become independent, to adapt quickly to scenarios unforeseen by their creators, then it’s essential that they learn to simulate themselves,” says Hod Lipson, professor of mechanical engineering, and director of the Creative Machines lab, where the research was done. 

Image

For the study, Lipson and his PhD student Robert Kwiatkowski used a four-degree-of-freedom articulated robotic arm. Initially, the robot moved randomly and collected approximately one thousand trajectories, each comprising one hundred points. The robot then used deep learning, a modern machine learning technique, to create a self-model. The first self-models were quite inaccurate, and the robot did not know what it was, or how its joints were connected. But after less than 35 hours of training, the self-model became consistent with the physical robot to within about four centimeters. The self-model performed a pick-and-place task in a closed loop system that enabled the robot to recalibrate its original position between each step along the trajectory based entirely on the internal self-model. With the closed loop control, the robot was able to grasp objects at specific locations on the ground and deposit them into a receptacle with 100 percent success.  

Image

Even in an open-loop system, which involves performing a task based entirely on the internal self-model, without any external feedback, the robot was able to complete the pick-and-place task with a 44 percent success rate. “That's like trying to pick up a glass of water with your eyes closed, a process difficult even for humans,” observed the study’s lead author Kwiatkowski, a PhD student in the computer science department who works in Lipson’s lab.
 

An image of the intact robotic arm used to perform all of the tasks

An image of the deformed robotic arm, deformed by using a 3d printed piece as a replacement for one of the standard arm parts to extend and change the angle of the gripper.

An image of the deformed robotic arm in multiple poses as it was collecting data through random motion.

If we want robots to become independent, to adapt quickly to scenarios unforeseen by their creators, then it’s essential that they learn to simulate themselves

The self-modeling robot was also used for other tasks, such as writing text using a marker. To test whether the self-model could detect damage to itself, the researchers 3D-printed a deformed part to simulate damage and the robot was able to detect the change and re-train its self-model. The new self-model enabled the robot to resume its pick-and-place tasks with little loss of performance. 

Image

Lipson, who is also a member of the Data Science Institute, notes that self-imaging is key to enabling robots to move away from the confinements of so-called “narrow-AI” towards more general abilities. “This is perhaps what a newborn child does in its crib, as it learns what it is,” he says. “We conjecture that this advantage may have also been the evolutionary origin of self-awareness in humans. While our robot’s ability to imagine itself is still crude compared to humans, we believe that this ability is on the path to machine self-awareness.”

Lipson believes that robotics and AI may offer a fresh window into the age-old puzzle of consciousness. “Philosophers, psychologists, and cognitive scientists have been pondering the nature self-awareness for millennia, but have made relatively little progress,” he observes. “We still cloak our lack of understanding with subjective terms like ‘canvas of reality,’ but robots now force us to translate these vague notions into concrete algorithms and mechanisms.”

Lipson and Kwiatkowski are aware of the ethical implications. “Self-awareness will lead to more resilient and adaptive systems, but also implies some loss of control,” they warn. “It’s a powerful technology, but it should be handled with care.”

The researchers are now exploring whether robots can model not just their own bodies, but also their own minds, i.e. whether robots can think about thinking.

Columbia Engineering

Columbia Engineering, based in New York City, is one of the top engineering schools in the U.S. and one of the oldest in the nation. Also known as The Fu Foundation School of Engineering and Applied Science, the School expands knowledge and advances technology through the pioneering research of its more than 220 faculty, while educating undergraduate and graduate students in a collaborative environment to become leaders informed by a firm foundation in engineering. The School’s faculty are at the center of the University’s cross-disciplinary research, contributing to the Data Science Institute, Earth Institute, Zuckerman Mind Brain Behavior Institute, Precision Medicine Initiative, and the Columbia Nano Initiative. Guided by its strategic vision, “Columbia Engineering for Humanity,” the School aims to translate ideas into innovations that foster a sustainable, healthy, secure, connected, and creative humanity.

Journal

Science Robotics

About the Study

The study is titled “Task-Agnostic Self-Modeling Machines.”

Authors are: Robert Kwiatkowski, Department of Computer Science, and Hod Lipson, Department of Mechanical Engineering, Columbia Engineering, and Data Science Institute, Columbia University.

The study was supported by the Defense Advanced Research Projects Agency (DARPA MTO HR0011-18-2-0020).

The study, led by students Pedro Piacenza and Keith Behrman, was published online in IEEE/ASME Transactions on Mechatronics. It demonstrates the two aspects of the underlying technology that combine to enable the new results. Firstly, in this project, the researchers use light to sense touch. Under the “skin,” their finger has a layer made of transparent silicone, into which they shined light from more than 30 LEDs. The finger also has more than 30 photodiodes that measure how the light bounces around. Whenever the finger touches something, its skin deforms, so light shifts around in the transparent layer underneath. Measuring how much light goes from every LED to every diode, the researchers end up with close to 1,000 signals that each contain some information about the contact that was made. Since light can also bounce around in a curved space, these signals can cover a complex 3D shape such as a fingertip.

“The human finger provides incredibly rich contact information--more than 400 tiny touch sensors in every square centimeter of skin!” says Ciocarlie. “That was the model that pushed us to try and get as much data as possible from our finger. It was critical to be sure all contacts on all sides of the finger were covered--we essentially built a tactile robot finger with no blind spots.”

Secondly, the team designed this data to be processed by machine learning algorithms. Because there are so many signals, all of them partially overlapping with each other, the data is too complex to be interpreted by humans. Fortunately, current machine learning techniques can learn to extract the information that researchers care about: where the finger is being touched, what it is touching the finger, how much force is being applied, etc.

“Our results show that a deep neural network can extract this information with very high accuracy,” says Kymissis, who is also a member of the Data Science Institute. “Our device is truly a tactile finger designed from the very beginning to be used in conjunction with AI algorithms.”

In addition, the team built the finger so it, and others, can be put onto robotic hands. Integrating the system onto a hand is easy: thanks to this new technology, the finger collects almost 1,000 signals, but only needs a 14-wire cable connecting it to the hand, and it needs no complex off-board electronics. The researchers already have two dexterous hands (capable of grasping and manipulating objects) in their lab being outfitted with these fingers--one hand has three fingers, and the other one four. In the next months, the team will be using these hands to try and demonstrate dexterous manipulation abilities, based on tactile and proprioceptive data.

“Dexterous robotic manipulation is needed now in fields such as manufacturing and logistics, and is one of the technologies that, in the longer term, are needed to enable personal robotic assistance in other areas, such as healthcare or service domains,” Ciocarlie adds.

Columbia Engineering

Columbia Engineering, based in New York City, is one of the top engineering schools in the U.S. and one of the oldest in the nation. Also known as The Fu Foundation School of Engineering and Applied Science, the School expands knowledge and advances technology through the pioneering research of its more than 220 faculty, while educating undergraduate and graduate students in a collaborative environment to become leaders informed by a firm foundation in engineering. The School’s faculty are at the center of the University’s cross-disciplinary research, contributing to the Data Science Institute, Earth Institute, Zuckerman Mind Brain Behavior Institute, Precision Medicine Initiative, and the Columbia Nano Initiative. Guided by its strategic vision, “Columbia Engineering for Humanity,” the School aims to translate ideas into innovations that foster a sustainable, healthy, secure, connected, and creative humanity.

About the Study

The study is titled “A Sensorized Multicurved Robot Finger with Datadriven Touch Sensing via Overlapping Light Signals.”

Authors are: Pedro Piacenza and Matei Ciocarlie, Mechanical Engineering; Keith Behrman and Ioannis Kymissis, Electrical Engineering; and Benedikt Schifferer, Computer Science.

The work was sponsored in part by the National Science Foundation, under its CAREER program (grant IIS-1551631) and a National Robotics Initiative (grant CMMI-1734557).

The authors declare no financial or other conflicts of interest.

About the Study

Journal: Nature

Title: “Infrared nanosensors of piconewton to micronewton forces.”

Authors: Natalie Fardian-Melamed^1*, Artiom Skripka^2,3, Benedikt Ursprung¹, Changhwan Lee¹, Thomas P. Darlington¹, Ayelet Teitelboim², Xiao Qi², Maoji Wang⁴, Jordan M. Gerton⁴, Bruce E. Cohen^2,5, Emory M. Chan², P. James Schuck¹

1. Department of Mechanical Engineering, Columbia University
2. The Molecular Foundry, Lawrence Berkeley National Laboratory
3. Nanomaterials for Bioimaging Group, Departamento de Física de Materiales, Facultad de Ciencias, Universidad Autόnoma de Madrid
4. Department of Physics and Astronomy, University of Utah
5. Division of Molecular Biophysics and Integrated Bioimaging, Lawrence Berkeley National Laboratory, 

Acknowledgments: N.F.-M. gratefully acknowledges support from the European Union's Horizon 2020 research and innovation program under the Marie Skłodowska-Curie grant agreement No. 893439, the US Department of State Fulbright Scholarship Program, the Zuckerman-CHE STEM Leadership Program, the Israel Scholarship Education Foundation (ISEF) International Fellowship Program, and the Weizmann Institute’s Women’s Postdoctoral Career Development Award. B.U. and P.J.S. acknowledge support by the National Science Foundation under grant no. CHE-2203510. A.S. acknowledges the support from the European Union's Horizon 2020 research and innovation program under the Marie Skłodowska-Curie grant agreement No. 895809 (MONOCLE). Work at the Molecular Foundry was supported by the Office of Science, Office of Basic Energy Sciences, of the US Department of Energy under contract number DE-AC02-05CH11231. X.Q., B.E.C., and E.M.C. were supported in part by the Defense Advanced Research Projects Agency (DARPA) ENVision program under contract HR0011257070, and C.L. and P.J.S. under DARPA ENVision contract HR00112220006. T.P.D. and P.J.S. also acknowledge support for the scan-probe measurements from Programmable Quantum Materials, an Energy Frontier Research Center funded by the US DOE, Office of Science, Basic Energy Sciences (BES), under award DE-SC0019443. 

The authors declare no financial or other conflicts of interest.