To date, robots have operated by having a human explicitly model the robot. “But if we want robots to become independent, to adapt quickly to scenarios unforeseen by their creators, then it’s essential that they learn to simulate themselves,” says Hod Lipson, professor of mechanical engineering, and director of the Creative Machines lab, where the research was done. 

Image

For the study, Lipson and his PhD student Robert Kwiatkowski used a four-degree-of-freedom articulated robotic arm. Initially, the robot moved randomly and collected approximately one thousand trajectories, each comprising one hundred points. The robot then used deep learning, a modern machine learning technique, to create a self-model. The first self-models were quite inaccurate, and the robot did not know what it was, or how its joints were connected. But after less than 35 hours of training, the self-model became consistent with the physical robot to within about four centimeters. The self-model performed a pick-and-place task in a closed loop system that enabled the robot to recalibrate its original position between each step along the trajectory based entirely on the internal self-model. With the closed loop control, the robot was able to grasp objects at specific locations on the ground and deposit them into a receptacle with 100 percent success.  

Image

Even in an open-loop system, which involves performing a task based entirely on the internal self-model, without any external feedback, the robot was able to complete the pick-and-place task with a 44 percent success rate. “That's like trying to pick up a glass of water with your eyes closed, a process difficult even for humans,” observed the study’s lead author Kwiatkowski, a PhD student in the computer science department who works in Lipson’s lab.
 

An image of the intact robotic arm used to perform all of the tasks

An image of the deformed robotic arm, deformed by using a 3d printed piece as a replacement for one of the standard arm parts to extend and change the angle of the gripper.

An image of the deformed robotic arm in multiple poses as it was collecting data through random motion.

If we want robots to become independent, to adapt quickly to scenarios unforeseen by their creators, then it’s essential that they learn to simulate themselves

The self-modeling robot was also used for other tasks, such as writing text using a marker. To test whether the self-model could detect damage to itself, the researchers 3D-printed a deformed part to simulate damage and the robot was able to detect the change and re-train its self-model. The new self-model enabled the robot to resume its pick-and-place tasks with little loss of performance. 

Image

Lipson, who is also a member of the Data Science Institute, notes that self-imaging is key to enabling robots to move away from the confinements of so-called “narrow-AI” towards more general abilities. “This is perhaps what a newborn child does in its crib, as it learns what it is,” he says. “We conjecture that this advantage may have also been the evolutionary origin of self-awareness in humans. While our robot’s ability to imagine itself is still crude compared to humans, we believe that this ability is on the path to machine self-awareness.”

Lipson believes that robotics and AI may offer a fresh window into the age-old puzzle of consciousness. “Philosophers, psychologists, and cognitive scientists have been pondering the nature self-awareness for millennia, but have made relatively little progress,” he observes. “We still cloak our lack of understanding with subjective terms like ‘canvas of reality,’ but robots now force us to translate these vague notions into concrete algorithms and mechanisms.”

Lipson and Kwiatkowski are aware of the ethical implications. “Self-awareness will lead to more resilient and adaptive systems, but also implies some loss of control,” they warn. “It’s a powerful technology, but it should be handled with care.”

The researchers are now exploring whether robots can model not just their own bodies, but also their own minds, i.e. whether robots can think about thinking.

Columbia Engineering

Columbia Engineering, based in New York City, is one of the top engineering schools in the U.S. and one of the oldest in the nation. Also known as The Fu Foundation School of Engineering and Applied Science, the School expands knowledge and advances technology through the pioneering research of its more than 220 faculty, while educating undergraduate and graduate students in a collaborative environment to become leaders informed by a firm foundation in engineering. The School’s faculty are at the center of the University’s cross-disciplinary research, contributing to the Data Science Institute, Earth Institute, Zuckerman Mind Brain Behavior Institute, Precision Medicine Initiative, and the Columbia Nano Initiative. Guided by its strategic vision, “Columbia Engineering for Humanity,” the School aims to translate ideas into innovations that foster a sustainable, healthy, secure, connected, and creative humanity.

Journal

Science Robotics

About the Study

The study is titled “Task-Agnostic Self-Modeling Machines.”

Authors are: Robert Kwiatkowski, Department of Computer Science, and Hod Lipson, Department of Mechanical Engineering, Columbia Engineering, and Data Science Institute, Columbia University.

The study was supported by the Defense Advanced Research Projects Agency (DARPA MTO HR0011-18-2-0020).

Conventional transistors are made out of silicon, so they cannot function in the presence of ions and water, and in fact break down because of ion diffusion into the device. Therefore, the devices need to be fully encapsulated in the body, usually in metal or plastic. Moreover, although they work well with electrons, they are not very effective at interacting with ionic signals, which is how the body’s cells communicate. As a result, these properties restrict the abiotic/biotic coupling to capacitive interactions only on the surface of material, resulting in lower performance. Organic materials have been used to overcome these limitations as they are inherently flexible, but the electrical performance of these devices was not sufficient to perform real-time brain signal recording and processing.

Khodagholy’s team took advantage of both the electronic and the ionic conduction of organic materials to create ion driven transistors they call e-IGTs, or enhancement-mode, internal ion-gated organic electrochemical transistors, that have embedded mobile ions inside their channels. Because the ions do not need to travel long distances to participate in the channel switching process, they can be switched on and off quickly and efficiently. The transient responses depend on electron hole rather than ion mobility, and combine with high transconductance to result in a gain-bandwidth that is several orders of magnitude above that of other ion-based transistors.

The researchers used their e-IGTs to acquire a wide range of electrophysiological signals, such as in vivo recording of neural action impulses, and to create soft, biocompatible, long-term implantable neural processing units for the real-time detection of epileptic discharges.

“We’re excited about these findings,” says Gelinas. “We’ve shown that E-IGTs offer a safe, reliable, and high-performance building block for chronically implanted bioelectronics, and I am optimistic that these devices will enable us to safely expand how we use bioelectronic devices to address neurologic disease.”

Another major advance is demonstrated by the researchers in their Science Advances paper: enabling bioelectronic devices, specifically those implanted in the body for diagnostics or therapy, to interface effectively and safely with human tissue, while also making them capable of performing complex processing. Inspired by electrically active cells, similar to those in the brain that communicate with electrical pulses, the team created a single material capable of performing multiple, non-linear, dynamic electronic functions just by varying the size and density of its composite mixed-conducting particles.

“This innovation opens the door to a fundamentally different approach to electronic device design, mimicking biological networks and creating multifunctional circuits from purely biodegradable and biocompatible components,” says Khodagholy.

The researchers designed and created mixed conducting particulate (MCP)-based high performance anisotropic films, independently addressable transistors, resistors, and diodes that are pattern-free, scalable, and biocompatible. These devices carried out a variety of functions, including recording neurophysiologic activity from individual neurons, performing circuit operations, and bonding high-resolution soft and rigid electronics.

“MCP substantially reduces the footprint of neural interface devices, permitting recording of high-quality neurophysiological data even when the amount of tissue exposed is very small, and thus decreases the risk of surgical complications,” says Gelinas. “And because MCP is composed of only biocompatible and commercially available materials, it will be much easier to translate into biomedical devices and medicine.”

Both the E-IGTs and MCP hold great promise as critical components of bioelectronics, from wearable miniaturized sensors to responsive neurostimulators. The E-IGTs can be manufactured in large quantities and are accessible to a broad range of fabrication processes. Similarly, MCP components are inexpensive and easily accessible to materials scientists and engineers. In combination, they form the foundation for fully implantable biocompatible devices that can be harnessed both to benefit health and to treat disease.

Khodagholy and Gelinas are now working on translating these components into functional long-term implantable devices that can record and modulate brain activity to help patients with neurological diseases such as epilepsy.

“Our ultimate goal is to create accessible bioelectronic devices that can improve peoples’ quality of life,” says Khodagholy, “and with these new materials and components, it feels like we have stepped closer to that.”

Columbia Engineering

Columbia Engineering, based in New York City, is one of the top engineering schools in the U.S. and one of the oldest in the nation. Also known as The Fu Foundation School of Engineering and Applied Science, the School expands knowledge and advances technology through the pioneering research of its more than 220 faculty, while educating undergraduate and graduate students in a collaborative environment to become leaders informed by a firm foundation in engineering. The School’s faculty are at the center of the University’s cross-disciplinary research, contributing to the Data Science Institute, Earth Institute, Zuckerman Mind Brain Behavior Institute, Precision Medicine Initiative, and the Columbia Nano Initiative. Guided by its strategic vision, “Columbia Engineering for Humanity,” the School aims to translate ideas into innovations that foster a sustainable, healthy, secure, connected, and creative humanity.

More than 100 students participated in the weekend design challenge and many are continuing to refine their projects as the website grows. The four winning teams, which each received $500, were:

• Biggest Impact
  • Sentiment, a text messaging service that sends out daily texts to prompt users to check in with themselves and their loved ones. Founders Julia Sheth and Madeline Placik, both SEAS ’20, say they made Sentiment to strengthen emotional connections and build community in times of isolation, so “you can emote while you’re remote.” They’ve signed up more than 120 members so far and have received lots of positive feedback. They are now building out a website to create an online Sentiment community to expand their mobile community. They hope that the website will be a place “where people can respond to daily prompts and polls to check in with themselves and understand how the rest of the Sentiment community is feeling.”
• Best Focused Impact
  • Care from Home, an app for caregivers without medical training to help someone recover from COVID 19. Founder: Cassidy Gabriel, CC ’21
• Most Fun
  • Columbia@Home, an augmented reality overlay of Columbia campus scenes via a Snapchat lens. Founder: Lindsey Weiskopf, Barnard ’22
• Best Design Insight
  • LionClub, an app for maintaining your motivation by holding yourself accountable to your friends. Founders: Maggie Fei CC ’20, Lily He SEAS ’20, and Michelle Liu SEAS ’20

As part of the design challenge, students met over Zoom with several mentors and judges, who included:

• Sarah Morrison Smith, a Roman Family Teaching and Research Fellow at Barnard with a focus on human-centered computing;
• Yohana Tesfamariam Tekeste, an experienced staff associate who is interested in the development of various informative tools to help agricultural stakeholders in their decision making processes;
• Alex Weintraub, a core lecturer for art humanities with expertise in art history and archaeology;
• Harry West, a professor of practice in mechanical engineering and industrial engineering and operations research, who is interested in the intersection of design, data, and behavior change with a focus on developing new ways to reduce consumption; and
• Eugene Wu, assistant professor of computer science, who develops systems and algorithms for modern interactive data analysis.

“We saw 13 interesting ideas revealing the ingenuity of Columbia students and also their concerns during this time of social distancing,” says West. Several teams recreated virtually the Columbia campus--visually or even aurally--as a place to meet with new people, concentrate on work, play with friends, or show the campus off to admitted students. We were reminded of the importance of place in our community: our campus is not just a physical space, it is also a cognitive, social, and emotional space.”  

Image

One of the 16 competing teams, LionCraft, seems to have already won the popular vote. Four students--Evan Tilley SEAS ’22, Cindy Espinosa Barnard ’22, Hector Liang GS ’21, and Annie Sui CC ’22--worked with Chilton as their mentor to build an actual “virtual” campus in Minecraft, a game in which players can build a 3D world with blocks. All you need is a computer--you can work to create a world with others from anywhere around the globe.  

Image

The team invited all members of the Columbia community to play and help build, including incoming first years. LionCraft is even offering tours to admitted and prospective students on the virtual campus.

The community is growing fast: so far more than 500 people have signed up to participate, and over 400 have joined the LionCraft server. Alumni have also taken interest in the project--many have joined the server, and many others have seen their children having fun on LionCraft while “walking” around the virtual Columbia campus.

The entire campus exterior, including Barnard and parts of the surrounding city, has now been built and students have moved on to filling up the interiors of their favorite buildings, from dorms to Mudd, Joe’s Coffee, and Butler Library. One student even built the halal food truck always parked on Broadway at the 116th Street entrance to Morningside Campus.

“Meeting people you haven’t met before is a really nice way to stay connected,” says Tilley, who has wanted to create a project of this nature since his first year at Columbia. “It’s a fun, interactive way to feel that, even though we’re spread out all over the world, we’re all on campus together.”

“This was a great design challenge with a lot of really creative collaborations,” says Chilton, who works in human-computer interaction with a focus on computational design. “It was instantly immersive and that’s where the magic happened – students got together and were totally motivated to get things done. And they learned that, no matter the outcome, the most important thing is to keep building your idea, to get a prototype in front of people who can benefit from it--real users--and listen to their feedback. You never know what will catch on.”

From reluctant participant to returning champ and judge, Lucas Schuermann’s '20 journey to successful entrepreneur started with one Fast Pitch.

Currently VP of Engineering at Genesis Trading, in 2017 Schuermann cofounded Qu Capital, a quantitative hedge fund with powerful market research capabilities. What had begun a year before as an informal reading group with friends had evolved into an idea for a startup good enough to suddenly land him in front of major investors and venture capital firms downtown.

Luckily, Schuermann was ready for his big break. Having been a member of the Residential Incubator, or Res. Inc., which brings aspiring undergraduate entrepreneurs together to live and collaborate in the same dorm, and a frequent competitor at events like Fast Pitch and the Columbia Venture Competition, he’d been practicing for that moment from the moment he’d arrived in Morningside Heights.

But he never would have taken the first step without a few well-timed encouraging words from supporters like Columbia Entrepreneurship Director Ivy Schultz.

“To be honest, looking back I now think our entire idea was terrible,” says Schuermann of the proposed new social network he and neighbor Edward Yu ’18 (later Qu Capital’s cofounder) had dreamed up. “But Ivy said going through the motions of pitching would be fantastic practice. And she was right.”

Now in its ninth year, the School’s annual Fast Pitch competition has seasoned more than 300 alumni and students who have learned the ins and outs of creating startups and how to successfully pitch them to investors.

For Schuermann, that practice that quickly paid off: by his sophomore year, his team Laminar Dynamics reached the finals in the Columbia Venture Competition with a proposal to use streamlined long-range drones to transport time-sensitive medical supplies in areas with inhospitable terrain. The group later traveled to China to prototype their aircraft, and worked with several NGOs on bringing the idea to market.

As investors signed on at Qu Capital, Schuermann took a leave of absence from Columbia to oversee his rapidly growing management company and fund while leading a team building out systems and technical infrastructure. Key to everything he accomplished were contacts and connections made on campus, he notes. Over three quarters of his eventual colleagues were computer scientists, mathematicians, and statisticians from Columbia Engineering.

Earlier this year, Qu Capital was acquired by Genesis Trading, a subsidiary of Digital Currency Group (DCG), the largest investor in the blockchain space. In addition to serving as VP of Engineering, and leading a team continuing to develop sophisticated systems for trading, Schuermann has returned to Columbia Engineering to complete his undergraduate studies and mentor the next generation of rising entrepreneurs. He’s also been named Columbia Engineering’s Entrepreneur in Residence.

“As Entrepreneur in Residence, I’m essentially a jack-of-all-trades supporting the School’s entrepreneurship programs,” he says. “I’ve been lucky enough to go through the process of raising three capital rounds and finalizing an acquisition in the span of about two years. Since I recently went through the exact same opportunities as the students I mentor, I try to provide guidance and feedback drawing from the fantastic experiences I’ve had.”

That mentorship included returning as a judge at this year’s Fast Pitch, which brought together 35 early-stage student enterprises to make their best 60-second pitches for feedback and potential funding from a $5000 prize pool. This year, for the first time ever an entire engineering class—the Applied Physics Undergraduate Seminar on Entrepreneurship—revolved around tailoring innovation to market needs in order to launch viable tech start-ups. The course is taught by Professor Mike Mauel, who was among the crowd assembled at Davis Hall on November 21.

A team of Mauel’s students was among the undergraduate winners at night’s end: Quantum Data Defender, from James Borovilas ’20 and James Lee ’21, won second place and $650 for their proposed new method to better encrypt data by encoding it onto photons. Taking first and $1250 was a pitch from biomedical engineer and EMT Benjamin Greenfield ’20 for Dialetica, a small high-efficiency blood pump in development which can operate over days at a time to continuously remove excess fluid from human blood—promising to significantly reduce the need for dialysis. The endeavor grew out of R&D in Professor Ed Leonard’s lab. Additionally, Res. Inc. member Jared Gonzales ’23 earned the audience choice award and $100 for presenting CashClimate, a new financial management app that gamifies saving responsibly for young consumers.

Among the graduate winners, biomedical engineering PhD candidate Naveed Tavakol took first and $1500 for presenting InterOrgan, an "organ-on-a-chip" system utilizing patient-specific stem cells to model disease and advance drug development. He is part of a team of researchers from Professor Gordana Vunjak-Novakovic’s Laboratory for Stem Cells and Tissue Engineering. In second place, receiving $1000, was Basis Beauty from Columbia Business School MBA student Efi Turkson ’20BUS, who pitched a new multi-brand beauty retailer specifically catered to and personalized for consumers of color. Taking home third place and $500 was avoMD, a medical app from biomedical engineering master’s student Yiela Saperstein ’18BC MS’20 and her team. The interactive app personalizes medical guidelines and makes them conversational to help physicians consult with their patients at the point of care.

This year’s edition also debuted a new track, Urban Works India, devoted to innovations that will benefit residents of booming megacities in India and across the developing world. Organized in partnership with RMZ Foundation, the track will also be featured at the Venture Competition in the spring.

After every pitch, presenters took part in rapid-fire Q&A sessions with the diverse panel of judges, receiving invaluable tips and insights along the way to help hash out their proposals and hone their presentation skills.

“Fast Pitch is always invigorating,” Schuermann said. “Feeling the excitement, appreciating the amount of work that students put into their pitches, and seeing peers begin their own paths is an incredibly rewarding experience.”