More than 100 students participated in the weekend design challenge and many are continuing to refine their projects as the website grows. The four winning teams, which each received $500, were:

• Biggest Impact
  • Sentiment, a text messaging service that sends out daily texts to prompt users to check in with themselves and their loved ones. Founders Julia Sheth and Madeline Placik, both SEAS ’20, say they made Sentiment to strengthen emotional connections and build community in times of isolation, so “you can emote while you’re remote.” They’ve signed up more than 120 members so far and have received lots of positive feedback. They are now building out a website to create an online Sentiment community to expand their mobile community. They hope that the website will be a place “where people can respond to daily prompts and polls to check in with themselves and understand how the rest of the Sentiment community is feeling.”
• Best Focused Impact
  • Care from Home, an app for caregivers without medical training to help someone recover from COVID 19. Founder: Cassidy Gabriel, CC ’21
• Most Fun
  • Columbia@Home, an augmented reality overlay of Columbia campus scenes via a Snapchat lens. Founder: Lindsey Weiskopf, Barnard ’22
• Best Design Insight
  • LionClub, an app for maintaining your motivation by holding yourself accountable to your friends. Founders: Maggie Fei CC ’20, Lily He SEAS ’20, and Michelle Liu SEAS ’20

As part of the design challenge, students met over Zoom with several mentors and judges, who included:

• Sarah Morrison Smith, a Roman Family Teaching and Research Fellow at Barnard with a focus on human-centered computing;
• Yohana Tesfamariam Tekeste, an experienced staff associate who is interested in the development of various informative tools to help agricultural stakeholders in their decision making processes;
• Alex Weintraub, a core lecturer for art humanities with expertise in art history and archaeology;
• Harry West, a professor of practice in mechanical engineering and industrial engineering and operations research, who is interested in the intersection of design, data, and behavior change with a focus on developing new ways to reduce consumption; and
• Eugene Wu, assistant professor of computer science, who develops systems and algorithms for modern interactive data analysis.

“We saw 13 interesting ideas revealing the ingenuity of Columbia students and also their concerns during this time of social distancing,” says West. Several teams recreated virtually the Columbia campus--visually or even aurally--as a place to meet with new people, concentrate on work, play with friends, or show the campus off to admitted students. We were reminded of the importance of place in our community: our campus is not just a physical space, it is also a cognitive, social, and emotional space.”  

Image

One of the 16 competing teams, LionCraft, seems to have already won the popular vote. Four students--Evan Tilley SEAS ’22, Cindy Espinosa Barnard ’22, Hector Liang GS ’21, and Annie Sui CC ’22--worked with Chilton as their mentor to build an actual “virtual” campus in Minecraft, a game in which players can build a 3D world with blocks. All you need is a computer--you can work to create a world with others from anywhere around the globe.  

Image

The team invited all members of the Columbia community to play and help build, including incoming first years. LionCraft is even offering tours to admitted and prospective students on the virtual campus.

The community is growing fast: so far more than 500 people have signed up to participate, and over 400 have joined the LionCraft server. Alumni have also taken interest in the project--many have joined the server, and many others have seen their children having fun on LionCraft while “walking” around the virtual Columbia campus.

The entire campus exterior, including Barnard and parts of the surrounding city, has now been built and students have moved on to filling up the interiors of their favorite buildings, from dorms to Mudd, Joe’s Coffee, and Butler Library. One student even built the halal food truck always parked on Broadway at the 116th Street entrance to Morningside Campus.

“Meeting people you haven’t met before is a really nice way to stay connected,” says Tilley, who has wanted to create a project of this nature since his first year at Columbia. “It’s a fun, interactive way to feel that, even though we’re spread out all over the world, we’re all on campus together.”

“This was a great design challenge with a lot of really creative collaborations,” says Chilton, who works in human-computer interaction with a focus on computational design. “It was instantly immersive and that’s where the magic happened – students got together and were totally motivated to get things done. And they learned that, no matter the outcome, the most important thing is to keep building your idea, to get a prototype in front of people who can benefit from it--real users--and listen to their feedback. You never know what will catch on.”

After months of preliminary experimentation, the group boarded a plane for Europe. Two weeks and 75 million data points later, they returned to Morningside Heights with the raw files for a model that could not only represent the building from every angle, but also peel away additions and adjustments layered over the original cathedral, to reveal its underlying structure.

The team had to run a gauntlet of technical hurdles to create it. No one had yet designed a system for efficiently piecing together the hundreds of individual 3D scans into one coherent model. Researchers, who at the time also included Ioannis Stamos ‘01, made significant progress by developing algorithms that automatically merged the scans more efficiently than a human could, improving the model’s accuracy right out the gate. To make the model photorealistic, they developed a novel method for automatically mapping disparate camera images onto their 3D model for a fully textured effect.

But first, they had to rethink the scanning process itself. Pondering the massive number of scans required by their model led to the invention of computational tools for reducing the number of scans. These tools were the product of Blaer’s dissertation on devising optimal algorithms for scanner placement using War of 1812-era forts located on New York’s Governors Island as a case study.

From a technical perspective, old buildings have much to offer computer science—their handmade design and hidden nooks and crannies serving as ideal testbeds for resolving glitches in computer vision, notes Blaer, who’s also a lecturer at Columbia Engineering. And computer science has been returning the favor.

“Paper degrades. Buildings can only fight gravity for so long. But we can construct and store such massive datasets now that we’re able to virtually preserve things for the long run,” Blaer says.

Shaking up the Art World

On a daily basis, humans pummel our bridges, roads, and buildings with millions of pounds of force, a fact that civil engineers must factor in when maintaining infrastructure designed to last for centuries. Inside an art gallery, those same forces—foot traffic, nearby construction, pulsing sound—can cause delicate objects to flake, crack, bend, or even “walk” off their shelves as vibrations repeatedly nudge them toward the edge. In the spring of 2012, New York’s famed Metropolitan Museum of Art was faced with just such a shaky proposition. The time had come to conduct a complete renovation on their Costume Institute. The only problem—the Costume Institute sits directly beneath the Met’s wildly popular Egyptian wing, 27 galleries housing 20,000 of some of the most fragile specimens on-site, including mummies, dried flowers, and ruins from the two-thousand-year-old Temple of Dendur. De-installing the exhibit and shipping it off to storage simply wasn’t an option. Instead they called Andrew Smyth.

A civil engineering professor with expertise in using sensor networks to monitor and avert mechanical failures, Smyth deployed a collection of accelerometer devices programed to automatically alert key personnel whenever construction crews hammering away below exceeded preset vibration thresholds. But the odds were already stacked in the Egyptian’s favor. Based on data obtained from a series of pilot tests, Smyth’s team pre-installed several stabilizing strategies, such as targeted introduction of dampening materials that reduced an object’s shaking by up to 80%. For particularly vulnerable galleries, they went a step further, incorporating spring-loaded granite pedestals tuned to oscillate at the same frequency as the floor. “That’s a very different way of solving the problem,” Smyth says. “You’re actually altering the way the building vibrates.”

Many institutions have since applied Smyth’s techniques in scenarios from rap concerts at member events to shipping items on loan, while a former student on the Met team (now a faculty member at Oxford University) translated their research into developing tools for stabilizing hospitals, transportation systems, and even nuclear waste during earthquakes.

Image

Physics meets “bacteria poop”

In 1545, Henry VIII’s favorite warship—the Mary Rose—sank in the English Channel just off the coast of Portsmouth. In 1982, researchers located it, raised it and installed it in the eponymous museum where it’s now been viewed by more than 60 million people. Hauling the brittle remains of a 600-ton Tudor era warship up from beneath four stories of seawater was just the first set of engineering challenges facing conservators at the Mary Rose Trust, however. In 2014, materials expert Simon Billinge worked with colleagues at the Trust to identify a mysterious deterioration process which had begun threatening to transform this 500-year-old shipwreck into dust.

Billinge, whose day job as a professor includes advancing the physics behind cleaner energy and better medicine, has pioneered techniques for parsing the atomic structure of different materials. Billinge’s group cracked the Mary Rose case by imaging how x-rays scatter through sample cross sections at the smallest level—the researchers used that information to precisely characterize the nature of nano-scale materials hidden deep in the Tudor wood. Comparing the resulting images pixel by pixel allowed them to determine that over centuries the wood had become riddled with nanoparticles of zinc sulfide. Surfacing from the ocean floor’s anaerobic environment kickstarted an oxidizing process that transmuted that sulfide into zinc sulfate—and ultimately sulfuric acid that ate away at the hull. How this zinc alloy built up inside the cellulous wasn’t hard to deduce; it was clearly the byproduct of millions of microscopic sulfur-based organisms—bacteria poop in layman’s terms.

This discovery led to a new conservation method: application of strontium carbonate, which holds the promise to not only preserve the ship itself, but also organic materials among the 19,000 artifacts salvaged along with the wreckage. But “this isn’t just a big deal for conservators to understand,” Billinge notes. “It’s also a big deal for the study of bacteria ecology. These kinds of sulphur eating organisms are difficult to study and our map offers insights into how they self-organize in colonies and what they feed on.” 

Image

It was a master’s student continuing on from his undergraduate studies in the materials science and engineering program of the applied physics and applied math department who brought materials expert Katayun Barmak together with Alexis Hagadorn and Emily Lynch from Columbia University Libraries’ Conservation Lab. In designing his master’s research project, the student—Michael Berkson—had hoped to find a way to combine his research in materials science with his interest in art conservation, which had been sparked by a talk he heard about the work in Columbia’s Ancient Ink Laboratory. This lab, of which Hagadorn was a member, at the time was an offshoot of the university’s Nano Initiative run by electrical engineer James Yardley. Yardley’s lab used nanotechnology to elucidate the material properties of inks painted millennia ago, many of which were composed of unknown ingredients. Using spectroscopic signatures—which register the way a material reflects light—the Ancient Inks group could also determine the age of a given manuscript.

Berkson’s timing couldn’t have been better. Hagadorn, current head of Columbia’s conservation department, had identified the cause of a peculiar type of degradation manifesting in one of the library’s rare 15^th century encyclopedias. However, her research had raised further questions. Created at the dawn of the printing press, this volume contained a mix of mechanically produced text and hand painted blue and red initials. Oddly, only the latter were turning the paper beneath them brown before eating away at it. Hagadorn intuited that the blue ink involved—a common pigment derived from copper that pops up in everything from European illuminated manuscripts to Far Eastern scroll paintings—was somehow to blame. But since this ink is known for being extremely stable over long periods and varying conditions, a heretofore unknown cause must be at work. Having determined that a component in the blue ink was interacting with the original artist’s surface preparation of the paper, a previously unrecorded phenomenon, she set out to determine what mechanisms were at fault.

“I thought, I’d love to explore that, but I definitely need a scientist to help me,” she says. Luckily, at Columbia, where the arts and sciences are considered two sides of the same coin, such cross-disciplinary collaborations are easy to come by.

“Materials scientists are interested in conservation because they have a way of thinking they can bring to the problem,” says Barmak from her art-filled office on the 11^th floor of Mudd where she’s gathered with Hagadorn and Lynch. This manuscript degradation, she says “would be equivalent to the corrosion of metals causing your car to rust. It’s very much in the vein of materials science paradigm.”

Once Barmak signed on, the project quickly became a master class in experimental design. Together with her students—since the project’s inception, six in all have taken it up—she first had to craft a set of tests capable of narrowing down a multitude of parameters (such as ink recipes and base materials), all the while controlling for humidity, pH, temperature and a myriad of other factors. To do that, Barmak drew on her background in R&D at IBM to make use of a factorial design approach, in which multiple parameters are varied at a time to determine not only the most important parameters but also interactions between parameters. Barmak also drew on connections across the engineering school to access sophisticated microscopy and other state-of-the-art equipment. But in many cases, the group had to develop suitable protocols to ensure repeatability and minimize sample-to-sample variation. Being a small field, few tools are purpose-built for conservation and the team spent hours just brainstorming ways to ensure consistent application of the ink. “You’re imitating something that really wasn’t standardized,” Hagadorn says. “So, for instance, how uniform the paint application was wasn’t the concern of the creator. But it has to be one of ours.” There, Lynch contributed a bit of inspired improvisation. “I found these plastic spatulas used in the cosmetics industry,” she says. “They had the perfect width, so we just dragged them across in one swoop.”

Thus far, their work identified exposure to elevated temperatures as one of the parameters, though not the only one. Ultimately, the group believes the degradation factors they’re investigating, which they fully intend to publish, could have wide ramifications for the conservation field, considering how ubiquitous the ink they studied is. 

“I’ve worked on technologies that 30 years later have not yet hit the market,” says Barmak. “But here, you’re immediately connecting to the past and you’re preserving it for the future. To have played even a small part in protecting these beautiful books is very satisfying.”

Augmenting Reality

When historian Pamela Smith first set out to produce scholarly work on a previously unknown 16^th-century French manuscript, she knew the end result would live in a digital format; she just imagined it would be a fairly conventional one. Then she met computer scientist Steve Feiner, and began to imagine how augmented reality (AR) could turn that idea on its head.

“Once I talked to Steve, I realized how limited my conception of representing the manuscript for publication was,” she says. “He made me rethink what a ‘book’ could be.”

Smith’s plan already involved an ambitious program to co-create an open source, open access, open ended digital environment for users to manipulate and contribute to a translated, annotated, and deconstructed version of the text. Deploying such inventive strategies for preserving and advancing knowledge at the intersection of art, history and science has long been a focus for the professor. As founder of the Making and Knowing Project, Smith is Columbia’s only humanities faculty currently running a chemistry lab, where students across the disciplines can gather to investigate historical materials, techniques, and technologies.

Image

With Feiner’s input came the opportunity to layer on a whole other dimension. The 16^th-century manuscript currently at the heart of their project, an early artisan’s how-to on craft techniques to produce objects and materials in various media, immediately struck a chord with the computer science professor, whose own lab has been redefining what’s possible in AR for decades. “In my work I’m particularly interested in developing new tools to assist people in performing skilled tasks in a wide range of domains,” he says.

For historians, documenting bygone processes is as critical as chronicling artifacts themselves. So, Feiner’s group created an AR app that allows Making and Knowing researchers to spatiotemporally record phases of their experiments by overlaying artifacts with 3D photos, videos, and notes captured in the lab in real time. “We wanted to make their process richer,” he says. Data from that tool will come online when Smith’s group launches the first phase of their website this January; the group plans to roll out a more robust version that includes curricula, templates, case studies and additional pedagogical materials in subsequent months. In the meantime, Feiner’s group is also at work creating a 3D AR model of Smith’s lab that will allow users around the globe to step inside and explore.

However visitors engage with her lab, Smith hopes they’ll come away with a new appreciation for older ways of learning by doing. “There’s a really important aspect of human knowledge that we don’t give much attention to these days, and that’s hands-on knowledge,” she says. “Hands-on knowledge is part of how we learn innovation and improvisational creativity.”

For us humans, that creative essence has long united our seemingly disparate endeavors.

“We’ve come to think of science and art as two realms that are very far apart,” she says. “But art in the 16^th century was about investigating nature in order to make objects. We’re entering a period when both worlds are coming back together.”

The study, led by students Pedro Piacenza and Keith Behrman, was published online in IEEE/ASME Transactions on Mechatronics. It demonstrates the two aspects of the underlying technology that combine to enable the new results. Firstly, in this project, the researchers use light to sense touch. Under the “skin,” their finger has a layer made of transparent silicone, into which they shined light from more than 30 LEDs. The finger also has more than 30 photodiodes that measure how the light bounces around. Whenever the finger touches something, its skin deforms, so light shifts around in the transparent layer underneath. Measuring how much light goes from every LED to every diode, the researchers end up with close to 1,000 signals that each contain some information about the contact that was made. Since light can also bounce around in a curved space, these signals can cover a complex 3D shape such as a fingertip.

“The human finger provides incredibly rich contact information--more than 400 tiny touch sensors in every square centimeter of skin!” says Ciocarlie. “That was the model that pushed us to try and get as much data as possible from our finger. It was critical to be sure all contacts on all sides of the finger were covered--we essentially built a tactile robot finger with no blind spots.”

Secondly, the team designed this data to be processed by machine learning algorithms. Because there are so many signals, all of them partially overlapping with each other, the data is too complex to be interpreted by humans. Fortunately, current machine learning techniques can learn to extract the information that researchers care about: where the finger is being touched, what it is touching the finger, how much force is being applied, etc.

“Our results show that a deep neural network can extract this information with very high accuracy,” says Kymissis, who is also a member of the Data Science Institute. “Our device is truly a tactile finger designed from the very beginning to be used in conjunction with AI algorithms.”

In addition, the team built the finger so it, and others, can be put onto robotic hands. Integrating the system onto a hand is easy: thanks to this new technology, the finger collects almost 1,000 signals, but only needs a 14-wire cable connecting it to the hand, and it needs no complex off-board electronics. The researchers already have two dexterous hands (capable of grasping and manipulating objects) in their lab being outfitted with these fingers--one hand has three fingers, and the other one four. In the next months, the team will be using these hands to try and demonstrate dexterous manipulation abilities, based on tactile and proprioceptive data.

“Dexterous robotic manipulation is needed now in fields such as manufacturing and logistics, and is one of the technologies that, in the longer term, are needed to enable personal robotic assistance in other areas, such as healthcare or service domains,” Ciocarlie adds.

Columbia Engineering

Columbia Engineering, based in New York City, is one of the top engineering schools in the U.S. and one of the oldest in the nation. Also known as The Fu Foundation School of Engineering and Applied Science, the School expands knowledge and advances technology through the pioneering research of its more than 220 faculty, while educating undergraduate and graduate students in a collaborative environment to become leaders informed by a firm foundation in engineering. The School’s faculty are at the center of the University’s cross-disciplinary research, contributing to the Data Science Institute, Earth Institute, Zuckerman Mind Brain Behavior Institute, Precision Medicine Initiative, and the Columbia Nano Initiative. Guided by its strategic vision, “Columbia Engineering for Humanity,” the School aims to translate ideas into innovations that foster a sustainable, healthy, secure, connected, and creative humanity.

About the Study

The study is titled “A Sensorized Multicurved Robot Finger with Datadriven Touch Sensing via Overlapping Light Signals.”

Authors are: Pedro Piacenza and Matei Ciocarlie, Mechanical Engineering; Keith Behrman and Ioannis Kymissis, Electrical Engineering; and Benedikt Schifferer, Computer Science.

The work was sponsored in part by the National Science Foundation, under its CAREER program (grant IIS-1551631) and a National Robotics Initiative (grant CMMI-1734557).

The authors declare no financial or other conflicts of interest.