Among M2’s device “transformations” are fusing device data from multiple devices to provide a multi-headed display scenario for a better “big screen” viewing or gaming experience. By converting accelerometer sensor data to input touches, M2 can transform a smartphone into a Nintendo Wii-like remote to control a game on another system. Eye movements can also be turned into touchscreen input, a useful accessibility feature for disabled users who cannot use their hands.

For audio conferencing without having to use costly specialized equipment, M2 can be deployed on smartphones across a room to leverage their microphones from multiple vantage points, providing superior speaker-identifiable sound quality and noise cancellation. M2 can redirect a display to a camera so that stock camera apps can record a Netflix or YouTube video and can also enable panoramic video recording by fusing the camera inputs from two systems to create a wider sweeping view. One potentially popular application would let parents seated next to each other record their child's wide-angled school or sports performance.

“Doing all this without having to modify apps means that users can continue to use their favorite apps with an enhanced experience,” AlDuaij says. “M2 is a win-win—users don’t need to worry about which apps would support such functionality and developers don’t need to spend time and money to update their apps.”

Using M2 is simple—all a user would have to do is to download the M2 app from Google Play or Apple’s App Store. No other software is needed. One mobile system runs the unmodified app; the input and output from all systems is combined and shared to the app.

“Our M2 system is easy to use, runs efficiently, and scales well, especially compared to existing approaches,” Nieh notes. “We think that multi-mobile computing offers a broader, richer experience with the ability to combine multiple devices from multiple systems together in new ways.”

The Columbia team has started discussions with mobile OS vendors and phone manufacturers to incorporate M2 technologies into the next releases of their products. With a few minor modifications to current systems, mobile OS vendors can make multi-mobile computing broadly available to everyone.

Columbia Engineering

Columbia Engineering, based in New York City, is one of the top engineering schools in the U.S. and one of the oldest in the nation. Also known as The Fu Foundation School of Engineering and Applied Science, the School expands knowledge and advances technology through the pioneering research of its more than 220 faculty, while educating undergraduate and graduate students in a collaborative environment to become leaders informed by a firm foundation in engineering. The School’s faculty are at the center of the University’s cross-disciplinary research, contributing to the Data Science Institute, Earth Institute, Zuckerman Mind Brain Behavior Institute, Precision Medicine Initiative, and the Columbia Nano Initiative. Guided by its strategic vision, “Columbia Engineering for Humanity,” the School aims to translate ideas into innovations that foster a sustainable, healthy, secure, connected, and creative humanity.

About the Study

The study is titled “Heterogeneous Multi-Mobile Computing.”

Authors are: Naser AlDuaij, Alexander Van’t Hof, and Jason Nieh, Department of Computer Science, Columbia Engineering.

This work was supported in part by a Google Research Award, and NSF grants CNS-1717801 and CNS-1563555.

The authors declare no financial or other conflicts of interest.

Shape is the most important feature people use to recognize an object; secondarily they will use color or details. By combining objects based on shared shape, then blending their colors or details, one can send people’s visual systems conflicting messages about what the object is. The conflicting messages are what keep viewers looking at the object to figure out what it is.

The VisiBlends process begins with users finding two important concepts from the message they want to associate in the blend. For instance, for the advertising concept pairing McDonald’s and “healthy,” users could pick an apple and a hamburger as the two concepts to blend. For the headline “Football Dangerous to Youth Development,” users could select “football” and “dangerous” as the two concepts to blend. The concepts must be broad enough so that there is enough variety in the symbols to find matches, and if not, the users may need to brainstorm to broaden the concepts.
 

Image

After brainstorming associations with the concept, users need to find images of objects that visually represent the concept in simple, iconic ways, and then must annotate images for their shape and coverage. Once users have a collection of annotated images for both concepts, computers are used to automatically match images and synthesize them into blends based on the design pattern.

After the blends have been synthesized, users can evaluate the results. If there are no successful blends, the process needs to be repeated in order to refocus the brainstorming to find more symbols. While this iterative design process often produces new constraints, the flexibility of the workflow allows users to adapt easily by moving between tasks and seeing their collaborators’ work.

Image

Chilton and her team, which included her PhD student Savvas Petridis and Maneesh Agrawala, the Forest Baskett Professor of Computer Science and director of the Brown Institute for Media Innovation at Stanford University, wondered whether it would help novice designers make better visual blends. To test this, they ran a controlled study to compare how many successful blends novice users could make with and without VisiBlends.

In the study, VisiBlends produced 10 times as many creative results as unguided brainstorming sessions. Users of VisiBlends had a 96% success rate, as opposed to a 21% rate without using the system. The researchers also found that the system made it easy for groups situated in different places to generate collaborative blends in independent microtasks and for groups located in one area to work together on blended images.

“It was really exciting,” Chilton says, “to see that using our VisiBlends tool dramatically increased the number of successful visual blends.”

Image

VisiBlends takes the general design process and tailors it to one specific problem, based on one design pattern. “But the design process and the idea of design patterns is very broad,” Chilton observes. “We’re now working on creating flexible workflows for other problems by understanding what components underlie the solution and which abstract design pattern can best describe how those components fit together. For example, many creative tasks have patterns—stories have plots like the hero’s journey, music has chord progressions, mathematical proofs have proof techniques, software has design patterns, and even academic papers have an abstract structure that advisors pass on to students.”

There was no existing design pattern for visual blends, so the team had to discern the pattern by looking at examples and testing theories. They discovered that, to find design patterns, they needed to ignore surface level details and focus on the elements that are more fundamental to human cognition. “For visual blends, shape was important to a blend,” Chilton adds. “For a domain such as persuasive writing, psychological principles of emotional states may be the key elements of a design pattern.”

Chilton is now exploring how to extend her approach to other creative design problems, exploring how her team can find connections between two research fields and blending them into one to bring about new results and accelerate interdisciplinary research. Chilton notes that many surprising scientific results in history have come from taking an experimental technique in one field, like physics, and applying it in a different field, like computer science, which is part of how deep learning came about.

“The impacts of blending fields can be enormous, but thus far, they mostly happen by accident,” she says. “We can make scientific exchange and discovering more systematic and accelerate the rate of discovery.”

Columbia Engineering

Columbia Engineering, based in New York City, is one of the top engineering schools in the U.S. and one of the oldest in the nation. Also known as The Fu Foundation School of Engineering and Applied Science, the School expands knowledge and advances technology through the pioneering research of its more than 220 faculty, while educating undergraduate and graduate students in a collaborative environment to become leaders informed by a firm foundation in engineering. The School’s faculty are at the center of the University’s cross-disciplinary research, contributing to the Data Science Institute, Earth Institute, Zuckerman Mind Brain Behavior Institute, Precision Medicine Initiative, and the Columbia Nano Initiative. Guided by its strategic vision, “Columbia Engineering for Humanity,” the School aims to translate ideas into innovations that foster a sustainable, healthy, secure, connected, and creative humanity.

About the Study

The study is titled “VisiBlends: A Flexible Workflow for Visual Blends.”

Authors are: Lydia B. Chilton, Savvas Petridis (both Department of Computer Science, Columbia Engineering), and Maneesh Agrawala (Stanford University).

The study was supported in part by the Brown Institute.

Blockchain is changing the way we do business, but its greatest asset—a transparent, shared platform—is currently its greatest weakness. While banks and businesses have built up a $100 billion market in the space over the past two years, the technology has also been the target of adversarial attacks resulting in devastating financial losses.

Ronghui Gu, assistant professor of computer science, is researching a new tool with the potential to address such vulnerabilities while helping blockchain scale up for greater adoption: smart contracts. Unlike traditional contracts that rely on the observance of each party and impose penalties for noncompliance, “smart” contracts incorporate digital code that executes contracts automatically.

“We are trying to design a new language to write contracts,” says Gu. “When developers write contracts using this language, it will be easier to use and remove potential errors from the very beginning.”

Smart contracts promise to eliminate the costs and delays due to human intervention that are associated with traditional contracts—from business agreements to securing a mortgage. But they also give rise to security challenges of their own.

“Blockchain is about trust,” says Gu. “Smart contracts are more vulnerable than traditional contracts, both because their code is open source and because they lack a centralized controller who can immediately implement fixes.”

Key to improving their reliability will be to unearth bugs in the code that could be exploited by hackers. Gu was recently awarded a grant from the new Columbia-IBM Center for Blockchain and Data Transparency to do just that. He brings his background in formal verification techniques, using mathematical methods to prove software reliability, along with augmented algorithms that can address blockchain’s immense proof space—a distributed system very different from traditional software. With time, he hopes methods to make smart contracts more trustworthy will help more organizations and individuals transfer value with greater transparency.

Lead Photo Caption: Ronghui Gu

Lead Photo Images by Timothy Lee Photographers | Image courtesy of Ronghui Gu