Futurism logo

A treacherous black box

Deep learning" is currently helping AI achieve a breakthrough. But there's a catch: the technology is virtually impenetrable - and amazingly easy to fool.

By AddictiveWritingsPublished 3 years ago 12 min read
Like
A treacherous black box
Photo by Franki Chamaki on Unsplash

Dean Pomerleau can still remember the first time he struggled with the black box problem. The year was 1991, and Pomerleau was trying to teach a computer to drive. It's become more commonplace now, thanks to research on autonomous vehicles. At the time, it was a true pioneering act.

Pomerleau got behind the wheel of his specially converted Humvee and piloted the military truck through the city streets. Onboard, the researcher, then a robotics doctoral student at Carnegie Mellon University in Pittsburgh, had a computer that he had programmed to look through a camera, interpret what was happening on the road, and record every maneuver Pomerleau made. The hope was that eventually, the device would have made enough associations to steer the car on its own.

On each trip, he trained the system for a few minutes, then let it drive on its own. Everything seemed to be going fine - until one day when the Humvee suddenly swerved sideways in front of a bridge. Pomerleau was only able to prevent a crash by quickly grabbing the steering wheel.

Back in the lab, he tried to determine what had led the computer astray. "That was part of my doctoral thesis: to open the black box and find out what the device was thinking," Pomerleau explains. But how? He had programmed the computer with a "neural network" - an artificial intelligence (AI) process that is based on the structure of the human brain and is generally better at dealing with complex everyday situations than the standard algorithms of the time.

Just as opaque as the brain

Unfortunately, such networks are also as opaque as the brain. Instead of neatly storing what is learned as a block of data in digital memory, they distribute the information in a way that is extraordinarily difficult to decipher. Only after Pomerleau had thoroughly examined the reaction of his software to various visual stimuli was he able to pinpoint the problem: The neural network had been using the grassy verges as a guide to the course of the road. When a bridge suddenly appeared, the irritation was great.

Today, 25 years later, it is becoming increasingly difficult, but also increasingly important, to solve the riddle of the black box. In terms of complexity and fields of application, the technology of AI has developed at an almost explosive rate. Pomerleau, who now teaches robotics part-time at Carnegie Mellon University, describes his small system on the transport vehicle as a "poor man's version" of the giant neural networks running on computers today. There are now even initial commercial applications for the technique of "deep learning," in which neural networks are trained with the help of huge databases - from self-driving cars to websites that generate product recommendations from a user's browsing history.

The ubiquity of Deep Learning is also emerging in science. Future radio telescopes will hardly be able to identify meaningful signals in their enormous data volumes without the help of AI; in gravitational wave detectors, Deep Learning is expected to locate and help eliminate even the smallest sources of interference; publishing houses will sift through millions of scientific publications and books and keyword them using AI. And eventually, at least some researchers predict, Deep Learning-equipped computers could even develop imagination and creativity. "You would simply feed the computer large amounts of data and it would spit out the laws of nature," says physicist Jean-Roch Vlimant of the California Institute of Technology in Pasadena.

But such advances would only make the black box problem all the more pressing. How exactly does the machine actually find the signals that make sense? And how can we be sure it's right? To what point should we trust Deep Learning? "We're visibly losing ground to these algorithms," says Hod Lipson, a roboticist at Columbia University in New York. It's like encountering an intelligent alien species, he says. If their eyes not only have receptors for red, green, and blue, but also for a fourth color, then humans would have great difficulty understanding how the aliens see the world. And it would be just as difficult for the aliens, Lipson says. The computer would eventually face the same problem, he says. "It's then like trying to explain Shakespeare to a dog."

Faced with such challenges, AI researchers respond as Pomerleau did then - opening the black box and trying, almost like neuroscientists, to understand the networks inside. According to Vincenzo Innocente, a physicist at CERN in Geneva and one of the pioneers of AI in particle physics, the answers that such a network can provide do not in themselves represent a gain in knowledge: "As a scientist, it is not enough for me to be able to tell cats and dogs apart. I want to be able to say, 'This-and-that is the difference.'"

Layer by layer

The first artificial neural networks were developed in the early 1950s, shortly after computers existed that could perform the necessary computations. They are based on the idea of simulating small computational units - "neurons" - arranged in layers and connected by a multitude of "synapses." Each unit in the lowest layer takes a value from the outside, such as the brightness of a particular image pixel, and relays that information to some or all of the units in the next higher layer. Applying a simple mathematical rule, those units then integrate the inputs from the first layer and pass the results upward until finally, the last layer provides an answer - for example, by classifying the original image as a "cat" or a "dog."

The power of such networks lies in their ability to learn. If fed training datasets and their associated correct answers, they can gradually improve their performance by adjusting the strength of each connection until an input at the lowest level leads to correct results at the top level. This process, somewhat reminiscent of the learning processes in the brain, eventually results in a network that succeeds incorrectly classifying new inputs not included in the training data set.

This learning ability excited CERN physicists in the 1990s. They were among the first to use the networks routinely and at an appreciable scale in research; the technique later proved invaluable in reconstructing the trajectories of subatomic splinters produced by the Large Hadron Collider's billions and billions of particle collisions.

"We don't understand these networks any more than we understand the human brain"

(Jeff Clune)

However, this type of learning is also why information is so diffusely localized in the network: Memory content, as in the brain, is encoded in the strength of a multitude of connections, rather than in a defined location as in a traditional database. "Where does your brain store the first number of your phone number? Probably in a bunch of synapses, probably not too far from the other digits," explains Pierre Baldi, who researches machine learning at the University of California at Irvine. However, there is no well-defined bit sequence in which the number is encoded. That's why, adds computer scientist Jeff Clune of the University of Wyoming in Laramie, "we don't understand these networks, even though we build them ourselves, any more than the human brain does."

Answers without insight?

If you're a researcher dealing with Big Data, with massive amounts of data, you'll find Deep Learning is a tool that should be treated with caution. Computer scientist Andrea Vedaldi from the University of Oxford in the UK explains why with a comparison. Imagine a neural network equipped with Deep Learning that is trained with older mammography images. For each image, the information is available as to whether the woman in question later developed breast cancer or not. After such training, it could happen that tissue from an apparently healthy woman already appears to the system as highly suspicious, he said. "The neural network might have implicitly learned to recognize certain markers - features that we don't know about but that can be a portent of cancer," Vedaldi says.

But if the computer can't explain how it arrives at its conclusions, that would present doctors and patients with a serious dilemma. It is difficult enough for a woman to decide to have a preventive mastectomy if she is at increased risk of cancer due to a genetic predisposition. But making such a decision when you don't even know the risk factor, could be much more difficult - even if the device has always been right with its predictions so far.

"The problem is: knowledge is integrated into the network, not into us, Is it us who have understood something? Actually, no - it's the web that has understood something."

(Michael Tyka)

In 2012, several research groups began looking at this black box problem. At the time, a team led by Geoffrey Hinton of the University of Toronto demonstrated for the first time in a machine vision competition that Deep Learning could classify photos from a 1.2 million-image database far better than any previous AI approach.

Looking for the reason for this extraordinary performance, Vedaldi's team took algorithms that Hinton's group had used to train the networks and ran them in reverse, so to speak. Instead of teaching a net to correctly interpret an image, the team accessed already-trained nets and tried to reconstruct those images that the net was based on. This helped the scientists figure out how the net represented different features - it was like asking the hypothetical breast cancer AI the question, "What part of this mammogram do you think is indicative of high cancer risk?"

In AI dreamland

In 2015, Tyka and fellow researchers at Google took a very similar approach. Their algorithm, which they called Deep Dream, starts with an image - say, a flower or a beach - and modifies it until the response of a selected top-level neuron becomes as large as possible. For example, if the neuron prefers to categorize images as "birds," birds will eventually be everywhere in the modified image. The resulting images are reminiscent of LSD trips, with birds emerging from faces, buildings, and the like. "In my opinion, it's more of a hallucination than a dream," explains Tyka, who is also an artist. When he and his colleagues realized the algorithm's potential for creative purposes, they made it freely available to other users. Within a few days, Deep Dream became a viral hit.

Such aspects have led some computer scientists to conclude that perhaps they shouldn't put all their eggs in one basket when it comes to AI. Take Zoubin Ghahramani, for example, who conducts research at the University of Cambridge: "If AI is to give us comprehensible answers, then there are whole classes of problems for which Deep Learning is simply not the tool of choice." A comparatively transparent method that is also suitable for scientific purposes was first presented in 2009 by Hod Lipson together with bioinformatician Michael Schmidt, then at Cornell University in Ithaca, New York. The algorithm they developed, Eureqa, was able to re-derive Newton's laws of physics from the mere observation of a relatively simple mechanical object - a system of pendulums.

Starting with a random combination of basic mathematical elements, such as plus, minus, sine, and cosine, Eureqa, using a method of trial and error developed along the lines of Darwinian evolutionary theory modifies these elements until it reaches a formula that best reflects the data. The algorithm then proposes experiments to test its models. One of its advantages lies in its simplicity, Lipson says. "A model developed by Eureqa typically contains a dozen parameters, whereas a neural network contains millions."

The black box wanted it that way!

In 2015, Ghahramani published an algorithm that automates the work of a scientist evaluating data - from raw data analysis to writing a publication. The "Automatic Statistician" recognizes both trends and anomalies in data sets, draws conclusions from them, and also provides a detailed explanation of how it arrived at its result. Such transparency, Ghahramani says, is "absolutely critical" for scientific applications, but also many commercial applications. For example, in many countries, banks that refuse to grant a loan are required by law to provide a rationale for their decision - something a deep-learning algorithm would be hard-pressed to do.

Ellie Dobson, head of data science at Oslo-based Big Data company Arundo Analytics, sees similar concerns across a range of institutions. If the U.K. economy goes downhill because policy rates were set incorrectly, "the Bank of England can't just stand there and say, 'But the black box wanted it that way.'"

Despite these fears, many computer scientists insist that transparent AI should not be a substitute for Deep Learning, but only a complement to it. Some of these techniques would probably be most useful for questions that could be described in terms of abstract rules - and less so for perceptual tasks that involve discerning the facts in raw data in the first place.

Without the complex answers provided by machine learning, an important tool would be missing from the scientific toolbox, the researchers argue. After all, they say, the real world is complex: For some phenomena, such as the weather or the stock market, not even a reductionist and synthetic description may exist. "Some things simply cannot be verbalized," says Stéphane Mallat, a mathematician at the École Polytechnique in Paris. "Ask a doctor why he diagnoses this or that, and he'll give you some reasons. But why does it take 20 years to become a good doctor? Because the information you need is just not in books."

For Pierre Baldi, scientists should embrace Deep Learning without being "too pedantic" about the black box. After all, he said, each of them has a black box in his head. "You use your brain all the time, you trust it all the time - and you have no idea how it actually works."

artificial intelligence
Like

About the Creator

AddictiveWritings

I’m a young creative writer and artist from Germany who has a fable for anything strange or odd.^^

Reader insights

Be the first to share your insights about this piece.

How does it work?

Add your insights

Comments

There are no comments for this story

Be the first to respond and start the conversation.

Sign in to comment

    Find us on social media

    Miscellaneous links

    • Explore
    • Contact
    • Privacy Policy
    • Terms of Use
    • Support

    © 2024 Creatd, Inc. All Rights Reserved.