Imagine Everything That Could Go Wrong
Agents are a powerful tool, but it’s essential to consider the damage they can do.
By Kostis Kaffes
Mistakes aren’t the problem. Humans make them all the time.
For people and organizations that are implementing AI agents, the real threats are mistakes that happen quietly, quickly, and at scale. A hallucination sends funds to the wrong account or deletes important records could cause enormous damage.
By definition, AI agents take actions that can’t necessarily be undone. When an agent edits a database, sends an email, or initiates a credit card transaction, it makes a real-world change with real-world consequences.
When something goes wrong, it might take hours (or days) to notice.
To implement a responsible AI strategy, you have to work backward by imagining what can go wrong and taking steps to prevent unacceptable outcomes. But catching mistakes quickly isn’t enough. We also need to give agents structured ways to explore, experiment, and learn safely, so they can reduce the number of mistakes they make in the first place.
As a systems researcher, I’ve seen problems propagate through a network faster than the human brain could begin to understand what was going on. These issues can compromise fundamental requirements like data integrity, customer privacy, and legal obligations.
From my perspective, anyone implementing AI agents should take a few basic steps to prevent the worst outcomes:
- Track what changes. You need full lineage: what changed, when, by which agent, and who relied on it next. Without this, even basic troubleshooting becomes impossible.
- Simulate first. Don’t let agents take just one action. They need to test dozens before choosing. Your infrastructure should support isolated, fast simulations so they’re not experimenting directly on systems with real-world implications.
- Spread out your safeguards. There’s no single “safety layer.” You need checks at different levels of the stack that work together to catch problems early.
At Columbia Engineering, we’re working with partners across sectors to help systems handle these edge cases before they become headlines. Because in this next phase of AI, a robust system isn’t a plus, it’s essential.
In the next and final issue of this series, my colleague Eugene Wu will explore what it takes to build the kind of infrastructure that makes AI agents trustworthy.
If you’d like to learn more about partnering with Columbia researchers working at the forefront of applied research in AI, visit the DAPLab website or contact Kostis Kaffes.
Get in Touch
Image
Assistant professor of computer science at Columbia Engineering |