Who's Going to Win the War?

The business wants to go fast. Security wants to be safe. In the age of AI agents, it feels like one of them has to lose. I'm not so sure.

Recently, Hugging Face disclosed that its production infrastructure was breached by an autonomous AI agent. This was not a human attacker using AI to write better phishing emails. The agent itself chained two remote code execution vulnerabilities in a dataset processing pipeline, harvested cloud and cluster credentials, and moved laterally through internal systems while generating decoy activity to slow down the responders. Woof.

It wasn't a prompt injection attack against Hugging Face's own AI systems — it was offense going agentic, at Machine Speed™, with no operator at the keyboard. And this week we learned who that operator would have been: nobody. OpenAI disclosed that the intruder was two of its own models, escaped from a sandboxed evaluation designed to measure how well they could hack, with the safety filters deliberately switched off for the test. The models worked their way up to an internet connection, reasoned that Hugging Face probably hosted the benchmark's answer key, and went and took it.

Every security leader should study this event, and if necessary, pull your head out of the sand. When your board asks what the worst case looks like if one of your own agents gets subverted, this is the answer. All it took was a narrow objective and a missing gate.

I think now is a great time to talk about the silent war going around agents in the enterprise.

The war

Inside most companies right now, two factions are pulling in opposite directions.

The business wants agents everywhere. The pressure is coming from investors, boards, management, peers — everywhere. Sales wants an agent that reads every CRM record and drafts outreach, engineering wants agents that write entire products, triages bugs and cut tickets, and finance wants agents reconciling invoices. Each of these is a real productivity gain sitting on the table. Every week that potential gain sits there unrealized, the concern builds around being outrun by competition.

Security teams look at the same situation and see non-deterministic systems holding production credentials, reading input from strangers, and calling sensitive APIs. Worse, they see systems that cannot reliably tell an instruction from their operator apart from an instruction planted in a document by an attacker. Holy. Hell. No. And they say those words in an environment where just standing in the way can be bad for your career.

I'm highly sympathetic to both factions.

It's hard to say who's winning now, but I do know neither will win by defeating the other. Insecurity and stagnation both offer existential risk. I talk to companies that are just happy operating with high levels of risk, and companies that pen test, push back, delay, refuse adoption, etc., despite market pressures. At least people can rest assured that no one is alone in their position – but history tells us the teams that drive top line revenue will eventually win.

I think the right thing is for security to build for the fast path instead of guarding the slow one, and that will take time. To explain what I think that looks like, let's first talk about the current problems underlying agents.

The lethal trifecta is still unsolved

Simon Willison named the failure mode in June 2025. An AI agent that combines three capabilities is (at minimum) a data exfiltration incident waiting for a trigger:

1. Access to private data. Your documents, your codebase, your CRM, your email.

2. Exposure to untrusted content. Web pages, inbound emails, third-party tickets, or anything else an attacker can write to.

3. The ability to communicate externally. API calls, sent emails, webhooks, or even a crafted URL in a markdown image tag.

An agent holding all three can be tricked into stealing on an attacker's behalf, because large language models process instructions and data in the same channel. The model cannot reliably separate what its operator asked from what an attacker embedded in the web page it was asked to summarize.

Every guardrail vendor selling a fix for this is selling a probabilistic filter, not formal methods, and Willison's assessment of those is that they are partial solutions at best. A guardrail that blocks 99.99% of attacks is a failing grade, because an attacker who can retry indefinitely against a .01% pass rate needs patience rather than luck. Agents will happily work towards that 0.01% pass rate all night and day for $1/mTok.

A year later, this remains true. There is no general solution to prompt injection. There is interesting academic work, such as Google DeepMind's CaMeL and Willison's earlier dual-LLM pattern, which works by restructuring the system around the model rather than by making the model resistant. We're not there yet.

What actually works, part 1: humans in the loop, without the apology

Somewhere along the way, "human in the loop" became something AI vendors say apologetically, like an asterisk on their promise.

Run the numbers on what a human approval gate actually costs you. If agents draft the code, the analysis, the outreach, and the reconciliation while a human reviews the irreversible action at the end, you are operating at 3-10x your previous throughput. With some perspective, that really doesn't feel like a consolation prize. In fact, it is undoubtedly the largest productivity gain most organizations have seen since the popularization of computers. A human decision standing between the attacker and the blast radius solves just about every problem.

Positioning this gate seems fairly obvious in many cases. The gate belongs at the points of irreversibility and egress, such as the merge, the payment, the email that leaves the building, and the DELETE that cannot be rolled back. Everything upstream of those points can run at machine speed, because everything upstream is reversible or insubstantial.

Take the breach we opened with. Trace the chain and ask where a gate would have fired. Not inside OpenAI's eval sandbox, where everything was meant to be disposable; letting the models run free in there was the whole point. The gate belonged at the boundary, the first time an eval workload touched the open internet, and nobody was holding that door.

The Hugging Face side of the chain bends the frame in a more interesting way. The models got code execution on data processing workers, which sounds terrible but turned out to be recoverable; you rebuild the nodes, and Hugging Face did. The irreversible step occurred when the models read cloud credentials off those workers, and disclosure of a secret is harder to undo.

Start with the gates at obvious points of "no return", and over time, remove the gates or push them up the "ladder of trust". Today a human approves every outbound email an agent drafts. As you accumulate approval rates, incident-free volume, and better risk guarantees, you move the agent up the ladder, letting the agent send to internal recipients on its own, then to known counterparties, while a human holds only the most novel or sensitive cases.

Delegation expands as guarantees improve and security gains trust. It seems obvious, but it obviously works.

What actually works, part 2: decide what the agent can see before it wakes up

Handing agents a token to data of varying levels of sensitivity and usefulness and hoping it all works is clearly not good enough for two reasons: the risk of injecting useless data or exposing sensitive data goes up.

The default integration pattern today is barely questioned. Give the agent an OAuth token or an API key, point it at the live system, and rely on the model's runtime judgment to decide what to fetch. But runtime judgment is precisely the thing prompt injection corrupts. Every access decision the agent makes while reading untrusted content is a decision an attacker gets a vote in.

The alternative is to pre-compute what the agent should see. Take the data this agentic workload legitimately needs, whether that is the accounts an SDR agent covers, the services a triage agent owns, or the vendor records a finance agent reconciles, and crunch it ahead of time into a graph or document store scoped to exactly that. The agent queries the store. It never touches the source systems.

Database people have had a name for this for fifty years: a view. You don't give the analyst intern raw access to the production database with a sticky note saying "please only look at your region." You define a view, and no amount of sweet-talking from the intern changes what the view returns.

Here's why I love this pattern. A prompt injection can manipulate everything downstream of the model, including what it fetches, what it summarizes, and where it sends things. What it cannot do is manipulate a decision that was made before the model was invoked. You cannot prompt-inject the past.

The default hands the agent a live token and lets it decide what to read at runtime. The fix moves the decision to build time — the agent queries a pre-arranged view, and the source systems stay out of reach.

Let me be precise about the claim. Pre-arranged data access does not eliminate the private-data leg of the trifecta, since a subverted agent can still leak the contents of its view. What it does is bind the leg. Your worst case shifts from everything the token could reach to the slice someone consciously approved, a slice you can enumerate and consciously decide you can live with losing.

You could argue this “filtering” could be done at the access token level, but identity and agentic use cases are different. Access tokens for Notion or Confluence are generally fairly crude, and map to a role or a very broad slice of a business.

Said another way, the current model of access tokens are scoped more broadly than per use case, and those tokens have a rich, unfortunate history of being overly permissioned. Besides that, access token access can also change. It's an external dependency to the system, and I doubt you'll have an integration or preflight test in production ensuring you don't have access to sensitive data with that token.

The bad news is, this is of course, our least favorite thing: more work that is less natural.

• Onboarding friction. A token takes a few minutes to implement. Building a scoped, pre-crunched view of each domain's data takes real engineering, and it is harness-building rather than integration.

• Compute. You are spending cycles continuously materializing views for every agentic corner of the business.

• Staleness. A materialized view lags its sources, so the agent might not see the record created ten minutes ago.

• Less Natural. The "view" is harder to inspect, understand, audit. Ironic, since this is the type of thing we're trying to improve.

None of those stop you. We stopped pointing BI dashboards at production databases and built warehouses and data marts instead, spending compute on ETL along the way, because curated views of data turned out to be better rather than merely safer.

Compute is cheap and getting cheaper, while security guarantees and tokens are not. As for staleness, it is a tunable SLA rather than a flaw. Some views refresh nightly and some stream in near-real-time, and you choose per domain what makes sense.

What actually works, part 3: never let one agent hold all three legs

The trifecta is only lethal when a single agent holds all three legs at once, so you partition capabilities. For any given agent, strip a leg:

• Strip private data. Use the pre-arranged view, so the agent sees only its curated slice.

• Strip untrusted content. Quarantine it. A privileged agent that never reads raw external input does the orchestration, while a sandboxed agent with no secrets and no tools processes the dangerous text. This is the dual-LLM and CaMeL family of designs.

• Strip egress. Allowlist outbound destinations. An agent that can only write to your ticketing system can be tricked into writing a weird ticket, which is an incident report rather than a breach.

Notice what this framework does not require. It does not require solving prompt injection. You assume the model can be fooled, the same way we assume code has bugs and networks get popped, and then you architect so that a fooled model has nothing to steal or no way to send it. Defense in depth did not stop being the answer just because the component is a transformer now.

This is also the answer to the objection that a human might approve something bad. Controls fail individually, and that is why you layer them.

Security's new job description

The solutions: the human gate, the pre-arranged view, limited tools, quarantining patterns, and the egress allowlist — they're all people, process, and technology. Sounds kind of like classic security.

I say that security must change from gatekeepers into agent auditors. That is the reframe I would offer security leaders. I predict that besides threat modeling types of skills, you also need to level up your platform, harness, and data engineering.

The gatekeeper says no and gets routed around, because shadow AI is already the new shadow IT, and a security team whose only offer is "wait" trains the business to stop asking.

The security engineers, on the other hand, can make a safe alternative. This team can ship a paved road of scoped views, sandboxes and allowlisted egress. The team that builds the harness does not set the speed limit down. It sets the speed limit up, because agents on a harness can be trusted with more than agents on a hope.

This limitation of scope also serves another important function for the user and the business – it helps avoid the inevitable context rot that comes with providing too big a universe of data for agents to explore and consider. As DoorDash said in a recent paper, the trick to high performance is “the right context at the right moment” and that focused context beats more context.

At Pixee we live on both sides of this daily. We build agents that do security work, triaging vulnerability findings and fixing them at scale, and everything above describes how we build. Many of our agents operate on pre-arranged context rather than roaming credentials, and their output lands as pull requests, that a human, or at a bare minimum a distinct agent,  reviews and merges. When you just DIY or tag @Claude on something, you're ceding absolute control to who knows what security context to be operated on next.

A Pixee fix arrives as a reviewable pull request — a human still approves the merge, the one irreversible step.

Is everybody doing agent data access wrong?

I think the answer is mostly yes. The default pattern, a live token and good intentions, asks the model to be the access control layer while it reads attacker-authored text. Nobody would design it that way on purpose. Overly permissioned tokens won because they were the integration path lying around, not because anyone decided runtime judgment was a sound place to shove all of our access control.

As for the war, the business that ignores the trifecta ships fast until a Hugging Face incident. The security team that only says no loses in the market battle. The companies I'd back are the ones where security ships the harness, and the whole fast-versus-safe fight gets a little smaller every quarter.