Ask an engineering leader why their agent pilot never made it to production and you will usually get a technical answer: the model wasn’t reliable enough, the integration was messier than expected, the use case turned out to be narrower than it looked.
The aggregate data says something different. When Stanford’s AI Index 2026 asked what blocks organizations from reaching fully scaled agentic AI, the top answer by a wide margin was security and risk concerns, at 62%, nearly twenty-five points clear of the next item (Ch3, Fig 3.3.10).
| Obstacle to scaled agentic AI | Share |
|---|---|
| Security and risk concerns | 62% |
| Technical limitations | 38% |
| Regulatory uncertainty | 38% |
| Gaps in responsible-AI tooling and control | 36% |
| Resource or budget constraints | 34% |
| Unclear or insufficient business value | 32% |
| Immature vendor or ecosystem landscape | 28% |
| Organizational resistance | 23% |
| Lack of executive support | 9% |
That is not a procurement complaint. It is an architecture verdict, and it is usually delivered late, by a security review that the pilot was never designed to pass.
Why agents are a different security object
A chatbot answers questions. An agent takes actions. That distinction, which we covered in building AI agents that actually work, is also the entire security story: the moment a system can call tools, write to systems of record, or move money, its failure modes stop being embarrassing and start being expensive.
Four properties make agents genuinely harder than the software your review process was built for.
They take untrusted input as instructions. Every document, email, ticket, or web page an agent reads is potential instruction. Prompt injection is not an exotic attack; it is the default consequence of a system that cannot reliably separate data from commands.
Their defences degrade under pressure. The AI Index reports that frontier models score “Very Good” or “Good” on the AILuminate safety benchmark under normal use, and that safety performance dropped across every model tested when evaluated against adversarial jailbreak prompts (Ch3). Whatever your model does in the demo is its best-case behaviour, not its worst.
They fail at a rate you must design for. Agents on OSWorld, which tests real computer tasks across operating systems, improved from roughly 12% to 66.3% task success in a year (Ch2). Genuine progress, and still roughly one failed attempt in three. A system that assumes success is not shippable at that rate regardless of how good the successes look.
They are confidently wrong in a specific, exploitable way. On a new knowledge-versus-belief benchmark, hallucination rates across 26 top models ranged from 22% to 94%. Models handled false statements well when framed as another person’s belief, and collapsed when the same false statement was framed as something the user believes: GPT-4o fell from 98.2% to 64.4% accuracy, DeepSeek R1 from over 90% to 14.4% (Ch3). An agent that adopts the user’s false premise and then acts on it with real permissions is a very particular kind of problem.
And the assurance you get from vendors is thinning rather than improving: the Foundation Model Transparency Index rose from 37 to 58 between 2023 and 2024, then fell to 40 in 2025, with the largest gaps in training data, compute, and post-deployment impact (Ch3). Meanwhile documented AI incidents rose to 362 in 2025 from 233 the year before (Ch3).
The threat model, in the order it matters
Security reviews stall agent projects because the project has no threat model to review. Building one is a half-day exercise, and it is the cheapest half day in the entire engagement.
1. What can it read? Enumerate every source the agent ingests, and mark which of them can be influenced by someone outside your organization. Those are your injection surfaces. A support agent that reads customer emails has an injection surface with a ticket queue attached to it.
2. What can it do? List the tools, and for each one, ask what the worst legitimate-looking call does. “Issue refund” is different from “search knowledge base.” Write down the blast radius rather than assuming it.
3. Whose permissions does it hold? This is where most designs are quietly broken. An agent running with a service account has access to everything the service account can reach, which is typically far more than the user who invoked it. The agent should act as the user, scoped down, never as the platform.
4. What can it exfiltrate? Any channel where model output reaches an external destination (an email it drafts, a webhook it calls, a URL it renders) is a path for data to leave. Injection plus an outbound channel is exfiltration, and the two are usually reviewed by different people.
5. What happens when it is wrong? Not if. At a one-in-three failure rate on structured benchmarks, wrong is a normal operating condition. Which actions are reversible, which are logged, which require confirmation, and who gets paged.
6. Can you reconstruct it afterwards? Six months later, someone will ask why the system did a specific thing on a specific day. If you cannot answer with logs, you have an incident you cannot close.
The engineering that answers it
The good news in the 62% figure is that nothing on the list requires research. It requires building.
Scope permissions to the invoking user, per action. Short-lived, narrowly scoped credentials issued per task, not a standing service identity. If the user cannot approve a refund, neither can the agent acting for them.
Treat retrieved content as data, structurally. Keep untrusted content in a separate channel from instructions, strip or neutralise instruction-shaped text, and never let retrieved content expand the agent’s tool permissions. Defence in depth, because no single filter holds.
Put irreversible actions behind a human. Confirmation for anything that moves money, changes entitlements, contacts a customer, or deletes data, and record the human’s decision as a decision, so that “a person approved it” is a fact in the log rather than an assumption in a slide.
Constrain the outbound surface. Allowlist the destinations the agent can reach. Most exfiltration paths close with a network policy that nobody thought to write because the agent “only reads.”
Log model version, prompt version, retrieval corpus version, tool calls, and outcome, together, per inference. This single artifact serves security review, incident response, regulatory traceability, and debugging. It is the highest-leverage thing on this list and it is usually added last.
Run evals as gates, not reports. Including adversarial cases. If a model upgrade degrades your jailbreak resistance, you want CI to fail, not a customer to find out.
Write the runbook before launch. Who is paged, how the feature is disabled, what is communicated, how affected records are identified. Our LLM incident response for enterprise apps post covers the shape of this in detail.
Design for provider substitution. With six frontier labs now within 25 Elo points of each other (Ch2), your provider is a dependency, not a moat. Keep the abstraction thin but real.
Sequencing this without stopping the project
The failure pattern we see is a pilot built for the demo, followed by a security review that asks for all of the above at once, followed by a project that quietly loses its sponsor.
The alternative is to bring the review forward. Get the threat model written in week one and the security reviewer into the room while the architecture is still cheap to change. What you are buying is not safety theatre; it is the ability to widen the agent’s permissions incrementally, with evidence, instead of arguing for all of them at the end with a demo.
Start narrow: read-only, one function, a human on every write. Earn each expansion with logs. It is slower than the pilot, and it is the only version that reaches the 10% of organizations with AI actually in production, a gap we unpack in the AI execution gap.
For teams building agents into insurance, healthcare, or financial services workflows, the security work and the compliance work are largely the same artifacts pointed at different audiences. That is the subject of our AI for regulated industries practice, and of what enterprise buyers now ask about AI governance.
All figures cited from the Stanford HAI Artificial Intelligence Index Report 2026 (9th edition). Obstacle data originates in McKinsey’s 2025 survey and is self-reported. Model performance figures are a March 2026 snapshot.
