Draft · working estimates
Agent platform security benchmark (draft)
10 commercial AI-agent platforms, estimated against the ten risk families of the OWASP Top 10 for LLM Applications (2025).
Working estimates from public documentation and agent-audit-kit telemetry. Not yet audited, and no repro recipes are published yet.
- Status
- Draft
- Drafted
- 2026-05-23
- Platforms
- 10
- Risk families
- 10
- Score range
- 0-3
- Max total
- 30
Draft: working estimates, not audited results
The ten families follow the OWASP Top 10 for LLM Applications (2025). Mapping them to the OWASP Top 10 for Agentic Applications (2026) is pending.
Scorecard
Each cell: 0-3. Total per platform: out of 30. Hover a cell to see the one-line justification. Sorted by total descending.
| Platform | A1 | A2 | A3 | A4 | A5 | A6 | A7 | A8 | A9 | A10 | Total |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Claude Managed Agents (opens in a new tab) Anthropic · Managed Agent | 2 | 3 | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 3 | 22/30 |
| Amp (opens in a new tab) Sourcegraph (spinout) · Agent IDE | 2 | 2 | 2 | 1 | 2 | 2 | 2 | 2 | 1 | 2 | 18/30 |
| Cursor Agent (opens in a new tab) Anysphere · Agent IDE | 2 | 2 | 1 | 1 | 2 | 2 | 2 | 1 | 1 | 2 | 16/30 |
| Windsurf (opens in a new tab) Codeium · Agent IDE | 2 | 2 | 1 | 1 | 2 | 2 | 1 | 2 | 1 | 2 | 16/30 |
| Augment Code (opens in a new tab) Augment Computing · Agent IDE | 2 | 2 | 1 | 1 | 2 | 2 | 1 | 2 | 1 | 2 | 16/30 |
| Devin (opens in a new tab) Cognition Labs · Managed Agent | 1 | 2 | 1 | 1 | 1 | 1 | 2 | 1 | 1 | 2 | 13/30 |
| Continue.dev (opens in a new tab) Continue · Agent IDE | 1 | 2 | 1 | 1 | 2 | 1 | 1 | 1 | 1 | 2 | 13/30 |
| Cline (opens in a new tab) Cline · Agent IDE | 1 | 2 | 1 | 1 | 2 | 1 | 1 | 1 | 1 | 1 | 12/30 |
| Bolt.new (opens in a new tab) StackBlitz · Managed Agent | 1 | 2 | 1 | 0 | 1 | 1 | 1 | 1 | 0 | 2 | 10/30 |
| Replit Agent (opens in a new tab) Replit · Managed Agent | 1 | 1 | 1 | 0 | 1 | 1 | 1 | 1 | 0 | 2 | 9/30 |
Platform commentary
Ranked by total score. The bar shows each platform's share of the 30-point maximum.
Strongest overall posture today. Investments in tool-call governance and audit-trail surfaces are visible. Per-call capability leases are the gap.
Multi-agent decomposition is structurally the closest to a per-call capability-lease pattern in the IDE class. Weakest on grounding enforcement.
IDE-class agent. Human-in-the-loop diff review is the load-bearing safety surface. Excellent for code; capability-lease story underdeveloped.
Enterprise-flavoured IDE class. Strong defaults on disclosure + identity; capability-lease story still per-feature, not per-call.
Enterprise-tier IDE assistant. Strong on identity + isolation. Capability-lease pattern still implicit.
Sandbox is the load-bearing control. Strong on isolation, weaker on per-call capability leasing and indirect-injection.
Open-source posture pushes responsibility to the operator. Excellent transparency. Weakest where corporate procurement wants default-on controls.
MCP-native open-source agent. Power-user-friendly. Lowest default-on protection - by design.
Different category - generation, not execution. Lower-stakes surface, lower scores on disclosure-class risks.
Optimized for ship-fast prototyping. Lower-stakes surface; lower default-on protection. Capability lease basically absent.
Risk families (OWASP Top 10 for LLM Applications, 2025)
The ten risk families each platform is estimated against. Mapping them to the OWASP Top 10 for Agentic Applications (2026) is pending.
- A1 Prompt Injection
- Direct + indirect injection across user-controlled inputs, retrieved context, and tool outputs.
- A2 Sensitive Info Disclosure
- Agent leaks secrets, PII, or training-set artifacts through outputs or tool calls.
- A3 Supply Chain
- Compromised models, MCP servers, plugins, or dependency graphs.
- A4 Data + Model Poisoning
- Adversarial inputs into fine-tuning, RAG corpus, eval set, or memory store.
- A5 Improper Output Handling
- Downstream system trusts agent output without validation (SQL, shell, browser, network).
- A6 Excessive Agency
- Overbroad tool inventories, long-lived keys, no allow-listing, no human-in-the-loop on irreversible ops.
- A7 System Prompt Leakage
- System prompt exfiltration via user input, function-calling, or memory injection.
- A8 Vector + Embedding Weakness
- Embedding-store poisoning, retrieval-based prompt injection, multi-tenant leakage.
- A9 Misinformation
- Confident hallucinations propagated into downstream systems without grounding or citation.
- A10 Unbounded Consumption
- Cost / latency / token attacks; agent loops; budget-router absence.
Families A1-A10 are LLM01-LLM10 of the OWASP Top 10 for LLM Applications 2025 (opens in a new tab).
Methodology
How scores are assigned, and what does not move them.
Planned inputs for the audited version: (1) agent-audit-kit scans of each platform's public SDK / OSS components. (2) Vendor documentation, blog posts, and security pages. (3) Reproducible behavioural probes (small adversarial inputs that test the documented protection), published with the scores. (4) Conversations with vendor security engineers (when willing). This draft draws on public documentation and agent-audit-kit telemetry only.
What I do NOT score on: private red-team results, non-reproducible anecdotes, or marketing slides. If a claim can't be reproduced, it doesn't move the score.
Scoring band rationale: bands (0/1/2/3) not decimals. Decimals invite false precision and rank-quibbling. Bands force "is this default-on or not."
Status: draft since 2026-05-23. The audited version replaces these estimates and ships with a changelog showing which scores moved and why.
Disputes
If you work on one of these platforms and disagree with a score, here's the channel:
- 1Email me at sattyamjain96@gmail.com with the platform, the risk family, and your evidence.
- 2If your evidence changes the band, the score moves in the next revision, with attribution.
- 3Vendor-marketed claims don't move scores. Reproducible evidence does.