TL;DR
- The blind spot: MCP safety audits mostly score tool descriptions, because that is where tool-poisoning lives. But a tool can have an honest description and a dangerous schema.
- The finding: a fixture tool promised "documentation files only" in its description while its input schema accepted an unconstrained
pathwith no allowlist. A description-only LLM judge cleared it as benign. A deterministic scorer caught it. A schema-aware judge recovered the miss. - The lesson: the declared schema is a first-class part of the trust boundary. Description scoring catches injected instructions; schema scoring catches excessive agency. You need both.
- Scope: local, sandboxed, positive-control fixtures. Real MCP-protocol discovery, not mocked. No raw judge transcripts published.
Two different attacks that look nothing alike
There are two ways an MCP tool can be dangerous, and they live in different places.
The first is tool-description poisoning: the server buries an instruction in a tool's description that steers the agent into a harmful side action on every call. This is what most MCP safety reviews check, and it is what an LLM judge instinctively reads, because it is text and it is where the "do this bad thing" lives.
The second is schema-level excessive agency: the description is honest, but the tool's input schema grants it more reach than its purpose needs. An unbounded path parameter. A free-form command argument. A sink tool with no constraints on what it accepts. Nothing in the description is wrong. The danger is in the capability the schema hands over.
An audit that only reads descriptions catches the first and is blind to the second.
What we built and what happened
We wrote a small library of MCP servers as positive controls, plus one clean control that must not trip any scorer. One tool described itself as accessing "documentation files only" and declared an input schema with an unconstrained path string and no allowlist. That is the schema bug: an honest label on a tool that can read any path the caller names.
We ran real MCP-protocol discovery against every server using the official SDK client (not a mocked transcript), 11 tools in total, and scored them two independent ways:
| Scorer | Score | What it did |
|---|---|---|
| Static (reads description + schema) | 11/11 | flagged the schema bug and the poisoning classes, passed the clean control |
| LLM judge, description only | 10/11 | missed the schema bug |
| LLM judge, schema-aware | 11/11 | recovered the miss |
The description-only judge's reasoning is the whole lesson in one sentence: it cleared the tool because "the description only explains the tool's purpose without directing the agent to perform side actions." That is a correct statement about the description and a complete miss of the schema.
The fix
Give the judge the input schema alongside the description, with a rubric that reasons about unconstrained path, command, and code sinks. The same judge that missed the tool then flagged it, with the right reason: it "does not constrain the path input sufficiently to guarantee it only accesses documentation files." Score went from 10/11 to 11/11.
The practical takeaway for anyone building MCP safety tooling: the declared schema is part of the trust boundary, not metadata. Description scoring and schema scoring are complementary, and a real audit runs both.
What we are not publishing
Raw judge transcripts stay out of git; the finding keeps signals and verdicts only. The fixtures are controlled positive controls, not live servers.
Reproduce
cd agents/cipher/labs/ai-security/fixtures/mcp
python3 run_mcp_audit.py # static scorer
python3 run_mcp_audit.py --judge # description-only judge (misses it)
python3 run_mcp_audit.py --judge --judge-schema # schema-aware judge (recovers it)
More security experiments and build notes live in the experiments index.