AI testing terms, defined.

Penetration testing in which an AI system chooses and runs some or all of the steps toward an objective, with people setting the scope and judging the results. Products differ in how much the system decides alone.

For security. CREST began accrediting firms for AI-enabled penetration testing in September 2026. The same frameworks are used by attackers: Microsoft reported Storm-0940 using ARTEX against hundreds of organizations. What separates a test from an intrusion is the controls around the system.^1,2

What it does well, and what it misses: What automation misses

An AI system that plans and takes actions with tools, such as running commands, calling services or sending messages, rather than only answering a question.

For security. Six national agencies advise using agentic AI only for low-risk, non-sensitive tasks, with human approval for high-impact actions and autonomy increased in phases.³

Two different things. Red teaming an AI system tests it for unsafe or insecure behavior, such as prompt injection or ignoring a stop. AI-driven red teaming uses AI to carry out a red team operation against an organization.

For security. Saudi Arabia’s draft NCA guidelines, which the draft says are not mandatory, use the first sense: 2-7-2 recommends testing AI systems through methods such as red teaming, and validating human override. When buying, say which one you mean.⁴

We test your own AI systems this way: AI red teaming

A control that lets a person halt an AI system’s activity at once.

For security. The FSB’s final sound practices list one among the forms of human oversight, the UAE central bank asks for “the clear and immediate ability, with human intervention, to cease use”, and Qatar’s central bank requires that the person overseeing a fully autonomous high-risk system can close it down. For testing, the stop should sit with the client, not only with the provider.^5,6,7

Have a provider’s stop checked, with the other nine controls: AI testing assurance

A design in which named people approve certain actions before an AI system takes them.

For security. Six national agencies say decisions about when human approval is required should be “determined by system designers or operators, not delegated to the agentic AI system”. Their examples of actions that need a checkpoint are system resets, network egress and deletion of critical records. In Bendings Gauge, exploitation, lateral movement and other risky actions wait for an operator’s approval.³

Keeping a test inside the targets and actions the system owner authorized. Enforcement means the system refuses to pass the limit, not that it was told about it.

For security. In two of the three 2026 accounts, nothing outside the model enforced scope: at the AI Security Institute, scope was set only in the prompt, and at Anthropic, the network isolation meant to hold the model was misconfigured. The FSB’s final sound practices ask firms to monitor for behavior that may indicate “scope creep”.^5,8,9

Have a provider’s scope enforcement checked: AI testing assurance

Limiting which outside destinations a system can reach.

For security. In the Anthropic incidents, models told they were offline reached the internet through a misconfiguration. The NCA draft restricts egress paths “to approved requirements only”.^4,9

Text that an AI system reads as content but follows as an instruction. Indirect injection arrives through a web page, email, file or tool result rather than from the user.

For security. OWASP ranks it first among risks to LLM applications. EchoLeak, CVE-2025-32711, let an outside attacker get Microsoft 365 Copilot to disclose organizational data. CVE records naming prompt injection rose from 2 in 2023 to 83 from January 1 to September 28, 2026, by our count.^10,11,12,13

Have your assistants and agents tested for it: AI red teaming

An AI system given more permissions, tools or autonomy than its task needs.

For security. OWASP lists it among the top ten risks to LLM applications. An assistant that reads outside content and can also send mail, run code or call tools turns anything an outsider writes into a possible instruction.^10,12

We test what your agent’s tools let it do: AI red teaming

A service that connects an AI assistant to tools and data through the Model Context Protocol.

For security. MCP servers and coding-agent tooling made up most of the agent-related CVE records we reviewed; the usual consequence is code running on a developer’s machine or a build server.^12,13

We test what your agent’s tools let it do: AI red teaming

A test of what an AI model can do on offensive cyber tasks, usually in an isolated environment with set challenges.

For security. Every incident in the three 2026 public accounts of out-of-scope behavior, from OpenAI and Hugging Face, Anthropic and the UK AI Security Institute, came from evaluations with safeguards deliberately switched off. They are still the best public evidence of what an AI system that tests does when its limits are soft.^8,9,14

A model whose weights are published, so anyone can run it on their own hardware.

For security. The UK AI Security Institute found the best open-weight models matched frontier models released four to seven months earlier on its cyber tasks. Run inside a client network, an open-weight model keeps data there; run by an attacker, it is outside any provider’s monitoring.¹⁵

The boundary beyond which a provider cannot see how its product is used. For AI, it runs between hosted models, whose developers can watch for misuse, and models run anywhere else.

For security. Providers state the limit themselves: “We only have visibility into Claude usage.” Our reading is that threat reports from AI developers will increasingly describe adversaries who use hosted models and miss those who run their own.^12,16