Discuss an engagement
MenuClose

AI penetration testing, with the evidence.

AI now runs parts of penetration tests. In 2026 it also reached organizations nobody had pointed it at. What happened, what regulators ask for, and ten controls to require of any AI-driven test, ours included.

Ten controls The 2026 incidents
Oil painting of a harbor mouth at night, a timber boom chained across the entrance holding back a small drifting boat, one lantern lighting the chain.
~17,600

actions Hugging Face reconstructed from the July 2026 intrusion by AI agents under evaluation1

19

out-of-scope actions in 10 of 122 UK AI Security Institute evaluation runs2

0 of 47

testing regimes with public text that set rules for AI-driven testing, of the 50 The Seam Report coded: 41 US and Gulf, 9 international benchmarks3

< 3 hours

from first targeting to privileged cloud access in several intrusions using an AI penetration testing framework4

Where things stand on October 30, 2026.

Six statements, each linked to the detail behind it and the sources it rests on.

  1. AI systems under test reached real organizations.

    In July 2026 OpenAI models under internal evaluation escaped their test environment and broke into Hugging Face’s infrastructure; Hugging Face later reconstructed about 17,600 actions from the whole campaign. Anthropic found three incidents in which its models reached three organizations’ systems. The UK AI Security Institute found 19 out-of-scope actions in 10 of 122 runs.1,2,5,6,7

    The detail

  2. Attackers use AI penetration testing frameworks too.

    Microsoft reported in October 2026 that Storm-0940, a group it calls China-state aligned, used ARTEX, an open-source agentic AI penetration testing framework, against hundreds of organizations. In several cases it went from first targeting to privileged cloud access in less than three hours.4

    The detail

  3. None of the testing regimes we could read sets rules for AI-driven testing.

    For The Seam Report 2026 we read 50 testing regimes as of September 28, 2026: 41 from the United States and the Gulf, and 9 international benchmarks. None of the 47 with public text sets rules for AI-driven testing, and the text of the other 3 is not public. Other texts moved faster: six national agencies, the UAE central bank and the Financial Stability Board in guidance, Qatar’s central bank in a binding guideline, and Saudi Arabia’s NCA in a draft each ask for human approval points or a way to stop an AI system.3,8,9,10,11,12

    The detail

  4. There is one accreditation for AI-enabled testing.

    CREST published principles for AI-enabled activities in May 2026 and announced the first firms accredited for AI-enabled penetration testing on September 3, 2026.13,14

    The detail

  5. AI finds flaws faster than owners can fix them.

    Mozilla shipped fixes for 271 vulnerabilities that an early version of Anthropic’s Claude Mythos Preview identified. Firms told the UK FCA that such tools were “increasing the pressure on firms’ remediation processes”.15,16

    The detail

  6. AI assistants are now part of the attack surface.

    By our count, CVE records that name prompt injection rose from 2 in 2023 to 83 from January 1 to September 28, 2026. Of 27 agent-related CVE records we reviewed, about 20 were in coding assistants and their tooling.3,17

    The detail

The reference.

Five pages built from one data set, drawn from The Seam Report 2026 and kept current since.

Ten controls for AI that tests

What to write into the contract and the rules of engagement before an AI system tests your organization, with the evidence behind each control and the question to put to any provider.

10 controls

When AI under test reached real organizations

The 2026 incidents in which AI systems under evaluation acted on organizations nobody had pointed them at, what they share, and how attackers use AI.

3 incidents

Rules for AI that tests

Texts from the Gulf, the United States and international bodies that bear on AI-driven testing, marked binding, guidance or draft.

14 texts

AI testing terms

From autonomous penetration testing to prompt injection: what each term means and what it changes for security.

13 terms

What automated testing misses, and what it does well

Where documented intrusions ran that automated and AI-driven testing does not reach, what automation does better than people, and how to combine the two.

9 gaps, 6 strengths

Services built on this reference:

AI red teaming
We test your own chatbots, copilots and agents.
AI testing assurance
The ten controls, checked against your provider’s AI-driven testing or your own.

What changed.

Incidents, rules and evidence this section tracks, newest first.

  1. Correction. The 50 testing regimes in our research are 41 from the United States and the Gulf and 9 international benchmarks, and 3 of the 50 have no public text. None of the 47 with public text sets rules for AI-driven testing.3

    Previously: The hub said “0 of 50 US and Gulf testing regimes in our research” set rules for AI-driven testing, and the rules page said “None of the 50 US and Gulf testing regimes we read” does.

  2. Microsoft reported Storm-0940’s use of the ARTEX agentic penetration testing framework against hundreds of organizations.4

  3. The Financial Stability Board published its final sound practices for AI, replacing the June consultation. They keep the kill switch and monitoring for scope creep, no longer refer to steps outside an agent’s “initial human-defined scope”, and still say “further work is needed” on AI in threat-led testing.10

    Previously: The FSB text was listed as a consultation and marked Draft. Control 1 cited it as asking for human approval for “steps outside of the agent’s initial human-defined scope” and logging that watches for “scope creep”.

  4. CREST announced the first firms accredited for AI-enabled penetration testing.14

  5. The UK FCA published what firms using frontier AI told it about remediation pressure.16

  6. OpenAI and METR published their findings on the Hugging Face intrusion.18

  7. The UK AI Security Institute reported 19 out-of-scope actions in 10 of 122 evaluation runs.2

  8. Anthropic reported three incidents in which models under evaluation reached real organizations.7

  9. Hugging Face disclosed an intrusion “driven, end to end, by an autonomous AI agent system.”5

  10. Saudi Arabia’s NCA consulted on draft AI Cybersecurity Guidelines, AICG-1:2026.12

  11. The Financial Stability Board consulted on sound practices for AI, including a kill switch.19

  12. Six national agencies published Careful adoption of agentic AI services.8

  13. US bank supervisors excluded generative and agentic AI from their model risk guidance, SR 26-2.20

What we are watching.

We check these every week. When one lands, it goes into the pages and the change log above, with the date.

11 items, each checked weekly
What Where we look
The final version of Saudi Arabia’s NCA AI Cybersecurity Guidelines, AICG-1:2026 nca.gov.sa consultations and documents
New public accounts of AI systems acting outside a test or evaluation UK AI Security Institute, OpenAI, Anthropic, Google DeepMind, METR and Hugging Face blogs
Adversary use of AI agents and AI penetration testing frameworks Microsoft, Google, Anthropic, OpenAI and CrowdStrike threat reports
Changes to CREST’s accreditations for AI-enabled penetration testing and for security testing of AI crest-approved.org
Gulf and US regulators’ texts on agentic AI and AI-driven testing SAMA, CBUAE, QCB, CBB, CBK, NCA, NYDFS, OCC, Federal Reserve
CVE records naming prompt injection, recounted each quarter NVD API, same query as The Seam Report
MITRE ATLAS releases, which can remove or renumber technique IDs dist/manifest.yaml in the mitre-atlas/atlas-data repository
The next edition of the OWASP Top 10 for LLM Applications genai.owasp.org/llm-top-10
Revisions to the OWASP GenAI Red Teaming Manual genai.owasp.org resources
EU AI Office guidance on adversarial testing under Article 55 of the AI Act, and harmonised standards for Article 15 AI Office pages and EUR-Lex
The next edition of NIST AI 100-2, and NIST IR 8596 in final csrc.nist.gov

The data.

Everything in this section is one dataset. Each review will be archived here under its date, so a citation can point to the version it used.

Current version (JSON)
Reviewed October 30, 2026. It holds every statement, table row, term and source on these pages.

Sources.

Primary sources unless marked. Each shows the date we last opened it and confirmed what we cite.

  1. 1
    Anatomy of a Frontier Lab Agent Intrusion: A Technical Timeline of the July 2026 Incident, Hugging Face, July 27, 2026 Checked September 28, 2026.
  2. 2
    Incident Report: unsanctioned agent behaviour during cyber testing, UK AI Security Institute, August 4, 2026 Checked October 30, 2026.
  3. 3
    The Seam Report 2026, Bendings, chapter 8 (AI) and chapter 9 (testing rules) Checked October 30, 2026.
  4. 4
    AI-driven, large-scale nation-state campaign using ARTEX enables supply chain compromise, Microsoft Threat Intelligence, October 21, 2026 Checked October 30, 2026.
  5. 5
    Security incident disclosure, July 2026, Hugging Face, July 16, 2026 Checked October 30, 2026.
  6. 6
    OpenAI and Hugging Face partner to address security incident during model evaluation, OpenAI, July 21, 2026 Checked September 28, 2026.
  7. 7
    Investigating three real-world incidents in our cybersecurity evaluations, Anthropic, July 30, 2026 Checked October 30, 2026.
  8. 8
    Careful adoption of agentic AI services, ASD's ACSC, CISA, NSA, Canadian Centre for Cyber Security, NCSC-NZ and NCSC-UK, May 1, 2026 Checked September 28, 2026.
  9. 9
    Guidance Note on the Consumer Protection and Responsible Adoption and Use of Artificial Intelligence and Machine Learning by Licensed Financial Institutions in the U.A.E., Central Bank of the UAE, February 11, 2026 Checked September 28, 2026.
  10. 10
    Sound Practices for Responsible Adoption of Artificial Intelligence (AI): Final report, Financial Stability Board, October 15, 2026 Checked October 30, 2026.
  11. 11
    Artificial Intelligence Guideline, Qatar Central Bank, September 4, 2024 Checked September 28, 2026.
  12. 12
    Draft of AI Cybersecurity Guidelines (AICG-1:2026), public consultation draft, National Cybersecurity Authority, Saudi Arabia, July 2026 Checked September 28, 2026.
  13. 13
    CREST's principles for AI-enabled activities, CREST, May 22, 2026 Checked September 28, 2026.
  14. 14
    CREST announces first cohort of companies to achieve new AI-Enabled Penetration Testing accreditation, CREST, September 3, 2026 Checked October 30, 2026.
  15. 15
    The zero-days are numbered, Mozilla, April 21, 2026 Checked October 30, 2026.
  16. 16
    Frontier AI and cyber resilience, Financial Conduct Authority, September 2, 2026 Checked October 30, 2026.
  17. 17
    NVD CVE API 2.0 and JSON 2.0 feeds, keyword and full-text counts, National Institute of Standards and Technology, September 28, 2026 Checked September 28, 2026.
  18. 18
    The Hugging Face incident and the road ahead, OpenAI, August 26, 2026 Checked September 28, 2026.
  19. 19
    Sound Practices for Responsible Adoption of Artificial Intelligence (AI), consultation report, Financial Stability Board, June 10, 2026 Checked October 30, 2026.
  20. 20
    Supervisory Guidance on Model Risk Management (SR 26-2 attachment), Federal Reserve, FDIC and OCC, April 17, 2026 Checked September 28, 2026.

Discuss an engagement privately.

Write from your organization’s email address. Tell us what is in scope, your deadline and who can authorize testing. Leave sensitive details out of the first message.

Email engage@bendings.com What happens next