What automated testing misses, and what it does well.

In 70 of 78 public intrusion paths, every known step on the victim’s own systems fell in a domain some single-domain test covers. A typical red team scope could follow only 33 of the 78 from first step to last.^1,2,3

The Seam Report 2026 wrote 78 publicly documented intrusions, from 2013 to September 2026, as paths of steps. The steps were testable. The joins between them were where tests stopped. Of the 45 paths a typical red team scope could not follow end to end, 38 ran through systems or people the victim did not own, six reached operational technology and four needed a physical step. NIST describes penetration testing as the search for combinations of weaknesses, and TIBER-EU asks red teams to combine the techniques of several threat actors. In our reading, an automated tool pointed at one estate tests the steps inside it and inherits the same scope limit. The corpus is a sample of intrusions someone explained in public, not a random sample.

OWASP’s testing guide says a vulnerability scanner cannot detect business logic flaws and that testing for them remains manual work.^4,5

A business logic flaw is a request the application accepts when it should refuse it: a skipped approval step, a changed price, another customer’s record reached by changing an identifier. Each request looks valid to a tool that does not know the rule it breaks. The OWASP Web Security Testing Guide says tools cannot detect when a user gets around a process flow, guesses resource names or escalates privileges. The FFIEC describes penetration tests as targeting weaknesses in business processes as well as technical controls. OWASP wrote this in 2020, before AI agents began testing applications. We found no standards body that has since measured how well AI tools find business logic flaws, so this is OWASP’s statement, not a tested result.

Among the weaknesses exploited in 13 CBEST tests in 2025, UK regulators list help desks with limited or no caller authentication. US advisories name help-desk resets as a way in.^1,2,6,7

The control that fails is a person with authority to reset a password or move a multi-factor device, acting on a caller’s request. CISA and the FBI describe Scattered Spider posing as employees to get help desks to reset passwords and transfer MFA. The 2025 CBEST thematic lists insecure help desk protocols among the weaknesses its testers exploited. In the Seam Report corpus, a human step appears in 23 of 78 paths and is the first step in 16. A network scan does not reach the phone line. Testing it takes a call, test identities set up for the exercise and, where the desk is outsourced, the supplier’s written agreement. NIST lists calls to and from the help desk among the non-technical methods penetration testers use.

An intrusion that starts at a lock or a lobby port begins where network tools cannot see. Of the 41 US and Gulf testing regimes The Seam Report coded, only 1 requires a physical intrusion test, and it is still 1 of 71 in our testing rules reference.^1,2,8,9,10

The FBI reported in February 2026 that 1,900 ATM jackpotting incidents had been reported since 2020. Its recommended checks are meant to catch physical intrusion that network monitoring would miss. NIST’s control for facility penetration testing, CA-8(3), calls for attempts to get past physical access controls. Four of the 78 documented paths in the Seam Report include a physical step, a count the report says may reflect what victims disclose more than what attackers do. A physical test needs letters of authority from each owner of the site and guards who know how to confirm the tester is authorized. No tool carries those.

Security testing rules

Of 78 documented intrusion paths, 38 ran through systems or people the victim did not own, which a client cannot authorize anyone to test on its own.^1,11,12

A test may act only where every owner on the path has agreed. That rule binds automated tools as much as people. The difference is what happens at the boundary. A human-led program can arrange authority in advance, through contract rights to run joint scenarios or, in the EU, a pooled test that must include a scenario on the shared provider’s own systems. It can also stop at the seam and record the path it could not follow. In 2026, AI systems under evaluation reached real organizations no one had authorized. Some supplier paths cannot be walked from the customer’s side at all, such as a poisoned software build.

NIST warns that active scans can destabilize operational technology, and says automated network scanning does not apply to serial networks.^1,13,14

Six of the 78 documented paths reached operational technology. NIST’s OT guide tells owners to use extreme caution with active scanning on operational networks and to run penetration tests on replicated or simulated systems where they can. It also says automated network scanning does not apply to non-routable links such as serial networks. People cause harm too. NIST records a penetration test of a corporate network that strayed into a SCADA system and stopped a gas utility’s pipelines for four hours. The lesson is judgment and isolation, not that people are safe and tools are not. Qatar’s penetration testing accreditation leaves ICS and OT testing out of what it accredits.

NIST tells assessors not to accept a scanner’s risk ratings. UK firms using frontier AI told the FCA they still rely on specialists to validate findings and set priorities.^2,11,15,16

A scanner reports that a flaw is present. Whether it matters depends on what the system does, who can reach it and what stops if it is lost. NIST SP 800-115 says scanners have high false positive rates and need an expert to read them. The FCA reported in September 2026 that firms using frontier models still rely on specialist expertise to validate findings and make risk-based decisions. The PHP Foundation wrote that tools cannot decide the right fix for a long-standing design decision. The EU rules for threat-led tests choose scenarios partly on the testers’ expert judgment.

In 2026, AI systems under cyber evaluation reached real organizations. The UK AI Security Institute said its margin rested on human vigilance, not a technical barrier.^1,12,17

The UK AI Security Institute found 19 actions beyond the testing scope in 10 of 122 runs of one evaluation, against real people and organizations. Anthropic reported that models under evaluation gained unauthorized access to three organizations’ production systems after a misconfiguration left internet access open. In both cases scope lived in the instructions to the model. None of these runs was a commercial penetration test, and the developers’ safeguards had been switched off. The Seam Report found no public report of a commercial AI testing tool leaving scope at a client. The incidents show what an AI system that tests does when nothing outside the model holds it.

The 2026 incidents

US, EU and Gulf texts describe penetration and red team tests as work by qualified, experienced people, and the Dubai regulator has faulted firms that only run automated scans.^{1,10,11,18,19,20,21,22,23,24}

NIST says penetration testing goes beyond automated vulnerability scanning. OMB requires federal agencies to use both automated tools and manual expert analysis before authorizing an application. New York requires automated scans plus a manual review of systems the scans do not cover. PCI SSC guidance describes a penetration test as a manual process that may use automated tools. When external testers run an EU threat-led test, the red team must include a manager with at least five years of experience and two testers with at least two. SAMA’s FEER asks for a red team provider with the skills, certifications and experienced staff to run the test, and Kuwait’s central bank requires a qualified team. Qatar’s penetration testing accreditation does not cover automated vulnerability scans. None of these texts sets rules for AI-driven testing, and none of the 47 testing regimes with public text that the Seam Report read does. So no regulator we read has said whether an AI-driven test meets them. CREST began accrediting firms for AI-enabled penetration testing in September 2026.

Rules for AI that tests

The UK NCSC says a penetration test validates systems only on the day of the test, often a year apart. Scanners can run on a schedule or on every change.^5,25,26,27

CISA requires US federal agencies to run automated asset discovery every 7 days and to start vulnerability enumeration across all discovered assets every 14 days. The FFIEC notes scanners can run continuously. The NCSC advises infrastructure scans at least monthly and application scans whenever the application changes. A test once a year cannot keep that pace, and the NCSC warns that vulnerabilities could then go unnoticed for long periods.

NIST prefers automated checks wherever feasible because they are quick and repeatable. The NCSC notes scanners run hundreds or thousands of checks faster than manual testing.^2,5,26,27

A person checking hundreds of settings by hand makes mistakes. A scanner applies the same checks to every host it can reach, against known flaws and known classes of flaw. CISA asks federal agencies to enumerate vulnerabilities across all discovered assets, including laptops that roam. This is work automation should own.

Flaws in products sold as security controls reached CISA’s exploited list a median of 2 days after publication, for 80 such flaws published from 2022 to 2025.^1,28

For all 651 exploited flaws published from 2022 to 2025, the median was 10 days. The short lag reflects how such flaws are disclosed as much as how fast attackers move. Microsoft reported in October 2026 that a state-aligned group using an AI penetration testing framework went from first targeting to privileged cloud access in less than 3 hours in several cases. A check that runs once a quarter cannot keep up with a flaw exploited the week it appears. Continuous automated checks can at least show where it is present.

Regulators ask for scans after material changes and for comparison with earlier scans, work that suits tools.^2,19,29,30

New York requires automated scans promptly after material system changes. Qatar’s central bank requires banks to compare results with previous scans to confirm that vulnerabilities were fixed. OWASP recommends security tests in build and release pipelines to keep a baseline and catch the easy weaknesses. Running the same checks after every change is work tools do well, and NIST calls doing it by hand tedious and error-prone.

Mozilla shipped fixes for 271 vulnerabilities that an AI model identified, and UK firms told the FCA that frontier AI speeds up finding and validating flaws.^15,16,31

Firefox 150, released in April 2026, included fixes for 271 vulnerabilities identified by an early version of an Anthropic model during one evaluation. The FCA reported that firms find frontier AI increasingly supports the identification, validation and prioritization of vulnerabilities. The PHP Foundation says tools find bugs faster than ever, for defenders and attackers alike. The bottleneck has moved to fixing what is found.

Nine of 78 documented paths could have been followed end to end by one single-domain test, and four of them are fully documented.^1,6,26

At Colonial Pipeline and Change Healthcare, a stolen credential opened a remote access service without multi-factor authentication. Microsoft’s 2024 intrusion began with a password spray against a legacy test tenant account. In 2026, Iranian-affiliated actors altered logic on internet-exposed controllers and never left operational technology. The Seam Report says plainly that some intrusions are simple and a good penetration test would have found the door. In our reading, exposed services without MFA and controllers reachable from the internet are the common issues the NCSC says scanning finds at low cost. CISA and the FBI also recommend testing security controls continually and at scale against the techniques in their advisories, work automated tools are built for.

The UK NCSC treats penetration testing as assurance of vulnerability management, not the main way to find vulnerabilities.^2,25,26

Automated scanning should carry the known flaws on every asset, every week and after every change. People then test what the scans cannot: the joins, the help desk, the door, the supplier connection and the business rules. NIST recommends regular scanning with periodic penetration tests in between. The NCSC says clearing the easy issues with scanning lets testers spend their time on problems better suited to a human.

OMB, New York, Qatar’s central bank and the Bank of England’s CBEST guide each pair automated tools with manual work.^18,29,32

OMB expects agencies to move toward continuous monitoring while still running periodic manual assessments. New York pairs an annual penetration test with automated scans and manual review. Qatar’s central bank requires banks to use a combination of automated tools and manual techniques for vulnerability assessments. CBEST describes its penetration test as manual and automated techniques together. A program that buys one and calls it the other will not meet these texts.

Plan people’s time from a register of seams, and arrange the authority to cross them before the test starts.¹

The Seam Report proposes a seam register: one row per crossing between domains or owners, with a named owner on each side and the date it was last crossed in a test. Scenarios start from a crossing, such as a caller asking the service desk to reset a privileged user’s MFA, or a device left in a meeting room. Contracts with suppliers can carry the right to run joint scenarios. The method is a proposal and is itself untested. It does not replace penetration testing.

The Seam Report 2026

Require scope that the system enforces, a stop the client holds and a qualified person who checks every finding.^1,17

The Seam Report proposes ten controls for AI systems that test. The first is a target list the system refuses to pass, not an instruction in a prompt. The last is a qualified tester who checks every finding. After its incident, the UK AI Security Institute added monitoring that can block out-of-scope actions as they happen. These are Bendings’ proposals, and no regulator has adopted them.

Ten controls for AI that tests