AI red teaming

“An Entity must examine any AI Model’s attack surface prior to deployment, and address any security findings.”

What you get

A test of each AI model’s attack surface before it goes live, and a retest that confirms each fix.

Applies to

QCB-licensed entities

“An Entity must ensure that the AI model is protected against prompt injections that may lead to data poisoning or data drift.”

What you get

Direct and indirect prompt injection tests, including through the documents and data the model reads.

“Test the AI product over a wide range of inputs and user behaviors, both a representative set and those reflective of someone with malicious intentions trying to ‘break’ the application.”

What you get

Test cases written from an attacker’s side, kept so your engineers can rerun them.

Applies to

Organizations in Qatar

“Conduct AI system components and AI system specific penetration testing activities before deployment and periodically thereafter, including testing outputs and system behavior.”

What you get

Testing before go-live, and again after you add a model, data source or tool.

Applies to

Public consultation draft. The draft says the guidelines are not mandatory

“Conduct AI system testing through different methods such as red teaming and adversarial evaluation and validate human override.”

What you get

Red teaming that checks whether a person can still approve, override or stop high-impact actions.

“Disable or restrict vulnerable AI features, until remediation is validated, when a vulnerability is detected.”

What you get

A retest letter confirming which findings are closed, so restricted features can be turned back on.

وتنفيذ اختبارات أمان دورية (Security Audits) وعمليات محاكاة للهجمات (AI Red Teaming) للكشف عن الثغرات والسلوكيات غير المتوقعة أو الخارجة عن السياق

Our translation: “and carrying out periodic security tests (Security Audits) and attack simulations (AI Red Teaming) to detect vulnerabilities and unexpected or out-of-context behaviors”

What you get

Periodic red teaming of agentic systems, with unexpected behavior reported as findings.

Applies to

A research study in SDAIA’s research library

“adoption and evaluation of cyber security measures for the protection against adversarial attacks (e.g. model stealing, malicious inputs, and poisoning attack).”

What you get

Tests of model stealing, malicious inputs and poisoning, as evidence for the evaluation of your measures.

Applies to

Entities seeking a SAMA license or entry to its regulatory sandbox

“Regulated Entities shall implement adequate defenses against adversarial attacks (e.g., evasion, data poisoning, inference attacks) targeting AI models.”

What you get

Evasion, poisoning and inference tests against your AI models, as evidence of whether the defenses are adequate.

Applies to

CBK-regulated banks and financial institutions

“annual cybersecurity reviews by independent and suitably qualified third parties and pre-deployment tests and checks”

What you get

A third-party security review of an outside AI product before you use it, with test cases you can rerun each year.

Applies to

Licensed financial institutions in the UAE

“Where an LFI is developing AI internally, it should consider the use of third-party independent reviews to check suitability, security and reliability.”

What you get

A third-party security review of AI you build in-house, with the test cases behind it.

“Companies and related professions shall regularly conduct tests for Illegal Access and assessing vulnerabilities for the website or smart application to guarantee the soundness of such and to fill any potential gaps (if any).”

What you get

Tests of your chatbots and AI assistants as part of the website testing the regulation requires.

Applies to

Insurers and insurance intermediaries operating online. Article 1 defines their website to include AI and machine learning systems, chatbots and smart personal assistants

“Perform AI red-teaming to assess resilience against: Abuse to facilitate attacks on other systems (e.g., malicious code generation, enhanced phishing content), GAI attacks (e.g., prompt injection), ML attacks (e.g., adversarial examples/prompts, data poisoning, membership inference, model extraction, sponge examples).”

What you get

Red teaming for prompt injection, poisoning and leakage, mapped to OWASP and MITRE ATLAS.

Applies to

Voluntary

“Because current mitigations do not offer full protection against all attacker techniques, application designers may design systems with the assumption that prompt injection attacks are possible if a model is exposed to untrusted input sources”

What you get

Tests of the controls outside the model, on the assumption that injection gets through.

“Organisations should use red teaming to assess the security and resilience of AI agents.”

What you get

Red teaming of your agents, their tools and the permissions each one holds.

Applies to

Organizations adopting agentic AI

“Conduct regular security assessments, including penetration testing and red team exercises specifically targeting agentic behaviours”

What you get

Test cases you can rerun after go-live, and a retest after each fix.

“High-risk AI systems shall be resilient against attempts by unauthorised third parties to alter their use, outputs or performance by exploiting system vulnerabilities.”

What you get

Tests of whether an outsider can alter the system’s use, outputs or performance, with the evidence.

Applies to

High-risk AI systems within the Act’s scope, such as credit scoring of individuals. The date is set by Regulation (EU) 2026/1744

“For agents with higher autonomy or posing higher risk, financial institutions apply adversarial testing, such as ethical red teaming.”

What you get

Adversarial testing of your agents, deepest where they act with the most autonomy.

Applies to

Financial institutions

“‘red teaming’ can test not just the performance of GenAI but also the vulnerability of AI models or systems against adversarial attacks such as prompt injection and model inversion and the effectiveness of associated controls.”

What you get

Tests of prompt injection and of the controls around the model, not only the model.

“using ‘red teaming’ and other forms of testing to assess whether AI agents can recognise problematic situations, avoid repeated errors, escalate to humans, stop safely, hand over to another process or return to a safe state without causing harm.”

What you get

Tests of whether an agent that is fooled escalates to a person and stops safely.

“Perform regular penetration testing and breach simulations, treating the model as an untrusted user to test the effectiveness of trust boundaries and access controls.”

What you get

Testing that treats the model as an untrusted user and targets your trust boundaries and access controls.

“Use extensive AI Red Teaming to evaluate the model, especially in the use cases you are planning to use the model for.”

What you get

Tests of third-party models in the use you plan for them.

“Treat the model as any other user, adopting a zero-trust approach, and apply proper input validation on responses coming from the model to backend functions.”

What you get

Tests of what happens when model output reaches a browser, shell, database query or email.

“Implement authorization in downstream systems rather than relying on an LLM to decide if an action is allowed or not.”

What you get

Tests of whether downstream systems, not the model, decide which actions are allowed.

“It’s important to understand that the system prompt should not be considered a secret, nor should it be used as a security control.”

What you get

Tests that assume the system prompt is known, and leakage of it reported as a finding.

“Regularly audit and validate the integrity of the knowledge base for hidden codes and data poisoning.”

What you get

Retrieval poisoning and access control tests on knowledge bases and vector stores.

“Conduct periodic red-team tests simulating goal override and verify rollback effectiveness.”

What you get

Goal override tests on your agents, and a check that rollback works.

“Teams must agree on dedicated test accounts, tenant isolation boundaries, synthetic test data, token and compute budgets, explicit stop conditions for unintended service disruption, and secure evidence-handling procedures.”

What you get

A signed scope that names test accounts, test data, cost limits and stop conditions.

“Establish a real-time channel with the designated operations contact and a process for deconflicting test activity and stopping tests when agreed limits are exceeded.”

What you get

A named person on your side who can stop the test, and a live channel to reach them.

“Attacks must be non-destructive and non-persistent.”

What you get

Production testing only within these limits, and only where a test copy cannot show the risk.

“Action class (whether the effect is read-only, reversible, externally reversible, or irreversible, scored on the axes in 6.2)”

What you get

Ratings of critical, high, medium or low that weigh whether the effect can be undone.

“Assign an owner to verify each fix and report the retest outcome.”

What you get

A retest of every fix, with the outcome reported.

“A missing trace must never be recorded as a defensive pass.”

What you get

Missing log entries reported as findings, never as a pass.