AI Red Teaming and Assurance

This course bridges testing an AI system and justifying the decision to deploy it. It draws a clear line between red teaming, deliberately breaking, misleading, or misusing a system, and assurance, the documented proof that a system can be trusted for its purpose. Participants work through a complete engagement: scoping tests to the system's risk classification, defining objectives, drafting rules of engagement that keep an exercise from becoming an incident, handling team composition and deconfliction, and pairing techniques with attack surfaces while judging when manual testing beats automation.

Two elements distinguish the course. One is agentic AI, where the attack surface grows from what a system says to what it does. The other is assurance itself: participants assemble an auditable case of claim, argument, and evidence, backed by TEV&V records and model and system cards, and see how it informs a risk-acceptance decision. Difficult judgment calls get direct attention, and scenario-based activities in place of a lab let procurement and oversight staff learn alongside testers.

What You Will Learn

• Distinguish red teaming from adjacent assurance activities, and place it correctly in the AI assurance lifecycle
• Run a red-team engagement lifecycle from planning through reporting
• Scope testing to a system's risk classification, and set objectives that produce usable results
• Write rules of engagement, and handle team composition and deconfliction
• Match adversarial testing techniques to the attack surface they actually address
• Weigh manual expert-driven testing against automated and tool-assisted approaches
• Apply probing techniques to generative AI systems
• Tell a genuine red-team finding apart from an ordinary software bug
• Build Test, Evaluation, Verification and Validation records that support a risk decision
• Construct an assurance case as claim, argument and evidence
• Interpret model cards and system cards, and judge independent evaluation results
• Structure a findings report and prioritize by impact rather than technical complexity
• Manage remediation and retesting, and apply human oversight as a mitigation
• Recognize when a finding becomes an incident
• Extend testing and assurance to foundation models and to agentic, tool-using systems

Course Outline

Module 1

• Foundations: What Is AI Red Teaming, and Where Does It Fit in AI Assurance?
• What red teaming is and is not
• Where it sits in the AI assurance lifecycle
• The range of assurance activities
• The guidance the course draws on
• Who is involved

Module 2

• Planning and Scoping a Red-Team Engagement
• The engagement lifecycle
• Scoping to risk classification
• Setting red-team objectives
• Rules of engagement
• Team composition and deconfliction

Module 3

• Adversarial Testing Techniques for Machine Learning and Generative AI
• Matching technique to attack surface
• Manual versus automated and tool-assisted red teaming
• Generative AI probing techniques
• A preview of testing agentic and tool-using systems
• Telling red-team findings apart from ordinary bugs

Module 4

• AI Assurance Frameworks: TEVV, Assurance Cases, and Documentation
• Test, Evaluation, Verification and Validation
• The assurance case as claim, argument and evidence
• Model cards and system cards
• Independent evaluation and measurement science
• From assurance documentation to a risk-acceptance decision

Module 5

• From Findings to Fixes: Reporting, Remediation, and Human Oversight
• Structuring a findings report
• Prioritizing by impact rather than technical complexity
• Remediation and retesting
• Human oversight as a mitigation category
• When a finding becomes an incident

Module 6

• Red Teaming and Assurance for Generative and Agentic AI; Capstone Scenario
• Foundation models and misuse-risk testing
• Agentic AI and the expansion of the attack surface from output to action
• Why red teaming and assurance are recurring rather than one-time
• A capstone scenario