Auditing AI Systems

What you will be able to do

• Scope an AI audit using a risk-based approach mapped to the NIST AI Risk Management Framework and the EU AI Act risk tiers.
• Build an AI system inventory and pinpoint where model risk and data risk concentrate across the lifecycle.
• Assess training data for provenance, quality, representativeness, and consent, and flag the gaps that cause downstream harm.
• Test a model for bias using fairness metrics such as demographic parity, equalized odds, and the four-fifths disparate impact rule.
• Evaluate explainability with tools in the SHAP and LIME families and judge whether an explanation fits the decision it supports.
• Probe model robustness through adversarial and stress testing, drawing on MITRE ATLAS and the OWASP Top 10 for LLM Applications.
• Review model cards and datasheets for datasets, and test documented claims against the evidence you gathered.
• Write an AI audit report that ties findings to evidence, assigns risk ratings, and recommends clear remediation.

What is inside

4 modules, 8 lessons. Each module ends in a graded quiz and most carry an assignment.

1. 01

   Scoping the AI Audit

   Every credible AI audit starts with a scope that matches the system's risk. This module maps the AI lifecycle and the instruments that govern it (the NIST AI Risk Management Framework, ISO/IEC 42001, and the EU AI Act), then shows how to turn a vague request to "audit the AI" into a defined engagement with clear objectives, boundaries, and criteria. You finish able to plan an audit a board or regulator would take seriously.

   2 lessons · 5 quiz questions

2. 02

   Assessing Model and Data Risk

   Most AI failures trace back to data no one vetted or model risk no one owned. This module teaches you to inventory the models in scope, trace data provenance and quality, and evaluate the controls around training, versioning, and deployment, so you can find the weak points before they reach production.

   2 lessons · 5 quiz questions · assignment

3. 03

   Testing the Model: Fairness, Explainability, and Robustness

   Put the model under real scrutiny. You will learn to measure bias with fairness metrics such as demographic parity, equal opportunity, and equalized odds, to judge whether a model can justify its own decisions using the SHAP and LIME families, and to probe robustness through adversarial testing structured by MITRE ATLAS. By the end you can run a defensible fairness, explainability, and robustness assessment and read a vendor's claims with a critical eye.

   2 lessons · 5 quiz questions

4. 04

   Reviewing Documentation and Reporting Findings

   An audit stands or falls on its evidence and its report. This module trains you to read AI documentation (model cards, datasheets for datasets, system cards, and more) as testimony to be verified rather than proof to be trusted, and to build a claim register that maps every assertion to the evidence that would confirm it. You then convert corroborated and contradicted claims into rated findings using the five Cs, and write a report with severity ratings and SMART remediation that both a board and an engineering team can act on.

   2 lessons · 5 quiz questions · assignment