Production ML Operations

What you will be able to do

• You will be able to design a model serving architecture with explicit latency, throughput, and availability SLOs across online, batch, and streaming inference.
• You will be able to scale and cost-optimize inference using autoscaling, request batching, caching, and accelerator (GPU) utilization tuning.
• You will be able to instrument production models to detect data drift, concept drift, and prediction drift using tests such as PSI and Kolmogorov-Smirnov.
• You will be able to monitor real model quality when ground-truth labels arrive late or never by combining proxy metrics, sampling, and delayed-label reconciliation.
• You will be able to build automated retraining pipelines with clear triggers (scheduled, performance-based, and drift-based) and continuous-training orchestration.
• You will be able to promote new models safely using validation gates, shadow deployment, canary rollout, champion-challenger tests, and fast rollback.
• You will be able to operate a model registry with versioning, lineage, model cards, and approval workflows that stand up to NIST AI RMF and audit expectations.
• You will be able to run a model incident response process: classify severity, execute runbooks, roll back, find root cause, and write a blameless postmortem.

What is inside

4 modules, 8 lessons. Each module ends in a graded quiz and most carry an assignment.

1. 01

   Serving and Scaling the Model Platform

   You cannot operate a model you cannot serve reliably, so this module builds the platform underneath everything else. You will learn the three serving patterns (online, batch, and streaming) and when each is right, how to set latency, throughput, and availability SLOs against real traffic, and why tail latency and utilization decide your capacity. Then you will scale and optimize inference: autoscaling on the right signals, batching, quantization, and compiled runtimes, and right-sizing hardware so cost per prediction stays under control. By the end you can size, autoscale, and tune a serving stack that meets its SLOs without the bill running away.

   2 lessons · 5 quiz questions

2. 02

   Monitoring Drift and Performance

   A served model decays the moment it meets live data, so it needs a heartbeat. This module teaches ML observability: detecting data drift, concept drift, and prediction drift with the right statistical tests, and tracking true model quality even when ground-truth labels arrive late or never. You will build the monitoring that warns you a model is failing before your users do.

   2 lessons · 5 quiz questions · assignment

3. 03

   Closing the Loop with Automated Retraining

   Monitoring earns its keep only when a fired alert produces action. This module turns signals into a working retraining loop. You will design scheduled, performance-based, and drift-based triggers; build a continuous training pipeline that is reproducible and safe to run unattended; and promote a new model without breaking production by gating candidates on validation, proving them with shadow traffic, ramping them through canary stages, and rolling back on defined criteria.

   2 lessons · 5 quiz questions

4. 04

   Governance and Incident Response

   Operating ML at a real company means someone can always ask what shipped, why, and who approved it. This module builds the governance layer that answers those questions (model registry, immutable versioning, lineage, model cards, and approval workflows mapped to the NIST AI RMF) and then the discipline for when models break: classifying incidents by severity, running a structured response that mitigates before it diagnoses, and writing a blameless postmortem that feeds fixes back into governance. You finish able to make a model both auditable and recoverable.

   2 lessons · 5 quiz questions · assignment