MLOps & AI Production Operations: The 2026 Guide

How I think about taking AI from a working notebook to something that runs reliably in production — the full path from commit to monitoring, and the decisions that matter along the way.

01 — DevOps vs MLOps: What’s Different

MLOps extends DevOps principles to machine learning — but ML systems have fundamentally different properties that traditional CI/CD wasn’t built for. Everything here assumes the architecture upstream of it is already settled; that groundwork is the AI architecture field guide.

The fundamental difference: In software, code is the complete specification — if the code hasn’t changed, the behavior hasn’t changed. In ML, the data is part of the specification — the model can degrade silently even when no code changes, because the real-world data distribution shifted. This is why MLOps adds continuous training and continuous monitoring to the DevOps loops.

02 — The MLOps Pipeline (End-to-End)

The production MLOps pipeline:

Data ingestion (sources → lake)
  → Data validation (schema + quality)
  → Feature engineering (feature store)
  → Model training (GPU/TPU cluster)
  → Evaluation (metrics + tests)
  → Model registry (versioning)
  → CI/CD gate (approve / reject)
  → Deployment (container + endpoint)
  → Monitoring (drift + quality)
  → ⤴ Retrain

What each stage does

The 20/80 rule: Training a model is ~20% of the effort. The other 80% is testing, packaging, versioning, deploying, monitoring, and maintaining it. Teams that treat ML like one-off experiments instead of production software always fail at scale.

03 — CI/CD for ML: The Three Loops

Traditional DevOps has one CI/CD loop (code change → test → deploy). ML has three:

Loop 1 — Continuous Integration (code): code change → lint + unit tests → integration tests → data validation tests → model training tests (small subset)

Loop 2 — Continuous Training (model): new data arrives or drift detected → feature pipeline runs → training job executes → evaluation against baseline → register new model version

Loop 3 — Continuous Deployment + Monitoring: promote model to staging → canary/shadow deploy → monitor quality + drift → auto-rollback if degraded → trigger retraining if drift exceeds threshold

Google’s MLOps maturity levels

Target: Most teams start at Level 0. The goal is Level 2 — where new data automatically triggers retraining, evaluation gates ensure quality, and deployment is hands-free with auto-rollback. Level 1 (automated training, manual pipeline releases) is a practical intermediate milestone.

04 — DevOps / MLOps Tool Catalog

Organized by function. Pick one tool per row — don’t try to use all of them.

Core DevOps tools (you still need these)

ML-specific tools

LLM-specific tools

Starter stack (if you’re choosing today): Git + GitHub Actions + Docker + Terraform + MLflow + Evidently + vLLM + Langfuse + Qdrant covers 90% of production needs for both traditional ML and LLM systems. Add Kubernetes and Feast as you scale. Avoid tool sprawl — one tool per function.

05 — Cloud Infrastructure Selection

Training infrastructure

Inference infrastructure

Infrastructure decision tree

Trade-off — managed vs self-hosted serving: Managed endpoint (SageMaker AI/Agent Platform/Azure ML) = autoscaling, monitoring, blue-green deployment built in — but higher per-hour cost and less control. Self-hosted (vLLM on K8s) = cheapest at scale, full control over batching/quantization — but you build your own scaling, health checks, and rollback. Start managed; migrate to self-hosted when monthly inference spend exceeds ~$10K and you have dedicated ops capacity.

This section assumes a managed cloud underneath. For the hybrid and on-premises case — outbound-only delivery, cross-cluster connectivity, air-gapped installs — see Connecting Cloud and On-Premises.

06 — Model Deployment Patterns

The model packaging and deployment pipeline:

Trained model (from registry)
  → Containerize (Docker + serving runtime)
  → Push to registry (ECR / Artifact Registry)
  → Deploy to endpoint (real-time / batch / serverless)
  → Health check (smoke test + canary)
  → Live traffic

Serving runtimes for LLMs

07 — Release Strategies for Models

ML-specific release difference: In software, a canary catches code bugs. In ML, a canary catches accuracy degradation under real traffic — which is much harder to detect because the model returns HTTP 200 even when its predictions are wrong. You need quality metrics (accuracy, latency percentiles, business KPIs) in your canary monitoring, not just error rates.

08 — Monitoring & Observability

ML monitoring has four layers — each catches different failure modes.

Layer 1 — Infrastructure monitoring (DevOps standard): CPU/GPU utilization, memory, latency (p50/p95/p99), error rate, throughput (QPS), disk/network

Layer 2 — Model performance monitoring (ML-specific): prediction accuracy, precision / recall / F1, confidence distribution, prediction vs actual (when labels arrive)

Layer 3 — Data / feature monitoring (drift detection): input feature distributions, data schema validation, missing value rates, statistical drift tests (PSI, KL, JS)

Layer 4 — Business KPI monitoring (the only one that really matters): conversion rate, revenue impact, customer satisfaction, false positive cost, escalation rate

Monitoring tool stack

Common mistake: Teams monitor Layer 1 (infrastructure) and think they’re covered. The model returns HTTP 200 even when predictions are garbage. Infrastructure can be perfectly healthy while the model is silently failing. You need all four layers.

09 — Model Drift & Retraining

Three types of drift

Retraining strategies

Whichever trigger you pick, what actually runs is a training job — the techniques, data curation, and eval gates behind it are in model training, fine-tuning and evaluation.

Trade-off — retrain frequency: More frequent = more current model, less drift — but more compute cost, more operational complexity, and more risk of training on noisy/insufficient data. Less frequent = cheaper, simpler — but model degrades between retrains. Match frequency to data volatility: fraud models retrain weekly, product recs daily, medical models with regulatory review quarterly.

10 — LLMOps: What’s Different for LLMs

LLMs add new concerns that traditional MLOps doesn’t cover well. Think of LLMOps as an extension layer.

LLMOps pipeline additions

Prompt versioning (track + A/B test)
  → LLM evaluation (judge + metrics)
  → Prompt caching (90%+ cheaper reads)
  → Model gateway (route + failover)
  → Guardrails (input/output filters)
  → LLM observability (traces + quality scores)

LLM observability: what to monitor

• Quality metrics: faithfulness (is the output grounded in context?), relevance (does it answer the question?), safety (no toxicity/PII/bias).
• RAG metrics: retrieval precision, context relevance, groundedness — is the retrieval pipeline returning useful chunks?
• Operational metrics: tokens per request, time to first token (TTFT), cache hit rate, cost per session, agent step count.
• Drift metrics: embedding drift (semantic shift in queries/outputs), prompt regression (quality drops after prompt edit), model version regression.
• Agent metrics: tool selection accuracy, planning quality, loop count, escalation rate.

The key insight: LLMs fail silently — a hallucinated answer returns HTTP 200. Infrastructure monitoring cannot catch this. You need quality scoring on every trace (or a sample). The best practice is to run an LLM-as-judge on 5–10% of production traces and alert on statistically significant quality drops.

11 — Cost Optimization

Measure cost per outcome, not cost per token: A cheaper model that gets the answer wrong 30% of the time costs more than an expensive model that’s right 95% of the time — because you pay for retries, escalations, and lost customers. Track cost per successful resolution, not just token spend.

12 — Security & Governance

• Version everything: code (Git), data (DVC/LakeFS), models (MLflow registry), prompts (Langfuse/version control), infrastructure (Terraform). If you can’t reproduce it, you can’t audit it.
• Access control: RBAC on model registry (who can promote to production?), IAM on training resources, API auth on serving endpoints.
• Data lineage: track which data trained which model, which model is serving which endpoint. Metadata stores (MLflow, cloud registries) maintain this.
• Audit trail: log every training run, evaluation result, deployment, and rollback. Required for EU AI Act, HIPAA, SOC2.
• Model cards: document each model’s purpose, training data, known limitations, fairness evaluations, and intended use. This is both good practice and increasingly required by regulation.
• Supply chain security: scan model weights for trojans (backdoors injected during training). Don’t download untrusted models from the internet without verification. Use SafeTensors format, not pickle.
• Prompt injection defense: input validation + output verification + separation of trusted/untrusted content + guardrail hooks.

The EU AI Act: High-risk AI systems (Annex III) must demonstrate: transparency, explainability, human oversight, data governance, accuracy/robustness testing, and a conformity assessment. Note the timeline has shifted — the AI “Digital Omnibus” (agreed in May 2026, in force since July 27, 2026) postponed the high-risk (Annex III) obligations from August 2, 2026 to December 2, 2027, and product-embedded (Annex I) systems to August 2, 2028, so the near-term compliance deadline is no longer 2026. That’s more runway, not a reprieve: building MLOps pipelines that version, test, monitor, and audit is how you demonstrate compliance, and it’s far easier to bake in now than to retrofit later. If your pipeline can’t reproduce a training run and explain a prediction, you won’t be EU AI Act compliant when the obligations land.

13 — Trade-off Master Reference

Built from current industry practice and production patterns. Independent reference — not affiliated with any vendor. Tools and pricing change rapidly; verify before procurement decisions.