MLOps & LLMOps Engineering: A Practical 2026 Guide

The unit of work is the complete release

A model file is not an AI system. A production prediction also depends on the data snapshot, feature logic, source revision, dependencies, inference schema, preprocessing, container, serving runtime, resource configuration, route, and policy. An LLM response adds more moving parts: tokenizer and chat template, prompt version, model parameters, context assembly, retrieval evidence if present, adapter, output schema, safety controls, and gateway behavior.

MLOps and LLMOps are therefore release engineering disciplines. Their central question is not “Which model scored highest?” It is “Which exact, approved combination produced this behavior, what evidence allowed it to serve traffic, how do we detect deterioration, and how do we restore a known-good combination?” If the answer relies on a mutable file named final.pkl, a latest container tag, or an undocumented prompt pasted into code, the release is not reproducible.

The five-phase roadmap starts with lineage, adds data and delivery contracts, then serving and monitoring, LLM evaluation and tracing, and finally cost, rollout, governance, and decommissioning. The accompanying checks and projects are practical engineering exercises, not exam preparation.

Track experiments and lineage that can reproduce a decision

MLflow Tracking organizes work around experiments, runs, and models. A run can record parameters, metrics, start and end times, tags, and artifacts. That is the minimum structure for comparing training executions. A practical run also records the source revision, dataset identity, feature revision, dependency environment, random seeds, and hardware or execution context that materially affect the result.

MLflow 3 can link metrics to particular logged models and datasets. This matters because “accuracy 0.91” is ambiguous. Was it measured on the validation snapshot or training data? Which checkpoint produced it? Which slice and label version were used? Linking a metric to the evaluated dataset and model turns a loose number into evidence. Dataset-aware search also makes comparisons fairer than sorting every run by an unqualified metric name.

A dataset path is not a dataset version. A reliable identity includes an immutable snapshot, revision, or digest; schema; source and collection period; transformation revision; labeling process; and a card describing intended use, bias, limitations, and license. Time-aware prediction requires point-in-time correctness: features for a training row must be constructed only from information available at that row’s prediction time. Leakage can create spectacular offline results and immediate production failure.

Logged models should include an inference signature and representative input example. These form a testable boundary between clients and the serving system. They do not prove semantic correctness, but they catch common incompatibilities in names, types, shapes, and output contracts before a request reaches production.

Use registries for immutable versions and movable intent

MLflow Model Registry creates distinct model versions associated with logged models and source runs. Version descriptions, tags, and aliases provide lifecycle evidence. A validation tag can record whether a model passed a gate; a candidate alias can identify the version currently under release review; a champion alias can identify the approved version for the majority route.

Aliases and versions serve different purposes. The numeric version is immutable identity. The alias is movable intent. Rollback can reassign intent to a prior version, but a registry update alone does not prove serving pods reloaded the artifact. Deployment automation must reconcile the registry, route, workload, runtime cache, and live version telemetry.

MLflow documents model stages as deprecated. Modern workflows can use aliases, tags, and separate access-controlled registered models or environments. Mature promotion also moves source, training, inference, feature, monitoring, and infrastructure code through CI/CD—not just a binary model between labels.

Prompt Registry applies a similar discipline to LLM templates. Prompt versions are immutable and can be compared. Aliases permit A/B tests and rollback. The prompt object can include response format and model configuration, but teams should understand which fields are mutable and ensure release evidence captures the effective values. Cache behavior also matters: an alias can point to a new version while a process temporarily retains the previous resolved prompt.

Build CI/CD/CT as separate controlled boundaries

Continuous integration for ML combines ordinary software checks with data and statistical contracts. Unit tests validate transformations and utility code. Schema and quality tests validate types, ranges, nulls, categories, uniqueness, relationships, and freshness. Leakage tests use known traps. Training-serving parity tests send golden records through both transformation paths. A bounded training smoke test proves that the pipeline can produce a loadable model with a valid signature.

A candidate evaluation then compares the new release with an immutable baseline using declared tolerances. Aggregate metrics are not enough. Relevant segments, calibration, safety, resource use, and failure behavior may be release criteria. A candidate that gains one point of average accuracy while failing a protected segment or doubling memory should fail unless an accountable exception process explicitly accepts the residual risk.

Continuous delivery makes a passing release available to deploy. Continuous deployment may route it automatically when policy allows. Continuous training is different: a schedule, validated data arrival, drift signal, or performance signal can start a new training candidate. CT should be idempotent, deduplicate triggers, validate its input, and pass the same evaluation, registry, security, and release gates. Retraining completion is never automatic evidence that production should change.

Separate permissions strengthen the design. A training identity may write run artifacts but not reconfigure production traffic. A CI identity may register a passing version but not approve its own high-impact release. A deployment controller should consume a signed or reviewed release manifest instead of assembling mutable references at runtime.

Serve a pinned model and runtime contract

MLflow Models package model metadata, dependencies, and inference schema. Containers help reproduce the serving environment. Kubernetes then manages desired application state, while KServe provides inference-oriented resources and runtimes for multiple frameworks. KServe documentation recommends explicitly setting runtimeVersion for consistent production services. Otherwise, a platform upgrade can alter serving behavior while the model URI remains unchanged.

A release manifest should bind the model version or digest, image digest, serving runtime, tokenizer or preprocessing revision, feature contract, API protocol, and relevant configuration. Kubernetes requests and limits, startup and readiness behavior, timeouts, service identity, and network access are part of the release because they shape availability and latency.

Canary deployment limits blast radius. In KServe’s serverless canary flow, a percentage of traffic can reach the latest revision while the previous rolled-out revision remains available. Promotion removes the canary split and sends traffic to the new revision. Rollback can set the candidate’s percentage to zero and route requests to the previous model. Feature availability depends on deployment mode, so the exact KServe topology and version must be tested instead of assuming every manifest supports the same fields.

Kubernetes itself retains Deployment revision history and supports rollout status and undo. This is useful for serving containers, but it rolls back the Pod template—not external model registries, object storage, prompt aliases, or feature state. A complete AI rollback coordinates every external dependency and verifies the result with live requests and telemetry.

Monitor data quality, drift, service health, and outcomes separately

One dashboard cannot collapse every failure mode into a single “model health” light. Service metrics answer whether requests succeed within latency and capacity objectives. Data quality checks reveal schema breaks, null growth, stale inputs, category expansion, and invalid ranges. Drift compares distributions. Prediction monitoring shows output mix and confidence behavior where appropriate. Delayed labels reveal actual quality. Segment views show whether aggregate stability hides concentrated harm.

Evidently compares current and reference datasets using methods selected by column type and sample size unless customized. It can produce column drift and a dataset-level decision based on the share of drifting columns. Methods and thresholds have different semantics: a statistical test may flag a small p-value, while a distance method flags a value above a threshold. Teams should select a meaningful reference, calibrate methods against history, and document important columns and false-positive trade-offs.

Drift is not model failure. It says distributions changed under the configured comparison. The change may be benign, expected seasonality, a data bug, a new population, or a signal of performance decay. If labels are delayed, drift can prioritize investigation or labeling. It should not silently authorize retraining and deployment.

Missingness needs separate checks because drift calculations can filter empty values. A feature’s non-null distribution may remain stable while its null rate doubles. The same principle applies to schema and freshness: statistical similarity does not establish that the input pipeline is complete or timely.

Evaluate the complete LLM release

LLM evaluation begins with a versioned dataset. MLflow describes an evaluation through data, a predict function, and scorers. A useful corpus includes representative tasks, edge cases, structured-output cases, ambiguity, conflicting evidence, abstention cases, safety and refusal scenarios, prompt injections, excessive input lengths, and relevant language or accessibility slices. Development examples and held-out release cases should be separated where practical.

Scorers should reflect the task. Deterministic checks can validate exact values, schemas, citation identifiers, forbidden strings, length, and tool constraints. Human review remains important for nuanced quality and high-impact failures. LLM-as-a-judge can scale subjective review, but the judge is another versioned model-prompt-config system. Calibrate it against labeled examples, measure disagreement, inspect critical misses, and avoid treating one judge score as ground truth.

Evaluate quality, grounding, citation support, completeness, abstention, refusal, injection resistance, safety policy, structured-output validity, latency, errors, tokens, and cost. The release unit includes model, adapter, prompt, parameters, response schema, context builder, retrieval version, safety policy, and route. Comparing model A and model B while prompts differ does not identify the cause of an improvement.

Production-like feedback can expand the evaluation set, but it needs governance. Sampled traces should enter a review queue; outputs should not label themselves; personal or confidential data should not become permanent test fixtures by default. Every added case needs provenance, purpose, access, and retention.

Operate vLLM multi-model and adapter serving deliberately

vLLM provides an HTTP server compatible with OpenAI-style completion and chat interfaces and supports many model architectures. Compatibility is valuable, but API shape is not a complete security perimeter. Official vLLM documentation warns that API-key authentication does not protect every endpoint, so production-style designs should use an authenticated gateway, network restrictions, and explicit endpoint exposure.

Multiple model identities can use separate serving pools, which provides strong isolation but duplicates model memory. Compatible LoRA adapters can share a base model and reduce memory, but they introduce provenance, compatibility, routing, and administrative risk. Pin the base revision and adapter revision, record their relationship, test each combination, and set per-route quotas.

Dynamic LoRA loading is especially sensitive. vLLM states that runtime adapter loading is not a secure operation for deployments exposed to untrusted clients. Disable it, preload approved adapters, or isolate load and unload endpoints behind a trusted administrative identity, allowlisted storage, and network controls. A model prompt is not authorization to load an artifact.

Capacity depends on model size, quantization, prompt and output length, concurrency, batching, KV cache, adapters, hardware, and parallelism. Tensor parallelism shards parameters across GPUs and can reduce per-GPU memory pressure, but communication can hurt latency or throughput. Pipeline and data parallel approaches have different use cases. Measure before adding complexity.

Trace latency and connect efficiency to acceptable outcomes

OpenTelemetry traces model a request as related spans. An LLM trace can include gateway validation, prompt loading, retrieval, context assembly, scheduler queue, inference, streaming, output validation, and post-processing. Stable release IDs, prompt and adapter versions, safe token counts, queue time, time to first token, duration, status, retries, and sanitized errors make regressions diagnosable.

OpenTelemetry’s Generative AI semantic conventions are evolving and have moved to a dedicated repository. Record the convention and instrumentation versions used and expect migrations. Do not turn traces into an unrestricted transcript store. OpenTelemetry security guidance emphasizes data minimization and makes implementers responsible for sensitive data. Avoid credentials, tokens, raw personal data, complete retrieved documents, and unnecessary prompts or outputs. Use source prevention first, then allowlists, redaction, restricted access, sampling, and retention.

Cost needs workload context. Track GPU-hours, model and adapter memory, KV-cache headroom, requests and tokens per second, evaluation calls, artifact storage, and telemetry. Time to first token matters for interactive experience; output-token latency matters during streaming; end-to-end latency includes the complete application. Compare metrics under the same prompt/output and concurrency distribution.

Cost per request can reward cheap failures. Cost per acceptable outcome divides total cost by results that pass defined quality and safety gates. It connects efficiency to utility. Report both, along with capacity and rejection behavior, because a route that is inexpensive at low load may fail expensively under queue growth or repeated retries.

Use NIST AI RMF across release and retirement

NIST AI RMF organizes work into Govern, Map, Measure, and Manage. Govern establishes policy, accountability, training, risk tolerance, and oversight. Map defines the context, intended use, affected parties, dependencies, limitations, and impacts. Measure evaluates validity, reliability, safety, security, privacy, fairness, transparency, and other context-relevant risks. Manage prioritizes responses, monitors operation, handles incidents, and retires systems.

The functions are iterative. A new pretrained model, adapter, dataset, population, or use case can change the risk map and require new measurements. The NIST Generative AI Profile adds considerations relevant to generative systems. Use it as a companion for risks such as confabulation, information integrity, misuse, privacy, security, and third-party foundation models, while tailoring actions to the actual system.

Governance should be visible in engineering artifacts: named owners, intended and prohibited use, a model and dataset inventory, evaluation thresholds, approval records, incident procedures, kill-switch authority, access reviews, retention, and residual-risk decisions. The model cannot approve itself. A passing benchmark does not substitute for use-context review.

Retirement is a release operation. Remove routes, compute, aliases, identities, secrets, access, artifacts, caches, retained telemetry, alerts, and dependent automation according to policy. Preserve only required audit evidence. Verify inventory and billing; deleting DNS or a dashboard does not prove an endpoint, GPU, or credential is gone.

Release in stages and make rollback a tested path

Begin with offline gates on versioned data. For an LLM release, shadow traffic can send synthetic or policy-approved mirrored requests to a candidate without using its output. Score the candidate asynchronously against the champion under the same quality, safety, latency, token, and cost rules. Shadowing validates request shape and capacity with no decision impact, though it still consumes resources and creates telemetry.

A canary sends a bounded traffic share or an explicit cohort to the candidate. The release dashboard must distinguish champion and candidate model, prompt, adapter, image, and runtime versions. Stop conditions should be automatic for clear reliability and safety breaches, with a named person authorized to hold or abort ambiguous releases.

Promotion expands traffic only after the observation window passes. Rollback sends candidate traffic to zero or restores the known-good manifest. Exercise a bad image, incompatible signature, drifted preprocessing, unsafe prompt, failing adapter, memory regression, and cost regression in a synthetic environment. Measure detection and recovery time.

A kill switch is broader than rollback when the safe response is to disable a route, adapter, tool, or model family entirely. It should be available to trusted operators, auditable, and independent of the failing model path. Recovery verification checks live traffic and behavior, not only control-plane success messages.

Two projects that prove the lifecycle

Project 1: versioned ML model delivery and monitoring

Generate time-aware synthetic maintenance records with controlled missingness, drift, label delay, category changes, and leakage traps. Create immutable dataset manifests and a shared feature contract. Track training runs, datasets, metrics, model signatures, artifacts, and dependencies in MLflow. Register only passing candidates and assign tags and aliases.

Deploy the exact model and runtime through KServe. Instrument request validation, preprocessing, inference, and monitoring with OpenTelemetry. Run data quality and calibrated Evidently drift checks, then join delayed synthetic labels for aggregate and segment performance. Canary a candidate, intentionally breach a gate, return its traffic to zero, and reconcile registry and runtime state. Finish with a CT candidate trigger that cannot bypass release approval.

Project 2: multi-model LLM serving, evaluation, cost, and rollout

Build an invented support corpus with expected answers, citations, abstentions, structured outputs, injections, and safety cases. Pin two small license-compatible model identities or versions and optional LoRA adapters from Hugging Face. Create cards, checksums, a remote-code policy, and base-adapter lineage. Version prompts, response schemas, routing, scorers, judge configuration, and evaluation datasets in MLflow.

Serve approved routes through authenticated vLLM pools or a validated multi-LoRA setup. Trace prompt resolution, queueing, inference, streaming, and output policy while minimizing content. Benchmark TTFT, output-token latency, throughput, GPU memory, KV cache, adapter mix, and cost per acceptable outcome. Move through offline, shadow, and canary gates. Inject a safety or memory regression, activate the kill switch, restore the exact known-good route, and prove complete cleanup.

A twelve-week implementation sequence

1. Week 1: Define use context, owners, risks, release gates, artifact policy, and synthetic-data boundaries.
2. Week 2: Build immutable dataset identities, model and dataset cards, MLflow runs, dataset links, and signatures.
3. Week 3: Add schema, null, leakage, point-in-time, feature parity, software, and bounded-training tests.
4. Week 4: Create registry candidate gates, aliases, tags, artifact scans, environment separation, and rollback manifests.
5. Week 5: Deploy a pinned model, image, feature revision, and KServe runtime to a disposable cluster.
6. Week 6: Add OpenTelemetry service traces, governed prediction records, data quality, drift, and delayed-outcome monitoring.
7. Week 7: Canary, promote, inject a failing release, prove rollback, and add a controlled CT candidate workflow.
8. Week 8: Create the synthetic LLM evaluation corpus, pin Hugging Face model and adapter revisions, and version prompts.
9. Week 9: Deploy authenticated vLLM routes, restrict adapter administration, and run offline quality and safety gates.
10. Week 10: Add LLM traces, content policy, token metrics, TTFT, output-token latency, GPU memory, and capacity tests.
11. Week 11: Calculate acceptable-outcome cost and progress a candidate through shadow and canary evaluation.
12. Week 12: Exercise kill switch and rollback, complete 25 original checks, publish sanitized evidence, and decommission every resource.

Present evidence without overstating it

Publish a release manifest, lineage graph, data and feature contracts, model and dataset cards, evaluation report, live-version dashboard, trace example, drift analysis, capacity envelope, cost model, canary timeline, rollback timeline, risk register, and cleanup checklist. Remove secrets and synthetic personal-like content from every screenshot and export.

State the measured environment: versions, hardware, dataset size, token distribution, concurrency, thresholds, and retention. A local KServe or vLLM project does not prove production scale, regulatory compliance, universal fairness, or incident leadership. A passing evaluation does not prove future prompts or populations are safe. A drift threshold calibrated on synthetic history may need complete redesign for another use case.

The skills are relevant to ML platform, MLOps, LLMOps, AI platform, data, DevOps, SRE, and model-serving responsibilities, but no role, demand level, salary, interview, or hiring outcome is guaranteed. Use current role research to compare requirements and identify additional gaps.

Official references

• MLflow Tracking
• MLflow datasets
• MLflow model signatures
• MLflow Model Registry workflows
• MLflow deployment
• MLflow Prompt Registry
• MLflow LLM and agent evaluation
• MLflow tracing
• Kubernetes Deployments and rollback
• KServe serving runtimes
• KServe canary rollout
• OpenTelemetry traces
• OpenTelemetry sensitive-data handling
• Evidently data drift explainer
• Evidently drift customization
• Hugging Face repository versioning
• Hugging Face model cards
• Hugging Face dataset cards
• vLLM OpenAI-compatible server
• vLLM LoRA adapters
• vLLM security
• vLLM optimization and parallelism
• NIST AI Risk Management Framework
• NIST AI RMF Playbook
• NIST Generative AI Profile

Continue across every learning surface

• MLOps & LLMOps Engineering five-phase roadmap
• MLOps & LLMOps original knowledge checks
• MLOps & LLMOps flashcards
• MLOps & LLMOps portfolio projects
• DevOps for AI Apps
• OpenTelemetry Observability Engineering Guide
• AI Infrastructure Guide
• Explore related jobs and current role requirements
• PrepKloud editorial policy

Frequently asked questions