AI-Ready Kubernetes Platform: GPU, DRA, Scheduling, Security & Observability Checklist (2026)

AI-Ready Kubernetes Platform: GPU, DRA, Scheduling, Security & Observability Checklist (2026)

Kubernetes has become the de facto operating system for AI: CNCF reports 82% production adoption overall, with 66% of organizations hosting generative AI models using Kubernetes for at least some inference—but only 7% deploy AI models daily, revealing a large production‑readiness gap. This checklist helps you close that gap by turning a standard container platform into an AI‑ready Kubernetes platform with proper GPU handling, Dynamic Resource Allocation (DRA), advanced scheduling, multi‑tenant security, and AI‑specific observability.Why Most Kubernetes Platforms Are Not Yet AI‑Ready

Most platform teams have solid CI/CD, networking, and basic autoscaling, but AI workloads introduce new constraints:

• Specialized hardware: GPUs, NPUs, and high‑bandwidth memory require explicit discovery, allocation, and sharing.

• Gang and topology scheduling: Training jobs need multiple GPUs/nodes at once; inference needs low latency and predictable placement.

• Multi‑tenancy and cost control: Multiple teams share expensive accelerators; quotas, fair sharing, and chargeback matter more than for typical microservices.

• AI‑specific observability: You need accelerator utilization, model loading time, queue times, inference latency, and throughput—not just pod CPU/memory.

CNCF’s 2025 survey highlights this mismatch: infrastructure is ready (66% using K8s for inference), but operational maturity lags (only 7% deploying daily).

Checklist Overview

Use this as a gate before declaring your platform “AI‑ready”:

1. GPU & Accelerator Foundation

2. Dynamic Resource Allocation (DRA)

3. Advanced Scheduling (Gang, Topology, Queues)

4. Multi‑Tenant Security & Isolation

5. AI Observability & SLOs

6. Operational Playbooks (Scaling, Rollouts, Cost)

Each section below includes concrete checks and example commands or manifests.

1. GPU & Accelerator Foundation

Goal: GPUs are discovered, driver/toolkit installed, and exposed as schedulable resources with clear tiers (training vs inference).

Checks

• Kubernetes version ≥ 1.34 (DRA GA in 1.34, stable‑by‑default in 1.35).

• GPU nodes labeled consistently (e.g., gpu.vendor=nvidia, gpu.family=h100, gpu.usage=inference|training).

• NVIDIA GPU Operator (or vendor equivalent) installed with drivers, toolkit, and device plugin or DRA enabled.

• MIG (Multi‑Instance GPU) or time‑slicing configured where appropriate for multi‑tenant inference.

• Node autoscaler can provision GPU node pools with correct labels and taints.

Example: Install NVIDIA GPU Operator with DRA

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update

helm install gpu-operator nvidia/gpu-operator \
  --namespace gpu-operator --create-namespace \
  --set driver.enabled=true \
  --set toolkit.enabled=true \
  --set dra.enabled=true

Then verify:

kubectl get resourceslices
kubectl get deviceclass

You should see ResourceSlices per GPU node and DeviceClasses for your tiers.

2. Dynamic Resource Allocation (DRA)

Goal: Move from legacy nvidia.com/gpu requests to DRA ResourceClaim/DeviceClass so the scheduler understands device attributes, sharing, and preferences.

Checks

• DRA driver installed (e.g., NVIDIA DRA driver) and publishing ResourceSlices.

• DeviceClasses defined for GPU tiers (e.g., training-gpu, inference-gpu, mig-small).

• Workloads use ResourceClaimTemplate + pod resourceClaims instead of only resources.limits.

• Fallback policies documented (e.g., prefer H100, fall back to 2× A10).

• Legacy device‑plugin path still supported during migration, with a plan to retire it.

Example: DeviceClass + ResourceClaimTemplate

apiVersion: resource.k8s.io/v1
kind: DeviceClass
metadata:
  name: nvidia-inference-gpu
spec:
  selectors:
    - cel:
        expression: |
          device.attributes["nvidia.com"].type == "gpu" &&
          device.attributes["nvidia.com"].product =~ "H100.*"
---
apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: h100-inference-claim
spec:
  resourceClaimTemplateSpec:
    resourceClassName: nvidia-inference-gpu
    parameters:
      # vendor-specific params if needed

Pod snippet:

spec:
  resourceClaims:
    - name: gpu
      resourceClaimTemplateName: h100-inference-claim
  containers:
    - name: vllm
      image: vllm/vllm-openai:latest
      resources:
        requests:
          cpu: "4"
          memory: 16Gi

This lets the scheduler match claims to ResourceSlices with the right GPU attributes.

3. Advanced Scheduling: Gang, Topology, Queues

Goal: Training jobs start only when all GPUs/nodes are available; inference is placed to minimize latency and maximize utilization; teams share GPUs fairly.

Checks

• Gang scheduling enabled (e.g., KAI Scheduler, Volcano, or Kueue with co‑scheduling).

• PodGroups / JobSets used for multi‑worker training with all‑or‑nothing scheduling.

• Topology‑aware scheduling configured (NUMA, NVLink domains, NIC affinity) for multi‑GPU nodes.

• Queue‑based admission control (Kueue, queues, cohorts) with fair sharing and preemption policies across teams.

• PriorityClasses defined for critical inference vs batch training.

Example: Gang-Scheduled Training Job (Conceptual)

Using KAI or Volcano, you define a PodGroup and attach it to your Job/JobSet:

apiVersion: scheduling.volcano.sh/v1beta1
kind: PodGroup
metadata:
  name: train-llama-8b
spec:
  minMember: 8
  queue: ai-training
---
apiVersion: batch/v1
kind: Job
metadata:
  name: train-llama-8b
spec:
  parallelism: 8
  completions: 8
  template:
    spec:
      schedulerName: volcano
      podGroup: train-llama-8b
      # ... containers with DRA claims ...

The job only starts when 8 GPU workers can be placed together.

4. Multi‑Tenant Security & Isolation

Goal: Multiple teams share GPU nodes without interfering with each other’s workloads, data, or secrets, and with clear quotas and chargeback.

Checks

• Namespaces per team/project with ResourceQuotas and LimitRanges for CPU, memory, and GPU claims.

• NetworkPolicies restricting cross‑namespace traffic, especially to model endpoints and data stores.

• RBAC scoped to namespaces and AI‑specific roles (e.g., ai-developer, ai-operator).

• Secrets management for model weights, API keys, and credentials (e.g., external secrets operator, sealed secrets).

• GPU isolation via MIG, time‑slicing, or DRA attributes so one team cannot starve others.

• Admission controllers enforcing image policies, security contexts, and resource claim requirements for AI workloads.

Example: ResourceQuota Including GPU Claims

apiVersion: v1
kind: ResourceQuota
metadata:
  name: ai-team-quota
  namespace: ai-team-a
spec:
  hard:
    requests.cpu: "40"
    requests.memory: 160Gi
    resourceclaims.resource.k8s.io/nvidia-inference-gpu: "8"

This caps how many inference GPUs the team can claim.

5. AI Observability & SLOs

Goal: You can see not just pod health but also accelerator utilization, model loading, queue times, inference latency, and throughput—and tie them to SLOs.

Checks

• Metrics for GPU utilization, memory, and temperature exposed (DCGM exporter or vendor equivalent).

• Custom metrics for:

  • Model load time

  • Request queue time

  • Inference latency (p50/p95/p99)

  • Tokens/sec or requests/sec per model endpoint

• Dashboards per model/service showing:

  • Utilization vs quota

  • Latency SLO compliance

  • Error rates and timeouts

• Tracing for inference requests (e.g., OpenTelemetry) to identify bottlenecks across gateway, model server, and vector DB.

• Alerts on:

  • GPU utilization too low (waste) or too high (contention)

  • Latency SLO breaches

  • Queue depth growth and job starvation

Example: Key AI SLOs

• Inference latency: p95 < 200 ms for chat API.

• Model load time: < 30 s for cold start on new replica.

• GPU utilization: 40–80% sustained for production inference pools.

• Queue time: p95 < 50 ms under target load.

Instrument your serving stack (KServe, vLLM, TGI, etc.) to export these as Prometheus metrics and wire them into your alerting.

6. Operational Playbooks: Scaling, Rollouts, Cost

Goal: You have repeatable patterns for scaling models, rolling out new versions, and controlling cost per workload.

Checks

• HPA/VPA or KEDA configured for inference services based on QPS, latency, or queue depth.

• Cluster autoscaler tuned for GPU node pools (scale‑up/down thresholds, pod priority considerations).

• Rollout strategy for models:

  • Canary or blue/green with traffic splitting

  • Automated rollback on latency/error SLO breaches

• Cost attribution:

  • GPU hours per team/model

  • Storage for model weights and datasets

  • Network egress for inference endpoints

• Runbooks for:

  • Adding a new GPU node pool

  • Migrating a workload from legacy nvidia.com/gpu to DRA

  • Responding to GPU driver or operator upgrades

Example: DRA Migration Playbook (High Level)

1. Inventory GPU workloads by shape (whole‑GPU training, small inference, MIG‑friendly, multi‑GPU).

2. Upgrade control plane and nodes to ≥ 1.34/1.35.

3. Install/upgrade GPU Operator and DRA driver in an isolated GPU node pool.

4. Define DeviceClasses and publish a “how to request GPUs” guide.

5. Migrate one team/workload at a time using ResourceClaimTemplate; keep legacy path running in parallel.

6. Measure GPU utilization and cost before/after; adjust quotas and classes.

Mapping to the CNCF Production‑Readiness Gap

CNCF’s data shows strong infrastructure adoption but weak deployment maturity: 66% use K8s for inference, yet only 7% deploy daily. This checklist targets the missing pieces that block daily, confident deployments:

• Predictable scheduling (DRA + gang + topology) reduces “works on my node” surprises.

• Multi‑tenant controls (quotas, isolation, fair sharing) let multiple teams share GPUs safely.[

• AI observability turns vague “slow model” complaints into measurable SLOs and actionable alerts.

• Operational playbooks make GPU scaling and model rollouts routine instead of heroic efforts.