Google Unveils Open AI-Native Kubernetes Stack at KubeCon

At KubeCon North America 2026, Google Cloud introduced an open AI-native stack on Kubernetes to unify reinforcement learning post-training, distributed inference, and autonomous agent orchestration under a common cloud-native control plane. The software framework connects open-weight models such as Gemma with Cloud Native Computing Foundation (CNCF) Sandbox projects, including OpenRL for cooperative accelerator time-slicing, llm-d 1.0 for prefix-cache-aware inference routing, KubeRay for distributed compute jobs, and gVisor alongside Agent Substrate for sandboxed agent execution. By standardizing these operational layers on upstream Kubernetes, the stack addresses persistent infrastructure challenges around accelerator underutilization, key-value cache fragmentation, and multi-tenant security when hosting production AI workloads.

A unified open-source runtime for post-training and inference

Modern enterprise engineering teams are shifting away from closed, black-box APIs toward running domain-adapted, open-weight models on private infrastructure. While supervised fine-tuning (SFT) has become standard, complex reasoning, mathematical verification, and autonomous tool use require post-training through reinforcement learning (RL). Historically, operationalizing RL has introduced severe cluster inefficiencies. Workflows alternate between training steps and rollout generation, leaving expensive GPU and TPU accelerators idle during phase transitions. Operating siloed clusters—such as standalone Slurm or bespoke Ray deployments alongside standard container fleets—also fragments compute capacity and increases administrative overhead.

To eliminate this friction, Google created OpenRL and contributed it to the CNCF Sandbox. OpenRL presents machine learning developers with a Python abstraction built around four primitives: forward_backward, optim_step, sample, and save_state. Rather than requiring ML engineers to manage low-level Kubernetes manifests, OpenRL translates these primitives directly into Kubernetes scheduling workflows.

To address accelerator underutilization, OpenRL implements cooperative accelerator time-slicing, interleaving compute-intensive training steps and latency-sensitive rollout generation on the same physical GPUs and TPUs. Rather than transferring multi-gigabyte checkpoints over cluster networks after each optimization step, OpenRL uses delta-weight synchronization to transmit only parameter deltas to rollout workers. The platform also incorporates zero-configuration Dynamic Resource Allocation (DRA), profiling batch dimensions, context windows, and Low-Rank Adaptation (LoRA) settings to dynamically bin-pack workloads across heterogeneous accelerator pools.

Eliminating memory stalls and accelerator idle time

Serving post-trained models under interactive agent traffic introduces distributed systems hurdles that standard ingress controllers cannot handle. Conventional Kubernetes ingress mechanisms route incoming traffic using stateless round-robin or least-connections algorithms, scattering sequential conversational turns across random replicas and destroying key-value (KV) cache locality.

To solve this, Google Cloud co-founded llm-d 1.0 alongside Red Hat, IBM Research, CoreWeave, and NVIDIA. Contributed to the CNCF Sandbox, llm-d functions as a distributed inference control plane that coordinates standalone serving engines such as vLLM and SGLang into an integrated serving fabric.

Central to llm-d is its prefix-cache-aware Endpoint Picker. Integrated into the Kubernetes Gateway API Inference Extension and managed on Google Kubernetes Engine via GKE Inference Gateway, the Endpoint Picker monitors real-time KV cache residency across worker pods. Production data published by Google indicates that cache-aware routing doubled KV cache hit rates from 35% to 70% and reduced median time-to-first-token (TTFT) by 70% to 85% on multi-turn agent workloads.

To handle extended context windows, llm-d cascades KV cache blocks from accelerator high-bandwidth memory (HBM) to host CPU RAM, and subsequently to local NVMe SSDs, Google Cloud Managed Lustre, or Cloud Storage. Additionally, llm-d supports Encoding, Prefill, and Decode (EPD) disaggregation through DisaggregatedSet, an abstraction built atop Kubernetes LeaderWorkerSet that decouples computational phases onto specialized node pools. Hardware parity across accelerators is managed through TorchTPU, delivering native PyTorch support across TPU v5e, v6e, and v7x hardware alongside NVIDIA GPUs.

Domain benchmarking and multi-tenant isolation

As AI agents execute shell commands and orchestrate APIs, multi-tenant security becomes essential. Running untrusted code inside shared Kubernetes clusters introduces sandbox escape risks. The open stack mitigates these threats using gVisor, a CNCF Sandbox project providing a user-space kernel that intercepts and virtualizes host system calls.

Complementing gVisor, Google contributed Agent Substrate to the CNCF Sandbox to manage agent lifecycles and provide sub-second ephemeral execution targets. Agent Substrate decouples execution state from compute, enabling idle agents to suspend and resume within 500 milliseconds without consuming active cluster resources. This effort parallels broader ecosystem standardization, such as Kubernetes formalizing AI conformance under a dedicated SIG subproject.

To evaluate the stack under production conditions, Google tested Gemma 4 on the Legal Agent Benchmark (LAB) developed by Harvey.ai across 24 legal practice areas. While standard supervised fine-tuning plateaued around 50% accuracy on multi-step reasoning rubrics, executing reproducible RL training loops with OpenRL increased task accuracy to 70%. Throughout testing, OpenRL shared cluster capacity with concurrent LoRA fine-tuning and live inference workloads on GKE without manual partition reconfiguration.

What happens next

The open-source stack is being showcased through sessions and live demos at KubeCon NA 2026 in Salt Lake City, including demonstrations of Arm-based inferencing on GKE, gVisor agent sandboxing, and node auto-provisioning across spot accelerator pools.

Platform engineers can deploy these tools using sample manifests from upstream repositories. OpenRL fine-tuning recipes are accessible via openrl.dev, while llm-d 1.0 inference pools can be integrated through the Gateway API Inference extension. KubeRay documentation provides deployment guidance for managing distributed Ray jobs alongside Kueue queueing on production clusters.

While integrating OpenRL, llm-d, and gVisor offers a path toward reducing infrastructure spend, platform teams must still navigate operational complexity. Managing multi-tiered KV offloading, coordinating DRA accelerator drivers across mixed GPU and TPU fleets, and tuning preemption thresholds require careful validation before deploying production agent workloads.