How GPU Cloud Operators Isolate Tenant Workloads in Kubernetes

Summary

• Proper GPU tenant isolation is crucial for preventing security risks and performance issues like the "noisy neighbor" problem, with 74% of companies reporting dissatisfaction with current resource management tools.
• Achieving true isolation requires a multi-layered approach across the control plane (RBAC, quotas), data plane (NetworkPolicies), and the hardware itself.
• At the hardware level, NVIDIA MIG offers the strongest hardware-enforced partitioning for production and untrusted workloads, unlike software-based methods like time-slicing.
• For strong control plane isolation, vCluster provides each tenant with a fully isolated Kubernetes environment, preventing resource conflicts and ensuring a secure GPU architecture with tenant isolation.

Building a GPU cloud — or scaling AI infrastructure across multiple teams — means one thing quickly becomes non-negotiable: every tenant needs genuinely isolated resources. Shared GPU clusters without proper isolation create data leakage risks, noisy-neighbor performance problems, and compliance gaps that grow worse at scale.

This article covers the full isolation stack: control-plane isolation, data-plane isolation, and hardware-level GPU partitioning — and how to layer these together in a production Kubernetes environment.

Why Tenant Isolation for GPUs Isn't Optional

The cost of getting this wrong isn't abstract. A recent industry survey found that 74% of companies are dissatisfied with their current job scheduling and resource management tooling — a signal that most teams are still dealing with poorly isolated, contention-prone infrastructure.

The risks fall into two buckets: security and performance.

Security Risks

At the GPU level, traditional hardware lacks tenant-level memory isolation by default. Without explicit controls, workloads running on the same physical GPU are exposed to side-channel attacks and memory leakage — where one tenant's process can potentially read data left in GPU memory by another. This isn't a theoretical concern in regulated industries; it's a compliance blocker.

Beyond GPU memory, there are two other threat vectors worth taking seriously:

• Container breakout: A misconfigured or malicious container can escape its namespace and access host-level GPU resources directly, compromising every other tenant running on that node.
• Lateral movement: Without network-level isolation, a compromised workload can probe and attack other services inside the cluster. One bad tenant becomes a beachhead.

There's also a subtler risk that catches teams off guard: shared Kubernetes resources like CRDs and PersistentVolumes can create unintended cross-namespace access, especially when different tenants need different versions of the same operator.

Performance Risks

The noisy neighbor problem is the most common complaint on shared GPU clusters. One tenant's memory-heavy training job can saturate GPU compute or bandwidth, degrading throughput for every other workload on the same node. Without resource quotas, there's no ceiling — just first-come, first-served chaos.

Configuration conflicts compound this. When multiple teams need different versions of the NVIDIA GPU Operator, or different CRD schemas for their ML frameworks, a single shared cluster's global API surface becomes a liability.

The Isolation Stack: Three Layers, Not One Setting

Tenant isolation is a stack, not a setting. You can't solve this with a single Kubernetes feature. Real isolation requires coordinated controls at three distinct layers:

1. Control Plane Isolation — Who can do what, and in which boundary?
2. Data Plane Isolation — Can workloads communicate with each other?
3. Hardware Isolation — Are workloads sharing the actual silicon?

The rest of this article walks through each layer with practical implementation guidance.

Layer 1: Control Plane Isolation

Control plane isolation determines what each tenant can see, touch, and configure in Kubernetes. This layer has three tiers of increasing strength.

Namespaces — A Starting Point, Not a Finish Line

Kubernetes namespaces are the baseline unit of resource grouping. They let you organize workloads, apply quotas, and scope RBAC permissions per team. But a common misconception trips up many platform teams: namespaces are not a security boundary.

A namespace gives you logical separation. It does not prevent a container from escaping to the host, does not block cross-namespace network traffic by default, and does not scope cluster-level resources like nodes or CRDs. Treating namespaces as your primary isolation mechanism leaves real gaps that more determined or misconfigured workloads will find.

Use namespaces as the scaffolding, not the wall.

RBAC — Controlling the API Surface

Role-Based Access Control (RBAC) is how you enforce who can do what within a namespace. The key discipline here is scope: reach for Role and RoleBinding (namespace-scoped) instead of ClusterRole and ClusterRoleBinding (cluster-wide) wherever possible.

A tenant who only needs to deploy workloads in their namespace should have exactly that — not the ability to read secrets across the cluster, modify node labels, or view other tenants' resources. Principle of least privilege isn't just a compliance talking point; it's the practical difference between an incident that stays contained and one that doesn't.

Resource Quotas — Preventing the Noisy Neighbor Problem

ResourceQuota objects let you set hard limits on what a namespace can consume. For GPU clusters, this means capping not just CPU and memory, but GPU allocation directly via the nvidia.com/gpu resource type:

apiVersion: v1
kind: ResourceQuota
metadata:
 name: tenant-gpu-quota
 namespace: tenant-a
spec:
 hard:
   requests.nvidia.com/gpu: "4"
   limits.nvidia.com/gpu: "4"


LimitRanges complement this by setting default and maximum resource requests at the pod level, ensuring no single workload in the namespace can quietly claim unlimited GPU memory even if the quota hasn't been hit.

Together, these prevent the most common complaint in tenant isolation: one team's training run monopolizing the cluster and starving everyone else.

vCluster for True Control Plane Isolation

Namespaces, RBAC, and quotas go a long way — but they all share the same API server, the same etcd, and the same global CRD registry. For GPU-heavy platforms with multiple teams or external customers, that shared surface eventually creates problems.

vCluster gives each team or customer their own isolated tenant cluster — a fully functional Kubernetes environment with its own API server, etcd, RBAC, and CRDs — running inside your shared GPU host cluster. From the tenant's perspective, they have cluster-admin access and a complete Kubernetes environment. From the operator's perspective, you're running hundreds of tenant clusters on the same underlying GPU hardware.

This matters in GPU-heavy environments because custom operators — like the NVIDIA GPU Operator — deploy CRDs globally in a standard Kubernetes cluster, creating conflicts when multiple tenants need different configurations. With isolated tenant clusters, each tenant's operators and CRDs are fully scoped, eliminating the conflict entirely.

vCluster supports a range of isolation tiers designed for production GPU infrastructure:

• Shared nodes — multiple tenant clusters share the same GPU nodes, with namespace-level resource quotas enforced
• Private nodes — tenant clusters are assigned dedicated GPU nodes, providing stronger hardware-level separation
• vNode isolation — kernel-native workload isolation (seccomp, cgroups, AppArmor) per workload, without hypervisor overhead

GPU cloud providers including CoreWeave and Nscale run this architecture in production at 100K+ GPU nodes. See how vCluster works for AI cloud operators →

Layer 2: Data Plane Isolation

Locking down the control plane prevents tenants from interfering with cluster configuration. But without data plane controls, their workloads can still talk to each other freely — and that's a problem.

Network Policies — Default Deny, Explicit Allow

By default, every pod in a Kubernetes cluster can reach every other pod, regardless of namespace. In an environment with multiple tenants, this means a workload in tenant A's namespace can probe, call, or exfiltrate data from tenant B's services.

Kubernetes NetworkPolicies fix this by letting you define exactly which pods can communicate with which other pods, and over which ports and protocols. The recommended starting point for any tenant namespace is a default-deny ingress and egress policy, with explicit rules added only for traffic that's actually needed:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
 name: default-deny-all
 namespace: tenant-a
spec:
 podSelector: {}
 policyTypes:
 - Ingress
 - Egress


From that baseline, you layer in allowances — same-namespace communication, access to a specific inference endpoint, egress to a model registry. This is the best practice approach for infrastructure tenancy in Kubernetes deployments: start closed, open deliberately.

Note that NetworkPolicy enforcement depends on your CNI plugin — Calico, Cilium, and Weave all support it, but Flannel does not by default. Verify your CNI supports policy enforcement before relying on it.

Pod Security Standards — Blocking Privileged Escalation

Pod Security Standards (PSS) are enforced via the Pod Security Admission controller, which can be applied at the namespace level to restrict what security contexts tenants are allowed to run.

The relevant profiles for production environments with infrastructure tenancy are:

• Baseline — Prevents known privilege escalations (e.g., hostPID, hostNetwork, dangerous capabilities) while remaining compatible with most legitimate workloads.
• Restricted — A stricter profile that also requires non-root user execution, read-only root filesystems, and drops all Linux capabilities by default.

For any namespace serving external customers or untrusted teams, applying at minimum the Baseline profile blocks the most common container breakout vectors. Privileged containers — ones that can mount host paths, access raw devices, or run as root with full capabilities — are the primary route from "noisy tenant" to "cluster-wide incident."

Enforce PSS at the namespace level with a label:

apiVersion: v1
kind: Namespace
metadata:
 name: tenant-a
 labels:
   pod-security.kubernetes.io/enforce: baseline
   pod-security.kubernetes.io/enforce-version: latest


Combined with NetworkPolicies, this closes off the two most common data plane attack paths: unauthorized network access and host-level privilege escalation.

Layer 3: Hardware-Level GPU Isolation

Even with tight control and data plane controls, multiple tenants sharing the same physical GPU are still co-located at the silicon level. For many workloads, that's acceptable. For others — regulated ML pipelines, external customer environments, SLA-backed inference services — it isn't. This is where hardware-level GPU isolation comes in.

There's a fundamental trade-off to understand: stronger isolation typically means more predictable (but lower peak) utilization. The right model depends on your trust assumptions, not just your performance requirements.

Analysis of tenant isolation on GPU infrastructure shows the spectrum runs from full GPU passthrough (one workload per card) through hardware partitioning down to software-based sharing. Here's how the three main Kubernetes-native approaches break down.

Method 1: NVIDIA Multi-Instance GPU (MIG)

MIG is NVIDIA's hardware partitioning technology, available on Ampere (A100) and newer architectures including H100. It allows a single physical GPU to be divided into up to seven fully isolated GPU instances, each with its own dedicated:

• Compute engines
• Memory slice (with enforced bandwidth limits)
• L2 cache partition
• PCIe and NVLink fabric paths

The key word is hardware-enforced. A process crash or memory error in one MIG instance does not affect other instances. There is no shared state between them at the hardware level. One H100 running 7 MIG instances gives you 7 independently scheduled, fault-isolated GPU resources — each visible to Kubernetes as a distinct schedulable device via the NVIDIA GPU Operator.

Best for: External customers, regulated workloads, production inference with strict SLAs, any environment where tenants are not fully trusted.

Trade-off: MIG profiles are static and must be configured before workloads run. Changing profiles requires a brief reconfiguration window. Only available on newer, higher-cost GPU SKUs.

Method 2: Time-Slicing

Time-slicing is a software-based sharing approach configured through the NVIDIA device plugin. Multiple containers are exposed to a single GPU and take turns via context switching — the GPU runs one process for a time slice, then switches to the next.

It's easy to configure: you set a replicas count in the device plugin config, and Kubernetes sees that number of "virtual" GPU resources available for scheduling.

version: v1
sharing:
 timeSlicing:
   resources:
   - name: nvidia.com/gpu
     replicas: 4


Best for: Managed Kubernetes environments for development, trusted internal teams, workloads that are GPU-memory-light and don't need dedicated throughput.

Hard limitation: There is no memory isolation and no fault isolation with time-slicing. If one process writes to GPU memory incorrectly or crashes the driver, it affects every other process sharing that GPU. This makes it unsuitable for untrusted tenants or any workload with data compliance requirements.

Method 3: Multi-Process Service (MPS)

CUDA MPS sits between time-slicing and MIG. It allows multiple CUDA processes to share a single GPU concurrently rather than sequentially, reducing context-switch overhead and improving aggregate throughput for CUDA-heavy workloads.

Best for: Trusted internal teams running multiple CUDA inference replicas that don't saturate a full GPU.

Limitation: MPS improves concurrency but still offers limited memory isolation and shared fault risk. A process that triggers a fatal GPU fault can impact all other MPS clients. Not recommended for external or untrusted tenants.

Choosing the Right Method

The following table summarizes each method's trade-offs.

For most GPU cloud operators serving multiple customers, MIG is the right default for production nodes. Time-slicing is a reasonable fit for internal dev clusters where trust is high and cost efficiency is more important than strict isolation.

Operational Best Practices for GPU Clusters with Infrastructure Tenancy

Putting the three layers in place is the foundation. Keeping them working in production requires ongoing operational discipline. Here's the checklist that matters:

Enforce no privileged pods — without exception. Privileged containers are the primary container breakout vector. Apply Pod Security Standards at the Baseline level minimum to every tenant namespace, and automate this as part of your tenant onboarding pipeline so it can't be skipped.

Use minimal, hardened base images. The smaller the container image, the smaller the attack surface. Tenant workloads should be built from distroless or minimal images wherever possible, and your platform should enforce image scanning before admission.

Instrument GPU observability per tenant. NVIDIA's DCGM (Data Center GPU Manager) combined with Prometheus gives you per-device metrics on utilization, memory usage, temperature, and error counts. Label these metrics by tenant namespace or vCluster to enable accurate per-tenant chargeback, capacity planning, and performance debugging. This is one of the most commonly cited operational gaps on shared clusters — teams know their GPU cluster is struggling but can't attribute usage to a specific tenant.

Enforce policy as code. Manual RBAC and NetworkPolicy configuration doesn't scale and drifts over time. Use OPA Gatekeeper or Kyverno to enforce security policies automatically at admission time — ensuring that every tenant namespace created meets your baseline security posture without relying on human process.

Standardize and automate tenant onboarding. Every new tenant should get the same set of resources created in the same way: namespace (or vCluster), RBAC roles and bindings, NetworkPolicy defaults, ResourceQuota, and LimitRange. Manual onboarding creates inconsistency; inconsistency creates security gaps. Codify the onboarding template in Helm, Terraform, or your GitOps tooling.

Monitor GPU topology and scheduling. On multi-GPU nodes connected via NVLink, workloads that span GPU instances experience dramatically different bandwidth depending on how they're placed. Pay attention to node labels related to GPU topology and use affinity rules to ensure high-bandwidth training jobs are scheduled onto nodes where the required GPUs are NVLink-connected. Placing an isolation boundary across an NVLink fabric can silently destroy distributed training throughput.

Plan MIG profiles ahead of workload deployment. MIG profiles are not dynamic — switching profiles requires reconfiguration. Work with your tenants to understand their GPU memory and compute requirements before assigning MIG instances. Consider offering a small set of standard profiles (e.g., 3g.40gb and 1g.10gb on H100) rather than fully custom configurations that are harder to manage at scale.

Conclusion

Sharing GPU infrastructure across teams or customers is how you make expensive hardware economically viable. Getting the isolation right — at the control plane, data plane, and hardware layer — is what makes it safe to do at scale.

If you're building or operating GPU cloud infrastructure for multiple tenants, vCluster's isolation stack is designed for exactly this problem. Or request a demo to discuss your specific architecture.

Frequently Asked Questions

What is GPU tenant isolation?

GPU tenant isolation is the practice of creating secure, performance-guaranteed boundaries between different users or workloads (tenants) sharing the same physical GPU hardware in a cloud or Kubernetes environment. This involves preventing data leakage, side-channel attacks, and performance degradation caused by "noisy neighbors." True isolation requires a multi-layered approach that includes securing the control plane (who can do what), the data plane (how workloads communicate), and the hardware itself (how GPU resources are partitioned).

Why aren't Kubernetes namespaces enough for secure GPU isolation?

Kubernetes namespaces provide logical grouping for resources but are not a true security boundary by default. They do not prevent container breakouts, block network traffic between namespaces, or isolate tenants at the hardware level. While namespaces are a good starting point for organizing workloads and applying RBAC, they share the same underlying cluster API server and kernel. A compromised container can potentially escape its namespace and access host-level resources, including the physical GPU, impacting all other tenants on that node.

What are the three layers of tenant isolation for GPU clusters?

The three essential layers for comprehensive GPU tenant isolation are Control Plane Isolation, Data Plane Isolation, and Hardware-Level GPU Isolation.

• Control Plane Isolation uses tools like RBAC, ResourceQuotas, and virtual clusters (vCluster) to define what each tenant can see and configure.
• Data Plane Isolation uses NetworkPolicies and Pod Security Standards to control communication between workloads and prevent privilege escalation.
• Hardware-Level Isolation uses technologies like NVIDIA MIG to physically partition a single GPU into multiple, fully isolated instances.

How does NVIDIA MIG provide stronger isolation than time-slicing?

NVIDIA Multi-Instance GPU (MIG) provides hardware-enforced isolation, partitioning a single GPU into multiple independent instances, each with its own dedicated memory, compute, and cache. Time-slicing is a software-based approach that offers no memory or fault isolation. With MIG, a crash or memory error in one tenant's instance has no impact on others. In contrast, time-slicing allows multiple processes to share the entire GPU, so a single faulty process can crash the driver or corrupt memory for every other tenant on that GPU, making it unsuitable for untrusted workloads.

When should I use MIG versus time-slicing or MPS?

Use NVIDIA MIG for production environments, external customers, or any workload requiring strict security and performance guarantees. Use time-slicing or MPS for internal development clusters or trusted teams where cost-efficiency is prioritized over strict isolation. The choice depends on your trust model. MIG is the only method that offers true hardware-level fault and memory isolation, making it essential for regulated industries or public clouds with tenant isolation. Time-slicing and MPS are suitable for environments where tenants are trusted and the risk of interference is low.

What is the "noisy neighbor" problem in a GPU cluster?

The "noisy neighbor" problem occurs when one tenant's resource-intensive workload consumes an unfair share of GPU compute, memory, or bandwidth, degrading performance for all other tenants on the same hardware. Without proper controls like resource quotas and hardware partitioning, a single memory-heavy training job can saturate a GPU, causing latency spikes and throughput drops for other tenants' inference or training tasks. This is a common issue in poorly isolated shared clusters.

How does a virtual cluster (vCluster) improve control plane isolation?

A virtual cluster (vCluster) gives each tenant their own dedicated Kubernetes control plane (API server, etcd) that runs within a namespace on the host cluster. This provides much stronger isolation than shared namespaces. Unlike standard namespaces, which share a global API server and CRD registry, vCluster scopes these resources entirely within the virtual cluster. This prevents tenants from interfering with each other's configurations, allows them to install their own operators (like different versions of the NVIDIA GPU Operator) without conflict, and provides a true cluster-admin experience without granting access to the underlying physical infrastructure.

‍