vCluster vs Kamaji vs Namespace Isolation for NVIDIA DGX Kubernetes

Summary

• Isolating tenants on NVIDIA DGX Kubernetes is a critical production challenge, as standard namespaces offer only "soft" isolation insufficient for security or compliance demands.
• This guide compares three approaches across four key dimensions: namespace isolation (weak security, high overhead), Kamaji (isolated control planes, shared data plane), and vCluster (full-stack isolation).
• For production AI workloads, the choice directly impacts security posture and operational efficiency, with vCluster being explicitly named in the NVIDIA DGX SuperPOD reference architecture.
• vCluster Platform provides a complete solution with strong control plane and data plane isolation, fine-grained GPU management, and low Day 2 overhead for building secure AI platforms with tenant isolation at scale.

You've committed to tenant-isolated Kubernetes on NVIDIA DGX. The hardware is racked, the clusters are running, and now you're facing the question that actually determines whether this works in production: how do you isolate tenants?

This isn't academic. Architects in the field are hitting this wall right now. One common scenario: "customer contracts now require proof that data is isolated at the hardware level." Another: "namespaces are just logical separation — if someone compromises the node, they could access other tenants' data." And the uncomfortable truth that sits underneath both of those: "Kubernetes was designed for logical isolation, not security isolation."

This guide compares three leading approaches — namespace isolation, vCluster, and Kamaji — across four dimensions that matter specifically for DGX GPU environments:

1. Isolation Strength — control plane vs. namespace vs. separate clusters
2. GPU Scheduling Granularity & Quota Enforcement per tenant
3. Time-to-Tenant — how fast can a new AI team or customer get a cluster
4. Operational Overhead — Day 2: upgrades, observability, backups

The Multi-Layer GPU Isolation Problem

Before comparing tools, it's worth grounding the discussion in what GPU tenant isolation actually requires. As Red Hat outlines in their designing tenant-isolated GPU infrastructure guide, there are at least four distinct layers of GPU isolation:

• Hardware Isolation: Which tenant owns which physical GPU via PCI device assignment
• Fabric Isolation: Whether GPUs from different tenants can communicate over NVLink or NVSwitch
• Scheduler Isolation: Whether the Kubernetes scheduler respects hardware and fabric boundaries
• Virtualization Isolation: How GPU resources are presented — full GPUs, MIG partitions, or time-sliced shares

Most Kubernetes-native isolation approaches only address one or two of these layers. The architectural decision you make today determines how many you can cover — and which ones become permanent gaps.

Beyond the physical stack, GPU workloads introduce application-level complexity. As detailed in GPU Tenant Isolation in Kubernetes: Strategies, Challenges, and Best Practices, the core challenges for AI/ML tenants include:

• Contention: Noisy neighbors consuming GPU memory and compute cycles, degrading performance for others on the same node
• Isolation risks: Data leaks between workloads sharing the same underlying OS kernel
• Quota management: Kubernetes natively allocates whole GPUs — fine-grained enforcement requires additional tooling
• Visibility & cost attribution: Accurately metering GPU usage per tenant for billing and showback

With that framing in place, let's break down each approach.

Option 1: Namespace Isolation

How it works: You create a namespace per tenant and use ResourceQuota and LimitRange objects to cap CPU, memory, and nvidia.com/gpu requests. Network policies restrict cross-namespace traffic.

This is what the Kubernetes documentation on tenancy calls "soft" tenancy isolation — and the name is honest.

Isolation Strength: Weak

All tenants share the same control plane: one API server, one etcd, one controller-manager. A single misconfigured RBAC rule or a compromised workload can have cluster-wide consequences. Tenants cannot be true cluster-admins. There is no isolation for CRDs or cluster-scoped resources.

Secrets and ConfigMaps for all tenants are commingled in the same etcd — a detail many architects overlook until a contract audit surfaces it.

GPU Scheduling & Quota Enforcement: Coarse

Standard ResourceQuota counts whole GPUs. Enforcement is at the pod level, not the tenant level. A single tenant can exhaust cluster-wide GPU capacity by spinning up many small pods if quotas aren't meticulously managed. NVIDIA MIG can address hardware partitioning but quota enforcement still happens at namespace granularity.

Time-to-Tenant: Fast

kubectl create namespace team-a takes seconds.

Operational Overhead: High (and Hidden)

The platform team becomes the bottleneck for every CRD request, permission escalation, or cluster-scoped change. Delivering adequate isolation without breaking other tenants requires constant RBAC auditing, network policy management, and admission controller tuning. Teams often resort to "dedicated node pools with tainted nodes — not perfect, but better than purely shared infrastructure" — which adds infrastructure complexity without actually solving the isolation problem.

Option 2: vCluster

How it works: vCluster virtualizes Kubernetes itself. Each tenant receives a fully isolated, CNCF-certified Kubernetes cluster running as lightweight pods within the host cluster. Crucially, this isn't just a control plane manager — it provides full tenant cluster environments with their own API server, etcd instance, and cluster-admin access, while syncing only a minimal set of objects (pods, services) to the underlying host for scheduling.

Isolation Strength: Strong to Full-Stack

Control plane isolation is complete. Each vCluster maintains its own etcd, which means tenant secrets, ConfigMaps, and all Kubernetes objects are never commingled with other tenants — directly addressing the audit risk that namespace-based approaches carry.

The data plane isolation is where vCluster decisively separates itself from the alternatives. It offers a full spectrum:

• Shared Nodes: Default mode — similar to Kamaji's data plane model, workloads share underlying nodes
• Private Nodes: Dedicated host nodes are tainted and assigned to specific vClusters, achieving node-level hardware isolation without separate physical clusters
• Dedicated VMs + vNode: For the highest security tier, tenant clusters run on dedicated VMs. vNode (currently in private beta) adds kernel-native workload isolation — seccomp, cgroups, namespaces, and AppArmor per workload — delivering strong security without hypervisor overhead and without sacrificing bare metal GPU performance

This spectrum directly resolves the tradeoff that architects have been struggling with: "performance overhead is significant and we lose Kubernetes features" when using traditional virtualization. vNode eliminates that tax.

GPU Scheduling & Quota Enforcement: Fine-Grained

Quotas are enforced at the tenant cluster boundary — a hard resource cap that isn't shared with any other tenant. vCluster integrates with NVIDIA MIG so specific MIG partitions can be scoped to specific tenant clusters, providing true hardware-partitioned GPU access. Pre-validated Certified Stacks for Run:AI, Ray, and Jupyter mean the AI platform layer above the cluster is already tested and compatible.

vCluster Labs is also the only vendor in this space that delivers raw hardware → Kubernetes distribution → tenant clusters → workload isolation in a single integrated stack, through vMetal for zero-touch bare metal provisioning and vCluster Platform for tenant orchestration.

Time-to-Tenant: Seconds

New tenant clusters are created in seconds — comparable to namespace creation, but with full control plane isolation. This is a material advantage for service providers onboarding new AI teams or customers.

Operational Overhead: Low

vCluster Platform is engineered for Day 2 at scale: central fleet management UI, CLI and API, SSO, cluster templates, auto-sleep for cost reduction, and built-in observability, backup, and disaster recovery. Managing hundreds of tenant clusters doesn't require proportional headcount growth. Boost Run, for example, launched in under 45 days with zero new platform engineering hires.

At production scale, vCluster powers 100K+ GPU nodes and has created 40M+ tenant clusters across 50+ GPU cloud and Fortune 500 customers — and is explicitly named in the NVIDIA DGX SuperPOD reference architecture.

Option 3: Kamaji

How it works: Kamaji is an open-source project from Clastix that runs tenant Kubernetes control planes as pods inside a central admin cluster. Each tenant gets their own API server, controller-manager, and scheduler. It typically uses a shared etcd cluster to keep resource consumption low. It's CNCF-aligned, GitOps-native, and designed for building managed Kubernetes services.

Isolation Strength: Medium

Control plane isolation is real and meaningful. Each tenant operates their own API server, which eliminates the noisy neighbor problem at the API level and allows true CRD and RBAC isolation per tenant. This is a genuine step up from namespaces.

The data plane, however, is a different story. Tenant workloads still run as standard pods on shared worker nodes, subject to the same kernel-level risks that make namespace isolation inadequate for regulated workloads. Robust network policy enforcement and runtime security tooling are still required — and are your responsibility to configure and maintain.

GPU Scheduling & Quota Enforcement: Moderate

Having a per-tenant control plane means you can set cluster-scoped defaults and policies within each tenant environment. However, the scheduling of workloads onto physical GPUs still happens on shared worker nodes via shared kubelets. Kamaji doesn't inherently solve GPU fractioning, advanced gang scheduling, or NVIDIA MIG assignment without additional tooling layered on top.

Time-to-Tenant: Minutes

Creating a TenantControlPlane custom resource is faster than provisioning a physical cluster, but slower than a namespace. You wait for control plane pods to spin up and for kubeconfig to be issued.

Operational Overhead: Moderate

Kamaji simplifies control plane lifecycle management and integrates naturally with GitOps workflows via Flux or Argo CD. The platform team still owns the admin cluster, the shared etcd cluster, and all underlying worker nodes — including upgrades, security patching, and scaling. Backup strategies, tenant observability pipelines, and DR runbooks need to be built out separately. Kamaji is a solid foundation for a capable platform team, but it is explicitly a component, not a complete platform.

At-a-Glance Comparison

When to Choose Each

Choose Namespace Isolation if you are running an early-stage internal R&D environment with a high trust model between teams and no strict compliance or isolation contractual requirements. Treat it as a starting point, not a destination — plan to migrate before you onboard external users or regulated workloads.

Choose vCluster if you are building production-grade AI infrastructure on NVIDIA DGX — whether for internal tenants or paying customers. The decision becomes straightforward when any of the following are true:

• Customer contracts require demonstrable tenant isolation — vCluster's per-tenant etcd, isolated control planes, and the vNode isolation spectrum provide a clear, auditable answer
• You need to onboard new AI teams in seconds, not hours, with a self-service experience comparable to EKS or GKE
• You are running or plan to run at scale — 100K+ GPU nodes and 40M+ tenant clusters is a data point that namespaces and Kamaji cannot match in production
• Your platform team is small — the built-in Day 2 operations tooling means you aren't manually wiring together observability, backups, and upgrade pipelines for every new tenant cluster
• You need the full isolation spectrum — from shared nodes to dedicated hardware with kernel-native vNode protection (vNode is currently in private beta), without choosing between security and GPU performance

Choose Kamaji if you are building a managed Kubernetes platform from open-source components and your team prioritizes GitOps-native control plane management. Kamaji is a strong foundation for experienced platform engineers who are prepared to own the surrounding operational complexity: worker node upgrades, network policy enforcement, custom observability pipelines, and etcd backup strategies. It shines when you want control plane separation and have the team to build the rest of the stack around it.

Conclusion

For infrastructure architects building on NVIDIA DGX Kubernetes, the isolation choice is a foundational decision with long-term consequences for security posture, operational efficiency, and the ability to satisfy customer contracts.

Namespace isolation is a pragmatic starting point — fast to set up, but brittle at scale and inadequate for any environment where tenants need guarantees stronger than "we pinky-promise no one will misconfigure RBAC." Kamaji is a meaningful improvement for teams that want GitOps-native control plane management and are equipped to build the operational layer around it.

For serious AI/ML infrastructure, vCluster is the only option that delivers the complete picture: control plane isolation, flexible data plane isolation through vNode, fine-grained GPU quota enforcement, seconds-to-tenant onboarding, and production-proven Day 2 tooling — all in a stack that is explicitly validated for NVIDIA DGX SuperPOD environments.

Frequently Asked Questions

Why aren't Kubernetes namespaces enough for secure multi-tenancy on DGX?

Kubernetes namespaces provide only "soft" logical isolation, not true security boundaries. All tenants share the same control plane (API server, etcd), kernel, and cluster-scoped resources, creating significant security and compliance risks. With namespaces, a single compromised workload or misconfigured RBAC rule can impact the entire cluster. Secrets for all tenants are stored in the same etcd database, which is often a non-starter for contract audits. Furthermore, tenants cannot have cluster-admin privileges or manage their own CRDs, making the platform team a constant bottleneck.

What is the primary difference between vCluster and Kamaji for tenant isolation?

The primary difference is in data plane isolation and operational completeness. vCluster provides both control plane and strong data plane isolation options (like vNode), while Kamaji focuses on control plane isolation, leaving the shared data plane and operational tooling for the platform team to manage. While both offer isolated control planes, Kamaji workloads run on a shared kernel by default. vCluster offers a spectrum of data plane isolation, from shared nodes to dedicated nodes and even kernel-native isolation with vNode. Additionally, vCluster Platform is a complete solution with built-in fleet management, observability, and backup tooling, whereas Kamaji is a component that requires you to build these surrounding systems yourself.

How does vCluster manage GPU quotas and scheduling for different tenants?

vCluster enforces hard resource quotas at the virtual cluster boundary, providing much stronger enforcement than namespace-level quotas. It integrates directly with NVIDIA MIG to assign specific GPU partitions to specific tenants, ensuring true hardware-level resource allocation. Unlike standard Kubernetes quotas which apply to namespaces and can be difficult to manage, vCluster applies resource limits to the entire tenant cluster. This guarantees a tenant cannot exceed their allocated GPU budget and prevents resource contention and noisy neighbor problems.

What is vNode and how does it enhance security for GPU workloads?

vNode is a technology from vCluster Labs that provides kernel-native workload isolation without the overhead of traditional hypervisors. It secures the data plane by applying seccomp, cgroups, and AppArmor profiles per workload, preventing container escapes and lateral movement. For GPU workloads, this is critical because it provides strong isolation without sacrificing the bare-metal performance needed for AI/ML. It closes the final isolation gap that exists in shared-kernel multi-tenancy models.

How fast can a new tenant be onboarded with vCluster?

A new, fully isolated tenant cluster can be created in seconds with vCluster. This speed is comparable to creating a simple namespace but delivers a complete, secure Kubernetes environment with its own API server and etcd. The ability to provision a full-featured, CNCF-certified Kubernetes cluster on-demand accelerates development cycles for internal teams and improves the self-service experience for customers of AI service providers.

Why is vCluster part of the NVIDIA DGX SuperPOD reference architecture?

vCluster is named in the NVIDIA DGX SuperPOD reference architecture because it provides a proven, production-grade solution for multi-tenant isolation at scale. It effectively addresses the security, management, and performance requirements of enterprise AI workloads running on DGX systems. The reference architecture validates that vCluster meets NVIDIA's standards for building secure and scalable AI platforms, making it the recommended approach for maximizing the ROI on DGX investments by safely serving multiple teams or customers.

If you're ready to go from GPU racks to a secure AI platform with tenant isolation without building the plumbing yourself, vCluster Platform, vMetal, and vNode (currently in private beta) are the integrated path to get there. See how vCluster can secure and scale your DGX environment—schedule a demo today.

‍