Bare Metal Kubernetes for GPU Infrastructure: Tenant Isolation and Provisioning

Summary

• Standard Kubernetes namespaces are inadequate for GPU workloads requiring tenant isolation, failing to prevent resource contention, security risks, or conflicts over shared cluster resources like CRDs.
• The solution is a spectrum of isolation models that trade GPU utilization for security, from sharing nodes with isolated control planes to providing fully private, dedicated nodes per tenant.
• Giving each tenant an isolated control plane is key, as it allows them to install their own CRDs and run custom GPU schedulers without impacting others.
• The vCluster platform implements these isolation models, enabling GPU cloud providers to efficiently manage bare metal infrastructure for isolated tenant environments and offer tiered services.

Anyone who has tried to run Kubernetes on bare metal for GPU-intensive workloads knows the common refrains: it's complex, the tooling is confusing, and the fear of "reinventing the wheel" for operational tasks is real.

And yet, for AI workloads that demand peak throughput, low latency, and full hardware exclusivity, bare metal remains the right call. Managed GPU cloud services add cost and abstraction layers. Virtualized environments introduce overhead that matters when you're training LLMs on H100s. Bare metal gives you the full CUDA stack, full NVLink bandwidth, and full control over networking and storage.

The challenge isn't whether to run on bare metal. It's how do you efficiently share that expensive hardware across multiple teams or customers without compromising security, performance, or operational sanity?

Imagine you've just racked a cluster with 50 GPU nodes. Those nodes represent significant capital expenditure. You can't afford to hand one cluster to each tenant — that's 50 separate control planes to manage, 50 clusters to patch, 50 times the operational burden. But you also can't throw everyone into one big namespace soup and call it "tenant isolation."

Tenant isolation is the key, and while various open source projects can help, they often only solve part of the problem. This post will explore a complete, production-grade solution using the vCluster platform.

We'll walk through why standard Kubernetes tenant isolation falls short for GPU workloads, what a complete provisioning-to-isolation stack looks like, and the three isolation models that map to the full spectrum of tenant requirements.

Why Namespace-Based Tenant Isolation Fails for GPU Workloads

Before jumping to solutions, it's worth understanding exactly why the default Kubernetes answer — namespaces — is inadequate for production GPU infrastructure.

Namespaces are a grouping and scoping mechanism. They were never designed to be a security boundary. For internal dev environments with trusted users, they're fine. For a production AI cloud where tenants are external customers paying for isolated GPU access, they're a liability.

Limited Tenant Isolation

The most immediate problem is the noisy neighbor effect. When multiple tenants share the same physical nodes, one tenant running a memory-intensive training job can starve other pods on the same machine of GPU memory, CPU cycles, and network bandwidth. Kubernetes ResourceQuota and LimitRange objects help at the pod level, but they don't prevent a greedy workload from monopolizing NUMA nodes, PCIe bandwidth, or GPU memory fragmentation.

There's also a security dimension. A container breakout in one namespace doesn't stop at the namespace boundary — it reaches the host kernel. On a shared node, that means a compromised tenant workload could potentially access data from other tenants running on the same machine. Production-grade tenant isolation requires addressing four distinct layers: hardware, fabric, scheduler, and virtualization. Namespaces alone address none of them at a meaningful depth.

Restricted Tenant Autonomy

In a shared cluster, cluster-scoped resources like CRDs (Custom Resource Definitions) are global. If tenant-a wants to install Kubeflow and tenant-b wants to install KServe, they're competing for the same CRD namespace. A CRD version conflict between the two operators can break the entire cluster.

This forces the platform team to become a bottleneck. Every request to install a new operator, every CRD change, every cluster-level RBAC policy has to go through central approval. This kills developer velocity and creates exactly the kind of operational overhead that bare metal setups are already prone to.

Complex Compliance Requirements

Regulatory frameworks like SOC 2, HIPAA, and PCI-DSS require strong evidence that tenant data is isolated and cannot be accessed by unauthorized parties. A shared-namespace architecture makes this nearly impossible to certify. Without control plane separation, tenants share the same API server — meaning a misconfigured RBAC rule could expose one tenant's workload metadata to another.

Operational Complexity at Scale

Managing ResourceQuotas, NetworkPolicies, LimitRanges, and RoleBindings for dozens or hundreds of namespaces in a single cluster is a full-time job. It scales poorly. Each new tenant means more YAML, more potential for misconfiguration, and more blast radius when something goes wrong.

The Scheduler Problem: KAI Scheduler and Cluster Boundaries

The KAI Scheduler is a Kubernetes scheduler specifically designed to optimize GPU resource allocation for AI/ML workloads. It understands concepts like GPU fractioning, gang scheduling for distributed training jobs, and queue-based fairness across teams.

But it has a hard constraint: it operates within a single cluster. Its scheduling decisions, queue configurations, and resource policies are scoped to the cluster it runs in. If you need tenant-specific scheduling policies — for example, tenant-a gets priority scheduling for inference while tenant-b gets bulk-batch priority for training — you're back to either one cluster per tenant or complicated workarounds.

This is the core tension: the tools that make GPU scheduling smart are designed for clusters, not namespaces. And running one cluster per tenant on bare metal hardware is operationally unsustainable.

The Full-Stack Solution: From Bare Metal to Isolated Tenant Clusters

So, what's the alternative that blends the best of both?

A note on bare metal provisioning: Before you can run tenant clusters on bare metal, you need the hardware itself registered, networked, and running Kubernetes. vMetal handles exactly this — zero-touch PXE provisioning, OS installation, GPU node registration, and network automation (VLANs, VXLANs via Netris). vCluster Standalone then runs as a binary directly on those bare metal nodes, eliminating the need for a separate base Kubernetes distribution. Together, vMetal + vCluster deliver the complete path from GPU rack to isolated tenant clusters.

This matters because the common pain points of bare metal Kubernetes — the high barrier of entry for PXE boot setups, the confusion around DHCP, DNS, and network automation, the "reinventing the wheel" feeling of building operational runbooks from scratch — are all provisioning-layer problems that exist before you even get to tenant isolation. vMetal is designed to absorb that complexity, reducing what traditionally requires a $1M+ investment and 6–10 specialized engineers down to an automated, repeatable pipeline. It's also NVIDIA-endorsed for building Kubernetes-driven GPU infrastructure.

Once the physical cluster is up and nodes are registered, vCluster takes over for tenant isolation. And the key insight is that vCluster doesn't force you into a single isolation model — it gives you a spectrum.

Let's try to understand the three modes that are supported by vCluster, ranging from highest GPU utilization to maximum security isolation.

Isolation Mode 1: Shared Nodes with Control Plane Isolation

The first and most resource-efficient model gives each tenant their own fully isolated Kubernetes control plane — dedicated API server, controller manager, etcd — while the underlying worker nodes (and their GPUs) are shared across tenants.

How It Works

With vCluster, each tenant cluster runs its control plane as pods within a namespace on the host cluster. A syncer component translates workloads from the tenant cluster's API server into real pods on the host cluster, rewriting pod specs to maintain proper scoping and isolation. From the tenant's perspective, they have a full Kubernetes cluster. From the platform team's perspective, it's just a set of pods running in a namespace.

Creating an isolated environment for a new tenant is as simple as:

# Create a new tenant cluster for tenant-a
vcluster create tenant-a-vcluster --namespace tenant-a


That's it. The tenant gets a full Kubernetes API endpoint, their own kubeconfig, and complete autonomy within their tenant cluster.

What This Solves

Control plane separation immediately eliminates a major class of namespace-based problems. Tenants can now install their own CRDs without conflicting with other tenants. A team using Kubeflow and a team using KServe no longer fight over CRD versions. Each tenant can run a different Kubernetes version if needed. RBAC within the tenant cluster is the tenant's responsibility — the platform team is no longer a bottleneck.

Critically, each tenant can deploy their own instance of the KAI Scheduler inside their tenant cluster, with scheduling policies tuned to their specific workload mix. Problem solved.

GPU utilization remains high because the actual scheduling of GPU pods still happens on the shared host cluster nodes. You're not wasting hardware by reserving entire nodes per tenant.

Addressing Kernel-Level Isolation with vNode

This mode does have one acknowledged limitation: if two tenants' pods land on the same physical node, they still share the host kernel. For most enterprise use cases, the control plane separation is sufficient. But for high-security environments, a container breakout is a real threat model.

The use of vCluster and vNode together provides a strong solution for secure tenant isolation: workloads run inside isolated tenant clusters on virtual nodes, so even if a container breakout occurs, the attacker reaches the virtual node boundary — not the actual bare metal host.

vNode provides kernel-native workload isolation without the overhead of full VMs, making it the right choice when you need node-level security guarantees without sacrificing the performance characteristics of bare metal.

Isolation Mode 2: Dedicated Nodes with Control Plane Isolation

The second model builds on the first by adding hardware-level separation. Each tenant's tenant cluster still has its own isolated control plane, but now its workloads are also constrained to a dedicated pool of physical nodes — and the GPUs on those nodes.

How It Works

This is achieved using standard Kubernetes primitives applied at the host cluster level. Physical GPU nodes are labeled with a tenant identifier:

kubectl label node gpu-node-07 tenant=tenant-a
kubectl label node gpu-node-08 tenant=tenant-a
kubectl taint node gpu-node-07 tenant=tenant-a:NoSchedule
kubectl taint node gpu-node-08 tenant=tenant-a:NoSchedule


The tenant cluster for tenant-a is then configured with a nodeSelector and matching toleration so that all synced pods land exclusively on those labeled, tainted nodes. No other tenant's workloads can schedule there.

What This Solves

The noisy neighbor problem disappears entirely at the hardware level. When tenant-a is running a 4-GPU distributed training job on A100s, tenant-b's inference pods on the next node are completely unaffected. There is no shared PCIe bus contention, no NUMA interference, no GPU memory fragmentation from cross-tenant allocation.

This model is particularly valuable for:

• Predictable SLAs: Tenants who are paying for guaranteed GPU throughput need hardware exclusivity to deliver on those commitments.
• Data sovereignty requirements: Some regulated workloads require that customer data never touches hardware that another customer has used without a full sanitization cycle. Dedicated nodes make it straightforward to demonstrate this in a compliance audit.
• Sensitive AI training jobs: Foundation model training runs that take days or weeks cannot afford unexpected slowdowns caused by resource contention from neighboring tenants.

Flexible Resource Management

One of the operational advantages of this model over the "cluster-per-tenant" approach is that the platform team retains centralized control over node assignment. Need to scale tenant-b's GPU pool from 4 nodes to 8 for a one-week training sprint? Remove the taint from four additional nodes, re-label them, and update the vCluster's node affinity configuration. No new cluster to provision, no new control plane to manage.

This matches the mental model that experienced platform teams prefer: a single pane of glass for node lifecycle management, with vCluster handling the tenant isolation boundary above it.

Isolation Mode 3: Dedicated Clusters with Private Nodes

The third model represents the highest degree of isolation available: each tenant receives a fully dedicated tenant cluster with its own private nodes. This is a single-tenant environment in every meaningful sense — private control plane, private worker nodes, zero shared infrastructure with any other tenant.

How It Works

Unlike the previous two modes where host cluster nodes are shared or partitioned via labels and taints, Private Nodes mode provisions genuinely isolated node pools per tenant. The tenant's cluster has exclusive ownership of its nodes — they don't appear as schedulable resources in the host cluster for any other workload. The result is the logical equivalent of a dedicated physical cluster, but managed centrally through the vCluster platform.

This is a live, GA feature in vCluster — you can see it listed as available under "Choose the model that fits your infrastructure" on vcluster.com.

What This Solves

Private Nodes mode is the answer to zero-trust security requirements. There is no shared kernel, no shared hardware, and no shared API surface between tenants. A complete tenant environment can be spun up through the vCluster platform without requiring the platform team to provision a separate physical cluster, manage a separate etcd, or maintain a separate upgrade lifecycle.

This model is ideal for:

• Enterprise customers with strict compliance mandates: SOC 2 Type II, HIPAA, and FedRAMP frameworks become dramatically easier to satisfy when you can point to fully private infrastructure per customer.
• Government and defense workloads: Zero-tolerance for shared hardware between classification levels or organizational boundaries.
• Financial services: Trading algorithms, risk models, and customer data pipelines frequently require contractual guarantees of hardware exclusivity that shared-node models cannot provide.
• AI cloud operators offering premium tiers: If you're selling GPU cloud access and need a product tier that justifies a premium price point, Private Nodes gives you a genuinely differentiated offering — not marketing language, but actual hardware isolation.

Operational Efficiency Preserved

The key advantage over the "traditional cluster per tenant" approach is that vCluster Platform provides centralized lifecycle management. You're not managing 20 separate Kubernetes clusters — you're managing one physical infrastructure layer, with vCluster handling the tenant boundary above it. Upgrades, networking policies, observability, and cost allocation all flow through a single platform.

This is what makes the economics work. The operational cost of serving 50 tenants with dedicated environments through vCluster is far closer to serving 50 tenants in a shared cluster than it is to managing 50 independent bare metal Kubernetes deployments.

Choosing the Right Isolation Mode

The three modes aren't mutually exclusive — most production GPU clouds run all three simultaneously, matching isolation level to tenant type.

A practical deployment often looks like:

• Tier 1 (standard): Shared nodes with control plane isolation. Tenants get full Kubernetes autonomy, can install their own CRDs, run their own KAI Scheduler configuration, manage their own RBAC. GPU utilization is maximized across the pool.
• Tier 2 (performance): Dedicated nodes. Tenants who are running long-duration training jobs or require predictable throughput guarantees are assigned a fixed GPU node pool. No noisy neighbors.
• Tier 3 (enterprise/private): Private nodes. Tenants with compliance requirements, contractual hardware exclusivity clauses, or premium SLA tiers get fully isolated environments managed through the same vCluster Platform interface.

This tiered model is exactly what GPU cloud operators like CoreWeave and Nscale run in production. It makes commercial sense: you can price each tier accordingly, offer upsell paths as tenant requirements evolve, and avoid the operational explosion of managing separate physical clusters for every high-value customer.

The Architecture in Practice

To make this concrete, here's what the full stack looks like end-to-end for a GPU cloud operator:

Layer 1 — Physical Infrastructure (vMetal)

GPU servers are racked and connected. vMetal handles zero-touch PXE boot, OS installation, NVIDIA driver and GPU Operator deployment, and network automation via Netris (VLANs for tenant separation at Layer 2, VXLANs for overlay networking). Nodes register automatically into the host Kubernetes cluster. No manual cluster initialization, no per-node cloud-init scripting, no custom DHCP/DNS configuration required. The high barrier of entry consistently flagged around bare metal Kubernetes provisioning is handled at this layer.

Layer 2 — Host Cluster (Kubernetes)

A standard Kubernetes host cluster runs on the bare metal nodes provisioned by vMetal. This cluster is never directly exposed to tenants. It's the substrate. Platform engineers manage it; tenants don't interact with it. The NVIDIA GPU Operator runs here, exposing GPU resources as schedulable Kubernetes resources.

Layer 3 — Tenant Isolation (vCluster)

vCluster runs on top of the host cluster. Each tenant's tenant cluster runs its control plane as pods in a dedicated namespace. The syncer handles workload translation. Depending on the tenant's tier, their pods land on shared nodes, dedicated labeled nodes, or fully private node pools. Each tenant gets their own kubeconfig, their own API server endpoint, their own CRD namespace. The KAI Scheduler can run per-tenant-cluster for tenant-specific GPU scheduling policies.

Layer 4 — Kernel-Level Isolation (vNode, optional)

For tenants who require it, vNode adds a virtual node layer between the tenant's pods and the host kernel. Container breakouts reach the virtual node boundary, not the bare metal host. This is the equivalent of hypervisor-level isolation without the VM overhead.

This four-layer architecture addresses all of the key isolation dimensions necessary for production-grade GPU infrastructure: hardware isolation (dedicated nodes / private nodes), fabric isolation (VLAN/VXLAN via vMetal/Netris), scheduler isolation (per-tenant KAI Scheduler in tenant clusters), and virtualization isolation (vNode).

Conclusion

Running Kubernetes on bare metal GPU infrastructure is the right choice for AI workloads that demand maximum performance, data control, and hardware exclusivity. The challenge is tenant isolation at scale: bare metal is expensive, and a physical cluster per tenant is operationally unsustainable.

The three modes covered in this post — shared nodes, dedicated nodes, and private nodes with isolated control planes — map to a spectrum of isolation requirements. vCluster delivers the full range: control plane isolation as the baseline, dedicated GPU nodes for tenants that need hardware separation, and vNode for kernel-native workload isolation without VM overhead.

GPU cloud operators including CoreWeave and Nscale run this architecture in production. vMetal handles bare metal provisioning from rack to Kubernetes; vCluster handles tenant isolation from there. If you're building AI cloud infrastructure on bare metal, see how the full stack works →

Schedule a demo to see how vCluster can streamline your AI cloud operations.

Frequently Asked Questions

What is vCluster and how does it help with GPU workloads on Kubernetes?

vCluster is a platform that creates lightweight, tenant Kubernetes clusters inside a single namespace of a host cluster. For GPU workloads, it provides strong tenant isolation by giving each tenant their own dedicated control plane. This allows tenants to manage their own CRDs, scheduling policies (like the KAI Scheduler), and Kubernetes versions without interfering with others, enabling efficient and secure sharing of expensive bare metal GPU hardware.

Why is namespace-based tenant isolation not enough for GPU workloads?

Standard namespace-based tenancy fails for GPU workloads because it lacks strong security and resource isolation. Namespaces do not prevent "noisy neighbor" problems, where one tenant can monopolize GPU memory or PCIe bandwidth, nor do they stop a container breakout from reaching the host kernel. Additionally, all tenants share one API server, preventing them from installing their own CRDs or custom schedulers without creating cluster-wide version conflicts.

How does vCluster provide different levels of tenant isolation?

vCluster offers a spectrum of three main isolation models to match different tenant requirements, ranging from shared nodes for maximum efficiency to fully private nodes for maximum security and compliance.

1. Shared Nodes: Tenants get isolated control planes but share physical worker nodes and GPUs, maximizing hardware utilization.
2. Dedicated Nodes: Tenants get isolated control planes and a dedicated pool of physical nodes, eliminating hardware-level "noisy neighbor" problems.
3. Private Nodes: Each tenant receives a fully dedicated cluster with its own private worker nodes, offering the highest degree of isolation for compliance or zero-trust security.

When should I choose dedicated nodes over shared nodes for my tenants?

You should choose the dedicated nodes model when a tenant requires guaranteed performance, predictable SLAs, or has data sovereignty requirements. This model is ideal for production AI training jobs or latency-sensitive inference services that cannot tolerate resource contention from other tenants. By assigning a tenant to a dedicated pool of nodes, you isolate them at the hardware level, preventing any contention for GPU memory, NVLink bandwidth, or PCIe resources.

Can I run different Kubernetes versions or schedulers for different tenants?

Yes, absolutely. Because each tenant gets their own isolated control plane with vCluster, they have complete autonomy to run different Kubernetes versions or custom configurations. For example, one tenant can run a vCluster with Kubernetes 1.28 and a custom KAI Scheduler, while another on the same host runs Kubernetes 1.29 with a default scheduler, all without conflict.

What is the role of vMetal in this architecture?

vMetal is a bare metal provisioning tool that automates the initial setup of the physical GPU infrastructure. It handles the "Day 0" tasks of PXE booting servers, installing the OS, configuring networking (VLANs/VXLANs), and deploying a base Kubernetes distribution. vCluster then runs on top of the host cluster provisioned by vMetal to provide the tenant isolation layer.

What is vNode and do I always need it?

vNode is an optional component that provides strong kernel-level workload isolation, acting like a lightweight hypervisor without the performance overhead of a full VM. You don't always need it, but it's essential for high-security environments. If a container breakout occurs in a pod running with vNode, the breach is contained within the virtual node boundary and cannot compromise the underlying bare metal host.

VCLUSTER LABS

‍