How to Support Slurm and Kubernetes on Your AI Cloud

Summary

• Standardizing your AI cloud on one scheduler means turning away tenants who need the other one. Some workloads need Slurm's deterministic batch scheduling. Others need native Kubernetes for inference and cloud-native tooling. Both are legitimate, and both show up in your sales pipeline.
• Running both schedulers on the same platform is architecturally hard with conventional Kubernetes. Namespace isolation can't give each tenant their own Slurm or Kubernetes control plane without creating a single point of failure for every customer.
• The solution is per-tenant control plane isolation. The vCluster Platform gives each tenant their own fully isolated Kubernetes cluster, with Private Nodes as the production default, so they can run Slurm via Slinky, native Kubernetes, or Run:AI, all managed from one control plane. Production-proven at 100,000+ GPU nodes.

If you run an AI cloud, you have already had this conversation. A prospect asks for a managed Slurm environment for their distributed training jobs. The next one wants native Kubernetes with all the cloud-native tooling their MLOps team already knows. A third wants Run:AI for GPU fractionalization.

If your platform only supports one scheduler, you turn away the other two. Build separate infrastructure for each, and your operational overhead scales with every new customer.

The slurm vs kubernetes search results will tell you to evaluate your workload profile and pick one. That advice makes sense if you are a single team running your own GPU cluster. It falls apart the moment you are building a platform for multiple tenants, each with their own workload requirements, compliance needs, and tooling preferences.

The real question for AI cloud builders is not which scheduler to pick. It is how to support both Slurm and Kubernetes on the same platform without multiplying your operational burden.

Why One Scheduler Is Not Enough for an AI Cloud

Slurm: The HPC Workhorse Under Pressure

Slurm earned its reputation honestly. Born in high-performance computing, it delivers deterministic scheduling and tight hardware control that cloud-native orchestrators still struggle to match for pure batch throughput. For large-scale distributed training jobs with strict performance requirements, it remains the gold standard in many production environments.

But Slurm was built for a single tenant on dedicated hardware. Push it toward an AI cloud model serving multiple enterprise customers on shared infrastructure, and two structural problems surface.

The central controller becomes a chokepoint. At high job submission rates, typical for an AI cloud serving multiple tenants, Slurm's architecture funnels everything through a single slurmctld daemon. The volume of state changes at this scale creates scheduling delays and degrades overall throughput. NVIDIA has invested heavily in hybrid Slurm-on-Kubernetes architectures — their Slinky operator runs Slurm as pods inside Kubernetes, validated at over 8,000 GPUs in production, which points to where the industry is heading for cloud-scale GPU orchestration.

Tenant isolation was never part of the design. Slurm's resource allocation model assumes a trusted user base on dedicated hardware. But in an AI cloud, every enterprise tenant gets their own dedicated machines — provisioned from bare metal and allocated directly to whatever kind of cluster they need, whether that's Kubernetes or Slurm. Slurm wasn't built to operate in a world where the platform operator, not the end user, controls which nodes go to which tenant. GPUs get double-allocated. Users bypass restrictions using CUDA_VISIBLE_DEVICES, whether intentionally or by accident. Operators end up manually enforcing isolation through cgroup configurations that were never meant to carry this responsibility.

Kubernetes: The Cloud-Native Orchestrator Overwhelmed

Kubernetes brings genuine advantages to AI infrastructure: elasticity, a rich ecosystem, and the ability to manage training and inference workloads within a single platform. For teams building AI clouds, its declarative model and extensive tooling are hard to walk away from.

But the default Kubernetes scheduler was not designed for gang-scheduled, GPU-dense distributed training jobs. When you coordinate hundreds of multi-node pods that must launch simultaneously or not at all, scheduler latency compounds. Cluster utilization drops. Time between job submission and actual compute stretches out.

The tenant isolation story is equally troubled. Kubernetes namespaces provide logical separation, but they share the same API server, the same etcd, and the same control plane. A configuration error, a runaway workload, or a noisy-neighbor problem can cascade across every tenant on the cluster. Ensuring proper isolation while sharing GPU resources is genuinely difficult, especially once you need to support per-tenant CRDs, RBAC policies, and custom tooling.

The Hybrid Answer: Running Slurm on Kubernetes with Slinky

A growing number of teams are refusing to choose between Slurm and Kubernetes. Instead, they run Slurm on top of Kubernetes. The key enabler is Slinky, an open-source project that deploys and manages a complete Slurm cluster as pods inside Kubernetes.

The architecture has two core components:

• slurm-operator: Manages the full Slurm control plane (slurmctld, slurmd daemons) as Kubernetes-native resources
• slurm-bridge: Allows Slurm to schedule work as a first-class Kubernetes scheduler, with GPU visibility managed through the NVIDIA GPU Operator

According to NVIDIA's testing, this architecture scales beyond 8,000 GPUs while maintaining performance parity with bare-metal Slurm, a meaningful data point for anyone sizing production AI infrastructure.

The operational benefits are real:

• High availability for free: Instead of Slurm's complex HA configuration, Kubernetes restarts failed controller pods automatically
• Unified observability: Prometheus metrics and Grafana dashboards across the entire stack, so you monitor Slurm and Kubernetes from the same tooling instead of maintaining separate monitoring stacks
• Automated driver management: The NVIDIA GPU Operator handles driver installation fleet-wide, no more per-node configuration rituals

The Problem Slinky Doesn't Solve: The Control Plane at Hundreds of Tenants

Here's where the hybrid approach runs into its own ceiling.

Slinky resolves the scheduler-level tension between Slurm and Kubernetes. But it does not address what happens when you run dozens or hundreds of enterprise tenants on the same underlying cluster. Each tenant may want their own Slurm-on-Kubernetes instance. Each has different CRD requirements, different RBAC policies, different compliance needs. Some are running LLM training at scale. Others are running inference. Some need isolation guarantees that are contractual, not just operational preferences.

A single monolithic Kubernetes cluster cannot serve all of this simultaneously without becoming a liability:

• The API server becomes a bottleneck when thousands of nodes and hundreds of isolated tenant workloads are all routing requests through the same control plane
• The blast radius is total: a control plane failure, a misconfigured webhook, or an aggressive noisy neighbor can take down every customer at once
• RBAC and policy management at this scale is error-prone by design. You are managing hundreds of distinct permission contexts in a system that was never intended for that level of tenant isolation

This is the structural problem that the slurm vs kubernetes framing obscures entirely. The bottleneck is not the scheduler. It is the centralized control plane architecture that both systems share when deployed conventionally.

How to Support Both Schedulers: Per-Tenant Control Plane Isolation

The core architectural problem is not that Slurm and Kubernetes are hard to run together. Slinky already solves that. It deploys Slurm as pods inside Kubernetes and has been validated by NVIDIA at 8,000+ GPU scale.

The architectural problem is that conventional Kubernetes gives every tenant namespace isolation on a single shared control plane. That shared control plane becomes the single point of failure for every customer. And it forces every tenant onto the same scheduler stack. One Slurm configuration. One set of CRDs. One RBAC model. That does not work when Tenant A needs Slurm for HPC batch jobs and Tenant B needs native Kubernetes with Argo CD and custom operators.

vCluster takes a different approach. Instead of sharing one control plane across all tenants, vCluster virtualizes the control plane itself, giving each tenant their own fully isolated Kubernetes cluster. Each tenant gets their own API server, their own etcd, their own RBAC, and their own CRD space. It is the equivalent of giving every customer their own EKS cluster, without the cost of provisioning separate physical infrastructure.

This matters for scheduler choice because a tenant with their own control plane can run whatever scheduler they need inside it. One tenant deploys Slurm via Slinky for deterministic batch scheduling. Another runs native Kubernetes with Volcano for gang-scheduled distributed training. A third runs Run:AI for GPU fractionalization and quota management. Each tenant's scheduler stack is isolated from the others. A Slurm controller failure in Tenant A's cluster does not touch Tenant B's Kubernetes workloads.

Private Nodes: Hardware-Level Isolation Per Tenant

For production deployments, vCluster Platform's default is Private Nodes: dedicated worker nodes joined directly into each tenant cluster over encrypted WireGuard VPN, with per-tenant CNI and storage. This delivers hardware-level tenant isolation that approaches a dedicated physical cluster. No cross-tenant scheduling. No shared kernel. No noisy-neighbor GPU contention. Shared nodes remain available for dev, test, CI/CD, and trusted-team environments where density matters more than physical separation.

One Platform, Not Many

Spinning up a new tenant cluster takes seconds and consumes minimal overhead. The platform operator manages the entire fleet from one control plane: observability, updates, backups, compliance, and config management across every tenant. Tenants get the self-service experience they expect (their own API endpoint, their own cluster view, their own scheduler). The operator does not multiply their workload with every new customer.

vCluster Platform powers 100K+ GPU nodes in production, with customers including CoreWeave and Nscale. The vCluster architecture is named in the NVIDIA DGX SuperPOD reference architecture. Boost Run launched a managed Kubernetes service in under 45 days with zero new platform engineering hires on this foundation.

The Full Stack: Bare Metal to Tenant-Ready AI Environments

For AI cloud providers building from raw hardware, vCluster is part of a broader integrated stack:

vMetal handles zero-touch bare metal provisioning. PXE boot, OS installation, machine registration, and network automation. GPU racks go from unboxed to production-ready without manual intervention at each node.

vNode adds kernel-native workload isolation using seccomp, cgroups, namespaces, and AppArmor. Container breakout protection without the GPU performance overhead of a hypervisor. This completes the isolation stack: control plane isolation via vCluster, workload isolation via vNode.

Certified Stacks provide pre-validated AI environments. Run:AI, Ray, Jupyter, and Slurm-on-Kubernetes via Slinky. Each turns a tenant cluster into a production AI platform in minutes instead of months.

You Are Not Picking a Scheduler. You Are Building a Platform.

The slurm vs kubernetes debate makes sense if you are one team running one GPU cluster. Pick the scheduler that matches your workload. Run with it.

If you are building an AI cloud that serves paying tenants, the question is different. Your platform needs to support the scheduler each tenant requires. Slurm for HPC batch jobs. Kubernetes for cloud-native tooling. Run:AI for fractional GPU sharing. The architecture that makes this possible without multiplying your operational overhead is per-tenant control plane isolation.

That is what vCluster provides. Fully isolated Kubernetes clusters per tenant, with Private Nodes as the production default. Each tenant chooses their scheduler. You manage one platform. Start with the vCluster Platform to see what that looks like in practice.

Frequently Asked Questions

Should AI cloud providers standardize on one scheduler?

If you are a single team running your own GPU cluster, pick the scheduler that matches your workload. But AI cloud providers serve multiple tenants, each with different workload profiles. Standardizing on one scheduler means turning away tenants who need the other. The better approach is per-tenant control plane isolation, so each tenant runs the scheduler they need inside their own isolated environment.

What is the main limitation of Slurm for large-scale AI workloads?

Slurm's primary limitation in a multi-tenant context is that it was not designed for strong tenant isolation. Its resource allocation model trusts users on dedicated hardware. In a shared AI cloud, this leads to GPU double-allocation, users bypassing restrictions via CUDA_VISIBLE_DEVICES, and operators manually patching isolation through cgroup configurations. For the slurmctld controller specifically, running Slurm on Kubernetes via Slinky — as NVIDIA does at 8,000+ GPU scale — addresses high availability and operational concerns by letting Kubernetes handle controller restarts, rolling updates, and unified monitoring.

Why isn't default Kubernetes ideal for LLM training in isolated tenant environments?

The default Kubernetes scheduler struggles with the gang-scheduling requirements of large, distributed training jobs, leading to lower cluster utilization. More importantly, its namespace-based isolation is insufficient for true tenant isolation, as all tenants share a single API server and control plane, creating a single point of failure and a massive blast radius.

How does running Slurm on Kubernetes solve some of these problems?

Running Slurm on Kubernetes, typically with an operator like Slinky, combines the best of both systems. This hybrid model uses Kubernetes to manage the Slurm control plane for high availability and leverages Slurm's superior scheduling for high-performance batch jobs, all while integrating with the cloud-native ecosystem for observability and driver management.

What problem does Slurm-on-Kubernetes not solve for AI cloud providers?

Slurm-on-Kubernetes via Slinky solves the scheduler integration problem. It lets you run Slurm workloads inside Kubernetes. What it does not solve is per-tenant isolation. If all tenants share one Kubernetes cluster, they also share one API server, one failure domain, and one scheduler configuration. Each tenant cannot run their own Slurm instance or choose a different scheduler. Per-tenant control plane isolation solves this by giving every tenant their own fully isolated Kubernetes cluster.

How does per-tenant control plane isolation enable multiple schedulers?

Each tenant gets their own fully isolated Kubernetes control plane: their own API server, etcd, RBAC, and CRDs. Since each tenant has a complete Kubernetes cluster, they can install whatever scheduler they need inside it. Slurm via Slinky. Native Kubernetes with Volcano. Run:AI. One tenant's Slurm controller failure does not affect another tenant's Kubernetes workloads. The platform operator manages the fleet from one control plane while each tenant gets scheduler autonomy.

Is vCluster a replacement for Kubernetes namespace isolation?

Yes, vCluster provides a fundamentally stronger isolation model. Namespaces give tenants logical separation within a shared control plane. They still share the same API server, etcd, and failure domain. vCluster gives each tenant their own dedicated control plane with Private Nodes as the production default. This means hardware-level tenant isolation, per-tenant CNI and storage, and full scheduler autonomy. Namespaces give you a room in a shared house. vCluster gives every tenant their own house, with their own locks, on their own foundation.

Does this architecture work on bare metal?

Yes. vCluster is part of a full stack designed for bare metal AI clouds. vMetal handles zero-touch provisioning: PXE boot, OS install, and machine registration. vCluster Standalone runs as a binary directly on Linux with no k3s or kubeadm dependency. vNode adds kernel-native workload isolation. Together they deliver the complete path from raw GPU racks to tenant-ready Kubernetes environments with per-tenant scheduler choice.

‍