Scaling Kubernetes Without the Pain of etcd Sharding

Summary

• A shared Kubernetes etcd struggles at scale, with a benchmark showing that just 1,000 new objects can spike API latency by 650ms — a 10-13x increase that impacts all tenants.
• While etcd sharding is a common approach to mitigate this, it is operationally complex and fails to address the core issue of a shared API server bottleneck.
• A better architectural solution is to isolate control planes for each tenant, which eliminates the "noisy neighbor" problem by design rather than just mitigating it.
• vCluster Platform enables this tenant cluster architecture by giving each tenant a dedicated, lightweight Kubernetes control plane on shared infrastructure, ensuring predictable performance and scalability.

If you've spent any time scaling Kubernetes, you've probably run into an etcd wall. Not a theoretical one — a real one. The kind where kubectl starts dragging, deployments hang, and your on-call engineer starts getting paged at 2am. Many engineers report issues with slow etcd, even in moderately sized clusters. This frustration resonates with anyone who's tried to run multiple teams or applications on a single Kubernetes control plane, where teams are constantly chased by slow etcd events when scaling controllers.

Here's the hard truth: Kubernetes's scalability limits aren't just a Google or AWS problem. Any platform team operating more than a handful of tenants on shared infrastructure will eventually hit the ceiling of a single, monolithic etcd instance and API server.

In this blog, you will explore practical sharding strategies available when using vCluster tenant clusters to reduce the load on your etcd and Kubernetes API server — with a real-world demo so you are prepared as your infrastructure scales.

The article compares two fundamentally different approaches to solving this problem:

• etcd Sharding: A low-level approach that partitions the etcd keyspace across multiple etcd clusters.
• Control Plane Isolation via Tenant Clusters: A higher-level architectural solution — vCluster — that gives each tenant their own dedicated API server and etcd, eliminating competition at the root.

Let's start by understanding exactly why etcd breaks under pressure.

Why Your Kubernetes Cluster's etcd Is Slowing Down

Before jumping to solutions, it's worth being precise about the failure modes. etcd is a distributed key-value store that acts as the single source of truth for all Kubernetes cluster state — every Pod, Secret, ConfigMap, Deployment, and CRD lives in etcd. Every kubectl apply, every controller reconciliation loop, and every watch event is a read or write against it. At small scale, this is invisible. At larger scale, the cracks appear fast.

1. The 8GB Database Size Limit

etcd is not a general-purpose database. It is purpose-built for consistency, not for high-volume data storage. Official etcd documentation recommends keeping the database under 8GB — and that's a soft ceiling, not a hard one. In practice, clusters start showing degraded performance well before they hit that number. The more objects you store, the more compaction and defragmentation cycles are needed to keep the database healthy, and those cycles themselves consume I/O and CPU.

For GPU cloud operators or platform teams running hundreds of tenants, CRDs, and custom controllers, this limit isn't hypothetical — it's a countdown.

2. API Server Latency at Scale

As the number of objects in etcd grows — especially past the 30,000–40,000 object mark — API server latency climbs non-linearly. What starts as a 50ms API response time can balloon into 500ms+ for routine list and watch operations. This translates directly into degraded user experience: slow kubectl responses, deployment rollouts that stall, and pod scheduling delays that cascade through your system.

A major outage at a large AI service provides a stark reminder that API server instability at the control plane level can take down production services that millions of users depend on. Control plane health is not a luxury — it's load-bearing infrastructure.

3. Work Queue Latency

The Kubernetes controller manager uses internal work queues to process changes to cluster state. When etcd is healthy, items flow through these queues rapidly. When etcd is slow — due to high object count, compaction lags, or disk I/O pressure — the work queues back up. The result is that critical operations like pod scheduling, endpoint reconciliation, and autoscaling responses are delayed.

This is the pain behind the common observation of being "always chased by slow etcd events when scaling controllers." It's not just an inconvenience — in latency-sensitive environments, a backed-up work queue means missed SLAs.

4. The Noisy Neighbor Problem

In a shared Kubernetes cluster for multiple tenants, all tenants share the same etcd instance. This means one tenant's misbehaving controller — say, a buggy operator that creates thousands of CRDs in a loop — can consume all available etcd I/O, starving every other tenant on the cluster. Their pods slow down, their API responses degrade, and their SLAs get violated through no fault of their own.

Compounding this, storage misconfigurations are surprisingly common and dramatically worsen noisy neighbor effects. Using filesystems not optimized for database workloads, like ZFS or btrfs, can make IOPS much slower. When etcd calls fsync, even fast NVMe drives can struggle. When a tenant's activity spike hits an already I/O-constrained etcd, the blast radius expands cluster-wide.

The Conventional Approach: The Complexity of etcd Sharding

The natural first instinct when hitting etcd limits is to shard it — split the database across multiple etcd instances so no single one carries the full load. Kubernetes does offer a native mechanism for this: the --etcd-servers-override flag on the API server, which lets you route specific resource types (like Events) to a separate etcd cluster. This is the most commonly cited native sharding strategy in the community.

# Example: routing events to a dedicated etcd cluster
kube-apiserver \
 --etcd-servers=https://etcd-main:2379 \
 --etcd-servers-overrides=/events#https://etcd-events:2379


This approach works for offloading a single high-churn resource type (Events are the canonical example), but it doesn't scale to arbitrary keyspace partitioning. For more ambitious sharding, you need specialized tooling, such as a proxy layer that attempts to shard the full Kubernetes API keyspace across multiple etcd clusters.

Why etcd Sharding Is Not a Silver Bullet

Even with specialized tooling, low-level etcd sharding introduces a set of tradeoffs that make it risky in production:

• Operational overhead multiplies: Every etcd cluster you add requires its own backup strategy, monitoring, alerting, failure recovery, and upgrade path. You've traded one complex system for several.
• Storage overhead: Some sharding proxies introduce a per-value storage overhead to track cross-cluster metadata.
• Single-key transaction constraint: Transactions in a sharded model can only reference a single key. This breaks assumptions baked into Kubernetes's optimistic concurrency model.
• Watch latency degradation on failed writes: In failure scenarios, watch consumers may observe increased latency as the proxy layer handles retries across member clusters.
• Fan-out on range queries: Multi-key range queries must be fanned out to all member etcd clusters and merged, which can paradoxically degrade read performance in high-cardinality scenarios.

Beyond the technical tradeoffs, there's a deeper problem: etcd sharding treats the symptom, not the disease. The root cause of the noisy neighbor problem and API server saturation is that all tenants share a single control plane. Splitting the underlying database into shards doesn't change that architectural reality — it just moves the bottleneck around.

The watch event explosion is what catches most teams off guard, and no amount of etcd sharding eliminates watch storms generated by competing controllers on the same cluster. The API server itself remains the chokepoint.

For most teams, the conclusion is clear: low-level etcd sharding is operationally brittle, technically constrained, and architecturally incomplete. There's a better way.

Tenant Cluster Isolation: The Scalable Alternative to etcd Sharding

The solution to the performance problem is vCluster's tenant cluster architecture. Rather than splitting a single etcd instance, vCluster spins up dedicated, isolated control planes per tenant — each with its own API server and etcd — while keeping the underlying compute shared. While this technically introduces a form of control plane sharding, it operates at a higher level of abstraction than raw etcd partitioning, which avoids the durability and linearizability risks associated with low-level etcd sharding.

This architecture is how GPU cloud operators run hundreds of isolated Kubernetes environments on shared infrastructure at scale — each tenant gets cluster-admin access and a fully isolated control plane, without the cost of provisioning a separate physical cluster per customer.

How vCluster Tenant Clusters Work

A vCluster tenant cluster is a fully functional Kubernetes cluster that runs inside a namespace of a host (or "super") cluster. Each tenant cluster provisions its own:

• Dedicated API server: Tenant API requests never touch the host API server. Watch events, list operations, and write traffic are all processed within the tenant's own control plane.
• Dedicated data store: By default, vCluster uses a lightweight embedded data store (SQLite or etcd, depending on configuration). The host etcd is completely shielded from tenant activity.
• Dedicated controller manager: Tenant controllers run in isolation, so a misbehaving controller in one tenant cluster cannot affect scheduling, endpoint reconciliation, or any other control loop in another.

The control plane components run as pods within the host cluster namespace, completely transparent to the tenant. From the tenant's perspective, they have a full Kubernetes cluster with cluster-admin access. From the host's perspective, a vCluster is just another set of pods in a namespace.

This shifts the scaling model from vertical (scale a single, shared etcd) to horizontal (add more isolated tenant control planes on the same underlying nodes). It's a fundamentally more elastic approach — one that mirrors how the most demanding Kubernetes deployments in the industry are architected today.

Real-World Demo: Benchmarking etcd Load

Theory is one thing. Let's make it concrete with a benchmark that demonstrates the performance difference between a shared host cluster and an isolated vCluster tenant cluster under a controlled load spike.

Prerequisites

• A host Kubernetes cluster (any distribution)
• Helm installed
• Prometheus and Grafana deployed for metrics (optional but recommended for visualization)
• The vCluster CLI

Set Up Prometheus and Grafana

Before running any tests, deploy Prometheus and Grafana to collect API server and etcd metrics from your host cluster. The key metric to watch is apiserver_request_duration_seconds_bucket, which captures latency for all API server requests broken down by verb, resource, and response code.

Once the CLI is installed, you can create your first tenant cluster with the following command:

vcluster create my-vcluster -n my-vcluster --connect=false


This provisions a fully isolated Kubernetes control plane — with its own API server, controller manager, and embedded data store — running as pods inside the my-vcluster namespace on your host cluster.

Run the Benchmark: 1,000 Secrets on the Host Cluster

The benchmark script creates 1,000 Kubernetes Secrets in a tight loop, simulating the kind of high-churn object creation that a real tenant workload (say, an application that creates per-job secrets) might generate.

First, run it against the host cluster directly:

./secret-creation.sh


Watch your Grafana dashboard during this test. You'll see the API server latency climb sharply as etcd processes the burst of write operations. This is the shared control plane under stress.

Run the Benchmark: 1,000 Secrets Inside a vCluster Tenant Cluster

Now connect to the vCluster tenant cluster and run the identical test:

vcluster connect my-vcluster -n my-vcluster
./secret-creation.sh


This time, keep your Grafana dashboard focused on the host cluster metrics — not the vCluster tenant's internal metrics. You want to see what the host cluster experiences while the tenant generates the same workload.

The results are stark, and we'll dig into them in the next section.

Performance Comparison: Host Cluster vs Tenant Cluster etcd Load

Host Cluster: The Latency Spike

When 1,000 Secrets are created directly against the host cluster, the Grafana dashboard tells a clear story. The apiserver_request_duration_seconds_bucket metric shows a sharp, unmistakable spike. Peak API latency hit 650ms during the creation burst — a figure that represents approximately a 10–13x degradation from baseline latency for a healthy, lightly loaded cluster.

At this level of latency, production consequences are real: reconciliation loops fall behind, HPA decisions are delayed, and any tenant making routine API calls during this window experiences degraded responses. This is the noisy neighbor effect captured in metrics — one workload, one burst, and the entire shared control plane is affected.

That 650ms spike was caused by just 1,000 objects. Production environments with hundreds of tenants, continuous deployments, and CRD-heavy workloads can generate this kind of pressure continuously, not just in isolated bursts.

vCluster Tenant Cluster: Zero Host Impact

In contrast, when the same test was conducted inside a vCluster tenant cluster, no spike appeared on the host's metrics. All etcd operations were managed by the tenant cluster's isolated control plane and dedicated data store — the host etcd and API server remained completely unaffected. This is what control plane isolation looks like in practice: tenant workloads generate zero impact on host cluster stability.

The host cluster's API latency stayed flat throughout the entire test. From the host's perspective, nothing happened — because as far as the host etcd is concerned, nothing did.

Side-by-Side Comparison

The performance difference is not marginal — it's categorical. With a shared control plane, load spikes are a cluster-wide event. With isolated tenant control planes, they are contained by design.

Why Tenant Cluster Isolation Outperforms etcd Sharding

As the benchmark demonstrates, vCluster eliminates the performance bottlenecks of a shared etcd and API server by isolating each tenant's control plane. Here's what that means operationally:

1. High Performance and Efficiency — By Architecture, Not Configuration

With etcd sharding, you're still running a shared data layer and hoping that partitioning the keyspace distributes load evenly. The performance characteristics depend entirely on how well your workload happens to align with your shard boundaries — and in practice, it rarely does cleanly.

With vCluster, performance isolation is structural. Each tenant's API server and etcd handle only that tenant's objects. A tenant with 50,000 Secrets doesn't affect a tenant with 500. The noisy neighbor problem isn't mitigated — it's eliminated by design. Each tenant's control plane is fully independent, which means you get predictable, consistent performance regardless of what other tenants are doing.

2. Shared Resources with Strong Isolation

One of the primary objections to running separate clusters per tenant is cost: provisioning dedicated worker nodes, control plane VMs, and networking infrastructure for every customer is expensive and operationally complex. vCluster solves this by decoupling the control plane from the compute.

Tenant clusters share the underlying worker nodes (and their GPUs, in AI infrastructure scenarios) while maintaining completely isolated control planes. Tenants get cluster-admin access to their own full Kubernetes environment, including the ability to install CRDs, create ClusterRoles, and manage their own namespaces — without any ability to see or affect other tenants. This is the cost efficiency of shared infrastructure with the isolation guarantees of dedicated clusters.

3. Better Backup and Restore

With a monolithic shared etcd, backup and restore is an all-or-nothing operation at the cluster level. If you need to restore a single tenant's state, you're either doing complex point-in-time filtering or restoring the entire cluster — neither of which is fast or safe in a production environment.

With vCluster, each tenant cluster's state is self-contained. Backing up and restoring a vCluster is an independent operation that has zero impact on other tenants. This dramatically simplifies disaster recovery workflows and makes it practical to offer per-tenant backup SLAs.

4. Rate Limiting and Blast Radius Containment

When a tenant's application misbehaves on a shared cluster — a misconfigured controller enters a hot reconciliation loop, a CI system floods the API server with requests — the impact affects every other tenant. There's no built-in boundary.

vCluster tenant clusters enforce hard boundaries at the control plane level. A runaway controller in one tenant cluster consumes resources within that tenant's control plane only. The host cluster's etcd, API server, and all other tenant clusters are unaffected. This is not just a performance benefit — it's an operational safety property.

However, the work queue latency issue is also mitigated with vCluster tenant clusters — there's no load spike on the host cluster because each tenant's control plane handles its own queued operations independently. Tenant controller managers process their own work queues in isolation, so a backed-up queue in one tenant's environment has no effect on scheduling or reconciliation in any other.

etcd Sharding vs. Tenant Cluster Isolation — Summary

Final Thoughts

etcd bottlenecks are a production reality, not a theoretical concern. As the benchmark here shows, even 1,000 object creations spike host latency by 650ms. At the scale GPU cloud operators run — hundreds of tenants, thousands of concurrent workloads, tens of thousands of objects — a shared control plane is a single point of failure for the entire platform.

vCluster solves this by moving the problem: each tenant cluster gets its own isolated control plane and etcd, so tenant activity never touches host cluster stability. The architecture that made this demo work is the same one used by GPU cloud operators like CoreWeave and Nscale — running 100K+ GPU nodes across 50+ production deployments.

If you're building or operating GPU cloud infrastructure and hitting etcd or API server ceilings, this is the architecture worth evaluating. See how vCluster works for GPU cloud operators → or book a demo to walk through your specific scale requirements.

‍