Kubernetes teams automate deploys but not CPU: why?

Key Takeaways

• Most Kubernetes teams automate code deployment but manually manage resource allocation
• AI workloads with unpredictable resource demands are exposing this trust gap
• Overprovisioning to avoid outages wastes an estimated 40% of cloud spend

Kubernetes teams have a trust problem. They'll let automation deploy code to production a dozen times a day. But ask that same automation to adjust CPU or memory? Suddenly, everyone wants a human in the loop.

This asymmetry has existed for years, but AI workloads are making it impossible to ignore. Machine learning inference and training jobs have resource profiles that spike unpredictably, and manual scaling simply can't keep up.

Where the trust gap comes from

The logic behind the split makes sense if you think about risk. A bad code deployment can be rolled back in seconds. GitOps tooling like ArgoCD and Flux has made this so reliable that teams don't think twice about pushing changes.

Resource misallocation is different. Set CPU requests too low and your application crashes. Set them too high and you're burning money on idle compute. The feedback loop is slower. The consequences feel less recoverable.

So teams overprovision. They allocate 2x or 3x the resources they need because the cost of downtime outweighs the cost of waste. Flexera's State of Cloud Report estimates that 40% of cloud spend is wasted this way. For a company spending $1 million monthly on cloud infrastructure, that's $400,000 going nowhere.

Why AI workloads are forcing the issue

Traditional web applications have relatively predictable resource patterns. Morning traffic spikes, evening dips. Seasonal peaks you can plan for. Horizontal Pod Autoscaler (HPA) handles these well enough that teams can set it and forget it.

AI workloads don't follow those rules. A batch inference job might need 16 GPUs for 20 minutes, then nothing for hours. A training run could require scaling from 8 to 64 nodes mid-job. The resource profile changes based on model architecture, batch size, and data characteristics that vary between runs.

Manual scaling breaks down here. By the time an engineer notices a bottleneck, investigates, and increases allocation, the job has either failed or the window has passed.

The CNCF's 2023 survey found that 84% of organizations now run Kubernetes in production, with 61% operating 10 or more clusters. As these organizations add AI workloads, the operational complexity multiplies faster than headcount.

What's changing in the tooling

Kubernetes native tools have improved, but they still require significant configuration. HPA handles horizontal scaling based on metrics. Vertical Pod Autoscaler (VPA) adjusts resource requests, but in older versions it required restarting pods to apply changes. The newer in-place resource resize feature, which hit beta in Kubernetes 1.27, finally allows VPA to adjust without restarts.

Third-party tools are filling gaps. StormForge and CAST AI use machine learning to recommend and apply resource changes. Spot by NetApp and Karpenter focus on node-level scaling and spot instance management. These tools promise to close the trust gap by making resource decisions more predictable and auditable.

The pitch is straightforward: if you trust automation to deploy code because it's version-controlled and reversible, resource automation can work the same way. Record the decision, log the metrics that triggered it, make rollback easy.

The organizational shift that matters more

Better tooling helps, but the real blocker is organizational. Resource decisions often live in a different part of the org chart than deployment decisions. Platform teams own infrastructure costs. Application teams own uptime. When automation touches both, accountability gets murky.

Companies that have bridged this gap tend to share a few practices. They give application teams visibility into their resource costs, often through internal showback or chargeback. They establish guardrails, not gates. Automation can scale within bounds; exceeding those bounds triggers human review.

Most importantly, they treat resource automation failures the same way they treat deployment failures. Post-incident reviews, not blame. Improved automation, not reversion to manual control.

What this means for the next 18 months

The Kubernetes ecosystem is moving toward tighter integration between deployment and resource automation. Projects like Keda (Kubernetes Event-Driven Autoscaling) are gaining traction because they tie scaling decisions to application-level events rather than raw CPU metrics.

AI is accelerating this on two fronts. First, AI workloads demand better resource automation. Second, AI-powered tools promise smarter automation. Whether that second promise delivers is an open question. The feedback loop for resource optimization is measured in weeks, not milliseconds. Training models on that data takes time.

For now, the trust gap remains real. But the cost of maintaining it keeps rising.

Frequently Asked Questions

Why do Kubernetes teams trust CI/CD automation but not resource automation?

Code deployments can be rolled back in seconds with GitOps tooling. Resource misallocation has slower feedback loops and less obvious recovery paths, making teams default to manual control and overprovisioning.

How much cloud spend is wasted on overprovisioned Kubernetes resources?

Industry estimates suggest around 40% of cloud spend goes to unused or underutilized resources, largely due to teams allocating extra capacity to avoid outages.

What Kubernetes tools help automate resource scaling?

Native options include Horizontal Pod Autoscaler (HPA) and Vertical Pod Autoscaler (VPA). Third-party tools like StormForge, CAST AI, and Karpenter offer ML-based recommendations and automated optimization.

Why are AI workloads harder to manage on Kubernetes?

AI training and inference jobs have unpredictable resource spikes that don't follow typical application patterns. Manual scaling can't respond fast enough, and overprovisioning GPU resources is extremely expensive.

Source: The New Stack / Yasmin Rajabi