Kubernetes ResourceQuotas: Automating Governance with Kyverno

Three months ago, our production cluster went dark for 45 minutes because a developer deployed a batch job with no resource requests. The pod ballooned, starved the core ingress controller, and triggered a cascade of 503 errors across our primary microservices. We spent the better part of the morning cleaning up the mess. That was the day I stopped relying on developer "best practices" and started enforcing hard guardrails.

If you don't have a strategy for Kubernetes ResourceQuotas, you're just one rogue deployment away from a cluster-wide outage.

Implementing Kubernetes Governance at Scale

Native ResourceQuotas are the bedrock of cluster resource management. They prevent a single namespace from consuming all the CPU and memory on a node. However, managing these manifests manually across 50+ namespaces is an operational nightmare. You’ll forget to update them, or someone will create a new namespace and forget to attach the quota entirely.

I solved this by using Kyverno to automate the generation of these manifests. By treating our policies as code, we ensure that every namespace—whether it’s for a staging environment or a production workload—starts with a baseline set of constraints. If you’re already looking into Kubernetes policy management with Kyverno and GitOps, this is the logical next step.

Why I chose Kyverno over OPA/Gatekeeper

I’ve worked with both, and for most DevOps teams, Kyverno is the winner. It’s written in YAML, not Rego. When I’m on-call at 2:00 AM, I don’t want to debug a complex logic gate written in a language I rarely touch. Kyverno allows me to generate ResourceQuotas dynamically the moment a namespace is created.

Here is the policy I use to ensure every namespace has a default CPU and memory limit:

YAML
apiVersion: kyverno.io/v1
kind: GenerateRequest
metadata:
  name: generate-resource-quota
spec:
  policy:
    name: create-default-quota
  resource:
    kind: Namespace
    name: "{{request.object.metadata.name}}"
---
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: create-default-quota
spec:
  generateExisting: true
  rules:
    - name: generate-quota
      match:
        resources:
          kinds:
            - Namespace
      generate:
        apiVersion: v1
        kind: ResourceQuota
        name: default-quota
        namespace: "{{request.object.metadata.name}}"
        data:
          spec:
            hard:
              requests.cpu: "4"
              requests.memory: 8Gi
              limits.cpu: "10"
              limits.memory: 20Gi

The Tradeoff: Hard Limits vs. Flexibility

By enforcing these Resource Limits, I accepted one major trade-off: developer friction. Some teams will complain when their pods are evicted for hitting a limit. But here’s the reality: if you aren't hitting your limits, you’re either over-provisioning or your application isn't as critical as you think.

I paired this enforcement with Kubernetes VPA and Goldilocks: Master Resource Right-Sizing to give teams the data they need to request the correct amount of resources. Don't just set a wall; provide the telemetry so they can climb it.

Beyond Static Limits

Static quotas are just the start. Cluster Resource Management is a dynamic game. Once you have these quotas in place, you’ll want to ensure that pods actually have resource requests defined. If you don't enforce LimitRanges alongside ResourceQuotas, you’ll end up with pods that have zero requests, which messes with the Kubernetes scheduler's ability to bin-pack effectively.

I recommend checking out Kubernetes Resource Management: Using VPA Recommendation Mode to fine-tune your baseline requests before locking them down with Kyverno.

FAQ: Common Implementation Hurdles

Q: Will Kyverno break existing namespaces? A: Not if you configure the generateExisting flag correctly. However, test this in a non-production environment first. If a namespace already has a quota, Kyverno won't overwrite it unless you explicitly define an update rule.

Q: How do I handle emergency exceptions? A: Use Kyverno’s exclude field in your policy. We have a "privileged" label we add to specific namespaces that need higher quotas, and our policy is scoped to ignore namespaces with governance: skip.

Q: Does this add latency to the API server? A: Kyverno runs as a controller. While it does add a slight overhead during namespace creation, it’s negligible compared to the stability you gain by preventing resource exhaustion.

Final Thoughts

Automating Kubernetes ResourceQuotas with Kyverno isn’t just about control—it’s about predictability. When you move away from manual management, you stop being a ticket-taker for resource adjustments and start being an engineer who builds self-healing infrastructure. Start small, automate the baseline, and let your clusters scale without the constant fear of a noisy neighbor taking everything down.