Admission Controllers 101: How To Block Risky Deploys Before They Run

If you’ve run Kubernetes in production for more than five minutes, you already know this truth: Admission Controllers 101: How To Block Risky Deploys Before They Run

1. Most outages don’t come from Kubernetes being “down.”
2. They come from bad workloads getting into the cluster.

• A privileged pod.
• A deployment with no resource requests.
• An image tagged
• A “quick fix” pushed during an incident that quietly makes things worse.

I’ve seen all of these cause real outages. Not hypotheticals. Real pages. Real Slack meltdowns.

Admission controllers are one of the few tools Kubernetes gives you to stop those mistakes before they ever run. But they’re also easy to misuse, easy to overdo, and very easy to roll out in a way that makes your developers hate you. It’s what admission controllers actually look like under pressure, based on real clusters, real failures, and real on-call rotations.

By the end, you should have a clear mental model and a realistic plan for what you’d enforce first and how not to blow up your org doing it.

Why blocking bad deploys before they hit the cluster matters

Once a pod is running, you’ve already lost some leverage.

Sure, you can:

• Evict it

• Throttle it

• Kill it

• Roll it back

But by then:

• Nodes may already be under pressure

• Other workloads may already be impacted

• The incident clock is ticking

Admission controllers let you fail fast at the API boundary, when the cost of saying “no” is lowest.

A very real failure example

One of the earliest outages I remember clearly:

• New service deployed

• No resource requests

• JVM app, memory-hungry

• Scheduler packed it onto already-busy nodes

• Node memory pressure → kubelet starts evicting

• Suddenly unrelated pods are crashing

The deploy “worked.”
Until it didn’t.

Nothing was technically broken. Kubernetes did exactly what we told it to do. The problem was we never should have let that workload in without requests in the first place.

Admission controllers are how you encode that lesson so you don’t have to relearn it at 2am.

What an admission controller actually

Forget the formal definitions for a second.

In practice, an admission controller is:

• A piece of logic that gets a vote on every API request before Kubernetes stores it.
• If it says “no,” the object never exists.

Where admission fits in the request flow

Roughly:

1. Authentication

   who are you?

2. Authorization

   are you allowed to do this kind of thing?

3. Admission

   is this object acceptable as written?

Admission is the last gate before reality.

Validating vs mutating

There are two types:

• Validating admission

  “This is allowed” / “This is rejected, here’s why”

• Mutating admission

  “I’m going to silently change this object”

In theory, mutation is powerful.
In practice, it’s dangerous.

I’ve seen mutation:

• Add default resource requests that developers never knew existed

• Inject sidecars that broke startup ordering

• Modify security contexts in ways that confused debugging

My rule of thumb:

Validate aggressively. Mutate sparingly

If developers don’t know what’s running because something rewrote their manifests, you’ve created a new class of incidents.

Admission vs RBAC

This trips people up.

• RBAC answers

  “Who is allowed to do this action?”

• Admission answers

  “Is this a sane thing to run at all?”

RBAC can’t say:

• “You must set resource requests”

• “Privileged pods are banned here”

Admission can.

What most people misunderstand about admission controllers

A few common misconceptions I’ve seen repeatedly:

We’ll just add policies and be safer

Policies don’t enforce themselves culturally.

If you roll them out badly, people will:

• Bypass them

• Copy-paste exceptions

• Or pressure you to disable them during incidents

Admission doesn’t replace judgment. It encodes the judgment you already agree on.

“We should start with everything locked down”

No. Please don’t.

The fastest way to kill adoption is to:

• Block half the cluster on day one

• Break existing workloads

• Provide useless error messages

You’ll spend weeks firefighting your own controls.

What you should enforce first

If you only enforce a few things at the beginning, enforce these.

Resource requests

If I could only pick one admission rule forever, this would be it.

Requests matter more than limits early on

Why?

• Scheduler uses requests, not limits

• Requests define how workloads are packed

• Missing requests lead directly to noisy-neighbor problems

I’ve seen clusters where:

• CPU limits existed everywhere

• Requests were missing

• Nodes were constantly under pressure

• Autoscaling behaved erratically

Block workloads with no requests.
Be flexible on limits at first.

This single rule prevents an entire class of outages.

Blocking truly dangerous privileges

Not everything “scary” is actually dangerous.

Things I don’t block immediately:

• Capabilities unless you know why

• Read-only root FS everywhere (great goal, painful first step)

Be precise. Over-broad security rules just create exceptions.

Image pinning

causes real pain:

• Rollbacks become impossible

• Rebuilds change behavior without config changes

• Debugging becomes “what even ran?”

Allow it in:

• Dev

• Sandboxes

• Experimental namespaces

Context matters.

Health probes

Lack of probes won’t crash the cluster but it will:

• Break rollouts

• Hide deadlocks

• Make “it worked locally” meaningless

I usually:

• Warn first

• Enforce later

• Allow opt-outs with justification

Be honest about edge cases and exceptions

Every rule has exceptions.

• DaemonSets often need host access

• Init containers may need different resources

• One-off jobs don’t behave like services

Pretending otherwise just pushes people to fight the system.

Design your policies with escape hatches but make them visible and intentional.

How I’ve seen admission rollouts go wrong

Breaking existing workloads

The classic mistake:

• Enable enforcement

• Suddenly dozens of old deployments are non-compliant

• Now nothing can be updated

Always audit first. Always.

Blocking emergency fixes

If your admission policies block:

• Scaling

• Image rollbacks

• Hotfixes during incidents

People will disable them under pressure. Permanently.

Have an emergency bypass that:

• Is explicit

• Is logged

• Requires intent

Over-strict too early

• Perfect is the enemy of adopted.
• A policy nobody can satisfy helps no one.

How to roll out admission controllers without making enemies

Start in audit mode

Always.

Let people see:

• What would be blocked

• How often

• By whom

This builds trust and surfaces edge cases early.

Scope deliberately

Use:

• Namespaces

• Labels

• Operations

You don’t have to apply everything everywhere on day one.

Exceptions that don’t rot

Avoid:

• “Just add this label forever”

• Global allowlists

Prefer:

• Time-bound exceptions

• Explicit annotations with reasons

• Periodic reviews

Exceptions should feel slightly uncomfortable.

Communicate like a human

Explain:

• Why the rule exists

• What outage it prevents

• How to comply

• How to get help

People tolerate friction when they understand the reason.

Tooling reality check

Gatekeeper vs Kyverno vs custom webhooks

Gatekeeper

• Powerful

• Rego is… Rego

• Easy to shoot yourself in the foot with complexity

Kyverno

• More YAML-native

• Easier for most teams

• Great for common policy patterns

Custom webhooks

• Only if you really need them

• You own scaling, latency, failure modes

• Debugging them at 3am is not fun

For most teams:

Start with Kyverno or Gatekeeper. Avoid custom code unless forced.

Operational risks people underestimate

• Admission adds latency to every API request

• Webhook outages can block the entire cluster

• Bad upgrades can brick deploys

Always:

• Set timeouts

• Use initially

• Monitor webhook latency and error rates

Admission controllers are part of your control plane. Treat them that way.

A realistic rollout plan

Week 2: Observe

• Audit-only policies

• Resource requests

• Privilege usage

• Image tags

Week 4: Enforce the obvious

• Block no-requests

• Block in prod

• Block privileged pods outside system namespaces

Week 5: Tighten and refine

• Improve error messages

• Add probes policy

• Review exceptions

What success actually looks like

Not:

• “100% compliant cluster”

But:

• Fewer noisy-neighbor incidents

• Faster rollouts

• Fewer “how did this get here?” moments

• Developers fixing issues before merge

Metrics to watch

• Admission rejections over time

• Webhook latency

• Policy exception counts

• Incidents caused by bad deploys (this should go down)

---

You Might Be Interested In

• Multi-tenant Saas Isolation: Patterns For Data, Compute, And Queues
• Kubernetes Rbac Explained: Roles, Bindings, And Least Privilege
• Cloud Iam Cleanup: Removing Unused Permissions With Audit Logs
• Best Cloud Gpu Options For Beginners
• Api Rate Limits And Scaling Basics

---

Conclusion

Admission controllers aren’t about locking Kubernetes down or proving how “secure” your platform is. They’re about taking the lessons you’ve already learned the hard way and making sure you don’t have to relearn them during the next incident. Every cluster has a small set of mistakes that cause outsized damage, and admission is where you stop those mistakes cheaply, early, and predictably.

What matters most is how you roll them out. Start with policies that prevent real outages, not theoretical ones. Observe before enforcing. Be explicit about exceptions. Treat admission like part of the control plane, not a YAML experiment. If developers understand what’s being blocked and why, admission controllers stop feeling like bureaucracy and start feeling like guardrails.

If you’re wondering what to do Monday morning: turn on audit mode, look at missing resource requests, scan for privileged pods, and see how much is actually in use. That data will tell you exactly where to start. Keep it practical, keep it humane, and your cluster and your on-call rotation will thank you.

FAQs  about Admission Controllers 101: How To Block Risky Deploys Before They Run

Should we block everything in prod but allow anything in dev?

This is a really common instinct, and I get why people reach for it. Production feels scary, dev feels disposable, so you draw a hard line between them. The problem is that this almost always pushes failures later in the lifecycle, when they’re more expensive and more stressful. If developers only discover missing requests, bad images, or broken probes when something hits prod, you’ve already lost most of the benefit of admission control.

In my experience, a better approach is graduated enforcement. Dev should be more flexible, but not lawless. Warn instead of block, or enforce only the most critical rules. The goal is that by the time something reaches production, it already looks like a production workload. Prod shouldn’t be a surprise test environment; it should just be stricter about rules everyone already understands.

What about performance impact?

Yes, admission controllers add latency. Every create or update goes through them, and badly written policies or slow webhooks absolutely can be felt at scale. I’ve seen clusters where a misbehaving admission webhook added hundreds of milliseconds to every deploy and made CI pipelines crawl. That’s not theoretical it happens.

The key is that policy complexity matters more than policy count. Simple checks are cheap. Overly clever logic, external lookups, or huge mutation chains are what get you into trouble. Set reasonable timeouts, watch webhook latency like you would any other control-plane component, and treat admission as production infrastructure. If you wouldn’t run it unmonitored, don’t put it in the API path.

Can admission replace code review?

No. And anyone telling you otherwise is selling something.

Admission controllers are great at enforcing mechanical truths: requests exist, tags are pinned, privileges are sane. They are terrible at understanding intent, architecture, or business context. They won’t tell you that a change is risky, that a rollout plan is bad, or that a dependency graph is about to explode.

What admission does well is take boring, repetitive checks out of code review so humans can focus on harder questions. In healthy teams, admission makes reviews faster and more focused not obsolete.

What’s the biggest mistake teams make?

Rolling out enforcement before they understand their own cluster.

I’ve watched teams flip policies to “enforce” and immediately discover that half their workloads violate them. Suddenly even small changes are blocked, people are scrambling for exemptions, and trust evaporates. At that point, admission controllers feel like an obstacle, not a safety net.

Audit-first rollout isn’t optional it’s how you learn what “normal” actually looks like in your environment. Until you’ve seen real data on what would be denied, you’re guessing. And guessing in the API server path is a bad habit.

How do we handle emergencies without bypassing everything?

This is where a lot of well-intentioned setups fall apart. If your only option during an incident is “disable the admission controller,” that’s exactly what will happen and it may never get turned back on. I’ve seen that movie more than once.

You need a deliberate, narrow escape hatch. Something that allows critical changes, requires explicit intent, and leaves a trail. That might be a specific annotation, a tightly scoped role, or a temporary namespace rule. The important part is that emergencies are possible without tearing down the guardrails entirely. If bypassing policy feels routine, your policies are too rigid or poorly designed.