Wiz Research cloned gpt2, modified it so that loading the model ran their code, and uploaded it to Hugging Face's inference API. That gave them execution inside a pod on Amazon EKS. What followed is four steps, and none of them are exotic.

The pod reached 169.254.169.254, the EC2 Instance Metadata Service, and pulled the worker node's IAM role credentials. With the node's AWS identity it ran aws eks get-token and received a valid Kubernetes token carrying the Node role. It found the cluster name by reading EC2 tags through DescribeInstances. Then it listed pods and read the secrets mounted to its own pod. Separate research against Hugging Face Spaces reached the shared container registry with write access, which means overwriting the images other customers pull.

Wiz called the metadata access a common EKS misconfiguration and left it there. It is worth being specific about what the setting actually is, because it is one field.

The IMDS hop limit governs how many network hops a metadata request survives. A process inside a container sits one hop from the host, so a limit of 2 lets it talk to link-local metadata as if it were the node itself. Set it to 1 and the request dies in the container. We built our reproduction with the limit at 2 and IMDSv1 permitted, and the pivot worked first time.

It defaults permissively, it lives in a launch template nobody reads twice, and almost nobody checks it until someone has already walked out of a pod wearing the node's identity.

This is not a Hugging Face problem. It is what multi-tenant inference looks like when the tenancy boundary is a container and the credential boundary is a hop count.

The part everyone wants to detect is the part you cannot see

The instinct is to go after the model. Scan the pickle, block the upload, alert on the deserialization.

From your cluster logs, you cannot. Not poorly. Not late. At all.

Pickle is not a data format, it is a small stack language whose opcodes rebuild objects, and rebuilding an object means calling something. A .pt or .bin file is an archive of pickled tensors, so a model file is executable content and torch.load is the interpreter. Python's own documentation says it plainly: never unpickle data you do not trust.

Now look at what that generates. The deserialization makes no Kubernetes API call, so there is no audit event. It performs no authentication, so there is no authenticator event. It does not touch CloudTrail either, and neither does the metadata request that follows it. Two full steps of this chain, the initial access and the credential theft, produce exactly nothing in the control plane.

Seeing the model execute requires a sensor inside the container. Falco, GuardDuty Runtime Monitoring, an EDR agent, something watching processes rather than API calls. If you do not have one, no amount of detection engineering against your audit logs will help, because the evidence does not exist.

So stop trying to catch the model.

What we measured

We stood up an EKS cluster to reproduce this properly, hop limit left at 2 on purpose, audit and authenticator logging on, both shipped into the SIEM. Then we ran two things in one window and compared them against what the control plane actually recorded.

First, a Caldera operation against an agent in a pod. Seven steps, all successful: T1033, T1087.001, and T1057 five times over. Host discovery, the sort of thing an operator runs in the first minute on a box.

It produced zero Kubernetes audit events.

That is worth sitting with if you plan to emulate container attacks with off-the-shelf adversary tooling. Those abilities run shell inside the container. They never touch the API, so the control plane never hears about them, and any Kubernetes detection content you validate that way is validated against silence.

Second, the same compromised pod talking to the Kubernetes API with its own service account token. Five calls, five audit events. Two allowed, both self-subject reviews, which is the API asking itself what it is permitted to do. Three refused: list secrets, list pods cluster-wide, read self-subject reviews.

That shape is the tell. Orientation succeeds because self-review is granted to every authenticated principal in the cluster. The escalation that follows it fails. An attacker who has just landed somewhere unfamiliar asks what it can do before trying to do anything, and the asking is permitted.

For scale, the same cluster generated 3,643 audit events in that six-minute window while otherwise doing nothing at all. Five of them were the attack.

The audit record names the pod

Here is the field almost everyone skims past. A Kubernetes audit event for a service account carries this:

user.username  system:serviceaccount:h4x-demo:default
user.extra
  authentication.kubernetes.io/pod-name   model-server
  authentication.kubernetes.io/node-name  ip-10-60-11-82.us-west-2.compute.internal
  authentication.kubernetes.io/pod-uid    6c6225d1-8a1d-4e26-b064-88e9f9d0a4aa
userAgent      Wget
sourceIPs      10.60.11.174

authentication.kubernetes.io/pod-name. The control plane tells you which pod made the call.

That changes the question you are able to ask. You are not stuck reasoning about a service account that fifty workloads might share. You know the workload. And once you know the workload, you can ask whether that workload has any business talking to the Kubernetes API at all.

A model server does not. It loads weights and serves predictions. It does not enumerate its own permissions, it does not list pods, and it does not read secrets. Nothing in an inference workload's job description involves the control plane.

There is a second signal sitting in the same record, and it is free. Look at the user agent. Legitimate in-cluster clients use client-go and say who they are, something like kubectl/v1.36 or a named controller. Our compromised pod presented Wget, because that was what the container had. Somebody hand-rolling HTTP requests against the API from inside a container is not your platform team.

The detection

Fire when a pod identity belonging to an inference workload touches the Kubernetes API, and qualify it with either an orientation call or a hand-rolled client.

In plain terms, four conditions:

• the principal is a service account, so system:serviceaccount: prefix on the account name
• the pod name matches your model-serving workloads
• and either the resource is selfsubjectaccessreviews, selfsubjectrulesreviews or selfsubjectreviews
• or the user agent is Wget, curl, python-requests or a bare HTTP library

The client check is an OR rather than a requirement, deliberately. An attacker who imports a Kubernetes client library presents a perfectly normal user agent, and the pod identity on its own should still fire.

The pod name pattern is the only part that is yours to tune. It is your naming convention, and if your inference workloads are called something else, the rule is inert until you say so. Where naming carries no meaning, invert it: list the workloads that are expected to call the API and alert on everything else. That is a better rule and a harder one to maintain, which is the usual trade.

None of this needs a runtime sensor. It runs on control plane logs you are probably already paying to store.

What to actually do

Set the IMDS hop limit to 1 on every node group running untrusted workloads. One field. It breaks the step that turns a container compromise into a cluster compromise, and if something in your cluster genuinely needs the node's IAM role from inside a pod, you have a bigger problem than a hop limit.

Turn on audit and authenticator logging and keep them. The authenticator log is the one people skip, and it is the one that records an AWS IAM principal becoming a Kubernetes identity. That is the Hugging Face pivot written down in a single line.

Stop treating model files as data. A .pt is a program. Anything that loads an untrusted one is running untrusted code, and the mitigation is safetensors or a format that does not execute on load, not a scanner that tries to guess which pickles are hostile.

And write the detection for the consequence. You will not see the model execute. You will see what it does afterwards, because the attacker has to orient, and orienting means asking the API questions it logs.

The uncomfortable part is how ordinary the chain is. A permissive default, a credential sitting at a link-local address, a token exchange that is documented behaviour, and an API that answers honestly when asked what you can do. No exploit, no CVE, nothing that a patch cycle would have caught. Every step is a supported feature working exactly as designed.

That is the shape of cloud compromise now. Are we still pretending the container is a security boundary?