name: Kubernetes Node Identity Used by a Non-Kubelet Client Analytic
signatureid: CAS-AUD02-RUN
category: 'Privilege Escalation'
threatname: 'Valid Accounts: Cloud Accounts'
functionality: 'Containers As A Service'
description: |
Detects a Kubernetes node identity being used to read cluster data in a way
the node authoriser does not grant a kubelet.
An EKS worker node authenticates to the API server as
system:node:<hostname>, in the system:nodes group, by presenting a token
signed with the node's IAM role. Exactly one program is supposed to do
that. Any other client holding a node identity is holding credentials it
was not issued.
The path to those credentials is short. A pod that can reach the instance
metadata service at 169.254.169.254 can read the worker node's IAM role
credentials, which is possible whenever the metadata hop limit is 2: the
container network namespace is one hop and the host is the second. Those
credentials are then exchanged locally for a Kubernetes bearer token.
Nothing in that exchange touches the cluster, so the first evidence of it
is the moment the token is used.
This is the step that made the Wiz Hugging Face research consequential. The
pod's own service account was permitted almost nothing and its escalation
failed. Repeating the same requests as the node succeeded, because the node
role is scoped for a kubelet's needs rather than a tenant's.
The analytic keys on the SHAPE of the request rather than on the client that
claims to have made it. The node authoriser requires a kubelet to scope its
pod queries to its own node, and grants no collection read of secrets at
any scope. An attacker has to break both rules for the access to be worth
having, because a pod list scoped to the node they already own returns
nothing new and a scoped read of one named secret is not what they came
for. Those are properties of the attack. A user agent is a string they
choose.
A denial does not mean the attempt failed harmlessly. A 403 is returned
AFTER authentication succeeded, so it proves the credential theft worked
and only authorisation stopped it. On a cluster whose node role has been
widened, a common outcome of debugging a CNI or CSI problem, the same
request returns data. Both outcomes matter, so this analytic does not
filter on the decision.
This is the node-identity half of the chain CAS-AUD01-RUN covers the
service-account half of. They cannot overlap: ObjectName is populated from
the pod-name claim on a service account token and is null on every
node-identity event, so CAS-AUD01-RUN's workload condition can never match
one.
reference:
- https://attack.mitre.org/techniques/T1078/004/
- https://attack.mitre.org/techniques/T1552/005/
- https://attack.mitre.org/techniques/T1613/
- https://www.wiz.io/blog/wiz-and-hugging-face-address-risks-to-ai-infrastructure
- https://kubernetes.io/docs/reference/access-authn-authz/node/
labels:
- attack.privilege_escalation
- attack.t1078.004
- attack.credential_access
- attack.t1552.005
- attack.discovery
- attack.t1613
logsource:
product: AWS
service: AWS EKS Audit
detection:
selection_node_identity:
AccountName|startswith: 'system:node:'
# Raw:
# user.username
#
# The identity a worker node authenticates as. A pod's own identity is
# system:serviceaccount:..., and a human is an IAM principal, so this
# prefix is specific to a node credential being presented. Normalization
# uppercases the value; Securonix comparisons are case insensitive.
selection_collection_never_granted:
Method: 'list'
SourceServiceName:
- 'secrets'
- 'configmaps'
- 'serviceaccounts'
# Raw:
# verb -> Method objectRef.resource -> SourceServiceName
#
# The node authoriser grants a kubelet a named read of a secret and
# nothing wider. It grants no collection list, at any scope, which the API
# server states in its own denial: "can only read namespaced object of
# this type". There is no scoping that makes this legitimate and nothing
# for an attacker to add.
#
# Measured against a deliberate control: a kubelet's own access to
# secrets, configmaps and service accounts is a NAMED watch or a token
# create, never a list. 16 such requests, 0 lists. See the control below.
selection_pod_enumeration:
Method:
- 'list'
- 'watch'
SourceServiceName: 'pods'
filter_scoped_to_own_node:
RequestUrl|contains: 'fieldSelector=spec.nodeName'
# Raw:
# requestURI -> RequestUrl, query string included
#
# A kubelet may list and watch pods, but only scoped to itself, and the
# authoriser says so: "can only list/watch pods with spec.nodeName field
# selector". Measured: 106 kubelet pod requests over six hours, every one
# of them carrying this selector. An attacker must drop it, because a
# list scoped to the node they already control returns nothing new and a
# selector naming another node is refused.
#
# THE Method CONSTRAINT IS LOAD BEARING, do not drop it as redundant.
# A kubelet also makes many NAMED pod requests - get and patch against
# /api/v1/namespaces/<ns>/pods/<name> and .../status - and those carry no
# field selector because a named object is already scoped by being named.
# Measured: 73 such requests in the same six hours. Without list|watch
# they all match this branch.
selection_tenant_data:
SourceServiceName:
- 'secrets'
- 'pods'
- 'configmaps'
- 'serviceaccounts'
filter_kubelet:
RequestClientApplication|startswith: 'kubelet'
# Breadth only. Catches a non-kubelet client making a request the two
# shape conditions permit, for example a named read of a single secret.
# This is the evadable branch and it is deliberately not load bearing.
condition: >
selection_node_identity and (
selection_collection_never_granted
or (selection_pod_enumeration and not filter_scoped_to_own_node)
or (selection_tenant_data and not filter_kubelet)
)
criticality: High
saveasthreat: true
violation_summary:
grouping_attribute: 'AccountName'
level2_attribute: 'SourceAddress'
level2_metadata_attributes:
metadata_attributes:TECHNICAL DETAILS
DATA SOURCE
Amazon EKS control plane audit logs, the same datasource and parser as
CAS-AUD01-RUN: SCNX_AMAZON_AWSEKSAUDIT_CAAS_AWS_JSO_COMM, filtered to the
kube-apiserver-audit- log stream prefix, datasource timezone UTC.
READ THIS BEFORE CONCLUDING A FIELD IS UNMAPPED
The default Activity_All search view does NOT display every parsed
attribute. Method and RequestUrl are both populated on this datasource
and neither appears in it. They show under "Activity All Attributes-OOTB"
and both confirm in the parser checker.
This analytic was first written WITHOUT them, on the evidence of a
rendered event, and the result was a weaker detection keyed on the user
agent. A field missing from a displayed event is not evidence the parser
dropped it. Test for presence directly, for example with
"@requesturl not null", rather than inferring it from a rendering.
WHY THE SHAPE IS THE SIGNAL
Both shape conditions come from the node authoriser's own refusals,
captured during emulation rather than inferred from documentation:
pods "can only list/watch pods with spec.nodeName field selector"
secrets "can only read namespaced object of this type"
So a kubelet may enumerate pods only when scoped to itself, and may read
a secret only by name. An attacker must break both to get value. Scoping
a pod list to the node they already control returns nothing they did not
have, and a field selector naming a different node is refused by the same
authoriser. Those are constraints on the attack. The user agent is not.
MEASURED, six hours on a live cluster:
106 kubelet pod requests, method watch, EVERY one carrying
fieldSelector=spec.nodeName%3D<its own node>
1 list pods, no selector, by a node identity presenting Wget
1 list secrets, cluster scope, by the same
Note the selector value is URL encoded, spec.nodeName%3D<node>. The
condition matches on 'fieldSelector=spec.nodeName' so the encoding of the
following character does not matter.
Method IS THE AUDIT VERB. verb IS NOT.
Method (requestmethod) carries the Kubernetes audit verb: list, watch,
create. There is a separate verb field parsed from the REQUEST OBJECT,
which on a selfsubjectaccessreview reads "get" while the audit verb was
"create", because the request body asked whether the caller could get
secrets. It populates only when the body carries resourceAttributes.
Write conditions against Method.
TRIAGE STARTS WITH SourceAddress
A node's username encodes its own address, and a real kubelet connects
from it:
kubelet SYSTEM:NODE:IP-10-60-11-241... sourceaddress 10.60.11.241
kubelet SYSTEM:NODE:IP-10-60-10-55... sourceaddress 10.60.10.55
THEFT SYSTEM:NODE:IP-10-60-10-55... sourceaddress 10.60.10.189
10.60.10.189 is the POD's address. The attacker holds the node's
credential but is still executing inside a container. To make that field
agree they would have to run in the host network namespace, at which
point they own the node and do not need the credential.
This is NOT a condition, because policies cannot compare one field
against another and the comparison needed is whether the dotted IP in
SourceAddress equals the dashed IP inside AccountName. A CIDR test is not
a substitute: the EKS VPC CNI allocates pod addresses from the same
subnet as the nodes. It is the level 2 attribute on the violation so an
analyst sees it first, and it resolves most triage in one line.
WHAT AN ATTACKER CAN STILL DO
Branch three, the user agent filter, is evadable by setting the user
agent to kubelet/v1.36.4. It is breadth only.
The two shape branches are not evadable without giving up the objective.
What they do NOT cover is a patient attacker who reads one named secret
at a time, which the node authoriser permits for secrets mounted by pods
on that node. That access is legitimate for a kubelet and indistinguishable
from it on shape alone, so only branch three sees it, and only while the
user agent is honest. SourceAddress is the backstop in triage.
FIELDS CONFIRMED POPULATED, FROM INGESTED EVENTS
accountname SYSTEM:NODE:IP-10-60-10-55.US-WEST-2...
sourceusername system:node:ip-10-60-10-55.us-west-2...
sourceaddress, ipaddress 10.60.10.189
method list | watch
requesturl /api/v1/pods?...fieldSelector=spec.nodeName%3D...
requestclientapplication Wget | kubelet/v1.36.4 (linux/amd64) ...
usergroup system:authenticated~system:nodes
sourceservicename secrets | pods
deviceaction forbid | allow
eventoutcome 403 | 200 | 201
message can only list/watch pods with spec.nodeName...
objectname NULL on every node-identity event
And these, present on node-identity events and ABSENT on service account
events, because the parser folds the authenticator correlation into the
audit record:
sessionid i-072109e9e6572f176 the EC2 instance id
arnrole arn:aws:iam::<acct>:role/<cluster>-node-role
arn arn:aws:sts::<acct>:assumed-role/<cluster>-node-role/i-...
accessid ASIAWI7AZSA6FJGHA2S5 the STS access key id
sourceuserid aws-iam-authenticator:<acct>:AROA...
An analyst gets the IAM role, the session and the access key from the
audit event alone, with no join to the authenticator datasource.
THE AUTHENTICATOR LOG CANNOT CARRY THIS DETECTION
Worth recording because it is the obvious place to look. The entry
produced by the stolen credentials is identical to the legitimate
kubelets' in every field except the timestamp, the ephemeral source port,
and which node is named. Same role arn, same groups, same uid, same
method, same path.
It also fires on token VALIDATION rather than use, so a token is
validated once and reused: two API calls as the node produced one
authenticator entry. And the whole cluster emitted three authenticator
records against 4,716 audit events in eight minutes, which is too sparse
for a behavioural analytic and too uniform for a rule.
THE CONTROL, AND WHAT IT SHOWED
A detection validated only against traffic containing no legitimate
instance of what it detects has not been validated. So a second pod was
built into the range with a mounted secret, for no purpose other than to
make the kubelet fetch one, and the run repeated.
The kubelet fetched it when the pod was scheduled:
16:46:58 verb=watch name=feature-store-credentials code=200
ua=kubelet/v1.36.4
A watch, on one named secret. Not a list of the collection.
Every access a node identity made to secrets, configmaps or service
accounts across the cluster's life:
watch configmaps kubelet 8
create serviceaccounts kubelet 7
watch secrets kubelet 1
Sixteen legitimate requests and ZERO lists. This analytic ignored all
sixteen and fired only on the illegitimate one.
So selection_collection_never_granted's true negative is TESTED AND
PASSED, against real kubelet behaviour rather than against a window that
happened to contain none of it.
A CORRECTION, recorded because it nearly became a published finding. An
earlier revision of this file stated that no kubelet secret access
appeared in this datasource at all. That was wrong. The search behind it
used a CloudWatch server side --filter-pattern on the secret's hyphenated
name, which silently returned nothing, and the absence was read as a
result. Pulling the window whole and selecting locally found the record
immediately. The access also happened at pod creation, eleven minutes
before the capture window opened, so the window itself genuinely held
none of it.
Do not use a server side filter pattern on this log group for evidence.
It has returned wrong subsets more than once, and a filter whose result
cannot be verified will report that an event did not happen.
TUNING AND PRE-PROMOTION CHECKS
Both shape branches were validated against six hours of live data.
branch 1 index=activity and resourcegroupname="<rg>" and accountname
CONTAINS "SYSTEM:NODE:" and method="list" and
sourceservicename IN ("secrets","configmaps","serviceaccounts")
-> 1 record, the emulation. No kubelet traffic at all.
branch 2 ... and method IN ("list","watch") and
sourceservicename="pods" and @requesturl not contains
"fieldSelector=spec.nodeName"
-> the emulation's single unscoped list.
Run branch 2 WITHOUT the method constraint and it returns 73 additional
records, all of them the kubelet's named get and patch operations against
individual pods. That is the check that proves the constraint is doing
work, and it is worth repeating in any environment before promotion.
Run both against a window long enough to include a kubelet restart, since
a client-go reflector lists before it watches and a short sample will not
contain one. That list will be scoped, so it should not match, but
confirm it rather than assume.
Components other than the kubelet may legitimately present a node
identity in some distributions. List the distinct
RequestClientApplication values seen with an AccountName beginning
system:node: over an upgrade cycle, and add any legitimate ones to
filter_kubelet.
Policy building walkthrough can be found in this previous post: