Azure workload identity on self-managed RKE2

Managed Kubernetes hands you workload identity. AKS runs an OIDC issuer for the cluster, publishes its discovery documents, and Entra already trusts them — a pod gets an Azure token and nobody thinks about it.

On plain VMs, none of that exists. The nodes have no Azure identity to assume, and the workloads still need Key Vault and Blob Storage. The default answer at that point is a service principal secret in a Kubernetes Secret: a static cloud credential, sitting on a node, rotated by nobody.

There is a better answer, and the cluster already does most of it. Every Kubernetes cluster signs service-account tokens with a private key. Those tokens are already OIDC-shaped JWTs. Publish the public half over HTTPS, point Entra at it, and the cluster becomes an identity provider it always was.

This is what that took on a production RKE2 cluster running on bare VMs.

What is actually missing

Piece	AKS	Self-managed
Signing key	Cluster has one	Cluster has one — same key
Public discovery endpoint	Provided	You serve it
Trust registered in Entra	Provided	You register it

No new key material is introduced. You are not standing up Dex or Keycloak. You are exposing the public half of a key RKE2 already uses to sign every service-account token in the cluster.

Serving discovery from the cluster itself

Entra fetches the discovery document and the JWKS over the public internet when it validates a token. It has no VPN and no route into your network — and on this cluster the Kubernetes API is restricted at the provider firewall and reachable only over VPN, so the API server cannot serve them.

The common workaround is a public storage bucket. The alternative — and what this cluster does — is to serve discovery from the cluster itself: an nginx:alpine pod in an oidc-issuer namespace, serving two files out of ConfigMaps. One fewer external dependency, and one fewer place for the JWKS to go stale.

server {
    listen 8080;

    location = /.well-known/openid-configuration {
        default_type application/json;
        alias /etc/oidc/openid-configuration.json;
    }

    location = /openid/v1/jwks {
        default_type application/json;
        alias /etc/oidc/jwks.json;
    }
}

Two static files and no application code. The discovery document is five fields:

{
  "issuer": "https://oidc.example.com/",
  "jwks_uri": "https://oidc.example.com/openid/v1/jwks",
  "response_types_supported": ["id_token"],
  "subject_types_supported": ["public"],
  "id_token_signing_alg_values_supported": ["RS256"]
}

The trailing slash on issuer is load-bearing. It must match, character for character, the service-account-issuer on the API server and the issuer on every federated credential. Mismatch it and Entra returns an error about no matching federated identity record, which reads like a subject problem and is not.

This cluster runs no ingress controller — NGINX Gateway Fabric owns ports 80 and 443 — so the endpoint is exposed with an HTTPRoute, not an Ingress:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: oidc-issuer
  namespace: oidc-issuer
spec:
  parentRefs:
    - name: ngf-gateway
      namespace: nginx-gateway
  hostnames:
    - oidc.example.com
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - name: oidc-issuer
          port: 80

Building the JWKS from RKE2's own key

RKE2 keeps the service-account signing key at /var/lib/rancher/rke2/server/tls/service.current.key. Extract the public half, and build a JWKS with azwi:

# Public key only — the private key never leaves the node
openssl rsa -in service.current.key -pubout -out sa.pub

azwi jwks --public-keys sa.pub --output-file jwks.json

In the automation, the private key is slurped from the master, written to a 0600 temp file purely so openssl can read it, and deleted in the next task. It has no reason to exist on an operator's laptop for longer than one command.

TLS: the cert-manager flag that is not optional

Entra requires HTTPS for the issuer, with a publicly valid certificate. A self-signed cert fails validation.

cert-manager solves that with Let's Encrypt — but because there is no ingress controller, the HTTP-01 challenge has to be solved through the Gateway:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: platform@example.com
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
      - http01:
          gatewayHTTPRoute:
            parentRefs:
              - name: ngf-gateway
                namespace: nginx-gateway
                kind: Gateway

The non-obvious part: Gateway API support in cert-manager needs the --enable-gateway-api controller flag on the deployment. Setting the ExperimentalGatewayAPISupport feature gate alone is not enough — the ACME challenge fails with gateway api is not enabled, which sounds like a cluster-level problem and is a missing argument.

kubectl -n cert-manager get deploy cert-manager \
  -o jsonpath='{.spec.template.spec.containers[0].args}'

That must contain --enable-gateway-api, and the startup log must list gateway-shim under enabled controllers — not skipping disabled controller controller="gateway-shim". After adding the flag, delete the stuck Certificate, Order and Challenge so cert-manager re-issues; it will not retry a wedged order on its own.

One more guard, before any of this runs. The automation resolves the OIDC hostname against a public resolver and asserts it points at the master:

- name: Verify OIDC domain resolves to this master before touching cert/ingress
  ansible.builtin.assert:
    that:
      - rke2_wi_resolved_ip == ansible_host
    fail_msg: >-
      {{ rke2_oidc_domain }} resolves to {{ rke2_wi_resolved_ip }},
      expected {{ ansible_host }}. Fix DNS before continuing.

Let's Encrypt rate-limits failed authorizations. A DNS record that has not propagated burns attempts you may want later the same day, and the failure surfaces minutes after the mistake rather than at the point of it.

The dual issuer, and the outage it prevents

This is the step that decides whether enabling federation is a maintenance window or a blip.

service-account-issuer can be passed more than once. The first value mints new tokens; all values are accepted when validating:

# /etc/rancher/rke2/config.yaml
kube-apiserver-arg:
  - "service-account-issuer=https://oidc.example.com/"
  - "service-account-issuer=https://kubernetes.default.svc.cluster.local"

The external issuer is primary, so new projected tokens carry the iss Entra expects. RKE2's default issuer stays accepted, so every pod that started before the switch keeps authenticating.

Drop that second line and the API server 401s every existing service-account token the moment it restarts. Controllers crash-loop with the server has asked for the client to provide credentials, and it presents as a cluster-wide outage — because it is one.

Changing this line restarts rke2-server on the master: roughly a one to two minute API blip. Running pods and ingress traffic are unaffected.

Confirm the change took:

kubectl create token default --duration=10m \
  | cut -d. -f2 | base64 -d 2>/dev/null | jq '{iss, aud, sub}'

The Azure side, once

The azure-workload-identity webhook mutates any pod labelled azure.workload.identity/use: "true", injecting the projected token volume and the AZURE_* environment variables. Install it with the tenant ID and the pod spec stays clean.

Then, per workload identity:

az identity federated-credential create \
  --name eso-keyvault \
  --identity-name "$MANAGED_IDENTITY" \
  --resource-group "$RG" \
  --issuer "https://oidc.example.com/" \
  --subject "system:serviceaccount:external-secrets:external-secrets" \
  --audiences "api://AzureADTokenExchange"

Adding a second workload is four steps and no secrets: App Registration or managed identity, a federated credential whose subject is system:serviceaccount:<ns>:<sa>, the Azure role assignment, then annotate the service account with azure.workload.identity/client-id and label the pod.

Do not wildcard the subject. One credential per service account. Otherwise any service account created in that namespace inherits access to your production Key Vault.

External Secrets Operator, with no bootstrap secret

Key Vault was the first consumer. ESO speaks workload identity natively, so the operator whose job is eliminating secrets no longer needs one of its own to start:

apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: azure-keyvault-app-dev
spec:
  provider:
    azurekv:
      authType: WorkloadIdentity
      vaultUrl: "https://<vault>.vault.azure.net"
      serviceAccountRef:
        name: external-secrets
        namespace: external-secrets

The same credential reaches any Azure resource behind Entra — Blob Storage, Service Bus, Cosmos. DefaultAzureCredential picks up the injected environment with no code change. Federation is configured once per workload, not once per Azure service.

When it does not work

InvalidProviderConfig, or a 400 from login.microsoftonline.com. Check, in order: the federated credential's subject is exactly system:serviceaccount:<ns>:<sa>; the issuer matches including the trailing slash; discovery returns 200 over valid HTTPS from outside your network:

curl -s https://oidc.example.com/.well-known/openid-configuration | jq .

The service account is missing its annotations. The webhook injects nothing without azure.workload.identity/client-id and tenant-id on the SA. Roll the deployment after adding them — the mutation happens at pod creation, so existing pods keep the old spec.

Signing keys rotated and the JWKS went stale. This one bites months later. Re-publishing the JWKS belongs in the same automation that rotates the key, not in a runbook.

Verify end to end with a real ExternalSecret against a known Key Vault entry and check status.conditions reports Ready=True / SecretSynced. A ClusterSecretStore can report healthy while no secret has actually been fetched.

What this does not solve

One static credential survives: the container registry.

Pulling images from a private registry is containerd's job, and containerd takes a static username and password. There is no federated path — the AKS-style keyless mechanism needs the nodes to be Azure VMs with IMDS, and these are not.

Worth stating plainly rather than implying the cluster is credential-free. It is not. It has exactly one static credential, in one place, with one narrow purpose, instead of a service principal secret in every namespace that needs a config value.

Why it is worth the day

No static cloud credential exists for any workload on these nodes. Not in a Secret, not in a config file, not in a CI variable. A compromised pod yields a token scoped to one service account, valid for an hour, useless anywhere else.

That property is what makes a self-managed cluster on bare VMs defensible to a security review that expects managed Kubernetes. The infrastructure is more exposed; the credential posture is better.

Managed Kubernetes gives you this for free, and free is the right price when it is available. When it is not — data residency, sovereignty, a provider with no managed offering — this is about a day of work, and it is the difference between "we handle secrets properly" and a service principal password in a YAML file somewhere.