Self-managed Kubernetes as a cloud-agnostic alternative to managed AKS

Why not just use managed Kubernetes

The platform ran on managed Kubernetes and ran well. The problem was commercial, not technical: enterprise prospects with data-residency and sovereignty requirements needed the same product running outside a hyperscaler. "We only run on one cloud" was costing deals.

The constraint that shaped everything: the target infrastructure was plain VMs from a provider offering no private network between them. Every node sat on a shared public segment. Managed Kubernetes hands you a private network, a load balancer and an identity system. Without those, each has to be designed.

The security model

Three layers, because no single one was sufficient:

• Admin plane. The Kubernetes API is restricted at the cloud security-group layer and reachable only over VPN. Authentication is never the only thing standing in front of the API server.
• Host. A default-deny firewall on every node, with cluster-internal ports scoped to peer node addresses rather than left open. Because scaling up runs a limited playbook against one new node, the automation pushes that node's firewall entry onto every existing peer directly — otherwise the new node would join a cluster that refuses to talk to it.
• Network. Node-to-node traffic is WireGuard-encrypted. On managed Kubernetes this is optional; when the pod overlay crosses a public segment, it isn't.

Then the harder problem: workloads still needed secrets from the managed cloud secret store, but these VMs have no cloud identity to assume. The answer was to self-host an OIDC issuer — reusing the cluster's own service-account signing key rather than introducing new key material — and register it as a federated identity provider. Workloads authenticate with their own projected service account tokens. No static cloud credentials exist anywhere on these nodes.

The API server runs with both the external and internal issuers accepted, so introducing federation didn't invalidate tokens already held by running pods.

Operations, which is where self-managed actually costs you

Standing a cluster up is the easy part. Keeping it healthy for years is the work, so the automation covers the unglamorous paths:

• Rolling OS patching one node at a time, refusing to start unless every node is healthy, and waiting for the ingress data plane on each node to be ready again before moving on — not merely for the node to report Ready.
• Upgrades that take an etcd snapshot and verify it exists on disk before touching anything. A scheduled snapshot may be hours stale; a pre-upgrade one is not.
• Offsite etcd backup on a timer to a machine outside the cluster, over a restricted key.
• A CNI state drift guard that runs on every converge. When a node leaves and rejoins outside the normal path, it can come back holding pod-IP reservations from its previous address range, which produces intermittent unreachable-pod failures that look like application bugs. The automation detects reservations outside the node's current range and clears them.

That last one is the kind of failure you only automate away after it has cost you an afternoon.

The outcome

The platform runs on bare VMs alongside the managed cluster estate, under the same GitOps control plane and the same delivery pipeline. Provider lock-in stopped being a commercial blocker, and the deployment model is repeatable across development, staging and production.

Ingress evolved across the three deployments — the earlier two use the bundled ingress controller, the newest uses the Gateway API via NGINX Gateway Fabric. Worth naming rather than hiding: platforms of different ages rarely converge on the same answer at the same time.

What I'd do differently

Two things, honestly.

The control plane is single-node. One etcd member, no quorum to recover into. That was a deliberate trade for the size of the workload, mitigated with verified pre-upgrade snapshots and offsite backups — but it means control-plane maintenance is a planned outage rather than a rolling one. A three-member control plane is the real fix.

I forked instead of templating. Each environment got its own copy of the Ansible codebase. That was fast, and it drifted: a node-identity fix landed in one copy across several task files and was never backported to another, and one environment grew playbooks the others lack. Forking bought speed and the bill arrived as divergence. Shared roles parameterized per environment would have cost a day up front and saved considerably more since.