A GitOps control plane managing 1,000+ applications from one root file

The situation

A commercial multi-tenant PaaS serving enterprise clients across Europe and North America. Dozens of services, several tenants, and a growing estate of Kubernetes clusters spanning development, staging and production.

Every service needed deployment manifests in every environment. That meant near-identical YAML duplicated per service, per cluster — and duplicated YAML drifts. Tenant onboarding was manual and slow, deployments diverged between clusters, and the platform team was on call for routine changes that should have been self-service.

The problem with the obvious fix

The usual answer is templating: generate the manifests, commit the output. That trades one duplication problem for a different one — the generated files still live in Git, still drift, and now there are two things to keep in sync instead of one.

What the platform actually needed was for the desired state to be derived, continuously, from a small set of declarations. Add a cluster, and every service that should run there appears. Add a service, and it appears everywhere it belongs. Nothing generated, nothing committed twice.

What I built

A central ArgoCD control plane on the app-of-appsets pattern. A single root Application bootstraps the whole system and fans out through three tiers:

1. Discovery. Git directory generators watch a handful of directories — one per cluster, one per tenant, one per service repo. Adding a directory is the onboarding action; the control plane notices and reconciles.
2. Fan-out. Each service repo carries one declaration listing its services and target clusters. A matrix generator expands that across services × clusters into individual Applications. One file can become dozens of deployments.
3. Reconciliation. Automated sync with prune and self-heal, so a manual change to a live cluster is corrected rather than silently kept.

Configuration is split across two repositories: the control plane holds the ArgoCD objects and knows nothing about application values; a second repo holds Helm values, one directory per cluster. ArgoCD's multi-source Applications stitch them together at sync time. The separation matters operationally — changing a service's memory limit touches the values repo and cannot accidentally restructure the control plane.

Tenancy is enforced through ArgoCD Projects, each generated from one template with four graduated roles: admin, developer (create and sync, but no delete), a read-and-sync role for CI automation, and viewer. Roles map to identity-provider teams by naming convention, so access follows team membership rather than a hand-maintained list.

Sync waves order the bootstrap so dependencies land before the things that need them — secrets tooling, DNS and certificates, then ingress, then workloads. And cluster credentials were removed from the system entirely: instead of storing bearer tokens, the control plane authenticates to each managed cluster via workload identity federation, with connection details materialized from a managed secret store at runtime.

The outcome

The control plane manages 1,000+ applications across the estate from a single ArgoCD instance. Onboarding a new service is one declaration plus a values file — minutes, not a ticket. New tenants get their Project, RBAC and namespace conventions generated from the same template every other tenant uses, so no tenant is a special case.

The less visible win: because desired state is derived rather than written, cluster-to-cluster drift stopped being a category of problem. If a service runs in staging and not production, that is now a visible, reviewable difference in one file.

What I'd do differently

The layering convention that distinguishes platform building blocks from business applications is real and useful, but it lives in directory naming rather than in documentation. A newcomer infers it. I would write that down as an ADR early, because conventions that exist only in file names are conventions that quietly stop being followed.

I would also wire observability into the bootstrap from day one rather than leaving it as a later addition. A control plane that manages a thousand applications should be able to answer "what changed, and did it work" without anyone opening a terminal.