On plain VMs, none of that exists. The nodes have no Azure identity to assume, and the workloads still need Key Vault and Blob Storage. The default answer at that point is a service principal secret in a Kubernetes Secret: a static cloud credential, sitting on a node, rotated by nobody.

There is a better answer, and the cluster already does most of it. Every Kubernetes cluster signs service-account tokens with a private key. Those tokens are already OIDC-shaped JWTs. Publish the public half over HTTPS, point Entra at it, and the cluster becomes an identity provider it always was.

This is what that took on a production RKE2 cluster running on bare VMs.

What is actually missing​

No new key material is introduced. You are not standing up Dex or Keycloak. You are exposing the public half of a key RKE2 already uses to sign every service-account token in the cluster.

Serving discovery from the cluster itself​

Entra fetches the discovery document and the JWKS over the public internet when it validates a token. It has no VPN and no route into your network — and on this cluster the Kubernetes API is restricted at the provider firewall and reachable only over VPN, so the API server cannot serve them.

The common workaround is a public storage bucket. The alternative — and what this cluster does — is to serve discovery from the cluster itself: an nginx:alpine pod in an oidc-issuer namespace, serving two files out of ConfigMaps. One fewer external dependency, and one fewer place for the JWKS to go stale.

server {
    listen 8080;

    location = /.well-known/openid-configuration {
        default_type application/json;
        alias /etc/oidc/openid-configuration.json;
    }

    location = /openid/v1/jwks {
        default_type application/json;
        alias /etc/oidc/jwks.json;
    }
}

Two static files and no application code. The discovery document is five fields:

{
  "issuer": "https://oidc.example.com/",
  "jwks_uri": "https://oidc.example.com/openid/v1/jwks",
  "response_types_supported": ["id_token"],
  "subject_types_supported": ["public"],
  "id_token_signing_alg_values_supported": ["RS256"]
}

The trailing slash on issuer is load-bearing. It must match, character for character, the service-account-issuer on the API server and the issuer on every federated credential. Mismatch it and Entra returns an error about no matching federated identity record, which reads like a subject problem and is not.

This cluster runs no ingress controller — NGINX Gateway Fabric owns ports 80 and 443 — so the endpoint is exposed with an HTTPRoute, not an Ingress:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: oidc-issuer
  namespace: oidc-issuer
spec:
  parentRefs:
    - name: ngf-gateway
      namespace: nginx-gateway
  hostnames:
    - oidc.example.com
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - name: oidc-issuer
          port: 80

Building the JWKS from RKE2's own key​

RKE2 keeps the service-account signing key at /var/lib/rancher/rke2/server/tls/service.current.key. Extract the public half, and build a JWKS with azwi:

# Public key only — the private key never leaves the node
openssl rsa -in service.current.key -pubout -out sa.pub

azwi jwks --public-keys sa.pub --output-file jwks.json

In the automation, the private key is slurped from the master, written to a 0600 temp file purely so openssl can read it, and deleted in the next task. It has no reason to exist on an operator's laptop for longer than one command.

TLS: the cert-manager flag that is not optional​

Entra requires HTTPS for the issuer, with a publicly valid certificate. A self-signed cert fails validation.

cert-manager solves that with Let's Encrypt — but because there is no ingress controller, the HTTP-01 challenge has to be solved through the Gateway:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: platform@example.com
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
      - http01:
          gatewayHTTPRoute:
            parentRefs:
              - name: ngf-gateway
                namespace: nginx-gateway
                kind: Gateway

The non-obvious part: Gateway API support in cert-manager needs the --enable-gateway-api controller flag on the deployment. Setting the ExperimentalGatewayAPISupport feature gate alone is not enough — the ACME challenge fails with gateway api is not enabled, which sounds like a cluster-level problem and is a missing argument.

kubectl -n cert-manager get deploy cert-manager \
  -o jsonpath='{.spec.template.spec.containers[0].args}'

That must contain --enable-gateway-api, and the startup log must list gateway-shim under enabled controllers — not skipping disabled controller controller="gateway-shim". After adding the flag, delete the stuck Certificate, Order and Challenge so cert-manager re-issues; it will not retry a wedged order on its own.

One more guard, before any of this runs. The automation resolves the OIDC hostname against a public resolver and asserts it points at the master:

- name: Verify OIDC domain resolves to this master before touching cert/ingress
  ansible.builtin.assert:
    that:
      - rke2_wi_resolved_ip == ansible_host
    fail_msg: >-
      {{ rke2_oidc_domain }} resolves to {{ rke2_wi_resolved_ip }},
      expected {{ ansible_host }}. Fix DNS before continuing.

Let's Encrypt rate-limits failed authorizations. A DNS record that has not propagated burns attempts you may want later the same day, and the failure surfaces minutes after the mistake rather than at the point of it.

The dual issuer, and the outage it prevents​

This is the step that decides whether enabling federation is a maintenance window or a blip.

service-account-issuer can be passed more than once. The first value mints new tokens; all values are accepted when validating:

# /etc/rancher/rke2/config.yaml
kube-apiserver-arg:
  - "service-account-issuer=https://oidc.example.com/"
  - "service-account-issuer=https://kubernetes.default.svc.cluster.local"

The external issuer is primary, so new projected tokens carry the iss Entra expects. RKE2's default issuer stays accepted, so every pod that started before the switch keeps authenticating.

Drop that second line and the API server 401s every existing service-account token the moment it restarts. Controllers crash-loop with the server has asked for the client to provide credentials, and it presents as a cluster-wide outage — because it is one.

Changing this line restarts rke2-server on the master: roughly a one to two minute API blip. Running pods and ingress traffic are unaffected.

Confirm the change took:

kubectl create token default --duration=10m \
  | cut -d. -f2 | base64 -d 2>/dev/null | jq '{iss, aud, sub}'

The Azure side, once​

The azure-workload-identity webhook mutates any pod labelled azure.workload.identity/use: "true", injecting the projected token volume and the AZURE_* environment variables. Install it with the tenant ID and the pod spec stays clean.

Then, per workload identity:

az identity federated-credential create \
  --name eso-keyvault \
  --identity-name "$MANAGED_IDENTITY" \
  --resource-group "$RG" \
  --issuer "https://oidc.example.com/" \
  --subject "system:serviceaccount:external-secrets:external-secrets" \
  --audiences "api://AzureADTokenExchange"

Adding a second workload is four steps and no secrets: App Registration or managed identity, a federated credential whose subject is system:serviceaccount:<ns>:<sa>, the Azure role assignment, then annotate the service account with azure.workload.identity/client-id and label the pod.

Do not wildcard the subject. One credential per service account. Otherwise any service account created in that namespace inherits access to your production Key Vault.

External Secrets Operator, with no bootstrap secret​

Key Vault was the first consumer. ESO speaks workload identity natively, so the operator whose job is eliminating secrets no longer needs one of its own to start:

apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: azure-keyvault-app-dev
spec:
  provider:
    azurekv:
      authType: WorkloadIdentity
      vaultUrl: "https://<vault>.vault.azure.net"
      serviceAccountRef:
        name: external-secrets
        namespace: external-secrets

The same credential reaches any Azure resource behind Entra — Blob Storage, Service Bus, Cosmos. DefaultAzureCredential picks up the injected environment with no code change. Federation is configured once per workload, not once per Azure service.

When it does not work​

InvalidProviderConfig, or a 400 from login.microsoftonline.com. Check, in order: the federated credential's subject is exactly system:serviceaccount:<ns>:<sa>; the issuer matches including the trailing slash; discovery returns 200 over valid HTTPS from outside your network:

curl -s https://oidc.example.com/.well-known/openid-configuration | jq .

The service account is missing its annotations. The webhook injects nothing without azure.workload.identity/client-id and tenant-id on the SA. Roll the deployment after adding them — the mutation happens at pod creation, so existing pods keep the old spec.

Signing keys rotated and the JWKS went stale. This one bites months later. Re-publishing the JWKS belongs in the same automation that rotates the key, not in a runbook.

Verify end to end with a real ExternalSecret against a known Key Vault entry and check status.conditions reports Ready=True / SecretSynced. A ClusterSecretStore can report healthy while no secret has actually been fetched.

What this does not solve​

One static credential survives: the container registry.

Pulling images from a private registry is containerd's job, and containerd takes a static username and password. There is no federated path — the AKS-style keyless mechanism needs the nodes to be Azure VMs with IMDS, and these are not.

Worth stating plainly rather than implying the cluster is credential-free. It is not. It has exactly one static credential, in one place, with one narrow purpose, instead of a service principal secret in every namespace that needs a config value.

Why it is worth the day​

No static cloud credential exists for any workload on these nodes. Not in a Secret, not in a config file, not in a CI variable. A compromised pod yields a token scoped to one service account, valid for an hour, useless anywhere else.

That property is what makes a self-managed cluster on bare VMs defensible to a security review that expects managed Kubernetes. The infrastructure is more exposed; the credential posture is better.

Managed Kubernetes gives you this for free, and free is the right price when it is available. When it is not — data residency, sovereignty, a provider with no managed offering — this is about a day of work, and it is the difference between "we handle secrets properly" and a service principal password in a YAML file somewhere.

The migration is genuinely worth doing. It also fails, almost always, at exactly one step.

What is actually happening​

Four parties, in order:

1. Your CI job asks its provider for an identity token.
2. The CI provider issues a signed JWT describing the job: which repository, which branch, which environment.
3. Your cloud verifies that signature against the provider's public keys, then checks the claims inside against a trust policy you configured.
4. If the claims match, the cloud issues short-lived credentials.

Step 4 is where people expect trouble. Step 3 is where trouble actually is.

The step that breaks it​

The trust policy matches on the token's subject claim (sub). That claim is a structured string, and its exact shape depends on what triggered the workflow.

For GitHub Actions, the same repository produces different subjects depending on context:

repo:my-org/my-repo:ref:refs/heads/main          # push to main
repo:my-org/my-repo:pull_request                 # pull request
repo:my-org/my-repo:environment:production       # job with an environment
repo:my-org/my-repo:ref:refs/tags/v1.2.3         # tag push

Configure the trust policy for ref:refs/heads/main, and everything works. Then someone opens a pull request, and the deploy job fails — because that job's token now says pull_request, which matches nothing in your policy.

The failure arrives as a permission error. That is misleading: permissions are fine, the role was never assumed. You will spend an hour widening IAM policies that were correct from the start.

Print the claim before you debug anything else.

permissions:
  id-token: write     # without this, no token is issued at all
  contents: read

steps:
  - name: Show the actual subject claim
    run: |
      TOKEN=$(curl -sS \
        -H "Authorization: bearer $ACTIONS_ID_TOKEN_REQUEST_TOKEN" \
        "$ACTIONS_ID_TOKEN_REQUEST_URL&audience=api://AzureADTokenExchange" \
        | jq -r '.value')
      echo "$TOKEN" | cut -d. -f2 | base64 -d 2>/dev/null | jq '{sub, aud, repository, ref}'

That prints what your cloud will see. Match the trust policy to that string, not to the one in the documentation example.

Configuring the trust side​

AWS, with a condition per context you intend to allow:

{
  "Effect": "Allow",
  "Principal": {
    "Federated": "arn:aws:iam::123456789012:oidc-provider/token.actions.githubusercontent.com"
  },
  "Action": "sts:AssumeRoleWithWebIdentity",
  "Condition": {
    "StringEquals": {
      "token.actions.githubusercontent.com:aud": "sts.amazonaws.com"
    },
    "StringLike": {
      "token.actions.githubusercontent.com:sub": "repo:my-org/my-repo:environment:production"
    }
  }
}

Azure, as a federated credential on an app registration:

az ad app federated-credential create \
  --id "$APP_OBJECT_ID" \
  --parameters '{
    "name": "github-prod",
    "issuer": "https://token.actions.githubusercontent.com",
    "subject": "repo:my-org/my-repo:environment:production",
    "audiences": ["api://AzureADTokenExchange"]
  }'

One credential per subject. A pull-request workflow and a production deploy are two different identities, and that is the feature, not an inconvenience — it is what stops a PR from a fork reaching production.

Three more things that will catch you​

permissions: id-token: write is required, and its absence is silent. Without it the runner cannot request a token at all. Worse, setting any permissions: block replaces the defaults rather than extending them, so adding id-token: write on its own can quietly remove the contents: read your checkout needed.

Wildcards in sub are a security decision, not a convenience. repo:my-org/my-repo:* will make your pipeline pass. It also means any branch, in that repo, from anyone who can push one, can assume a production role. If you need several contexts, write several credentials.

Forked pull requests do not get a token. By design — a fork could otherwise mint credentials against your cloud. Any workflow that needs cloud access on PRs has to run under pull_request_target, with all the care that carries, or not run on forks at all.

Do it in this order​

1. Add the federated credential for one workflow, in a non-production account.
2. Run it and print the subject claim. Fix the mismatch.
3. Add credentials for the remaining contexts, one per context.
4. Run both auth paths in parallel — OIDC and the old key — for one release cycle.
5. Delete the static key. If you skip this, you have added a system and removed no risk.

Step 5 is the one that gets postponed indefinitely. The migration only pays for itself when the old credential no longer exists.

A comprehensive learning resource for DevOps professionals, security teams, and developers new to automated security scanning

📋 Table of Contents​

1. Introduction to DevSecOps
2. Your Current Pipeline Analysis
3. Core Security Scanning Types
4. DevSecOps Tools Deep Dive
5. Security Trends & Best Practices
6. Implementation Guidelines
7. Pipeline Enhancement Recommendations
8. Metrics & Monitoring
9. Troubleshooting & Common Issues
10. Future-Proofing Your Security

🔐 Introduction to DevSecOps​

What is DevSecOps?​

DevSecOps (Development, Security, Operations) is a cultural and technical approach that integrates security practices throughout the software development lifecycle (SDLC). Unlike traditional approaches where security is an afterthought, DevSecOps embeds security from the very beginning.

🎯 Key Principles​

🚀 Benefits of DevSecOps​

🔍 Your Current Pipeline Analysis​

Based on your GitHub Actions workflow, here's an analysis of your current security implementation:

Current Security Tools in Your Pipeline​

Security Coverage Assessment​

🛡️ Core Security Scanning Types​

SAST vs DAST vs SCA vs IAST Comparison​

1. Static Application Security Testing (SAST)​

When to Use: Early in development, during code commits and builds

Tools in Your Pipeline: SonarQube

What it Does:

• Analyzes source code without executing it
• Identifies vulnerabilities like SQL injection, XSS, buffer overflows
• Provides exact line numbers and fix recommendations

Best Practices:

# Example SonarQube configuration enhancement
sonar.security.hotspots.maxFileSize=1000000
sonar.security.review.rating.A=80
sonar.security.review.rating.B=70
sonar.security.review.rating.C=50
sonar.security.review.rating.D=30

2. Dynamic Application Security Testing (DAST)​

When to Use: After application deployment, in testing environments

Current Gap: Missing from your pipeline

What it Does:

• Tests running applications from external perspective
• Simulates real-world attacks
• Finds runtime vulnerabilities missed by SAST

Recommended Tools:

• OWASP ZAP (Open source)
• Burp Suite (Commercial)
• Rapid7 AppSpider

3. Software Composition Analysis (SCA)​

When to Use: Throughout development lifecycle

Tools in Your Pipeline: OWASP Dependency Check, OWASP Dependency Track

What it Does:

• Scans third-party libraries and dependencies
• Identifies known vulnerabilities (CVEs)
• Tracks license compliance
• Monitors supply chain security

4. Container Security Scanning​

When to Use: Before and after container deployment

Tools in Your Pipeline: Trivy

What it Does:

• Scans container images for vulnerabilities
• Checks base image security
• Identifies misconfigurations
• Monitors runtime behavior

🔧 DevSecOps Tools Deep Dive​

Secret Detection: Gitleaks​

Purpose: Prevents sensitive information leaks in repositories

Configuration Example:

# .gitleaks.toml
[extend]
useDefault = true

[[rules]]
description = "Custom API Key Pattern"
regex = '''(?i)api[_-]?key[_-]?=?['"]\s?[0-9a-zA-Z]{32,}'''
tags = ["key", "API"]

[allowlist]
paths = [
    '''go\.sum''',
    '''\.git/'''
]

Best Practices:

• Run as pre-commit hook
• Scan entire Git history
• Use custom regex patterns for organization-specific secrets
• Integrate with CI/CD pipeline

Dependency Management: OWASP Tools​

OWASP Dependency Check​

Purpose: Identifies known vulnerabilities in dependencies

Enhanced Configuration:

<plugin>
    <groupId>org.owasp</groupId>
    <artifactId>dependency-check-maven</artifactId>
    <version>8.4.0</version>
    <configuration>
        <failBuildOnCVSS>7</failBuildOnCVSS>
        <suppressionFile>suppressions.xml</suppressionFile>
        <format>ALL</format>
        <nodeAnalyzerEnabled>true</nodeAnalyzerEnabled>
        <ossindexAnalyzerEnabled>true</ossindexAnalyzerEnabled>
    </configuration>
</plugin>

OWASP Dependency Track​

Purpose: Continuous monitoring of component risks

Features:

• Software Bill of Materials (SBOM) management
• Risk assessment and scoring
• Policy enforcement
• Vulnerability notifications

Code Quality & Security: SonarQube​

Enhanced Security Rules Configuration:

# sonar-project.properties
sonar.security.hotspots.inherit=NONE
sonar.security.review.rating=A
sonar.coverage.exclusions=**/test/**,**/config/**
sonar.security.inclusions=**/src/**

# Custom security rules
sonar.java.checkstyle.reportPaths=target/checkstyle-result.xml
sonar.java.pmd.reportPaths=target/pmd.xml
sonar.java.spotbugs.reportPaths=target/spotbugsXml.xml

Container Security: Trivy​

Enhanced Configuration:

# Enhanced Trivy scanning
- name: Run Trivy vulnerability scanner
  uses: aquasecurity/trivy-action@master
  with:
    image-ref: ${{ env.REGISTRY }}:latest
    format: 'cyclonedx'
    output: 'trivy-results.json'
    severity: 'CRITICAL,HIGH,MEDIUM'
    ignore-unfixed: false
    scanners: 'vuln,secret,misconfig,license'
    
- name: Upload Trivy scan results to GitHub Security tab
  uses: github/codeql-action/upload-sarif@v2
  with:
    sarif_file: 'trivy-results.sarif'

📈 Security Trends & Best Practices​

Key Trends Shaping DevSecOps​

Modern DevSecOps emphasizes automation, AI-driven security, and policy-as-code frameworks, with organizations adopting shift-left security practices and continuous monitoring to ensure that security vulnerabilities are identified and addressed during the coding phase rather than post-deployment.

1. Shift-Everywhere Security​

2. AI-Powered Security Analysis​

AI and machine learning are transforming DevSecOps by making security smarter and faster, with generative AI becoming the backbone of automation to help teams test more efficiently and zero in on potential risks.

AI-Enhanced Features:

• Intelligent vulnerability prioritization
• Automated fix suggestions
• Anomaly detection in runtime
• False positive reduction
• Predictive threat modeling

3. Policy-as-Code Implementation​

Organizations are turning to policy as code (PaC) where security policies are written as code, allowing for automated enforcement and real-time validation within the DevSecOps pipeline.

Example OPA Policy:

# security-policy.rego
package main

# Deny containers running as root
deny[msg] {
    input.spec.securityContext.runAsUser == 0
    msg := "Container must not run as root user"
}

# Require security scanning labels
deny[msg] {
    not input.metadata.labels["security-scan"]
    msg := "Container must have security-scan label"
}

Security Tool Categories​

🛠️ Implementation Guidelines​

Phase 1: Foundation (Weeks 1-2)​

1. Enhance Secret Scanning​

# Add to your pipeline
- name: Install Gitleaks
  run: |
    wget https://github.com/gitleaks/gitleaks/releases/download/v8.18.0/gitleaks_8.18.0_linux_x64.tar.gz
    tar -xzf gitleaks_8.18.0_linux_x64.tar.gz
    chmod +x gitleaks

- name: Run Gitleaks with custom rules
  run: |
    ./gitleaks detect --source . --config .gitleaks.toml --report-format json --report-path gitleaks-report.json

2. Improve SonarQube Configuration​

- name: Enhanced SonarQube Analysis
  uses: SonarSource/sonarqube-scan-action@master
  env:
    SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}
    SONAR_HOST_URL: ${{ vars.SONAR_HOST_URL }}
  with:
    args: >
      -Dsonar.projectKey=springboot-project
      -Dsonar.projectName=springboot-project
      -Dsonar.sources=src/main/java
      -Dsonar.tests=src/test/java
      -Dsonar.java.binaries=target/classes
      -Dsonar.coverage.jacoco.xmlReportPaths=target/site/jacoco/jacoco.xml
      -Dsonar.dependencyCheck.jsonReportPath=target/dependency-check-report.json
      -Dsonar.security.hotspots.inherit=NONE
      -Dsonar.qualitygate.wait=true

Phase 2: Infrastructure Security (Weeks 3-4)​

1. Add Infrastructure as Code Scanning​

iac-security:
  needs: checkout
  runs-on: ubuntu-latest
  steps:
    - uses: actions/checkout@v4
    
    - name: Run Checkov
      uses: bridgecrewio/checkov-action@master
      with:
        directory: .
        framework: terraform,dockerfile,kubernetes
        output_format: sarif
        output_file_path: checkov-results.sarif
        
    - name: Upload Checkov results to GitHub Security tab
      uses: github/codeql-action/upload-sarif@v2
      with:
        sarif_file: checkov-results.sarif

2. Add Kubernetes Security Scanning​

k8s-security:
  needs: build
  runs-on: ubuntu-latest
  steps:
    - uses: actions/checkout@v4
    
    - name: Run Kubesec
      run: |
        docker run -v $(pwd):/app kubesec/kubesec:latest scan /app/k8s/*.yaml

Phase 3: API Security (Weeks 5-6)​

1. Add DAST Scanning​

dast-scan:
  needs: smoke-test
  runs-on: ubuntu-latest
  steps:
    - name: Checkout code
      uses: actions/checkout@v4
      
    - name: Run OWASP ZAP Baseline Scan
      uses: zaproxy/action-baseline@v0.7.0
      with:
        target: 'http://localhost:8080'
        rules_file_name: '.zap/rules.tsv'
        cmd_options: '-a'

2. API Security Testing​

api-security:
  needs: dast-scan
  runs-on: ubuntu-latest
  steps:
    - name: API Security Scan
      run: |
        docker run --rm -v $(pwd):/app \
          -e TARGET_URL=http://localhost:8080/api \
          pynt-io/pynt:latest scan

Phase 4: Runtime Security (Weeks 7-8)​

1. Add Runtime Monitoring​

runtime-security:
  if: github.ref == 'refs/heads/main'
  needs: deploy
  runs-on: ubuntu-latest
  steps:
    - name: Deploy Falco Runtime Security
      run: |
        kubectl apply -f https://raw.githubusercontent.com/falcosecurity/falco/master/examples/k8s_audit_config/audit-policy.yaml
        helm repo add falcosecurity https://falcosecurity.github.io/charts
        helm install falco falcosecurity/falco

🎯 Pipeline Enhancement Recommendations​

Enhanced Security Pipeline Architecture​

Security Gates and Quality Criteria​

Tool Integration Matrix​

📊 Metrics & Monitoring​

Key Security Metrics to Track​

1. Vulnerability Metrics​

# Example metrics configuration
security_metrics:
  vulnerability_density:
    formula: "total_vulnerabilities / lines_of_code * 1000"
    target: "< 1 vulnerability per 1000 LoC"
    
  mean_time_to_fix:
    formula: "sum(fix_time) / count(vulnerabilities)"
    target: "< 48 hours for critical, < 7 days for high"
    
  security_debt:
    formula: "open_vulnerabilities * severity_weight"
    target: "< 100 security debt points"

2. Pipeline Security Metrics​

3. Security Dashboard Configuration​

# Grafana dashboard configuration example
security_dashboard:
  panels:
    - title: "Vulnerability Trends"
      type: "graph"
      targets:
        - expr: 'vulnerability_count{severity="critical"}'
        - expr: 'vulnerability_count{severity="high"}'
    
    - title: "Security Scan Results"
      type: "stat"
      targets:
        - expr: 'scan_success_rate'
        - expr: 'scan_duration_avg'
    
    - title: "Top Vulnerable Components"
      type: "table"
      targets:
        - expr: 'topk(10, vulnerability_count by (component))'

Monitoring and Alerting​

1. Critical Alert Rules​

# Prometheus alerting rules
groups:
  - name: security.rules
    rules:
      - alert: CriticalVulnerabilityFound
        expr: vulnerability_count{severity="critical"} > 0
        for: 0m
        labels:
          severity: critical
        annotations:
          summary: "Critical vulnerability detected in {{ $labels.component }}"
          
      - alert: SecurityScanFailure
        expr: security_scan_success_rate < 0.95
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Security scan success rate below threshold"

2. Integration with Incident Response​

🚨 Troubleshooting & Common Issues​

Common Security Tool Issues​

1. High False Positive Rates​

Problem: Security tools generating too many false positives Solution:

# SonarQube false positive management
- name: Configure SonarQube exclusions
  run: |
    echo "sonar.security.excludeUnknown=true" >> sonar-project.properties
    echo "sonar.exclusions=**/test/**,**/mock/**" >> sonar-project.properties

Trivy False Positive Suppression:

# .trivyignore
CVE-2023-12345  # False positive - patched in our base image
CVE-2023-67890  # Not applicable to our use case

2. Performance Issues​

Problem: Security scans taking too long Solutions:

# Parallel security scanning
security-scans:
  strategy:
    matrix:
      tool: [sast, sca, container, secrets]
  runs-on: ubuntu-latest
  steps:
    - name: Run ${{ matrix.tool }} scan
      run: |
        case ${{ matrix.tool }} in
          sast) run-sonarqube-scan ;;
          sca) run-dependency-check ;;
          container) run-trivy-scan ;;
          secrets) run-gitleaks-scan ;;
        esac

Incremental Scanning:

# Only scan changed files
- name: Get changed files
  id: changed-files
  uses: tj-actions/changed-files@v37
  
- name: Run incremental SAST
  if: steps.changed-files.outputs.any_changed == 'true'
  run: |
    sonar-scanner -Dsonar.inclusions="${{ steps.changed-files.outputs.all_changed_files }}"

3. Integration Failures​

Problem: Tools not integrating properly with CI/CD Debug Steps:

1. Check Tool Versions

- name: Debug tool versions
  run: |
    echo "SonarQube Scanner version:"
    sonar-scanner --version
    echo "Trivy version:"
    trivy --version
    echo "Maven version:"
    mvn --version

2. Validate Configuration

- name: Validate security tool configs
  run: |
    # Validate SonarQube connection
    curl -u ${{ secrets.SONAR_TOKEN }}: ${{ vars.SONAR_HOST_URL }}/api/system/status
    
    # Test OWASP Dependency Track connection
    curl -X GET "${{ vars.OWASP_DTRACK_HOST_URL }}/api/version" \
      -H "X-Api-Key: ${{ secrets.OWASP_DTRACK_KEY }}"

Security Tool Configuration Best Practices​

1. Error Handling and Retry Logic​

- name: Run security scan with retry
  uses: nick-invision/retry@v2
  with:
    timeout_minutes: 10
    max_attempts: 3
    retry_on: error
    command: |
      trivy image --exit-code 1 --severity HIGH,CRITICAL ${{ env.REGISTRY }}:latest

2. Graceful Degradation​

- name: Security scan with fallback
  run: |
    # Try primary scanner
    if ! trivy image ${{ env.REGISTRY }}:latest; then
      echo "Primary scanner failed, trying backup"
      # Use alternative scanner or skip with warning
      echo "⚠️ Security scan failed - manual review required"
      exit 0
    fi
  continue-on-error: true

🔮 Future-Proofing Your Security​

Emerging Technologies to Watch​

1. AI-Powered Security Tools​

Generative AI is moving DevSecOps from shift-left to shift-everywhere, with AI-assisted tooling helping teams test more efficiently and focus on the risks that matter.

Integration Example:

ai-security-analysis:
  runs-on: ubuntu-latest
  steps:
    - name: AI-Powered Vulnerability Analysis
      uses: codacy/codacy-analysis-cli-action@master
      with:
        project-token: ${{ secrets.CODACY_PROJECT_TOKEN }}
        upload: true
        max-allowed-issues: 2147483647

2. Supply Chain Security Evolution​

SLSA (Supply-chain Levels for Software Artifacts) Implementation:

- name: Generate SLSA Provenance
  uses: slsa-framework/slsa-github-generator/.github/workflows/generator_generic_slsa3.yml@v1.7.0
  with:
    base64-subjects: "${{ steps.hash.outputs.hashes }}"

Sigstore Integration:

- name: Sign container image
  uses: sigstore/cosign-installer@v3.0.5
  
- name: Sign the published Docker image
  run: |
    cosign sign --yes ${{ env.REGISTRY }}:${{ github.sha }}

3. Zero Trust Architecture Implementation​

Roadmap for Next 6 Months​

Quarter 1: Foundation Enhancement​

• Implement pre-commit security hooks
• Add IaC security scanning (Checkov/Terrascan)
• Enhance secret management with HashiCorp Vault
• Implement security metrics dashboard

Quarter 2: Advanced Protection​

• Add DAST scanning with OWASP ZAP
• Implement API security testing
• Add runtime security monitoring (Falco)
• Integrate with SIEM solution

Continuous Improvement Framework​

📚 Additional Resources​

🔗 Essential Links​

• OWASP DevSecOps Guideline
• NIST Secure Software Development Framework
• CIS Controls for DevSecOps
• Cloud Security Alliance DevSecOps Working Group

📖 Recommended Reading​

1. "DevSecOps for Dummies" - IBM Limited Edition
2. "Securing DevOps" by Julien Vehent
3. "Container Security" by Liz Rice
4. "Infrastructure as Code Security" by Various Authors

🎓 Training and Certifications​

🛠️ Tool-Specific Resources​

SonarQube​

• Official Documentation
• Security Rules Reference

OWASP Tools​

• Dependency Check Documentation
• ZAP Getting Started

Trivy​

• Official GitHub Repository
• Configuration Examples

🤝 Contributing to Security​

Security Champion Program​

Consider establishing a Security Champion program in your organization:

1. Identify Champions: Select 1-2 developers per team
2. Provide Training: Regular security training and updates
3. Assign Responsibilities: Code review, tool configuration, incident response
4. Regular Meetings: Monthly security discussions and knowledge sharing

Community Participation​

• Contribute to open-source security tools
• Participate in security conferences and meetups
• Share your experiences through blog posts or presentations
• Join DevSecOps communities and forums

📧 Questions or Feedback?

This guide is a living document. Please contribute improvements, report issues, or suggest enhancements to help the entire DevSecOps community.

Written while teaching DevOps to 800+ engineers across 12 cohorts.

I audit these estates for a living. The findings repeat, and they repeat in a predictable order of severity. Here they are, with the reasoning behind each.

1. Requests are a guess that nobody revisited​

Someone shipped a service. They needed numbers for resources.requests, so they picked numbers. Those numbers are still there.

This matters more than it sounds, because requests — not actual usage — are what the scheduler packs against. A pod requesting 2 CPU and using 0.2 occupies 2 CPU worth of schedulable space on that node. Ten of those, and you are paying for a node that is 90% idle and 100% full.

Start by finding the gap between requested and used:

# What each pod is actually consuming
kubectl top pods --all-namespaces --sort-by=cpu

kubectl top is a point-in-time reading, so do not resize off one sample. Use a week of Prometheus history and size against a high percentile — not the peak, not the mean:

quantile_over_time(0.95,
  rate(container_cpu_usage_seconds_total{namespace="prod",container!=""}[5m])[7d:5m]
)

P95 over a week gives you a request that absorbs normal spikes without paying for the one pathological Tuesday. On an audit of an enterprise SaaS estate, rightsizing against observed utilization rather than provisioning-time guesses was the single largest line item in a ~40% reduction — and no capacity was removed to get it.

Where this goes wrong: rightsizing memory is not symmetric with CPU. A container over its CPU request gets throttled; a container over its memory limit gets OOM-killed. Be aggressive on CPU requests, conservative on memory limits.

2. Non-production runs at 3am​

Development and staging clusters typically serve people in roughly one working day, in roughly one set of time zones. Left alone, they bill for all 168 hours in the week.

A five-day, ten-hour working window is 50 of those 168 hours. Everything else is a nightly charge for an idle environment.

# Scale a non-prod namespace to zero outside working hours
apiVersion: batch/v1
kind: CronJob
metadata:
  name: scale-down-nonprod
  namespace: platform
spec:
  schedule: "0 20 * * 1-5"   # 20:00, weekdays — cluster timezone
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: scaler
          restartPolicy: OnFailure
          containers:
            - name: kubectl
              image: bitnami/kubectl:latest
              command: ["/bin/sh", "-c"]
              args:
                - kubectl scale deploy -l cost.scaledown=allowed --replicas=0 -n staging

Pair it with a scale-up job at the start of the working day. Two caveats before you ship this:

• Do not use --all. Anything running a migration, a queue consumer draining a backlog, or a nightly integration suite will be stopped mid-flight. Select on a label, as above, and let teams opt a workload out by removing it.
• Scaling deployments to zero does not release nodes. The cluster autoscaler has to reclaim them, and it will not evict pods with local storage or restrictive PodDisruptionBudgets. If your node count does not follow your replica count down, that is where to look.

3. Steady baseline load paid at on-demand rates​

Every cluster has a floor — the capacity that is running at 4am on a Sunday because it always is. Paying on-demand rates for that floor is paying a premium for flexibility you provably are not using.

Three tiers, mapped to three kinds of workload:

The third row is where the largest untapped savings usually sit, and the reason is cultural rather than technical: the first time a spot node is reclaimed mid-job, someone files an incident, and the team quietly moves everything back to on-demand. Set the expectation before you migrate, not after the first reclaim.

Separate the tiers with node pools and let workloads choose:

spec:
  tolerations:
    - key: "spot"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: "workload-class"   # label your spot pool with this
                operator: In
                values: ["spot"]

4. Untagged resources, which is the one that compounds​

The first three are sizing problems. This one is different in kind, and it is why estates regress after a successful optimization.

Without ownership metadata, "is this safe to delete?" has no answer. Nobody deletes anything that might belong to someone else, so nothing gets deleted, and waste accumulates permanently. On an estate of 100+ projects, untagged resources were the root difficulty — not the biggest single number, but the reason the other numbers kept coming back.

The fix is a provisioning requirement, not a cleanup script. Make ownership a condition of creation:

# A variable with no default cannot be omitted
variable "owner_team" {
  type        = string
  description = "Team accountable for this resource's cost."
}

resource "azurerm_resource_group" "app" {
  name     = var.name
  location = var.location

  tags = {
    owner       = var.owner_team
    environment = var.environment
    cost_center = var.cost_center
    managed_by  = "terraform"
  }
}

Enforce it at the policy layer too, so resources created outside Terraform cannot skip it — Azure Policy, AWS SCPs, or an admission controller such as Kyverno for in-cluster objects.

The part that actually determines whether it lasts​

An audit is the easy half. Any competent engineer with read access to a billing console can find the waste in a week.

The reason most cost programmes regress within two quarters is that the finding was delivered as a number, and a number is not an owner. What stops the climb is making spend visible and attributable continuously — in the format the finance and delivery teams already use, not a dashboard someone has to remember to open. Adoption beats elegance, every time.

On the estate above, spend is still ~33% below the original baseline while the platform's workload has grown. That durability came from the tagging policy and the cost reporting, not from the rightsizing. The rightsizing produced the drop; the governance is why it stayed down.

If you cut before you can attribute, you will cut again next year.

1. Why Kubernetes?
2. Kubernetes Basics
3. Kubernetes Architecture
4. Kubernetes Components
5. Local Installation with Minikube
6. Local Installation with Kind
7. Cloud Installation
8. Practical Examples
9. Best Practices
10. Resources and Next Steps

Why Kubernetes?​

The Container Challenge​

Before diving into Kubernetes, let's understand why it exists. Modern applications are increasingly built using containers, but managing containers at scale presents several challenges:

Problems with Manual Container Management:​

• Scaling Issues: Manually starting/stopping containers based on demand
• Service Discovery: How do containers find and communicate with each other?
• Load Balancing: Distributing traffic across multiple container instances
• Health Monitoring: Detecting when containers fail and replacing them
• Resource Management: Efficiently utilizing CPU, memory, and storage
• Configuration Management: Managing secrets, config files, and environment variables
• Rolling Updates: Deploying new versions without downtime

What Kubernetes Solves​

Key Benefits of Kubernetes:​

Kubernetes Basics​

What is Kubernetes?​

Kubernetes (often abbreviated as "K8s") is an open-source container orchestration platform that automates the deployment, scaling, and management of containerized applications.

Key Concepts​

1. Cluster​

A set of machines (physical or virtual) that run Kubernetes. A cluster consists of:

• Control Plane: Manages the cluster
• Worker Nodes: Run your applications

2. Pod​

• The smallest deployable unit in Kubernetes
• Contains one or more containers
• Containers in a pod share storage and network
• Pods are ephemeral (temporary)

3. Node​

• A physical or virtual machine in the cluster
• Can be a control plane node or worker node
• Runs pods and Kubernetes components

4. Service​

• Provides stable networking for pods
• Acts as a load balancer
• Enables service discovery

5. Deployment​

• Manages pod replicas
• Handles rolling updates
• Ensures desired state

Kubernetes vs. Traditional Deployment​

Kubernetes Architecture​

Kubernetes follows a distributed architecture pattern with clear separation between the control plane and data plane.

High-Level Architecture​

Control Plane vs Data Plane​

Kubernetes Components​

Control Plane Components​

1. API Server (kube-apiserver)​

• Purpose: The front-end of the Kubernetes control plane
• Functions:
  • Exposes the Kubernetes API
  • Handles authentication and authorization
  • Validates and processes API requests
  • Serves as the communication hub for all components

2. etcd​

• Purpose: Distributed key-value store for cluster data
• Stores:
  • Cluster configuration
  • Pod states and metadata
  • Secrets and ConfigMaps
  • Network policies
• Features:
  • High availability
  • Consistent data storage
  • Backup and restore capabilities

3. Scheduler (kube-scheduler)​

• Purpose: Assigns pods to nodes
• Decision Factors:
  • Resource requirements (CPU, memory)
  • Node capacity and availability
  • Affinity and anti-affinity rules
  • Quality of Service requirements
  • Custom scheduling policies

4. Controller Manager (kube-controller-manager)​

• Purpose: Runs controller processes that regulate cluster state
• Key Controllers:
  • Node Controller: Monitors node health
  • Deployment Controller: Manages deployments
  • Service Controller: Manages services
  • Endpoint Controller: Manages endpoints
  • ReplicaSet Controller: Ensures desired pod replicas

5. Cloud Controller Manager​

• Purpose: Integrates with cloud provider APIs
• Functions:
  • Node management (create/delete cloud VMs)
  • Load balancer management
  • Volume management
  • Route management

Worker Node Components​

1. Kubelet​

• Purpose: Node agent that communicates with control plane
• Responsibilities:
  • Register node with cluster
  • Watch for pod assignments
  • Start and stop containers
  • Report node and pod status
  • Execute health checks

2. Kube-proxy​

• Purpose: Network proxy that maintains network rules
• Functions:
  • Implement service networking
  • Load balance traffic to pods
  • Maintain iptables rules
  • Enable service discovery

3. Container Runtime​

• Purpose: Software responsible for running containers
• Options:
  • containerd: Most popular, lightweight
  • CRI-O: Kubernetes-native runtime
  • Docker Engine: Traditional but being phased out

Component Communication Flow​

Local Installation with Minikube​

Minikube is the most popular tool for running Kubernetes locally. It implements a local Kubernetes cluster on macOS, Linux, and Windows, with primary goals to be the best tool for local Kubernetes application development and to support all Kubernetes features that fit.

System Requirements​

Installation​

Linux Installation​

# Download latest minikube
curl -LO https://github.com/kubernetes/minikube/releases/latest/download/minikube-linux-amd64

# Install minikube
sudo install minikube-linux-amd64 /usr/local/bin/minikube && rm minikube-linux-amd64

# Verify installation
minikube version

macOS Installation​

# Using Homebrew
brew install minikube

# Or direct download
curl -LO https://github.com/kubernetes/minikube/releases/latest/download/minikube-darwin-amd64
sudo install minikube-darwin-amd64 /usr/local/bin/minikube

Windows Installation​

# Using Chocolatey
choco install minikube

# Or download from GitHub and add to PATH

Install kubectl​

# Linux
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
chmod +x kubectl
sudo mv kubectl /usr/local/bin/

# macOS
brew install kubectl

# Windows
choco install kubernetes-cli

Starting Your First Cluster​

# Start minikube with default settings
minikube start

# Start with specific resources
minikube start --cpus=4 --memory=8192mb --disk-size=50gb

# Start with specific Kubernetes version
minikube start --kubernetes-version=v1.32.0

# Start with specific driver
minikube start --driver=docker

Available Drivers​

Minikube Commands Cheat Sheet​

# Cluster Management
minikube start                    # Start cluster
minikube stop                     # Stop cluster
minikube delete                   # Delete cluster
minikube status                   # Check cluster status
minikube pause                    # Pause cluster
minikube unpause                  # Unpause cluster

# Information
minikube ip                       # Get cluster IP
minikube dashboard                # Open Kubernetes dashboard
minikube service <service-name>   # Access service URL

# Add-ons
minikube addons list              # List available add-ons
minikube addons enable ingress    # Enable ingress
minikube addons enable dashboard  # Enable dashboard

# Advanced
minikube ssh                      # SSH into minikube node
minikube logs                     # View minikube logs
minikube tunnel                   # Create tunnel for LoadBalancer services

Useful Minikube Add-ons​

# Enable essential add-ons
minikube addons enable dashboard
minikube addons enable ingress
minikube addons enable metrics-server
minikube addons enable storage-provisioner

# View all available add-ons
minikube addons list

Local Installation with Kind​

Kind (Kubernetes in Docker) is a tool for running local Kubernetes clusters using Docker container "nodes". Kind was primarily designed for testing Kubernetes itself, but may be used for local development or CI.

Why Choose Kind?​

Installation​

Prerequisites​

# Install Docker first
# Ubuntu/Debian
sudo apt update && sudo apt install docker.io

# macOS
brew install docker

# Start Docker service
sudo systemctl start docker
sudo usermod -aG docker $USER  # Add user to docker group

Install Kind​

# Linux/macOS
curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.29.0/kind-linux-amd64
chmod +x ./kind
sudo mv ./kind /usr/local/bin/kind

# macOS with Homebrew
brew install kind

# Using Go
go install sigs.k8s.io/kind@v0.29.0

Creating Clusters​

Single Node Cluster​

# Create a simple cluster
kind create cluster

# Create cluster with custom name
kind create cluster --name my-cluster

# Verify cluster
kubectl cluster-info --context kind-kind

Multi-Node Cluster​

# cluster-config.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
- role: worker
- role: worker

# Create multi-node cluster
kind create cluster --config cluster-config.yaml --name multi-node

# View nodes
kubectl get nodes

Advanced Configuration​

# advanced-config.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: advanced-cluster
nodes:
- role: control-plane
  kubeadmConfigPatches:
  - |
    kind: InitConfiguration
    nodeRegistration:
      kubeletExtraArgs:
        node-labels: "ingress-ready=true"
  extraPortMappings:
  - containerPort: 80
    hostPort: 80
    protocol: TCP
  - containerPort: 443
    hostPort: 443
    protocol: TCP
- role: worker
  extraMounts:
  - hostPath: /path/to/my/files
    containerPath: /files
- role: worker

Kind Commands​

# Cluster Management
kind create cluster --name <name>     # Create cluster
kind delete cluster --name <name>     # Delete cluster
kind get clusters                     # List clusters

# Working with Images
kind load docker-image <image>        # Load Docker image into cluster
kind load docker-image nginx:latest --name my-cluster

# Export cluster logs
kind export logs ./logs --name my-cluster

Setting up Ingress with Kind​

# 1. Create cluster with port mapping
cat <<EOF | kind create cluster --config=-
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  kubeadmConfigPatches:
  - |
    kind: InitConfiguration
    nodeRegistration:
      kubeletExtraArgs:
        node-labels: "ingress-ready=true"
  extraPortMappings:
  - containerPort: 80
    hostPort: 80
  - containerPort: 443
    hostPort: 443
EOF

# 2. Install ingress controller
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/kind/deploy.yaml

# 3. Wait for ingress controller
kubectl wait --namespace ingress-nginx \
  --for=condition=ready pod \
  --selector=app.kubernetes.io/component=controller \
  --timeout=90s

Cloud Installation​

Cloud providers offer managed Kubernetes services that handle the control plane for you. The three main contenders are Amazon Web Services (AWS) with EKS, Microsoft Azure with AKS, and Google Cloud with GKE.

Comparison of Cloud Providers​

Amazon EKS (Elastic Kubernetes Service)​

Prerequisites​

# Install AWS CLI
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install

# Configure AWS credentials
aws configure

# Install eksctl
curl --silent --location "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz" | tar xz -C /tmp
sudo mv /tmp/eksctl /usr/local/bin

Create EKS Cluster​

# Simple cluster creation
eksctl create cluster --name my-cluster --region us-west-2

# Advanced cluster with specific configuration
eksctl create cluster \
  --name production-cluster \
  --version 1.32 \
  --region us-west-2 \
  --nodegroup-name workers \
  --node-type m5.large \
  --nodes 3 \
  --nodes-min 1 \
  --nodes-max 10 \
  --managed

EKS Configuration YAML​

# cluster.yaml
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: production-cluster
  region: us-west-2
  version: "1.32"

managedNodeGroups:
  - name: worker-nodes
    instanceType: m5.large
    minSize: 1
    maxSize: 10
    desiredCapacity: 3
    volumeSize: 100
    ssh:
      allow: true
    iam:
      withAddonPolicies:
        autoScaler: true
        cloudWatch: true
        ebs: true

# Create cluster from config
eksctl create cluster -f cluster.yaml

Azure AKS (Azure Kubernetes Service)​

Prerequisites​

# Install Azure CLI
curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash

# Login to Azure
az login

# Create resource group
az group create --name myResourceGroup --location eastus

Create AKS Cluster​

# Simple cluster creation
az aks create \
  --resource-group myResourceGroup \
  --name myAKSCluster \
  --node-count 3 \
  --enable-addons monitoring \
  --generate-ssh-keys

# Get credentials
az aks get-credentials --resource-group myResourceGroup --name myAKSCluster

# Advanced cluster with specific features
az aks create \
  --resource-group myResourceGroup \
  --name production-cluster \
  --node-count 3 \
  --node-vm-size Standard_D4s_v3 \
  --kubernetes-version 1.32.0 \
  --enable-addons monitoring,http_application_routing \
  --enable-autoscaler \
  --min-count 1 \
  --max-count 10 \
  --network-plugin azure \
  --service-cidr 10.0.0.0/16 \
  --dns-service-ip 10.0.0.10

Google GKE (Google Kubernetes Engine)​

Prerequisites​

# Install Google Cloud SDK
curl https://sdk.cloud.google.com | bash
exec -l $SHELL

# Initialize gcloud
gcloud init

# Set default project and zone
gcloud config set project PROJECT_ID
gcloud config set compute/zone us-central1-a

# Enable required APIs
gcloud services enable container.googleapis.com

Create GKE Cluster​

# Simple cluster creation
gcloud container clusters create my-cluster \
  --num-nodes=3 \
  --zone=us-central1-a

# Get credentials
gcloud container clusters get-credentials my-cluster --zone=us-central1-a

# Advanced cluster with auto-scaling and specific machine types
gcloud container clusters create production-cluster \
  --machine-type=e2-standard-4 \
  --num-nodes=3 \
  --enable-autoscaling \
  --min-nodes=1 \
  --max-nodes=10 \
  --enable-autorepair \
  --enable-autoupgrade \
  --disk-size=100GB \
  --image-type=COS_CONTAINERD \
  --cluster-version=1.32 \
  --zone=us-central1-a

Cost Optimization Tips​

Cost Comparison Table​

Practical Examples​

Example 1: Deploying a Simple Web Application​

# nginx-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.24
        ports:
        - containerPort: 80
        resources:
          requests:
            memory: "64Mi"
            cpu: "250m"
          limits:
            memory: "128Mi"
            cpu: "500m"
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-service
spec:
  selector:
    app: nginx
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
  type: ClusterIP

# Deploy the application
kubectl apply -f nginx-deployment.yaml

# Check deployment status
kubectl get deployments
kubectl get pods
kubectl get services

# Scale the deployment
kubectl scale deployment nginx-deployment --replicas=5

# View deployment details
kubectl describe deployment nginx-deployment

Example 2: ConfigMap and Secret Usage​

# app-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
data:
  database_url: "postgresql://localhost:5432/myapp"
  debug_mode: "true"
  app_name: "My Application"
---
apiVersion: v1
kind: Secret
metadata:
  name: app-secrets
type: Opaque
data:
  database_password: cGFzc3dvcmQxMjM=  # base64 encoded "password123"
  api_key: YWJjZGVmZ2hpams=  # base64 encoded "abcdefghijk"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: web-app
  template:
    metadata:
      labels:
        app: web-app
    spec:
      containers:
      - name: web-app
        image: nginx:alpine
        env:
        - name: DATABASE_URL
          valueFrom:
            configMapKeyRef:
              name: app-config
              key: database_url
        - name: DEBUG_MODE
          valueFrom:
            configMapKeyRef:
              name: app-config
              key: debug_mode
        - name: DATABASE_PASSWORD
          valueFrom:
            secretKeyRef:
              name: app-secrets
              key: database_password
        - name: API_KEY
          valueFrom:
            secretKeyRef:
              name: app-secrets
              key: api_key

Example 3: Persistent Storage​

# persistent-storage.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: web-app-storage
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: standard
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-app-with-storage
spec:
  replicas: 1
  selector:
    matchLabels:
      app: web-app-with-storage
  template:
    metadata:
      labels:
        app: web-app-with-storage
    spec:
      containers:
      - name: web-app
        image: nginx:alpine
        volumeMounts:
        - name: web-storage
          mountPath: /usr/share/nginx/html
        ports:
        - containerPort: 80
      volumes:
      - name: web-storage
        persistentVolumeClaim:
          claimName: web-app-storage

Example 4: Ingress Configuration​

# ingress-example.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: web-app-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - myapp.example.com
    secretName: tls-secret
  rules:
  - host: myapp.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: nginx-service
            port:
              number: 80
      - path: /api
        pathType: Prefix
        backend:
          service:
            name: api-service
            port:
              number: 8080

Best Practices​

1. Resource Management​

# Always set resource requests and limits
resources:
  requests:
    memory: "64Mi"
    cpu: "250m"
  limits:
    memory: "128Mi"
    cpu: "500m"

2. Health Checks​

# Implement liveness and readiness probes
livenessProbe:
  httpGet:
    path: /health
    port: 8080
  initialDelaySeconds: 30
  periodSeconds: 10
  timeoutSeconds: 5
  failureThreshold: 3

readinessProbe:
  httpGet:
    path: /ready
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 5
  timeoutSeconds: 3
  failureThreshold: 3

3. Security Best Practices​

# Use security contexts
securityContext:
  runAsNonRoot: true
  runAsUser: 1000
  fsGroup: 2000
  capabilities:
    drop:
    - ALL
  readOnlyRootFilesystem: true
  allowPrivilegeEscalation: false

4. Namespace Organization​

# Create namespaces for different environments
kubectl create namespace development
kubectl create namespace staging
kubectl create namespace production

# Use namespace in deployments
kubectl apply -f app.yaml -n development

5. Monitoring and Logging​

# Add labels for monitoring
metadata:
  labels:
    app: my-app
    version: v1.0.0
    environment: production
    team: backend

Resources and Next Steps​

Essential kubectl Commands​

# Cluster Information
kubectl cluster-info
kubectl get nodes
kubectl describe node <node-name>

# Working with Pods
kubectl get pods
kubectl describe pod <pod-name>
kubectl logs <pod-name>
kubectl exec -it <pod-name> -- /bin/bash

# Working with Deployments
kubectl get deployments
kubectl scale deployment <deployment-name> --replicas=5
kubectl rollout status deployment/<deployment-name>
kubectl rollout history deployment/<deployment-name>

# Working with Services
kubectl get services
kubectl port-forward service/<service-name> 8080:80

# Configuration
kubectl apply -f <file.yaml>
kubectl delete -f <file.yaml>
kubectl get all
kubectl get all -n <namespace>

Learning Path​

Recommended Tools​

Next Steps​

1. Hands-on Practice: Set up a local cluster and deploy sample applications
2. Certification: Consider CKA (Certified Kubernetes Administrator) or CKAD (Certified Kubernetes Application Developer)
3. Advanced Topics: Learn about service mesh, operators, and custom resources
4. Production Readiness: Study monitoring, logging, security, and backup strategies
5. Community: Join Kubernetes community, attend meetups, contribute to open source

Useful Resources​

• Official Documentation: https://kubernetes.io/docs/
• Interactive Tutorials: https://kubernetes.io/docs/tutorials/
• Kubernetes by Example: https://kubernetesbyexample.com/
• CNCF Landscape: https://landscape.cncf.io/
• Kubernetes Slack: https://kubernetes.slack.com/

Troubleshooting Common Issues​

Pod Issues​

# Pod stuck in Pending state
kubectl describe pod <pod-name>
# Look for events: insufficient resources, scheduling issues

# Pod CrashLoopBackOff
kubectl logs <pod-name>
kubectl logs <pod-name> --previous
# Check container logs and exit codes

# ImagePullBackOff
kubectl describe pod <pod-name>
# Check image name, registry access, credentials

Network Issues​

# Service not accessible
kubectl get endpoints <service-name>
kubectl describe service <service-name>

# DNS resolution issues
kubectl run test-pod --image=busybox -it --rm -- nslookup <service-name>

Resource Issues​

# Check resource usage
kubectl top nodes
kubectl top pods

# Check resource quotas
kubectl describe resourcequota -n <namespace>

This comprehensive guide provides a solid foundation for understanding and working with Kubernetes. Start with local installations, practice with the examples, and gradually move to cloud deployments as you gain confidence.​

Written while teaching DevOps to 800+ engineers across 12 cohorts.

Orchestration Made Easy with Kubernetes

📚 Table of Contents​

1. Introduction to Kubernetes Package Management
2. Helm Charts Fundamentals
3. Helm for Microservices
4. GitOps Concepts
5. ArgoCD Configuration
6. Complete Workflow Example
7. Best Practices

🎯 Introduction to Kubernetes Package Management​

The Problem: Managing Complex Kubernetes Applications​

Imagine you're organizing a massive event. You need:

• Venue booking
• Catering
• Entertainment
• Decorations
• Security

Without a coordinator, you'd manually manage each vendor, track changes, and ensure everything works together. This is what deploying applications on Kubernetes feels like without Helm.

What is Kubernetes?​

Analogy: Think of Kubernetes as a smart apartment building manager that:

• Assigns apartments (containers) to tenants (applications)
• Ensures everyone has electricity and water (resources)
• Handles complaints and repairs (self-healing)
• Manages the lobby, elevators, and common areas (networking)

Key Kubernetes Concepts​

🎁 Helm Charts Fundamentals​

What is Helm?​

Helm = The Package Manager for Kubernetes

Analogy: Helm is like a meal kit delivery service (HelloFresh, Blue Apron):

• Without Helm: You manually buy ingredients, write recipes, measure quantities, adjust for servings
• With Helm: You get pre-packaged ingredients, tested recipes, and easy customization

Why Do We Need Helm?​

Before Helm:

# You need to manually apply each file
kubectl apply -f deployment.yaml
kubectl apply -f service.yaml
kubectl apply -f configmap.yaml
kubectl apply -f secret.yaml
kubectl apply -f ingress.yaml
# Repeat for each environment (dev, staging, prod)
# Manually manage versions, rollbacks, dependencies

With Helm:

# One command to install everything
helm install my-app ./my-chart

Helm Architecture​

Core Helm Concepts​

Helm Chart Structure​

my-awesome-chart/
├── Chart.yaml              # Chart metadata (name, version, description)
├── values.yaml             # Default configuration values
├── charts/                 # Dependency charts (sub-charts)
├── templates/              # Kubernetes resource templates
│   ├── deployment.yaml     # Pod deployment template
│   ├── service.yaml        # Service template
│   ├── ingress.yaml        # Ingress template
│   ├── configmap.yaml      # ConfigMap template
│   ├── _helpers.tpl        # Reusable template functions
│   └── NOTES.txt           # Post-installation instructions
├── .helmignore             # Files to ignore (like .gitignore)
└── README.md               # Documentation

Understanding Chart.yaml​

apiVersion: v2                      # Helm API version
name: my-awesome-chart              # Chart name
description: A simple Helm chart    # What this chart does
type: application                   # Type: application or library
version: 0.1.0                      # Chart version (SemVer)
appVersion: "1.0"                   # Version of the app being deployed
keywords:
  - nginx
  - web
maintainers:
  - name: Your Name
    email: you@example.com
dependencies:                       # Other charts this needs
  - name: redis
    version: "~10.5.0"
    repository: "https://charts.bitnami.com/bitnami"

Understanding values.yaml​

Analogy: values.yaml is like a customization form at a restaurant:

• Want extra cheese? ✓
• How spicy? Medium
• Portion size? Large

# values.yaml - Configuration options

# Number of replicas
replicaCount: 3

# Container image
image:
  repository: nginx
  tag: "1.25.0"
  pullPolicy: IfNotPresent

# Service configuration
service:
  type: LoadBalancer
  port: 80
  targetPort: 8080

# Resource limits
resources:
  limits:
    cpu: 500m
    memory: 512Mi
  requests:
    cpu: 250m
    memory: 256Mi

# Environment-specific settings
environment: production

# Feature flags
features:
  monitoring: true
  logging: true