kubernetes-security

Chris Binnie - Kubernetes Security: Container Orchestration Hardening & Protection

Secure your Kubernetes infrastructure with this comprehensive security guide from my working notes. Learn essential hardening techniques including RBAC, network policies, and defence strategies for production clusters. This guide covers critical security domains from basic setup to advanced threat protection across all Kubernetes environments.

Kubernetes security is fundamental for protecting your containerised workloads from cyber threats, data breaches, and unauthorised access. This comprehensive guide covers essential security practices, from basic cluster hardening to advanced threat protection, ensuring your Kubernetes environment remains secure and compliant with industry standards.

Whether you’re managing self-hosted clusters, EKS, GKE, or AKS environments, these security principles apply across all Kubernetes distributions and will help you build a robust defence against modern cyber threats in cloud-native environments.

Table of Contents

Cluster Hardening Fundamentals

The foundation of Kubernetes security begins with proper cluster configuration. Never expose the API server without proper authentication and authorisation:

# Secure API server configuration
apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
apiServer:
  extraArgs:
    anonymous-auth: "false"
    audit-log-maxage: "30"
    enable-admission-plugins: "NodeRestriction,ResourceQuota"
    tls-cipher-suites: "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256"
    encryption-provider-config: "/etc/kubernetes/encryption-config.yaml"

Configure encryption at rest for etcd:

# Encryption configuration
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
  - secrets
  providers:
  - aescbc:
      keys:
      - name: key1
        secret: <32-byte-base64-encoded-key>
  - identity: {}

Secure kubelet configuration:

# /etc/kubernetes/kubelet/kubelet-config.yaml
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
authentication:
  anonymous:
    enabled: false
  webhook:
    enabled: true
authorization:
  mode: Webhook
readOnlyPort: 0
protectKernelDefaults: true

Security Tip: Always run CIS Kubernetes Benchmark assessments to identify configuration weaknesses and ensure compliance with security best practices.

Authentication and Authorisation

Implement robust RBAC (Role-Based Access Control) policies:

# Service account with minimal privileges
apiVersion: v1
kind: ServiceAccount
metadata:
  name: app-service-account
  namespace: production
automountServiceAccountToken: false
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: production
  name: pod-reader
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "watch", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: read-pods
  namespace: production
subjects:
- kind: ServiceAccount
  name: app-service-account
  namespace: production
roleRef:
  kind: Role
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io

Configure OIDC integration:

# API server OIDC configuration
apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
apiServer:
  extraArgs:
    oidc-issuer-url: "https://accounts.google.com"
    oidc-client-id: "kubernetes"
    oidc-username-claim: "email"
    oidc-groups-claim: "groups"

Network Security and Policies

Deploy comprehensive network policies:

# Default deny all traffic
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: production
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress
---
# Allow specific application communication
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: web-app-policy
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: web-app
  policyTypes:
  - Ingress
  - Egress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: frontend
    ports:
    - protocol: TCP
      port: 8080
  egress:
  - to:
    - podSelector:
        matchLabels:
          app: database
    ports:
    - protocol: TCP
      port: 5432

Configure secure ingress with TLS:

# Secure ingress configuration
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: secure-ingress
  namespace: production
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/enable-modsecurity: "true"
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
spec:
  tls:
  - hosts:
    - app.example.com
    secretName: app-tls-secret
  rules:
  - host: app.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: web-app-service
            port:
              number: 80

Security Tip: Always test network policies in a staging environment before applying to production, as overly restrictive policies can break application functionality.

Pod Security Standards

Implement Pod Security Standards to enforce security baselines:

# Restricted pod security profile
apiVersion: v1
kind: Pod
metadata:
  name: secure-pod
  namespace: production
spec:
  serviceAccountName: app-service-account
  securityContext:
    runAsNonRoot: true
    runAsUser: 10001
    fsGroup: 10001
    seccompProfile:
      type: RuntimeDefault
  containers:
  - name: app-container
    image: registry.example.com/secure-app:v1.2.3@sha256:abcd1234...
    securityContext:
      allowPrivilegeEscalation: false
      readOnlyRootFilesystem: true
      runAsNonRoot: true
      capabilities:
        drop:
        - ALL
    resources:
      limits:
        cpu: "500m"
        memory: "512Mi"
      requests:
        cpu: "250m"
        memory: "256Mi"
    livenessProbe:
      httpGet:
        path: /health
        port: 8080
      initialDelaySeconds: 30
      periodSeconds: 10
    readinessProbe:
      httpGet:
        path: /ready
        port: 8080
      initialDelaySeconds: 5
      periodSeconds: 5

Configure namespace-level security:

# Pod Security Standards enforcement
apiVersion: v1
kind: Namespace
metadata:
  name: secure-namespace
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/warn: restricted
---
# Resource quotas
apiVersion: v1
kind: ResourceQuota
metadata:
  name: production-quota
  namespace: production
spec:
  hard:
    requests.cpu: "10"
    requests.memory: 20Gi
    limits.cpu: "20"
    limits.memory: 40Gi
    pods: "50"
    secrets: "20"

Secrets and Configuration Management

Secure secrets management:

# Kubernetes secret with proper labelling
apiVersion: v1
kind: Secret
metadata:
  name: database-credentials
  namespace: production
  labels:
    app.kubernetes.io/name: database
    app.kubernetes.io/component: credentials
type: Opaque
stringData:
  username: dbadmin
  password: super-secret-password-123
---
# Secure secret mounting
apiVersion: v1
kind: Pod
metadata:
  name: app-pod
  namespace: production
spec:
  containers:
  - name: app
    image: app:latest
    env:
    - name: DB_USERNAME
      valueFrom:
        secretKeyRef:
          name: database-credentials
          key: username
    volumeMounts:
    - name: secret-volume
      mountPath: /etc/secrets
      readOnly: true
  volumes:
  - name: secret-volume
    secret:
      secretName: database-credentials
      defaultMode: 0400

External secrets integration:

# External Secrets Operator
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: vault-secret
  namespace: production
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-secret-store
    kind: SecretStore
  target:
    name: app-secret
    creationPolicy: Owner
  data:
  - secretKey: password
    remoteRef:
      key: secret/data/database
      property: password

Container Image Security

Implement image scanning and vulnerability management:

# Install and use Trivy for scanning
curl -sfL https://raw.githubusercontent.com/aquasecurity/trivy/main/contrib/install.sh | sh -s -- -b /usr/local/bin
trivy image --severity HIGH,CRITICAL registry.example.com/app:latest

# Deploy Trivy Operator for continuous scanning
kubectl apply -f https://raw.githubusercontent.com/aquasecurity/trivy-operator/main/deploy/static/trivy-operator.yaml

Image signing with Cosign:

# Generate keys and sign images
cosign generate-key-pair
cosign sign --key cosign.key registry.example.com/app:v1.2.3
cosign verify --key cosign.pub registry.example.com/app:v1.2.3

Secure image usage:

# Secure image configuration
apiVersion: v1
kind: Pod
metadata:
  name: secure-app
spec:
  containers:
  - name: app
    image: registry.example.com/secure-app:v1.2.3@sha256:abcd1234...
    imagePullPolicy: Always
    securityContext:
      readOnlyRootFilesystem: true
      runAsNonRoot: true
      runAsUser: 10001
  imagePullSecrets:
  - name: registry-credentials

Security Tip: Always use specific image tags or SHA digests instead of ‘latest’ to ensure consistency and prevent supply chain attacks.

Runtime Security

Deploy Falco for runtime monitoring:

# Falco DaemonSet
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: falco
  namespace: falco
spec:
  selector:
    matchLabels:
      app: falco
  template:
    metadata:
      labels:
        app: falco
    spec:
      serviceAccount: falco
      containers:
      - name: falco
        image: falcosecurity/falco:latest
        securityContext:
          privileged: true
        volumeMounts:
        - name: proc
          mountPath: /host/proc
          readOnly: true
        - name: boot
          mountPath: /host/boot
          readOnly: true
      volumes:
      - name: proc
        hostPath:
          path: /proc
      - name: boot
        hostPath:
          path: /boot

Custom Falco rules:

# Security monitoring rules
- rule: Shell in Container
  desc: Detect shell spawned in container
  condition: spawned_process and container and shell_procs
  output: "Shell in container (user=%user.name container=%container.name)"
  priority: WARNING

- rule: Sensitive Mount
  desc: Detect sensitive filesystem mount
  condition: mount and container and sensitive_mount
  output: "Sensitive mount (container=%container.name mount=%fd.name)"
  priority: ERROR

Monitoring and Auditing

Enable comprehensive audit logging:

# Audit policy configuration
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
- level: RequestResponse
  resources:
  - group: ""
    resources: ["secrets", "configmaps"]
  namespaces: ["production"]
- level: Request
  users: ["system:anonymous"]
  verbs: ["create", "update", "patch", "delete"]
- level: Metadata
  namespaces: ["kube-system"]

Set up monitoring and alerting:

# Security alerting rules
groups:
- name: kubernetes-security
  rules:
  - alert: UnauthorizedAPICall
    expr: increase(apiserver_audit_total{verb!="get"}[5m]) > 10
    for: 2m
    labels:
      severity: warning
    annotations:
      summary: "High number of API modification calls"
  
  - alert: PodSecurityViolation
    expr: increase(pod_security_policy_violations_total[5m]) > 0
    labels:
      severity: critical
    annotations:
      summary: "Pod security policy violation detected"

Compliance and Governance

Implement CIS Kubernetes Benchmark compliance:

# Install and run kube-bench
kubectl apply -f https://raw.githubusercontent.com/aquasecurity/kube-bench/main/job.yaml
kubectl logs job/kube-bench

Deploy policy engines:

# OPA Gatekeeper policy
apiVersion: templates.gatekeeper.sh/v1beta1
kind: ConstraintTemplate
metadata:
  name: k8srequiredsecuritycontext
spec:
  crd:
    spec:
      names:
        kind: K8sRequiredSecurityContext
  targets:
  - target: admission.k8s.gatekeeper.sh
    rego: |
      package k8srequiredsecuritycontext
      violation[{"msg": msg}] {
        container := input.review.object.spec.containers[_]
        not container.securityContext.runAsNonRoot
        msg := "Container must run as non-root user"
      }

Incident Response

Create automated incident response:

#!/bin/bash
# Kubernetes incident response script
NAMESPACE=${1:-default}
POD_NAME=${2:-}

echo "=== Kubernetes Security Incident Response ==="
echo "Timestamp: $(date)"

# Isolate affected workload
kubectl patch networkpolicy default-deny-all -n $NAMESPACE --type='merge' \
  -p='{"spec":{"podSelector":{"matchLabels":{"incident":"isolated"}}}}'
kubectl label pod $POD_NAME -n $NAMESPACE incident=isolated

# Preserve evidence
kubectl logs $POD_NAME -n $NAMESPACE --previous > /tmp/pod-logs-$(date +%Y%m%d).log
kubectl describe pod $POD_NAME -n $NAMESPACE > /tmp/pod-details-$(date +%Y%m%d).log
kubectl get events -n $NAMESPACE --sort-by='.lastTimestamp' > /tmp/events-$(date +%Y%m%d).log

echo "Initial response complete. Evidence preserved in /tmp/"

Configure alerting systems:

# Alertmanager configuration for security incidents
global:
  slack_api_url: 'https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK'

route:
  group_by: ['alertname', 'cluster']
  receiver: 'security-team'
  routes:
  - match:
      severity: critical
    receiver: 'security-team-urgent'

receivers:
- name: 'security-team'
  slack_configs:
  - channel: '#security-alerts'
    title: 'Kubernetes Security Alert'
    
- name: 'security-team-urgent'
  slack_configs:
  - channel: '#security-critical'
    title: 'CRITICAL: Kubernetes Security Incident'
    color: 'danger'

Warning: Always test incident response procedures in a controlled environment to ensure they function correctly without causing additional service disruptions.

Best Practices Summary

Kubernetes security requires a comprehensive, multi-layered approach combining cluster hardening, workload protection, and continuous monitoring. Regular security assessments, automated threat detection, and incident response capabilities are essential for maintaining a secure container orchestration environment.

Key Kubernetes security principles include:

By implementing these security measures and maintaining them consistently, you’ll significantly reduce your Kubernetes infrastructure’s attack surface and improve your overall security posture.

Remember that container security is a shared responsibility model and an ongoing process, not a one-time setup. Regular reviews, updates, and improvements to your security configuration are essential for staying ahead of evolving threats in the cloud-native environment.

Frequently Asked Questions

What is the most critical Kubernetes security vulnerability?

The most common vulnerabilities involve misconfigured RBAC permissions, exposed API servers, and containers running as root. Always implement least privilege principles and use Pod Security Standards.

How often should I scan container images for vulnerabilities?

Scan images during the CI/CD pipeline before deployment, and continuously scan running containers. Set up automated scanning with tools like Trivy or Snyk for daily vulnerability checks.

What’s the difference between Pod Security Policies and Pod Security Standards?

Pod Security Policies (PSPs) are deprecated as of Kubernetes v1.21. Pod Security Standards (PSS) replaced them, providing built-in security profiles: Privileged, Baseline, and Restricted.

How do I secure inter-pod communication in Kubernetes?

Use Network Policies to control traffic flow between pods, implement service mesh like Istio for mTLS encryption, and apply zero-trust networking principles with default deny-all policies.

What should I do if a security incident occurs in my cluster?

Immediately isolate affected workloads using network policies, preserve evidence through logging, follow your incident response plan, and conduct a post-incident review to improve security posture.

Expert Kubernetes and Cloud Security Resources

Visit Chris Binnie - Cloud Native Security and Linux Server Security for expert insights and practical guides on Kubernetes security, container orchestration, and cloud-native infrastructure hardening.

For comprehensive cloud security practices that complement your Kubernetes security strategy, refer to the AWS Cloud Security Guide.

For foundational server security that supports your Kubernetes nodes, refer to the Linux Server Security Guide.

About the Author

Author of Cloud Native Security and other cybersecurity books, with extensive experience in enterprise container security implementations and Kubernetes security assessments.

Linux® is the registered trademark of Linus Torvalds. Kubernetes® is a registered trademark of The Linux Foundation. Use the information from my notes found on these pages at your own risk.

Kubernetes Security, Container Security, RBAC, Network Policies, Pod Security Standards, Runtime Security, Falco, OPA Gatekeeper, Service Mesh Security, Istio Security, Container Image Scanning, DevSecOps, Cloud Native Security, Compliance Automation, Threat Detection, Incident Response, CIS Kubernetes Benchmark, Container Orchestration Security, Kubernetes Hardening, Cloud Security