DevOps¶

What experienced DevOps engineers forget before an interview, grouped by subtopic. For AWS service behavior see aws.md; for Linux internals see os.md.

Deployment and release¶

Deployment strategies¶

• Every strategy needs health checks and an automatic stop/rollback condition.

Feature flags¶

• Flags decouple deploy from release: code ships dark and is turned on per user, percentage, or region — and turned off without a deploy.
• Stale flags are debt: give each one an owner and a removal date.

Rollbacks¶

• Roll back by shifting traffic or redeploying the previous immutable artifact; it never undoes data changes already made.
• So the previous version must run against the new schema — keep migrations backward compatible.

Zero-downtime schema changes¶

• Expand-and-contract: add the new schema (backward compatible) → deploy code that handles both → backfill → remove the old schema in a later release.
• A destructive migration (dropping a column old code still reads) makes rollback impossible.
• Keep each DDL short and lock-safe (lock_timeout, concurrent index builds) — see postgresql.md.

Incident response¶

• Assign roles: an incident commander coordinates while others handle communications and operations.
• Mitigate first (roll back, flip a flag, fail over); find the root cause later.
• Write a blameless postmortem: timeline, contributing factors, and action items with owners.
• Track time to detect and time to recover (MTTD, MTTR); link a runbook from every alert.

Containers¶

How containers work¶

• A container is a Linux process with namespaces (isolated view: PID, network, mount, UTS, IPC, user) and cgroups (resource limits: CPU, memory, I/O).
• The filesystem is image layers stacked with OverlayFS plus a thin writable layer; all containers share the host kernel, unlike VMs.
• The OCI runtime (runc) sets this up; containerd/CRI-O manage images and lifecycles above it.

Dockerfile practices¶

• Multi-stage builds keep compilers out of the final image; pin base images by digest; copy dependency manifests before source to reuse cached layers.
• Run as non-root; never put secrets in ARG, ENV, or copied files — layers keep them even if a later layer deletes them (use BuildKit secret mounts).

PID 1 and signals¶

• PID 1 gets no default signal handlers, so an app not written for it may ignore SIGTERM and be killed after the grace period.
• Use the exec form (CMD ["app"], not CMD app, which wraps it in a shell) and a tiny init (tini, --init) to forward signals and reap zombies.

Kubernetes architecture¶

Control plane¶

• The API server is the only component talking to etcd; the scheduler assigns Pods to nodes; controllers reconcile actual state toward desired state.
• Everything is declarative and level-triggered: kubectl apply succeeding means the object was stored, not that the app is healthy.
• A bare Pod has no controller, so nothing recreates it after a node failure — run workloads through a Deployment, StatefulSet, DaemonSet, or Job.

Node components¶

• kubelet runs the node's Pods through the container runtime (CRI) and reports status.
• kube-proxy or an eBPF CNI such as Cilium implements Service virtual IPs.

Packaging and extension¶

• Helm templates charts with values and tracks releases; Kustomize patches plain YAML with overlays, no templating.
• CRDs + operators extend the API: an operator's controller reconciles custom resources (databases, certificates) like built-in objects.

Kubernetes workloads¶

Rolling update tuning¶

• maxSurge (extra Pods above desired) and maxUnavailable (Pods allowed down) — both default 25%.
• New Pods count as available only after readiness passes (plus minReadySeconds), so a broken readiness probe stalls the rollout safely.

Sidecars and init containers¶

• Init containers run to completion, in order, before app containers start.
• Native sidecars (an init container with restartPolicy: Always, GA in 1.33) start before the app and stop after it — fixing Jobs that never finished because a sidecar kept running.

Scheduling constraints¶

• Taints on nodes repel Pods unless they have a matching toleration (dedicated GPU nodes).
• Node affinity attracts Pods to labeled nodes; pod anti-affinity or topology spread constraints spread replicas across nodes and zones.

PodDisruptionBudgets¶

• A PDB (minAvailable or maxUnavailable) limits only disruptions that go through the Eviction API — node drains, autoscaler scale-down.
• It doesn't stop crashes, direct Pod deletes, or Deployment rolling updates (those follow maxUnavailable).
• A PDB that allows zero disruptions blocks node drains forever.

Persistent volumes¶

• A PVC requests storage; a StorageClass provisions a matching PV dynamically.
• Access modes: ReadWriteOnce (one node — most block storage) vs ReadWriteMany (shared file systems such as EFS or NFS).
• WaitForFirstConsumer binding delays provisioning until the Pod is scheduled, so a zonal disk lands in the Pod's zone.
• Dynamic volumes default to reclaim policy Delete (the disk goes with the PVC); Retain keeps it.

Kubernetes networking¶

Services¶

• A Service gives a changing set of Pods (chosen by label selector) a stable virtual IP and DNS name: ClusterIP (internal), NodePort, LoadBalancer (cloud LB).
• No traffic? Check the selector and readiness first: kubectl get endpointslices (the Endpoints API is deprecated since 1.33).
• General service-discovery patterns: see system.md.

Ingress and Gateway API¶

• Ingress and Gateway API route HTTP by host and path — neither does anything without a controller.
• ingress-nginx was retired in March 2026 (no more fixes); new setups use Gateway API implementations (Envoy Gateway, Cilium, cloud controllers).

NetworkPolicy¶

• By default all Pods can reach all Pods; a NetworkPolicy selecting a Pod switches it to deny-by-default for the listed direction.
• Enforcement needs a CNI that supports it (Calico, Cilium) — otherwise policies are silently ignored.

Kubernetes operations¶

Probes¶

• Readiness gates Service traffic; liveness restarts a stuck container; startup holds off both during slow boots.
• A liveness probe that checks dependencies or times out under load causes restart cascades — keep it a cheap local check.

Requests, limits, QoS¶

• Requests drive scheduling; limits cap usage — over the CPU limit the container is throttled.
• At the memory limit the kernel reclaims (page cache) first and OOMKills only if usage still can't stay under it.
• Requests vs limits set the QoS class (Guaranteed / Burstable / BestEffort), which decides eviction order under node pressure.
• In-place Pod resize (GA in 1.35) changes CPU/memory requests without restarting the Pod.

Autoscaling¶

• HPA scales replicas from metrics; VPA adjusts requests; Cluster Autoscaler/Karpenter add nodes when Pods can't be scheduled.
• HPA utilization targets are a percentage of requests, so Pods without CPU requests can't scale on CPU; scale-down waits out a 5-minute stabilization window by default.
• KEDA scales on external events (queue length, consumer lag), down to zero; Karpenter also consolidates underused nodes.
• More replicas don't help when the bottleneck is a saturated downstream database.

Graceful Pod shutdown¶

• On deletion, endpoint updates (EndpointSlice marks the Pod terminating, not ready) and the kubelet's shutdown run concurrently, so the Pod may still receive traffic briefly.
• The kubelet runs preStop first, then sends SIGTERM — a short preStop sleep covers that race.
• terminationGracePeriodSeconds (default 30 s) covers preStop plus shutdown; then the kubelet sends SIGKILL. App-side steps are in backend.md.

Debugging Pod states¶

• Pending: unschedulable (resources, quota, taints, unbound volume). ImagePullBackOff: wrong image or registry credentials. CrashLoopBackOff: the process or its liveness probe keeps failing.
• kubectl describe pod shows events; kubectl logs --previous shows the crashed container's last output.

Kubernetes security¶

Pod security¶

• Pod Security Admission enforces the privileged, baseline, or restricted profile per namespace via labels; PodSecurityPolicy was removed in 1.25.
• A hardened securityContext: runAsNonRoot, readOnlyRootFilesystem, allowPrivilegeEscalation: false, drop all capabilities, seccompProfile: RuntimeDefault.
• privileged: true, hostPath mounts, and hostNetwork/hostPID effectively hand over the node.
• Kyverno or OPA Gatekeeper enforce custom rules (allowed registries, no latest tags, required labels).

ServiceAccounts and workload identity¶

• Each Pod runs as a ServiceAccount; RBAC Roles bound to it limit what it may do through the Kubernetes API.
• Its tokens are projected, short-lived, and audience-bound; set automountServiceAccountToken: false when the app doesn't call the API.
• Workload identity maps a ServiceAccount to a cloud role (EKS IRSA or Pod Identity, GKE Workload Identity) — no static cloud keys in Secrets.

ConfigMaps and Secrets¶

• Secret values are only base64-encoded; enable encryption at rest (KMS), restrict access with RBAC, or sync from a vault (External Secrets Operator).
• Env vars from a ConfigMap don't update in a running Pod; mounted files do (eventually) — most apps still need a restart.

Terraform¶

Plan and apply¶

• init (providers, backend) → plan (diff) → apply; review every plan for replacements — replacing a stateful resource usually means data loss.
• lifecycle { prevent_destroy = true } and create_before_destroy guard critical resources.

State and locking¶

• State maps resource addresses to real objects and can hold secrets — keep it in a remote backend with locking, never in Git.
• Drift: out-of-band changes show up in the next plan, which proposes reverting them.

count vs for_each¶

• count addresses instances by index (aws_instance.web[2]): removing one element shifts every later index, so each later instance takes its neighbor's config and is updated in place or replaced, and the last one is destroyed.
• for_each keys instances by stable map or set keys (aws_instance.web["api"]), so adding or removing one touches only that one.
• for_each keys must be known at plan time, not computed during apply; switching between the two needs moved blocks.

Refactoring resources¶

• moved blocks record a renamed address, so the plan doesn't destroy and recreate it.
• import blocks (1.5) adopt existing resources into state.
• removed blocks (1.7) drop a resource from config; the default destroys it — only lifecycle { destroy = false } keeps the real object.

Modules, workspaces, OpenTofu¶

• Modules package resources behind inputs and outputs.
• Workspaces give one configuration several states but are not an access boundary — isolate environments with separate backends or root modules.
• OpenTofu is the open-source fork of Terraform with the same workflow.

CI/CD and GitOps¶

CI pipeline practices¶

• Trunk-based development: short-lived branches merged at least daily, unfinished work behind flags.
• Build once, promote the same image digest through environments — never rebuild per environment.
• Cache dependencies keyed on the lock-file hash, run fast checks first, and pin third-party CI actions by commit SHA.

Pull-based reconciliation¶

• A controller (Argo CD, Flux) watches a Git repo of desired state and reconciles the cluster — the cluster pulls, so CI needs no cluster credentials.
• Manual kubectl changes are drift: Flux reapplies every interval, Argo CD reverts them only with selfHeal on; emergency fixes must land in Git too.