Skip to main content

Deployment Readiness Checklist

A checklist of actions to review before deploying an app to production.

How to use it: copy the checklist into your ticket or merge request and tick items off as you go. If you skip an item, note why (e.g. "not applicable: no DB"), so the decision is visible to reviewers.

Non-negotiables​

If time is short, these matter most:

README.md​

  • Describes the app's purpose
  • Documents how to build
  • Documents how to start
  • Lists dependencies (DB, external services)
  • Links to the OpenAPI/Swagger spec (or generated docs), if the app exposes an API
  • Links to the architecture diagram / ADRs for anything non-trivial
  • Links to the runbook and the on-call/escalation contact for this service
  • Documents a reproducible local development setup (e.g. docker-compose up, seed data, required tooling versions)

Configuration​

  • The app is configured via environment variables plus a .env or config file
  • Environment variables override config files, and command-line arguments override environment variables
  • The app validates all required config and secrets at startup and fails fast with a clear error, instead of failing on the first request that needs them
  • Static frontend apps read variables at runtime rather than at the CI step (see Frontend apps)
  • The app exposes these configurable environment variables:
    • SERVER_HOST & SERVER_PORT / SERVER_ADDR (e.g. 0.0.0.0:8080)
    • HTTP_READ_TIMEOUT & HTTP_WRITE_TIMEOUT / HTTP_TIMEOUT (see Timeouts)

Secrets​

  • Secrets have no default values (JWT secret, DB password, API keys): a missing secret in production must stop the app from booting, not fall back to a dev default
  • Secrets are stored in a dedicated secret manager (Vault, AWS/GCP Secrets Manager, sealed-secrets, K8s Secrets + encryption at rest) and never committed to git, even in "private" repos
  • Secrets differ per environment (dev/staging/prod); no environment reuses another's credentials
  • Secrets and API keys are rotated on a defined schedule, and rotation doesn't require a full redeploy
  • Access to production secrets is scoped and audited (who can read/write them, and when they last did)

Docker​

  • The app is Dockerized, with a docker-compose.yml for dependencies, if any

  • A .dockerignore file is present:

    .git
    .gitlab-ci.yml
    Dockerfile
    docker-compose*.yml
    node_modules
    .env*
    *.md
    coverage

    .env* and node_modules matter most: without them, local secrets and host-built dependencies can leak into the image.

  • The container runs as a non-root user (e.g. USER node)

  • Only one process runs per container, started with exec directly (not via npm run)

  • A docker-entrypoint.sh is used if the app has separate runtime logic (e.g. web + workers) or runs migrations

  • The container filesystem is read-only where possible (readOnlyRootFilesystem: true), with explicit writable mounts (e.g. /tmp) only where needed

Docker image​

  • The image is as small as possible (under 200 MB is acceptable):
    • Alpine-based, if possible
    • Multi-stage (separate builder stage)
    • Build cache is cleaned (npm cache clean --force)
    • Only production dependencies are installed: prefer npm ci --omit=dev over npm install --omit=dev or a later npm prune --omit=dev
  • The image is scanned for known CVEs (Trivy/Grype) as a CI gate, not just ad hoc
  • The base image is kept up to date and rebuilt periodically, even without app code changes, to pick up OS-level security patches

CI​

  • Builds run inside Docker containers
  • Images are tagged uniquely per build:
    • GitLab CI: DOCKER_IMAGE_TAG="$CI_COMMIT_REF_SLUG-$COMMIT_DATE-$CI_PIPELINE_ID"
    • GitHub Actions: DOCKER_IMAGE_TAG="${GITHUB_REF_SLUG}-${COMMIT_DATE}-run-${GITHUB_RUN_NUMBER}" with github-slug-action
  • Lint and test steps are included
  • Images are published to a private container registry (e.g. DockerHub repos are public by default, so set the repository to private)
  • (GitLab) Deployments set environment.name and environment.url
  • A cleanup step removes created images and intermediate build images
  • Dependency versions are pinned (lockfile committed: go.sum, package-lock.json, etc.)
  • A dependency vulnerability scan (npm audit, govulncheck, Snyk/Trivy) runs as a blocking or at least visible step
  • Dependabot/Renovate (or equivalent) is enabled, so dependency updates are proposed automatically rather than discovered during an incident
  • Static analysis / SAST scanning runs in the pipeline (e.g. Semgrep, gosec, ESLint security rules)
  • Secrets scanning (e.g. gitleaks, trufflehog) runs on every commit/MR to catch accidentally committed credentials before merge

CD​

  • CI and CD are separate: CI builds, tests, and publishes immutable artifacts; CD promotes and deploys an already-built artifact without rebuilding it
  • Only artifacts built and tested by CI are deployed; nothing is built on a developer machine or deployed from an unverified/untagged branch
  • Production deployments reference an immutable artifact (image digest or immutable version tag), never latest or another mutable tag
  • The artifact running in production is traceable to the source commit, CI pipeline, and test results that produced it
  • Deploys to production pass through lower environments first (dev → staging → prod), except for documented emergency-fix paths
  • Configuration changes are version-controlled and reviewed; production configuration is never changed manually without an audit trail
  • The rollout strategy is explicit (rolling update, recreate, blue-green, or canary), with maxSurge/maxUnavailable (or equivalent) tuned to the service's traffic pattern; recreate is used only with documented, accepted downtime and an availability plan
  • The rollout keeps enough healthy capacity: new instances become ready before healthy old ones are terminated, and the service never intentionally drops below its required availability
  • Rollout progress and deployment health are observable (ready replicas, error rate, latency, saturation, app-specific health signals)
  • Automated smoke tests run right after deploy, before the deploy is considered complete
  • Critical user journeys are verified after deployment, not only that the process/container started
  • Rollback is a tested, one-step operation (redeploy the previous immutable image or revert the deployment manifest) that never requires a database rollback (see DB), rather than something improvised for the first time during an incident
  • The previous known-good artifact stays available, so rollback never depends on rebuilding it
  • Rollback is automated or triggered on defined signals (error-rate spike, failed smoke test, crash-loop, readiness failure, significant latency regression), not solely dependent on a human noticing in time
  • Deployments have a defined timeout; a rollout that doesn't become healthy in time is considered failed
  • Failed deployments never leave production partially updated without being clearly reported and handled
  • Pipeline results and deploys are visible to the team (e.g. Slack notification, changelog, release notes); notifications include at least the service, environment, version/artifact, source commit, deployer/trigger, and status
  • Risky deploys avoid peak-traffic windows and periods of reduced on-call coverage (e.g. Friday evening, holidays), unless there is a documented reason to deploy
  • Each production deployment has an identified owner/on-call engineer who can monitor the rollout and perform a rollback
  • A documented emergency deployment path bypasses normal promotion requirements while preserving auditability
  • Deployment frequency and change-failure rate are tracked, giving the team a baseline for whether the process is getting safer or riskier over time

App runtime​

  • The app is horizontally scalable
  • The app is stateless: no in-memory session/cache that would break when a second replica starts (use Redis/DB for shared state)
  • Sticky sessions (session affinity) are not used: any replica can serve any request
  • If the app has separate logic (e.g. web + worker), each runs as its own process so they can scale independently
  • Endpoints don't contain unique path segments (e.g. POST /users/{UUID}/resend-email), as they spam metrics and traces; otherwise, unique path segments and query parameters are stripped from metrics and traces
  • A load/performance test has been run at least once before launch, with a documented baseline (requests/sec, p95/p99 latency, breaking point) to compare against future regressions
  • CORS is explicitly configured with an allow-list of origins (not *) if the API is called from a browser
  • Feature flags exist for risky changes, so a bad feature can be turned off without a redeploy
  • Retries on transient failures use exponential backoff with jitter and are capped, so a retry storm can't take down a struggling dependency
  • A circuit breaker (or equivalent) protects calls to unreliable external dependencies, so one slow dependency can't exhaust the app's connection pool or threads
  • The app degrades gracefully when a non-critical dependency is unavailable (e.g. serves cached/stale data or a reduced feature set) instead of failing the whole request

Timeouts​

  • All outbound calls (HTTP, DB, queue) have an explicit timeout; no call can block indefinitely
  • The server's HTTP read and write timeouts are configurable (HTTP_READ_TIMEOUT, HTTP_WRITE_TIMEOUT), and the app cancels DB queries and closes connections when a timeout occurs

Server timeout notes:

  • HTTP_READ_TIMEOUT is the maximum duration for reading the entire request, including the body. It covers only the client request read and validation, and should always be fast (e.g. ~1s).
  • HTTP_WRITE_TIMEOUT is the maximum duration before timing out the response write. It resets whenever a new request's headers are read, and covers the whole response time (e.g. ~10s).

Node.js equivalents: server.requestTimeout (entire request read), server.headersTimeout (headers read), server.keepAliveTimeout (idle keep-alive connections), and server.timeout (socket inactivity). Node has no built-in write/response timeout, so enforce one with middleware (e.g. connect-timeout) or at the proxy/ingress level.

References​

Frontend apps​

  • Variables are read at runtime from a config.json file, not at the CI step
  • Static files are requested with a code-version param, which resets the browser cache after a new release: image123.webp?v=v1.4.10
  • Static files are served via a CDN
  • A built-in global maintenance page can be enabled via the backend-for-frontend (BFF) API or config.json

API design​

  • Every list endpoint is paginated; never return an unbounded collection. Prefer page/page_size (or cursor-based pagination for very large/real-time tables) with a sane default and a hard max page size
  • Input is validated at the boundary (schema/struct validation) before reaching business logic: reject bad input with a 400 instead of letting it become a 500 three layers down
  • PUT/DELETE are idempotent (calling twice has the same effect as once); consider idempotency keys for POST on payment/order-creation-type endpoints
  • Rate limiting/throttling is applied per client (API key or user), on top of any infra-level limiting, so the app stays protected if the edge limiter is misconfigured or bypassed
  • Response compression (gzip) is enabled for larger payloads
  • Filtering/sorting on list endpoints uses an explicit allow-list of fields; never pass a raw client-supplied field name into an ORDER BY
  • The API versioning strategy is explicit (URL path, header, etc.), so breaking changes can ship without breaking existing consumers
  • All endpoints are case-sensitive (/users ≠ /Users), avoiding duplicate URLs and routing/auth mismatches with the proxy. Express: app.set('case sensitive routing', true) and express.Router({ caseSensitive: true }) on every router (routers don't inherit the app setting)

Observability​

Health endpoints​

  • /health is the readiness probe and checks critical dependencies (DB); if the app has several dependencies, consider extra checks such as /health/db, /health/kafka, etc.
  • / or /status is the liveness probe (confirms the app is alive and not stuck, otherwise it will be restarted) and exposes some static app/build info
  • /health/external checks third-party service availability

Logs and errors​

Log enough to debug an incident without a redeploy.

  • All requests are logged, with an extra message for responses
  • Log levels are used consistently: debug, info, warn, error
  • Logs are written to STDOUT/STDERR only, never to files (structured JSON in deployed environments; human-readable pretty-printing is fine for local development)
  • Logs are structured (JSON), not freeform strings, so they stay queryable in ELK/Loki. Prefer an established format such as GELF over inventing a custom structure; whichever format is used, field names stay consistent (e.g. timestamp, level, message, request_id)
  • Errors are sent to Sentry (with a SENTRY_ENV variable for the environment tag)
  • No PII or secrets (passwords, tokens, full card numbers) land in logs; redact or mask before logging
  • Log volume and sampling are considered for high-throughput endpoints, so verbose logging doesn't overwhelm the logging pipeline or blow the budget
  • A log retention period is defined and enforced by the logging backend, consistent with the data retention policy

Metrics​

  • /metrics exposes Prometheus metrics
  • A request/response counter is labeled with method, endpoint, and status code
  • A request duration histogram is exposed (preferred over summaries, which can't be aggregated across replicas)
  • App-specific measurements cover the core logic (e.g. job status and duration)
  • Key business/product metrics are tracked alongside technical ones where relevant (e.g. signups, payments processed), so a regression in business impact is visible, not just infra health
  • Dashboards exist for the service (not just raw metrics) and are linked from the README/runbook, so an on-call engineer isn't building queries from scratch during an incident

References​

Alerts​

Alerts check for anomalies against a baseline. Prefer app-level metrics; use infra (e.g. nginx) metrics only as a fallback.

  • An alert exists for request-rate anomalies
  • An alert exists for error rate
  • An alert exists for latency
  • An alert exists for CPU/RAM
  • An alert exists for connection-pool saturation (DB pool, HTTP client pool); pool exhaustion is often the first sign of an incident, appearing before latency or error rate move
  • An alert exists for jobs that don't run or fail
  • SLOs (service level objectives) are defined for the service's key user journeys (e.g. "99.9% of checkout requests succeed in under 500ms"), and alerts fire on error-budget burn rate, not just raw threshold breaches
  • Alerts are actionable and routed to the right on-call rotation; an alert nobody can act on, or that pages the wrong team, trains people to ignore alerts
  • Alert thresholds are reviewed periodically to reduce noise, since flapping alerts erode trust in the whole alerting system

Tracing​

  • The app handles the X-Request-ID header and generates a random string or UUID if it's missing
  • The request ID is added to the request_id field on all logs
  • Application Performance Monitoring (APM) is in place
  • All external calls (other apps, DBs, queues, etc.) are covered by traces
  • Trace context propagates across service boundaries (e.g. W3C Trace Context headers), so a single request can be followed end-to-end across the whole call graph

References​

Graceful shutdown​

  • SIGINT (Ctrl+C) and SIGTERM are handled: in-flight requests complete, keep-alive connections close, and DB connections close
  • The app exits on critical/fatal errors
  • The readiness probe starts failing as soon as shutdown begins, so the load balancer/ingress stops routing new traffic before the process exits
  • In-flight background jobs/queue consumers finish or checkpoint their current unit of work before exiting, rather than being killed mid-task

DB​

  • DB connections use a pool
  • Pool size is managed via environment variables
  • All DB resources (tables, indexes, functions, policies, etc.) are created via a DB migration job
  • Migrations are forward-only: production migrations don't rely on down/rollback migrations
  • Migrations are backward-compatible with the previous app version (expand/contract pattern: add columns before removing them, deploy code before dropping columns), so a rolling deploy never has old code hitting a new schema mid-rollout
  • Destructive database changes are isolated in a separate migration and run only after the application no longer depends on the old schema
  • Slow-query logging is enabled; N+1 queries are checked for on hot paths (executed frequently or performance-critical)
  • The app separates read and write operations so it can use read replicas
  • Read replicas (if used) have documented replication-lag expectations, and the app tolerates eventually-consistent reads where they're used

Kubernetes / orchestration​

  • Separate readinessProbe, livenessProbe, and (for slow-starting apps) startupProbe are configured; conflating them causes either premature restarts or traffic sent to a not-yet-ready pod
  • CPU and memory requests are set, along with memory limits; set CPU limits only in edge cases where throttling the app is preferable to starving neighboring workloads
  • terminationGracePeriodSeconds matches how long graceful shutdown actually takes; if it's too short, SIGKILL cuts off in-flight requests
  • lifecycle.preStop.exec.command: ["sleep", "3"] is set, to avoid 502s while the ingress-nginx upstream syncs (every 1s)
  • A PodDisruptionBudget is set, so voluntary disruptions (node drains, cluster upgrades) don't take out every replica at once: e.g. minAvailable: 2 for services with 3+ replicas, or maxUnavailable: 1 (or a percentage) otherwise. A minAvailable equal to the replica count blocks node drains entirely
  • A HorizontalPodAutoscaler (or equivalent) is configured for services with variable load, with sane min/max replica bounds
  • Pod anti-affinity (or topology spread constraints) spreads replicas across nodes/availability zones, so a single node or AZ failure doesn't take out every replica

Backups & disaster recovery​

  • Automated, encrypted backups exist for every stateful dependency (DB, object storage, message queue state, if it matters)
  • Backups are stored somewhere that survives losing the primary environment (a different region/account)
  • A restore has been performed end-to-end at least once; untested backups routinely turn out to be broken
  • RTO (recovery time objective) and RPO (recovery point objective) are documented, even informally: "how much data/downtime can we tolerate" should be a decision, not a surprise

Compliance & data retention​

  • A data retention policy is defined: how long data is retained, what user data is stored in each type of storage, and whether an automated deletion process prevents uncontrolled data growth
  • PII is identified and specifically protected (encrypted at rest where appropriate, excluded from logs; see Logs and errors)
  • A documented process exists for handling a user's data-deletion request (GDPR "right to be forgotten" or equivalent), where applicable to your jurisdiction/users