Skip to main content

Deployment Readiness Checklist

A checklist of actions to review before deploying an app to production.

README.md​

  • About the app (its purpose)
  • How to build
  • How to start
  • Dependencies (DB, external services)
  • Link to the OpenAPI/Swagger spec (or generated docs), if the app exposes an API
  • Link to the architecture diagram / ADRs for anything non-trivial
  • Link to the runbook and on-call/escalation contact for this service
  • Local development setup is documented and reproducible (e.g. docker-compose up, seed data, required tooling versions)

Variables & secrets​

  • The app is configured via environment variables plus a .env or config file

  • Environment variables take priority over configuration files (but have lower priority than command-line arguments)

  • For static frontend apps, variables are configured at runtime via a config.json file - avoid setting them at the CI step

  • The app validates all required config/secrets at startup and fails fast with a clear error, instead of failing on the first request that needs them

  • No default values for secrets (JWT secret, DB password, API keys) - a missing secret in production should refuse to boot, not silently fall back to a dev default

  • Secrets are stored in a dedicated secret manager (Vault, AWS/GCP Secrets Manager, sealed-secrets, K8s Secrets + encryption at rest) - never committed to git, even in "private" repos

  • Secrets differ per environment (dev/staging/prod); no environment ever reuses another's credentials

  • Secrets and API keys are rotated on a defined schedule, and rotation doesn't require a full redeploy

  • Access to production secrets is scoped and audited (who can read/write them, and when they last did)

  • The app exposes the following configurable environment variables:

    • SERVER_HOST & SERVER_PORT / SERVER_ADDR (e.g. 0.0.0.0:8080)
    • HTTP_READ_TIMEOUT & HTTP_WRITE_TIMEOUT / HTTP_TIMEOUT

Note:

  • HTTP_READ_TIMEOUT - the maximum duration for reading the entire request, including the body. Covers only client request read and validation, and should always be fast (e.g. ~1s).
  • HTTP_WRITE_TIMEOUT - the maximum duration before timing out the response write. It resets whenever a new request's headers are read, and covers the whole response time (e.g. ~10s).

The app should correctly handle timeouts: cancel DB queries and close connections when they occur.

References:

Docker​

  • The app is Dockerized, and a docker-compose.yml is present for dependencies, if any

  • A .dockerignore file is present:

    .git
    .gitlab-ci.yml
    Dockerfile

  • Avoid running as root (e.g. USER node)

  • Only one process runs - exec to the process directly (not via npm run)

  • A docker-entrypoint.sh is used if the app has separate runtime logic (e.g. web + workers) or runs migrations

  • The container filesystem is read-only where possible (readOnlyRootFilesystem: true), with explicit writable mounts (e.g. /tmp) only where needed

Docker image​

  • The image should be as small as possible - under 200 MB is acceptable.
    • Alpine-based, if possible
    • Multi-staged (separate builder stage)
    • Build cache is cleaned (npm cache clean --force)
    • Only production dependencies are installed - npm ci --production is preferred over npm install --only=prod or cleaning up afterward with npm prune --production
  • The image is scanned for known CVEs (Trivy/Grype) as a CI gate, not just ad hoc
  • The image is pinned to a digest or an immutable tag in production manifests - never :latest
  • The base image is kept up to date and rebuilt periodically, even without app code changes, to pick up OS-level security patches

App runtime​

  • The app is horizontally scalable
  • If the app has separate logic (e.g. web + worker), each runs as its own process so they can scale independently
  • Healthchecks are separated accordingly
  • Avoid unique endpoints like POST /users/{UUID}/resend-email, as they generate metric and trace spam - alternatively, strip unique path segments and query parameters from metrics and traces
  • For the frontend apps all static files should be requested with extra params based on code version, it allows to reset browser cache after releasing new version: image123.webp?v=v1.4.10
  • The frontend apps should utilize cdn for the static files
  • The app is stateless - no in-memory session/cache that would break when a second replica starts (use Redis/DB/sticky sessions deliberately, not by accident)
  • A load/performance test has been run at least once before launch, with a documented baseline (requests/sec, p95/p99 latency, breaking point) to compare against future regressions
  • CORS is explicitly configured (an allow-list of origins, not *) if the API is called from a browser
  • Feature flags exist for risky changes, so a bad feature can be turned off without a redeploy
  • The frontend apps should have a built-in global maintenance page that can be enabled via the BFE API or through config.json
  • All outbound calls (HTTP, DB, queue) have an explicit timeout - no call can block indefinitely
  • Retries on transient failures use exponential backoff with jitter, and are capped (a retry storm shouldn't be able to take down a struggling dependency)
  • A circuit breaker (or equivalent) protects calls to unreliable external dependencies, so one slow dependency can't exhaust the app's own connection pool or threads
  • The app degrades gracefully when a non-critical dependency is unavailable (e.g. serve cached/stale data or a reduced feature set) rather than failing the whole request
  • The app separates read and write operations, allowing it to use database read replicas

Health and metrics​

  • /health acts as a readiness probe and checks critical dependencies (DB); if the app has several dependencies, consider adding extra healthchecks such as /health/db, /health/kafka, etc.
  • / or /status acts as a liveness probe (confirms the app is alive and not stuck - otherwise it will be restarted) and exposes some static app/build info
  • /health-srv checks third-party service availability
  • /metrics exposes Prometheus metrics - requests per endpoint, error rate, request duration

Logs and errors​

More logging is better.

  • All requests are logged, with extra messages for responses
  • Log levels are used consistently: debug, info, warn, error
  • No logging to file
  • All logs are human-readable and written to STDOUT/STDERR
  • Errors are sent to Sentry (add a SENTRY_ENV variable for the environment tag)
  • GELF support, with JSON logs
  • Logs are structured (JSON) with consistent field names (timestamp, level, message, request_id), not freeform strings - this keeps them queryable in ELK/Loki
  • No PII or secrets (passwords, tokens, full card numbers) ever land in logs - redact or mask before logging
  • Log volume and sampling are considered for high-throughput endpoints, so verbose logging doesn't overwhelm the logging pipeline or blow the budget
  • A log retention period is defined and enforced by the logging backend, consistent with the data retention policy

Metrics​

  • Requests and responses counter, labeled with method, endpoint, and status code
  • Response quantile summary metric
  • Request duration histogram
  • Any other measurements relevant to the app's core logic (e.g. job status and duration)
  • Key business/product metrics are tracked alongside technical metrics where relevant (e.g. signups, payments processed), so a regression in business impact is visible, not just infra health
  • Dashboards exist for the service (not just raw metrics) and are linked from the README/runbook, so an on-call engineer isn't building queries from scratch during an incident

Client libraries: https://prometheus.io/docs/instrumenting/clientlibs/

Examples:

Alerts​

Check for anomalies against a baseline - preferably app-level metrics; infra (nginx/linkerd) metrics only as a fallback.

  • Request rate
  • Error rate
  • Latency
  • CPU/RAM
  • Saturation of connection pools (DB pool, HTTP client pool) - pool exhaustion is often the first sign of an incident, appearing before latency or error rate move
  • Job not running / job failures
  • SLOs (service level objectives) are defined for the service's key user journeys (e.g. "99.9% of checkout requests succeed in under 500ms"), and alerts fire on error-budget burn rate, not just raw threshold breaches
  • Alerts are actionable and routed to the right on-call rotation - an alert nobody can act on, or that pages the wrong team, trains people to ignore alerts
  • Alert thresholds are periodically reviewed to reduce noise (flapping alerts erode trust in the whole alerting system)

Tracing​

  • The app handles the X-Request-ID header; if not present, it generates a random string or UUID
  • The request ID is added to the request_id log field on all logs
  • Application Performance Monitoring (APM) is in place
  • All external calls are covered by traces (other apps, DBs, queues, etc.)
  • Trace context propagates across service boundaries (e.g. W3C Trace Context headers), so a single request can be followed end-to-end across the whole call graph, not just within one service

Elastic APM as an example: https://www.elastic.co/guide/en/apm/agent/index.html

Graceful shutdown​

  • SIGINT (Ctrl+C) and SIGTERM are handled - in-flight requests complete, keep-alive connections close, and DB connections close
  • The app exits on critical/fatal errors
  • The readiness probe starts failing as soon as shutdown begins, so the load balancer/ingress stops routing new traffic before the process actually exits
  • In-flight background jobs/queue consumers finish or checkpoint their current unit of work before exiting, rather than being killed mid-task

DB​

  • DB connections use a pool
  • Pool size is managed via environment variables
  • All DB resources (tables, indexes, functions, policies, etc.) are created via a DB migration job
  • Database migrations are forward-only - production migrations must not rely on down/rollback migrations
  • Migrations are backward-compatible with the previous app version (expand/contract pattern: add columns before removing them, deploy code before dropping columns), so a rolling deploy never has old code hitting a new schema mid-rollout
  • Destructive database changes are isolated into a separate migration and are only executed after the application no longer depends on the old schema
  • Rollback is performed by deploying the previous application version, not by reverting the database schema
  • Slow-query logging is enabled; N+1 queries are checked for on hot paths (executed frequently or performance-critical)
  • Read replicas (if used) have documented replication-lag expectations, and the app tolerates eventually-consistent reads where they're used

CI​

  • Builds run inside Docker containers
  • The image tag is never latest:
    • GitLab CI: DOCKER_IMAGE_TAG="$CI_COMMIT_REF_SLUG-$COMMIT_DATE-$CI_PIPELINE_ID" for
    • GitHub actions: DOCKER_IMAGE_TAG="${GITHUB_REF_SLUG}-${COMMIT_DATE}-run-${GITHUB_RUN_NUMBER}" with github-slug-action
  • Lint and test steps are included
  • Images are published to DockerHub
  • Slack notifications are configured
  • The DockerHub repo is set to private (it's public by default)
  • GitLab environment variables are used: environment.name and environment.url
  • A cleanup step removes created images and intermediate build images
  • Dependency versions are pinned (lockfile committed: go.sum, package-lock.json, etc.)
  • A dependency vulnerability scan (npm audit, govulncheck, Snyk/Trivy) runs as a blocking or at least visible step
  • Dependabot/Renovate (or equivalent) is enabled so dependency updates are proposed automatically, rather than discovered during an incident
  • Static analysis / SAST scanning runs in the pipeline (e.g. Semgrep, gosec, ESLint security rules)
  • Secrets scanning runs on every commit/MR (e.g. gitleaks, trufflehog) to catch accidentally committed credentials before merge

CD​

  • CI and CD are separated — CI builds, tests, and publishes immutable artifacts; CD promotes and deploys an already-built artifact without rebuilding it
  • Only artifacts built and tested by CI are deployed - nothing is built directly from a developer machine or deployed from an unverified/untagged branch
  • Production deployments use an immutable artifact reference (image digest or immutable version tag), never latest or a mutable tag
  • The exact artifact deployed to production is traceable to the source commit, CI pipeline, and test results that produced it
  • Deploys to production require passing through lower environments first (dev → staging → prod), except for documented emergency-fix paths
  • Configuration changes are version-controlled and reviewed; production configuration is not changed manually without an audit trail
  • The rollout strategy is explicit (rolling update, recreate, blue-green, or canary), with maxSurge/maxUnavailable (or equivalent) tuned for the service's traffic pattern. If recreate strategy is used, the expected downtime is explicitly accepted and documented, and the service has an appropriate maintenance/availability plan
  • The deployment maintains sufficient healthy capacity throughout the rollout - a deployment must not intentionally take the service below its required availability level
  • The deployment waits for new instances to become ready before terminating healthy old instances
  • Rollout progress and deployment health are observable (ready replicas, error rate, latency, saturation, and application-specific health signals)
  • Automated smoke tests run against the environment immediately after deploy, before the deploy is considered complete
  • Critical user journeys are verified after deployment, not only that the process/container started successfully
  • Rollback is a tested, one-step operation (redeploy the previous immutable image / revert the deployment manifest) - not something improvised for the first time during an incident
  • The previous known-good artifact remains available for immediate rollback
  • Rollback does not depend on rebuilding the previous version
  • Rollback is automated or triggered on defined signals (error-rate spike, failed smoke test, crash-loop, readiness failure, or significant latency regression), not solely dependent on a human noticing in time
  • Database migrations are backward-compatible with the currently running application version, so application rollback does not require a database rollback
  • Destructive database/schema changes are deployed separately from the code that starts depending on them and follow the forward-only / expand-contract migration strategy
  • Deployments have a defined timeout; a rollout that does not become healthy within the expected time is considered failed
  • Failed deployments do not leave a partially updated production environment without being clearly reported and handled
  • Deploys are visible to the team (Slack notification, changelog, release notes) so people know what changed and when
  • Deployment notifications include at least the service, environment, version/artifact, source commit, deployer/trigger, and deployment status
  • Risky deploys avoid peak-traffic windows and periods of reduced on-call coverage (e.g. Friday evening, holidays), unless there is a documented reason to deploy
  • Production deployments have an identified owner/on-call engineer who can monitor the rollout and perform a rollback
  • There is a documented emergency deployment path that bypasses normal promotion requirements while preserving auditability
  • Deployment frequency and change-failure rate are tracked, so the team has a baseline for whether the deployment process is getting safer or riskier over time

API design​

  • Pagination on every list endpoint - never return an unbounded collection. Prefer page/page_size (or cursor-based pagination for very large/real-time tables) with a sane default and a hard max page size
  • Input is validated at the boundary (schema/struct validation) before reaching business logic - reject bad input with a 400, don't let it become a 500 three layers down
  • Idempotency for PUT/DELETE (calling twice has the same effect as once); consider idempotency keys for POST on payment/order-creation-type endpoints
  • Rate limiting/throttling per client (API key or user), on top of any infra-level limiting - this protects the app even if the edge limiter is misconfigured or bypassed
  • Response compression (gzip) is enabled for larger payloads
  • Filtering/sorting on list endpoints uses an explicit allow-list of fields - never pass a raw client-supplied field name straight into an ORDER BY
  • API versioning strategy is explicit (URL path, header, etc.), so breaking changes can ship without breaking existing consumers

Kubernetes / orchestration​

  • Separate readinessProbe, livenessProbe, and (for slow-starting apps) startupProbe are configured - conflating these causes either premature restarts or traffic being sent to a not-yet-ready pod
  • CPU and memory requests are set, along with memory limits. Set CPU limits only in edge cases where throttling the app is preferable to starving neighboring workloads
  • terminationGracePeriodSeconds matches how long graceful shutdown actually takes - too short, and SIGKILL cuts off in-flight requests
  • lifecycle.preStop.exec.command: ["sleep", "3"] is set, to avoid 502s while the ingress-nginx upstream syncs (every 1s)
  • podDisruptionBudget.minAvailable: 2 is set, so voluntary disruptions (node drains, cluster upgrades) don't take out every replica at once
  • HorizontalPodAutoscaler (or equivalent) is configured for services with variable load, with sane min/max replica bounds
  • Pod anti-affinity (or topology spread constraints) spreads replicas across nodes/availability zones, so a single node or AZ failure doesn't take out every replica

Backups & disaster recovery​

  • Automated, encrypted backups exist for every stateful dependency (DB, object storage, message queue state, if it matters)
  • Backups are stored somewhere that survives losing the primary environment (a different region/account)
  • A restore has actually been performed end-to-end at least once - untested backups routinely turn out to be broken
  • RTO (recovery time objective) and RPO (recovery point objective) are documented, even informally - "how much data/downtime can we tolerate" should be a decision, not a surprise

Compliance & data retention​

  • A data retention policy is defined - specify how long data is retained, what user data is stored in each type of storage, and whether an automated deletion process exists to prevent uncontrolled data growth
  • PII is identified and specifically protected (encrypted at rest where appropriate, excluded from logs - see the Logs section)
  • A documented process exists for handling a user's data-deletion request (GDPR "right to be forgotten" or equivalent), where applicable to your jurisdiction/users