How Secrets Management Actually Works

AWS access keys, DB passwords, OAuth client secrets, third-party API tokens — every production service consumes secrets. Where do they live? .env? git? Environment variables? Vault? How do you rotate them? This guide covers the #1 source of secret leaks all the way up to modern dynamic credentials.

#1 Source of Secret Leaks — git commits

The familiar story:
1. Put AWS keys in .env
2. Forget to add .env to .gitignore
3. git add . → commit → push
4. GitHub automated scanners (Amazon, Google) find it within 5 min
5. Amazon emails: "your key leaked, revoke now"
6. Meanwhile attackers also saw it → bitcoin miner spun up, $10K bill

Why it's bad:
- git history is forever — even after squash merge it's in the reflog
- public repos = GitHub crawl bots scan hourly
- private repos = any insider can leak

Defenses:
1. Add .env / *.pem / *.key to .gitignore upfront
2. pre-commit hooks (gitleaks, talisman) scan automatically
3. Make it a required PR review item
4. Already leaked? Revoke + rotate immediately; cleaning the repo isn't enough

Level 1 — Environment Variables (.env)

.env (not in git)
  DATABASE_URL=postgres://...
  AWS_ACCESS_KEY_ID=AKIA...

Application:
  process.env.DATABASE_URL

Pros: simple, universal, dev-friendly
Cons:
- Every process sees every secret (no least privilege)
- Logged dumps leak (printing process.env)
- Container env exposed via docker inspect
- Rotation requires restart

Level 2 — Cloud Secrets Manager

AWS Secrets Manager / GCP Secret Manager / Azure Key Vault.

Application:
  const secret = await secretsClient.getSecretValue({SecretId: "db-password"});

Pros:
- Central management + IAM-based access (which role reads which secret)
- Automatic rotation (the cloud refreshes some secrets directly)
- Audit log (who read what, when)
- Versioning (fallback to a previous secret)
- KMS encryption at rest

Cons:
- Cost ($0.4/secret/month + API calls)
- Cloud lock-in
- SDK + IAM setup per app

Level 3 — HashiCorp Vault

Open-source, self-host. Multi-cloud / on-prem.

Features:
- Secret storage (like cloud Secrets Manager)
- Dynamic credentials: generate DB users / IAM credentials on demand,
  short-lived
- Transit encryption (encryption as a service)
- PKI (own CA, issues certs)
- Many backends (KMS, AWS, GCP, Azure integrations)

Dynamic DB credential example:
  app → request Vault
  Vault → create temp user in DB (read perms, TTL 1h)
        → return username/password to app
  app → use that credential for 1 hour
  TTL expires → Vault deletes the user from DB

→ No long-lived credentials. Even if leaked, expires in 1h.

Envelope Encryption — A Secret for the Secrets

Wherever you store secrets — if that storage leaks?
→ All secrets leak.

Solution: envelope encryption.
1. Each secret encrypted with a random data key (DEK)
2. DEK encrypted with a master key (KEK, kept by KMS)
3. Store encrypted secret + encrypted DEK together

On decrypt:
1. Send encrypted DEK to KMS → receive plain DEK
2. Decrypt secret with plain DEK
3. Discard plain DEK from memory immediately

Benefits:
- Storage leak → only encrypted secrets exposed (cannot decrypt)
- KMS leak → all encrypted DEKs become decryptable (worst case)
- Both must leak for true danger

→ KMS master key sits inside a hardware HSM, leak is near impossible.

Rotation — Without Downtime

Naive:
  Change password → redeploy app → downtime

Approach 1 — Dual secret window:
  step 1: generate new password → app tries both (old + new)
  step 2: all clients move to new
  step 3: retire old

Approach 2 — Dynamic credentials (Vault):
  App constantly requests fresh credentials
  → Rotation is a backend concern, zero app impact

Approach 3 — Rolling restart:
  Cloud Secrets Manager promotes a new version
  → Apps restart one by one, picking up the new secret

→ Either way, apps must assume "secrets can change".

Secret Scanning — After-the-Fact Detection

Common Pitfalls

• Baking secrets into docker images — extractable via docker history / layers. Use multi-stage builds + runtime mounts.
• Logging process.env whole — every env shows up on errors. Redaction middleware required.
• Secrets in cloud-init / startup scripts — retrievable later via EC2 metadata. Call cloud Secrets Manager instead.
• No granular permissions — one IAM role reads all secrets. Per-service roles + secret-specific resource ARNs.
• No rotation — assume leakage. Force 90 / 180 day rotation.

Wrap-up

Evolution of secrets management — env file → cloud Secrets Manager → Vault dynamic credentials. Higher tier = less leak impact + more operational cost.

Practical start — git pre-commit hooks (gitleaks) + strict .gitignore + cloud Secrets Manager (the native one for AWS / GCP / Azure) + 90-day rotation. Vault when you need multi-cloud or on-prem.