diff options
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/runbook.md | 82 |
1 files changed, 82 insertions, 0 deletions
diff --git a/docs/runbook.md b/docs/runbook.md new file mode 100644 index 0000000..ddd1a8c --- /dev/null +++ b/docs/runbook.md @@ -0,0 +1,82 @@ +# Production operations runbook + +## HTTP boundary + +Production uses four listeners. Expose only `AIGW_PUBLIC_ADDRESS` and the Stripe path on +`AIGW_WEBHOOK_ADDRESS` to the internet. Put `AIGW_ADMIN_ADDRESS` behind SSO/VPN or an access +proxy. Keep `AIGW_OPERATIONS_ADDRESS` private to the orchestrator and monitoring network. +Set `AIGW_REQUIRE_HTTPS=true` behind TLS termination and list only the load balancer CIDRs in +`AIGW_TRUSTED_PROXY_CIDRS`; forwarding headers from other peers are discarded. + +## Alerts + +Page when `/readyz` is non-200 for five minutes, `aigw_ready == 0`, +`aigw_billing_settlement_spool_records > 0`, or the settlement backlog has an item older than +15 minutes. Warn when Redis is degraded, webhook events are unprocessed, +`aigw_stripe_refund_backlog` or `aigw_stripe_reconciliation_mismatches` is non-zero, disputes +need response, dead-letter mail exists, `aigw_billing_unmetered_successes` is non-zero, or +`aigw_billing_uncollected_micros` increases. + +## Stripe reconciliation + +Stripe only funds the prepaid wallet; inference never creates a Stripe charge. Run +`go run ./cmd/reconcile-billing` after deploys and investigate any non-zero exit. A paid Stripe +session with a local pending order is repaired through the same idempotent webhook transaction. +Never delete an orphaned local order. Resolve an uncredited missing session through the Admin UI, +or reverse an already credited missing session with the dedicated equal negative ledger action. +Both require `billing.adjust` and create an immutable resolution row. The maintenance CLI can do +the same only with the explicit `-resolve-confirmed-missing` flag and a written reason. + +The gateway restricted key should grant only Checkout Sessions Write, Customer Portal Write, +Customers Write, Charges and Refunds Write, Payment Intents Read, and Invoices Read. Use a separate +test key for Stripe CLI. Configure the production Webhook signing secret independently and alert on +signature failures or a five-minute unprocessed backlog. + +## Mail deliverability + +Use a production SMTP provider with STARTTLS/TLS and inject all endpoints, credentials, sender, and +`AIGW_MAIL_FEEDBACK_SECRET` through the deployment secret manager. Configure SPF, DKIM, and a DMARC +policy for the From domain in DNS. Map the provider's delivered/bounce/complaint event into the +normalized `/mail/feedback` payload and HMAC-sign the exact raw body; bounce/complaint events suppress +future delivery. Monitor dead-letter outbox rows. Low-balance and anomalous-spend notifications are +deduplicated per recipient and UTC day. + +## Database migrations + +Run `cmd/migrate` before deploying application instances and keep `auto_migrate=false` in +production. Migrations use a PostgreSQL advisory lock and a recorded checksum. Schema rollback +is always a reviewed forward migration; restore a database backup only for whole-release +rollback after stopping writers. Never edit an already applied migration body. + +## Backup and restore + +Run `scripts/backup-postgres.sh` from a host with `pg_dump`, encrypted storage, and a scoped +database credential. Test `scripts/restore-drill.sh` into a disposable isolated database at +least monthly. Record row-count evidence and application smoke tests before deleting the drill. + +Before each release, run `scripts/load-smoke.sh` against a non-production upstream, then +`scripts/redis-fault-drill.sh` to prove Redis is optional and PostgreSQL polling keeps readiness. +Exercise PostgreSQL failover separately and verify pending settlement jobs resume without duplicate +ledger entries. Archive the command output with the release evidence. + +## Credential rotation + +1. Put the new key in `AIGW_CREDENTIAL_KEY` and the old key in + `AIGW_CREDENTIAL_PREVIOUS_KEYS` on every instance. +2. Deploy and verify snapshot/MFA/mail decryption. +3. Run `go run ./cmd/rotate-credentials` once with both variables configured. +4. Restart with only the new key, then revoke the old key from the secret manager. + +Stripe credentials should be separate restricted keys per environment and service. Rotate the +Webhook endpoint secret with overlapping endpoints, then remove the previous endpoint after all +instances use the new value. + +## Retention + +Keep immutable billing ledger and Stripe financial event evidence for the statutory period set +with finance/legal. Partition and archive Usage by month. Audit events should be exported to +append-only object storage before database deletion. `admin.audit_retention_days` defaults to +2555 days and `admin.security_retention_days` defaults to 30 days; a daily worker enforces both. +Account action tokens, expired sessions, WebAuthn challenges, sent mail and login throttles are +security data covered by the shorter window. Set the audit period only after finance/legal and +incident-response owners approve the archive and retrieval procedure. |