feat: harden prepaid billing and commercial operations

1 files changed, 82 insertions, 0 deletions

diff --git a/docs/runbook.md b/docs/runbook.md
new file mode 100644
index 0000000..ddd1a8c
--- /dev/null
+++ b/docs/runbook.md
@@ -0,0 +1,82 @@
+# Production operations runbook
+
+## HTTP boundary
+
+Production uses four listeners. Expose only `AIGW_PUBLIC_ADDRESS` and the Stripe path on
+`AIGW_WEBHOOK_ADDRESS` to the internet. Put `AIGW_ADMIN_ADDRESS` behind SSO/VPN or an access
+proxy. Keep `AIGW_OPERATIONS_ADDRESS` private to the orchestrator and monitoring network.
+Set `AIGW_REQUIRE_HTTPS=true` behind TLS termination and list only the load balancer CIDRs in
+`AIGW_TRUSTED_PROXY_CIDRS`; forwarding headers from other peers are discarded.
+
+## Alerts
+
+Page when `/readyz` is non-200 for five minutes, `aigw_ready == 0`,
+`aigw_billing_settlement_spool_records > 0`, or the settlement backlog has an item older than
+15 minutes. Warn when Redis is degraded, webhook events are unprocessed,
+`aigw_stripe_refund_backlog` or `aigw_stripe_reconciliation_mismatches` is non-zero, disputes
+need response, dead-letter mail exists, `aigw_billing_unmetered_successes` is non-zero, or
+`aigw_billing_uncollected_micros` increases.
+
+## Stripe reconciliation
+
+Stripe only funds the prepaid wallet; inference never creates a Stripe charge. Run
+`go run ./cmd/reconcile-billing` after deploys and investigate any non-zero exit. A paid Stripe
+session with a local pending order is repaired through the same idempotent webhook transaction.
+Never delete an orphaned local order. Resolve an uncredited missing session through the Admin UI,
+or reverse an already credited missing session with the dedicated equal negative ledger action.
+Both require `billing.adjust` and create an immutable resolution row. The maintenance CLI can do
+the same only with the explicit `-resolve-confirmed-missing` flag and a written reason.
+
+The gateway restricted key should grant only Checkout Sessions Write, Customer Portal Write,
+Customers Write, Charges and Refunds Write, Payment Intents Read, and Invoices Read. Use a separate
+test key for Stripe CLI. Configure the production Webhook signing secret independently and alert on
+signature failures or a five-minute unprocessed backlog.
+
+## Mail deliverability
+
+Use a production SMTP provider with STARTTLS/TLS and inject all endpoints, credentials, sender, and
+`AIGW_MAIL_FEEDBACK_SECRET` through the deployment secret manager. Configure SPF, DKIM, and a DMARC
+policy for the From domain in DNS. Map the provider's delivered/bounce/complaint event into the
+normalized `/mail/feedback` payload and HMAC-sign the exact raw body; bounce/complaint events suppress
+future delivery. Monitor dead-letter outbox rows. Low-balance and anomalous-spend notifications are
+deduplicated per recipient and UTC day.
+
+## Database migrations
+
+Run `cmd/migrate` before deploying application instances and keep `auto_migrate=false` in
+production. Migrations use a PostgreSQL advisory lock and a recorded checksum. Schema rollback
+is always a reviewed forward migration; restore a database backup only for whole-release
+rollback after stopping writers. Never edit an already applied migration body.
+
+## Backup and restore
+
+Run `scripts/backup-postgres.sh` from a host with `pg_dump`, encrypted storage, and a scoped
+database credential. Test `scripts/restore-drill.sh` into a disposable isolated database at
+least monthly. Record row-count evidence and application smoke tests before deleting the drill.
+
+Before each release, run `scripts/load-smoke.sh` against a non-production upstream, then
+`scripts/redis-fault-drill.sh` to prove Redis is optional and PostgreSQL polling keeps readiness.
+Exercise PostgreSQL failover separately and verify pending settlement jobs resume without duplicate
+ledger entries. Archive the command output with the release evidence.
+
+## Credential rotation
+
+1. Put the new key in `AIGW_CREDENTIAL_KEY` and the old key in
+   `AIGW_CREDENTIAL_PREVIOUS_KEYS` on every instance.
+2. Deploy and verify snapshot/MFA/mail decryption.
+3. Run `go run ./cmd/rotate-credentials` once with both variables configured.
+4. Restart with only the new key, then revoke the old key from the secret manager.
+
+Stripe credentials should be separate restricted keys per environment and service. Rotate the
+Webhook endpoint secret with overlapping endpoints, then remove the previous endpoint after all
+instances use the new value.
+
+## Retention
+
+Keep immutable billing ledger and Stripe financial event evidence for the statutory period set
+with finance/legal. Partition and archive Usage by month. Audit events should be exported to
+append-only object storage before database deletion. `admin.audit_retention_days` defaults to
+2555 days and `admin.security_retention_days` defaults to 30 days; a daily worker enforces both.
+Account action tokens, expired sessions, WebAuthn challenges, sent mail and login throttles are
+security data covered by the shorter window. Set the audit period only after finance/legal and
+incident-response owners approve the archive and retrieval procedure.