1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
|
# Production operations runbook
## HTTP boundary
Production uses four listeners. Expose only `AIGW_PUBLIC_ADDRESS` and the Stripe path on
`AIGW_WEBHOOK_ADDRESS` to the internet. Put `AIGW_ADMIN_ADDRESS` behind SSO/VPN or an access
proxy. Keep `AIGW_OPERATIONS_ADDRESS` private to the orchestrator and monitoring network.
Set `AIGW_REQUIRE_HTTPS=true` behind TLS termination and list only the load balancer CIDRs in
`AIGW_TRUSTED_PROXY_CIDRS`; forwarding headers from other peers are discarded.
## Alerts
Page when `/readyz` is non-200 for five minutes, `aigw_ready == 0`,
`aigw_billing_settlement_spool_records > 0`, or the settlement backlog has an item older than
15 minutes. Warn when Redis is degraded, webhook events are unprocessed,
`aigw_stripe_refund_backlog` or `aigw_stripe_reconciliation_mismatches` is non-zero, disputes
need response, dead-letter mail exists, `aigw_billing_unmetered_successes` is non-zero, or
`aigw_billing_uncollected_micros` increases.
## Stripe reconciliation
Stripe only funds the prepaid wallet; inference never creates a Stripe charge. Run
`go run ./cmd/reconcile-billing` after deploys and investigate any non-zero exit. A paid Stripe
session with a local pending order is repaired through the same idempotent webhook transaction.
Never delete an orphaned local order. Resolve an uncredited missing session through the Admin UI,
or reverse an already credited missing session with the dedicated equal negative ledger action.
Both require `billing.adjust` and create an immutable resolution row. The maintenance CLI can do
the same only with the explicit `-resolve-confirmed-missing` flag and a written reason.
The gateway restricted key should grant only Checkout Sessions Write, Customer Portal Write,
Customers Write, Charges and Refunds Write, Payment Intents Read, and Invoices Read. Use a separate
test key for Stripe CLI. Configure the production Webhook signing secret independently and alert on
signature failures or a five-minute unprocessed backlog.
## Mail deliverability
Use a production SMTP provider with STARTTLS/TLS and inject all endpoints, credentials, sender, and
`AIGW_MAIL_FEEDBACK_SECRET` through the deployment secret manager. Configure SPF, DKIM, and a DMARC
policy for the From domain in DNS. Map the provider's delivered/bounce/complaint event into the
normalized `/mail/feedback` payload and HMAC-sign the exact raw body; bounce/complaint events suppress
future delivery. Monitor dead-letter outbox rows. Low-balance and anomalous-spend notifications are
deduplicated per recipient and UTC day.
## Database migrations
Run `cmd/migrate` before deploying application instances and keep `auto_migrate=false` in
production. Migrations use a PostgreSQL advisory lock and a recorded checksum. Schema rollback
is always a reviewed forward migration; restore a database backup only for whole-release
rollback after stopping writers. Never edit an already applied migration body.
## Backup and restore
Run `scripts/backup-postgres.sh` from a host with `pg_dump`, encrypted storage, and a scoped
database credential. Test `scripts/restore-drill.sh` into a disposable isolated database at
least monthly. Record row-count evidence and application smoke tests before deleting the drill.
Before each release, run `scripts/load-smoke.sh` against a non-production upstream, then
`scripts/redis-fault-drill.sh` to prove Redis is optional and PostgreSQL polling keeps readiness.
Exercise PostgreSQL failover separately and verify pending settlement jobs resume without duplicate
ledger entries. Archive the command output with the release evidence.
## Credential rotation
1. Put the new key in `AIGW_CREDENTIAL_KEY` and the old key in
`AIGW_CREDENTIAL_PREVIOUS_KEYS` on every instance.
2. Deploy and verify snapshot/MFA/mail decryption.
3. Run `go run ./cmd/rotate-credentials` once with both variables configured.
4. Restart with only the new key, then revoke the old key from the secret manager.
Stripe credentials should be separate restricted keys per environment and service. Rotate the
Webhook endpoint secret with overlapping endpoints, then remove the previous endpoint after all
instances use the new value.
## Retention
Keep immutable billing ledger and Stripe financial event evidence for the statutory period set
with finance/legal. Partition and archive Usage by month. Audit events should be exported to
append-only object storage before database deletion. `admin.audit_retention_days` defaults to
2555 days and `admin.security_retention_days` defaults to 30 days; a daily worker enforces both.
Account action tokens, expired sessions, WebAuthn challenges, sent mail and login throttles are
security data covered by the shorter window. Set the audit period only after finance/legal and
incident-response owners approve the archive and retrieval procedure.
|
