From 3f702084d20b3c3a3ea916f3110e99b22bda60b3 Mon Sep 17 00:00:00 2001 From: Chia Date: Thu, 6 Aug 2026 15:58:57 +1200 Subject: feat: complete commercial developer workflows Add tenant-safe usage observability, prepaid billing controls, API key lifecycle management, Embeddings metering, configurable billing alerts, and resilient provider health propagation. Harden Stripe failure handling, migrations, readiness, and the authenticated control-plane UI with end-to-end verification evidence. --- docs/runbook.md | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) (limited to 'docs/runbook.md') diff --git a/docs/runbook.md b/docs/runbook.md index ddd1a8c..6062e0c 100644 --- a/docs/runbook.md +++ b/docs/runbook.md @@ -44,7 +44,9 @@ deduplicated per recipient and UTC day. ## Database migrations Run `cmd/migrate` before deploying application instances and keep `auto_migrate=false` in -production. Migrations use a PostgreSQL advisory lock and a recorded checksum. Schema rollback +production. Migrations use one stable PostgreSQL advisory lock across all versions and a recorded +checksum. An already-applied matching version exits without replaying DDL, so concurrent process +starts do not contend with normal control-plane queries. Schema rollback is always a reviewed forward migration; restore a database backup only for whole-release rollback after stopping writers. Never edit an already applied migration body. @@ -55,7 +57,10 @@ database credential. Test `scripts/restore-drill.sh` into a disposable isolated least monthly. Record row-count evidence and application smoke tests before deleting the drill. Before each release, run `scripts/load-smoke.sh` against a non-production upstream, then -`scripts/redis-fault-drill.sh` to prove Redis is optional and PostgreSQL polling keeps readiness. +`scripts/redis-fault-drill.sh` to prove Redis is optional, PostgreSQL polling keeps readiness, +and the subscriber reconnects after Redis returns. Set `AIGW_READY_URL`, `AIGW_REDIS_CONTAINER`, +and, when shared provider health is enabled, `AIGW_METRICS_URL`; the metrics assertion verifies +`aigw_provider_health_shared_connected` transitions from 1 to 0 and back to 1. Exercise PostgreSQL failover separately and verify pending settlement jobs resume without duplicate ledger entries. Archive the command output with the release evidence. -- cgit v1.2.3