feat: complete commercial developer workflowspublish-commercial-control-plane

Add tenant-safe usage observability, prepaid billing controls, API key lifecycle management, Embeddings metering, configurable billing alerts, and resilient provider health propagation. Harden Stripe failure handling, migrations, readiness, and the authenticated control-plane UI with end-to-end verification evidence.

1 files changed, 70 insertions, 0 deletions

diff --git a/PROGRESS.md b/PROGRESS.md
new file mode 100644
index 0000000..111212c
--- /dev/null
+++ b/PROGRESS.md
@@ -0,0 +1,70 @@
+# Commercial Developer Experience Progress
+
+Last updated: 2026-08-06 (Pacific/Auckland)
+
+This file records implementation status and verification evidence for the
+ZenMux-aligned developer journey. A feature is only marked complete when its UI,
+API, PostgreSQL state, runtime behavior, and relevant billing path have been
+exercised with real data.
+
+## P0 Developer Observability And Spend Controls
+
+| Capability | Status | Current evidence | Remaining work |
+| --- | --- | --- | --- |
+| Request usage ledger | Complete | PostgreSQL `usage_events`, tenant-scoped filters, request details, token/cache/cost/status/latency/TTFT UI, stable cursor pagination and page controls; every request debit in the balance ledger opens the matching usage record; a real `gpt-5.5` request persisted with 2,693 ms TTFT and charged 1,249 micro-USD | Partition/archive policy is a later scale task |
+| Usage dashboard | Complete | Daily/model/provider/API-key aggregation from PostgreSQL with spend, tokens, success rate and P50/P95 total latency and TTFT; tenant responses redact upstream identity | Scheduled reports are a later product task |
+| API key lifecycle | Complete | Named keys, one-time plaintext reveal, persisted prefix plus six-character suffix display, model restrictions, expiry, tags, daily/monthly spend caps, RPM/TPM, current usage, disable/enable, revoke and atomic rotation. Legacy keys retain their honest prefix-only display because their suffix cannot be recovered from the digest | IP/CIDR conditions are a later enterprise task |
+| Project limits | Complete | PostgreSQL snapshot drives RPM, estimated TPM, concurrency and monthly spend enforcement; Redis failure falls back to local counters; combined project/key limits retested | None for the current fixed-window design |
+
+## P1 Public Model Discovery
+
+| Capability | Status | Current evidence | Remaining work |
+| --- | --- | --- | --- |
+| Public catalog API and UI | Complete | Anonymous `/admin/api/public/models` and `/admin/models`; service-rendered `/admin/models/{public_id...}` pages include unique metadata, canonical URL and only supported protocol examples; raw HTML and desktop/390 px Playwright checks passed with zero console errors; allowlisted models and internal route fields remain excluded | None for the current catalog scope |
+
+## P2 Billing And Balance Visibility
+
+| Capability | Status | Current evidence | Remaining work |
+| --- | --- | --- | --- |
+| Prepaid wallet and usage deduction | Complete | Real Responses request and local full-stack Embeddings request authorized, settled, ledgered and deducted; missing usage fails closed; operator release requires `billing.adjust`, reason and zero-amount audit evidence while preserving `usage_reported=false` | Continue regression coverage for future media protocols |
+| Top-up, refunds and reconciliation | Partially complete | Stripe Checkout/Webhook/auto-top-up/refund/dispute/reconciliation code and PostgreSQL integration tests exist. SDK/API version and hosted Checkout, dynamic payment method, explicit tax, SetupIntent, off-session PaymentIntent and idempotency contracts have unit coverage. A real PostgreSQL test proves an incomplete Checkout response becomes an auditable failed order instead of remaining pending. `cmd/stripe-preflight` safely validates the restricted key's required read permissions | Current development key is invalid (`BLK-002`); replace it and repeat the full live Stripe sandbox run before release |
+| Billing profile and exports | Complete | Tenant invoice profile persists locally and idempotently syncs Stripe Customer; ledger/invoice CSV and balance-ledger-to-request drill-down implemented with tenant-isolated exact-ID lookup | Production tax treatment remains a deployment/legal decision |
+| Email alerts | Partially complete | Encrypted outbox, retries, suppression, and real PostgreSQL notification scan/claim flow; tenant billing members can configure low-balance and anomalous-spend alerts with daily idempotency, while unconfigured tenants inherit `admin.mail` defaults | Production SMTP credentials and provider DNS/feedback wiring are deployment inputs |
+
+## P3 Protocol Expansion
+
+| Capability | Status | Current evidence | Remaining work |
+| --- | --- | --- | --- |
+| Embeddings | Partially complete | OpenAI-compatible endpoint/alias, wire routing, strict JSON success validation, capability filtering, catalog/Quickstart/Playground support, input-only reservation and usage settlement are implemented. Docker E2E request `req_cd938b7cb16c84664625337b5311c7e3` returned a 4-value vector, reported 6 input tokens, charged 6 micro-USD, reduced the wallet by 6 and wrote a -6 request ledger entry | The configured real provider account lacks any Embeddings model entitlement (`BLK-001`); repeat the same E2E after the upstream account is fixed |
+| Images and audio billing units | Partially complete | Generic fixed-point metering supports typed token/image/second quantities without changing existing token rounding | Images and Audio request/usage adapters and model price tables are not implemented |
+
+## P4 Provider Reliability
+
+| Capability | Status | Current evidence | Remaining work |
+| --- | --- | --- | --- |
+| Passive health, circuit breaking and failover | Complete | Request-derived route health, weighted routing, retryable failover and circuit cooldown tests; real transient upstream failure did not charge | None for passive path |
+| Active provider probes | Complete | Opt-in authenticated, non-inference `/models` probes deduplicate per provider, validate JSON, feed the existing route circuit, expose console timestamps/counts and Prometheus counters; live Docker test showed local route `healthy`, `active_probes=1`, 2 total probes and 0 failures | Per-provider custom probe paths may be needed for non-standard vendors |
+| Adaptive routing and shared history | Complete | Final-attempt TTFT EWMA and recent availability drive same-priority selection after a minimum sample floor; 5% weighted exploration and unknown-route participation prevent starvation. A two-upstream real HTTP test selected the faster route 16/20 times while preserving warm-up/exploration. A real Redis two-instance test proves outcome/TTFT propagation and restart replay; imported failures affect the receiving circuit without republishing, the console identifies shared samples, and Prometheus exposes connection/drop/failure counters | Throughput/cost-aware policies and durable customer-visible status history are later scopes |
+
+## Verification Baseline
+
+- `go test ./...`: passed on 2026-08-06 with loopback permission.
+- `go test -race ./...`: passed on 2026-08-06 with loopback permission.
+- `go vet ./...`: passed on 2026-08-06.
+- `CGO_ENABLED=0 go build -buildvcs=false ./cmd/...`: passed on 2026-08-06.
+- Frontend `node --check` for `app.js` and `models.js`: passed on 2026-08-06.
+- Docker PostgreSQL control-plane and billing integration suite: passed on 2026-08-06; isolated PostgreSQL schemas migrated idempotently through `2026080610`, enforced `released_unmetered` in the metering constraint, retained empty display suffixes for legacy keys, rejected malformed new suffixes, and persisted tenant anomaly alert settings. Migration calls share a version-independent advisory lock and take a checksum-only fast path after application; the full multi-package race suite passed without DDL/query lock conflicts.
+- Real PostgreSQL mail alert flow: tenant wallet/ledger data triggered low-balance and anomalous-spend notifications, tenant settings overrode deployment defaults, only verified tenant billing members received encrypted outbox rows, duplicate scans produced no second row, and `ClaimMail` decrypted the expected USD amounts.
+- Real tenant billing-alert UI flow: an emailed invitation was accepted through the browser, the new `tenant_billing` account received a device session, could not edit API defaults, and saved low-balance plus anomalous-spend settings. The management API and PostgreSQL both returned the exact fixed-point values; desktop/390 px rendering had no new console errors, then the original tenant settings were restored and the temporary account and outbox rows were removed. Password-manager `username` hints are stripped from form JSON, fixing strict-body failures in invite, reset, provider and MFA forms without weakening backend decoding.
+- Stripe contract and failure-path verification: current SDK pins API `2026-07-29.dahlia`; hosted Checkout/SetupIntent/off-session PaymentIntent contracts passed unit tests, and PostgreSQL tests passed for incomplete-Session failure persistence, signed Webhook exactly-once credit and automatic top-up idempotency. Live read-permission preflight remains blocked by `BLK-002`.
+- Real upstream Responses request: HTTP 200, 4,446 tokens reported, 2,693 ms TTFT, 1,249 micro-USD charged, wallet settled and no residual reservation.
+- Local full-stack Embeddings request `req_cd938b7cb16c84664625337b5311c7e3`: HTTP 200, four-value vector, 6 input/total tokens, 6 micro-USD charged, wallet and request-linked ledger differed by exactly 6.
+- Real billing UI drill-down: the `-6` micro-USD ledger debit for `req_cd938b7cb16c84664625337b5311c7e3` opened the matching PostgreSQL usage record with 6 input tokens, `reported` metering, project/key attribution and 12 ms latency. Desktop and 390 px dialog checks had zero browser errors or warnings; a PostgreSQL integration test proves another tenant cannot retrieve that detail by guessing its request ID.
+- Two historical successful responses without usage were released through the audited reservation API as `released_unmetered`; zero-amount release ledger entries preserved `usage_reported=false`, returned held funds and restored readiness without changing wallet balance.
+- Real Admin API key lifecycle: create with daily/RPM/TPM policy, disable, enable, rotate and revoke all persisted and hot-reloaded. A second real API/UI run created and rotated a temporary key, proved both six-character suffixes matched their one-time plaintext values, proved list responses contained no plaintext, observed `runtime_sync_status=applied`, rendered prefix+suffix on desktop/390 px without browser errors, and revoked the temporary credentials.
+- Raw server-rendered model detail plus desktop and 390 px mobile console checks: canonical metadata, protocol examples, responsive layout and zero browser console errors or warnings; no internal provider URL/model/weight leaked.
+- Runtime active-probe drill: an authenticated local `/models` probe produced `healthy`, `active_probes=1`, 2 total probes and 0 failures; disabled configuration was then restored.
+- Runtime Redis fault drill: the gateway stayed ready while Redis was stopped, `/readyz` reported optional Redis as `degraded`, shared provider-health connectivity changed from 1 to 0, and both control-plane propagation and shared history reconnected automatically after Redis restarted. Final readiness was fully `ok`; the shared failure counter retained one incident.
+- Bootstrap operator regression: the rebuilt Docker console loaded every permitted API with zero browser errors/warnings; PostgreSQL showed recent `actor_type='bootstrap'`, `actor_id IS NULL`, HTTP 200 audit rows, while financial resolution evidence retains the text actor ID `bootstrap`.
+- Local candidate image `aigw:20260806-goal-candidate` resolves to `sha256:8b2164912b942598eeb5a3f1d50c5f75deaa938bedaf550600ab2ca971aca95d`. It is intentionally not published or represented as release-ready while `BLK-001` and `BLK-002` remain open.
+- Docker readiness: PostgreSQL, optional Redis, snapshot, Stripe operations (disabled in this stack), settlement queue and mail queue healthy. Real-provider Embeddings remains `BLK-001` rather than being represented by the local upstream test.