path: root/PROGRESS.md

# Commercial Developer Experience Progress

Last updated: 2026-08-06 (Pacific/Auckland)

This file records implementation status and verification evidence for the
ZenMux-aligned developer journey. A feature is only marked complete when its UI,
API, PostgreSQL state, runtime behavior, and relevant billing path have been
exercised with real data.

## P0 Developer Observability And Spend Controls

| Capability | Status | Current evidence | Remaining work |
| --- | --- | --- | --- |
| Request usage ledger | Complete | PostgreSQL `usage_events`, tenant-scoped filters, request details, token/cache/cost/status/latency/TTFT UI, stable cursor pagination and page controls; every request debit in the balance ledger opens the matching usage record; a real `gpt-5.5` request persisted with 2,693 ms TTFT and charged 1,249 micro-USD | Partition/archive policy is a later scale task |
| Usage dashboard | Complete | Daily/model/provider/API-key aggregation from PostgreSQL with spend, tokens, success rate and P50/P95 total latency and TTFT; tenant responses redact upstream identity | Scheduled reports are a later product task |
| API key lifecycle | Complete | Named keys, one-time plaintext reveal, persisted prefix plus six-character suffix display, model restrictions, expiry, tags, daily/monthly spend caps, RPM/TPM, current usage, disable/enable, revoke and atomic rotation. Legacy keys retain their honest prefix-only display because their suffix cannot be recovered from the digest | IP/CIDR conditions are a later enterprise task |
| Project limits | Complete | PostgreSQL snapshot drives RPM, estimated TPM, concurrency and monthly spend enforcement; Redis failure falls back to local counters; combined project/key limits retested | None for the current fixed-window design |

## P1 Public Model Discovery

| Capability | Status | Current evidence | Remaining work |
| --- | --- | --- | --- |
| Public catalog API and UI | Complete | Anonymous `/admin/api/public/models` and `/admin/models`; service-rendered `/admin/models/{public_id...}` pages include unique metadata, canonical URL and only supported protocol examples; raw HTML and desktop/390 px Playwright checks passed with zero console errors; allowlisted models and internal route fields remain excluded | None for the current catalog scope |

## P2 Billing And Balance Visibility

| Capability | Status | Current evidence | Remaining work |
| --- | --- | --- | --- |
| Prepaid wallet and usage deduction | Complete | Real Responses request and local full-stack Embeddings request authorized, settled, ledgered and deducted; missing usage fails closed; operator release requires `billing.adjust`, reason and zero-amount audit evidence while preserving `usage_reported=false` | Continue regression coverage for future media protocols |
| Top-up, refunds and reconciliation | Partially complete | Stripe Checkout/Webhook/auto-top-up/refund/dispute/reconciliation code and PostgreSQL integration tests exist. SDK/API version and hosted Checkout, dynamic payment method, explicit tax, SetupIntent, off-session PaymentIntent and idempotency contracts have unit coverage. A real PostgreSQL test proves an incomplete Checkout response becomes an auditable failed order instead of remaining pending. `cmd/stripe-preflight` safely validates the restricted key's required read permissions | Current development key is invalid (`BLK-002`); replace it and repeat the full live Stripe sandbox run before release |
| Billing profile and exports | Complete | Tenant invoice profile persists locally and idempotently syncs Stripe Customer; ledger/invoice CSV and balance-ledger-to-request drill-down implemented with tenant-isolated exact-ID lookup | Production tax treatment remains a deployment/legal decision |
| Email alerts | Partially complete | Encrypted outbox, retries, suppression, and real PostgreSQL notification scan/claim flow; tenant billing members can configure low-balance and anomalous-spend alerts with daily idempotency, while unconfigured tenants inherit `admin.mail` defaults | Production SMTP credentials and provider DNS/feedback wiring are deployment inputs |

## P3 Protocol Expansion

| Capability | Status | Current evidence | Remaining work |
| --- | --- | --- | --- |
| Embeddings | Partially complete | OpenAI-compatible endpoint/alias, wire routing, strict JSON success validation, capability filtering, catalog/Quickstart/Playground support, input-only reservation and usage settlement are implemented. Docker E2E request `req_cd938b7cb16c84664625337b5311c7e3` returned a 4-value vector, reported 6 input tokens, charged 6 micro-USD, reduced the wallet by 6 and wrote a -6 request ledger entry | The configured real provider account lacks any Embeddings model entitlement (`BLK-001`); repeat the same E2E after the upstream account is fixed |
| Images and audio billing units | Partially complete | Generic fixed-point metering supports typed token/image/second quantities without changing existing token rounding | Images and Audio request/usage adapters and model price tables are not implemented |

## P4 Provider Reliability

| Capability | Status | Current evidence | Remaining work |
| --- | --- | --- | --- |
| Passive health, circuit breaking and failover | Complete | Request-derived route health, weighted routing, retryable failover and circuit cooldown tests; real transient upstream failure did not charge | None for passive path |
| Active provider probes | Complete | Opt-in authenticated, non-inference `/models` probes deduplicate per provider, validate JSON, feed the existing route circuit, expose console timestamps/counts and Prometheus counters; live Docker test showed local route `healthy`, `active_probes=1`, 2 total probes and 0 failures | Per-provider custom probe paths may be needed for non-standard vendors |
| Adaptive routing and shared history | Complete | Final-attempt TTFT EWMA and recent availability drive same-priority selection after a minimum sample floor; 5% weighted exploration and unknown-route participation prevent starvation. A two-upstream real HTTP test selected the faster route 16/20 times while preserving warm-up/exploration. A real Redis two-instance test proves outcome/TTFT propagation and restart replay; imported failures affect the receiving circuit without republishing, the console identifies shared samples, and Prometheus exposes connection/drop/failure counters | Throughput/cost-aware policies and durable customer-visible status history are later scopes |

## Verification Baseline

- `go test ./...`: passed on 2026-08-06 with loopback permission.
- `go test -race ./...`: passed on 2026-08-06 with loopback permission.
- `go vet ./...`: passed on 2026-08-06.
- `CGO_ENABLED=0 go build -buildvcs=false ./cmd/...`: passed on 2026-08-06.
- Frontend `node --check` for `app.js` and `models.js`: passed on 2026-08-06.
- Docker PostgreSQL control-plane and billing integration suite: passed on 2026-08-06; isolated PostgreSQL schemas migrated idempotently through `2026080610`, enforced `released_unmetered` in the metering constraint, retained empty display suffixes for legacy keys, rejected malformed new suffixes, and persisted tenant anomaly alert settings. Migration calls share a version-independent advisory lock and take a checksum-only fast path after application; the full multi-package race suite passed without DDL/query lock conflicts.
- Real PostgreSQL mail alert flow: tenant wallet/ledger data triggered low-balance and anomalous-spend notifications, tenant settings overrode deployment defaults, only verified tenant billing members received encrypted outbox rows, duplicate scans produced no second row, and `ClaimMail` decrypted the expected USD amounts.
- Real tenant billing-alert UI flow: an emailed invitation was accepted through the browser, the new `tenant_billing` account received a device session, could not edit API defaults, and saved low-balance plus anomalous-spend settings. The management API and PostgreSQL both returned the exact fixed-point values; desktop/390 px rendering had no new console errors, then the original tenant settings were restored and the temporary account and outbox rows were removed. Password-manager `username` hints are stripped from form JSON, fixing strict-body failures in invite, reset, provider and MFA forms without weakening backend decoding.
- Stripe contract and failure-path verification: current SDK pins API `2026-07-29.dahlia`; hosted Checkout/SetupIntent/off-session PaymentIntent contracts passed unit tests, and PostgreSQL tests passed for incomplete-Session failure persistence, signed Webhook exactly-once credit and automatic top-up idempotency. Live read-permission preflight remains blocked by `BLK-002`.
- Real upstream Responses request: HTTP 200, 4,446 tokens reported, 2,693 ms TTFT, 1,249 micro-USD charged, wallet settled and no residual reservation.
- Local full-stack Embeddings request `req_cd938b7cb16c84664625337b5311c7e3`: HTTP 200, four-value vector, 6 input/total tokens, 6 micro-USD charged, wallet and request-linked ledger differed by exactly 6.
- Real billing UI drill-down: the `-6` micro-USD ledger debit for `req_cd938b7cb16c84664625337b5311c7e3` opened the matching PostgreSQL usage record with 6 input tokens, `reported` metering, project/key attribution and 12 ms latency. Desktop and 390 px dialog checks had zero browser errors or warnings; a PostgreSQL integration test proves another tenant cannot retrieve that detail by guessing its request ID.
- Two historical successful responses without usage were released through the audited reservation API as `released_unmetered`; zero-amount release ledger entries preserved `usage_reported=false`, returned held funds and restored readiness without changing wallet balance.
- Real Admin API key lifecycle: create with daily/RPM/TPM policy, disable, enable, rotate and revoke all persisted and hot-reloaded. A second real API/UI run created and rotated a temporary key, proved both six-character suffixes matched their one-time plaintext values, proved list responses contained no plaintext, observed `runtime_sync_status=applied`, rendered prefix+suffix on desktop/390 px without browser errors, and revoked the temporary credentials.
- Raw server-rendered model detail plus desktop and 390 px mobile console checks: canonical metadata, protocol examples, responsive layout and zero browser console errors or warnings; no internal provider URL/model/weight leaked.
- Runtime active-probe drill: an authenticated local `/models` probe produced `healthy`, `active_probes=1`, 2 total probes and 0 failures; disabled configuration was then restored.
- Runtime Redis fault drill: the gateway stayed ready while Redis was stopped, `/readyz` reported optional Redis as `degraded`, shared provider-health connectivity changed from 1 to 0, and both control-plane propagation and shared history reconnected automatically after Redis restarted. Final readiness was fully `ok`; the shared failure counter retained one incident.
- Bootstrap operator regression: the rebuilt Docker console loaded every permitted API with zero browser errors/warnings; PostgreSQL showed recent `actor_type='bootstrap'`, `actor_id IS NULL`, HTTP 200 audit rows, while financial resolution evidence retains the text actor ID `bootstrap`.
- Local candidate image `aigw:20260806-goal-candidate` resolves to `sha256:8b2164912b942598eeb5a3f1d50c5f75deaa938bedaf550600ab2ca971aca95d`. It is intentionally not published or represented as release-ready while `BLK-001` and `BLK-002` remain open.
- Docker readiness: PostgreSQL, optional Redis, snapshot, Stripe operations (disabled in this stack), settlement queue and mail queue healthy. Real-provider Embeddings remains `BLK-001` rather than being represented by the local upstream test.