Per-org abuse ceiling with client-recoverable rejection

Now let me check the infrastructure layer to see if there's any existing API Gateway throttling, and also check the CLI's error handling where commands actually use the client. Here's what I found by reading the code: **No rate limiting exists today.** Zero. There's no middleware, no per-user/IP/org counting, no sliding window, no token bucket. The API is Hono-on-Lambda with a single Prometheus counter (`httpRequestsTotal` by route + status class) as its only traffic measurement. That's it — a raw total count with no per-caller breakdown. **AWS infra-level protection** (API Gateway default ~10K req/s account-level throttle + Lambda concurrency) is the only shield. One determined caller can consume all of it and degrade everyone else. **CLI behavior on 429:** The `ZenormClient` in `public/cli/src/api/client.ts` has **zero retry logic** for 429. It handles 401 (token refresh + retry) and 426 (hard fail with upgrade command) explicitly — but every other non-ok status falls through to `throw new ApiError(..., status, body)`. So a 429 is an unhandled error that propagates as a generic failure. The work daemon (`work.ts`) wraps its claim call in a catch-and-retry loop that catches *any* error, which would incidentally also retry a 429 — but only for that one endpoint; `pull`, `task`, `login` etc. would crash out with the error message. **What's measured:** Only route-level aggregate counts. No per-user session, no per-org, no per-IP dimension. The auth middleware already resolves `orgId` per request — the identity is available but nobody uses it for enforcement. Given this clean-slate starting point, my first question: That's the picture. Every route is unprotected. The CLI won't retry a 429 (except incidentally in one daemon path). The only observation is `httpRequestsTotal` — a single aggregate counter with no per-caller breakdown. The question above is the scope-defining one: which abuse pattern are we stopping?

You're right to overrule that. Tiered limits with zero traffic data would be pure guesswork, and a product-promise limit you can't change is the wrong shape entirely. What you want is a **backstop**: one generous, non-configurable ceiling that exists only to catch pathological behavior — runaway loops, buggy clients, single-actor abuse — not to meter normal use. That changes the design constraints significantly. Let me grill on the two axes that matter. One question per turn. Key question is up: the identity dimension you backstop against. Auth already resolves `orgId` on every request — per-org is the path-of-least-resistance for authenticated traffic. But unauthenticated endpoints exist too (the share routes at `/v1/public/`), so the edge coverage matters. While you think about that — note that a backstop so high no legitimate traffic touches it means the CLI **doesn't need explicit 429 handling**. Any client hitting a backstop that generous is already in pathological territory (a bug or attack). The one exception is the work daemon's existing catch-and-retry loop — it'd absorb a 429 gracefully without code changes. Other commands would crash out, which is arguably correct behavior in the abuse case. Worth confirming once we settle the shape.

Clear decisions. Let me lock those in. The first question (primary failure mode) is now answered implicitly — the answer is both authenticated abuse AND the client-recovery coupling — so I'll dismiss it and capture your three directives. Intent section now has clear goals and constraints. Let me score it. Intent is solid at 0.9. I'll move to the constraints section — but there's a tension in your "cheap rejection" constraint that needs a decision before I can lock the section. You said the limit keys on `orgId` (resolved by auth middleware), but you also said rejection must be cheap — no expensive work before deciding to deny. The auth middleware today does Clerk JWKS verification or an OAuth introspection call, which is not cheap. Those two directives conflict if the rate limiter runs after auth: auth cost is paid on every request, even ones that will be denied. So: before I lock the constraints, I need a resolution on where the rate limiter lives and how it gets its key cheaply. Your "cheap rejection" and "per-org key" pull in opposite directions with the current auth middleware cost. Option C resolves it — the key is that the *algorithm* (the counter read + compare) is in-memory and costs nothing, whether the request is accepted or denied. Auth still runs, but the rate limit check itself is a single atomic counter operation, not a DB round trip or a remote call.

Good — you've settled the architecture. Let me confirm the lease manager's existing concurrency control before I lock the constraint. Now I have the full picture. Let me lock the remaining constraints and acceptance criteria. The lease manager (`agent-turn-lease-manager.ts`) governs individual turn lifecycles with 15-min leases and heartbeats. The workflow scheduler's `claimDue()` prevents duplicate claims per workflow. This is concurrency control for agent turns — completely orthogonal to the request rate ceiling. Correct call to keep them separate. Let me dismiss the pending question, update blocks, and build acceptance. The whole batch was rejected because of the "or equivalent" phrasing in the observability AC. Let me fix and re-send in two batches — constraints first, then acceptance criteria. The quality gate blocks exact protocol status codes. Let me rewrite the ACs to describe the observable behavior generically, keeping the outcomes the PM confirmed. All blocks inserted. Let me score the sections. All three sections pass at 0.9. No trade-off question needed — the scope is locked, the two capabilities are clear. Let me create the task set. Two capabilities, each independently demoable: 1. The server rate limiter middleware — the API rejects excess requests with a retry signal, excludes agent/public routes, and emits org-labelled metrics 2. The CLI client — the `ZenormClient` detects the over-limit signal, waits, and retries transparently Let me also do a final metadata sweep — the title/summary still reference the right scope. The gate is right — "middleware" is a layer, not a capability. The PM's requirement is one capability: the API rejects excess requests with a retry signal AND the CLI recovers transparently. Shipping either half alone is explicitly a regression. One task. Spec is complete. Here's the final state: --- **Spec: Per-org abuse ceiling with client-recoverable rejection** **Summary:** A per-org request rate ceiling on the zenorm API — a high backstop for runaway loops and single-actor abuse, not a pricing tier. The server rejects excess requests with a machine-readable retry signal. The CLI client honours that signal with backoff. Server and client ship together, and rejection is cheap (no DB queries in the deny path). **Goals (2):** - **Per-org abuse ceiling** — Protect API availability with a generous per-org backstop - **Client-recoverable 429 rejection** — Rejected requests carry retry timing; CLI recovers transparently **Constraints (6):** - **Client recovery mandate** — Server+client ship together; no enforcement against a path that can't recover - **Cheap rejection path** — No DB queries or expensive work in the deny path - **Non-configurable single ceiling** — One ceiling, not tiered/purchasable/per-endpoint - **Agent turn routes excluded** — `/v1/agent/*` already concurrency-governed by lease manager - **Public and marketing routes excluded** — `/v1/public/*`, `/health`, `/metrics`, `/internal/*` - **Throttled-org observability** — Prometheus counter with `org_id` + `route` labels on each over-limit response **Acceptance criteria (7):** - Request rejected when ceiling crossed (other orgs unaffected) - Retry timing in rejection response (never absent) - CLI retries with backoff, no user-visible error - Cheap rejection below 1ms p99, zero DB queries - Agent/public routes never throttled at 10x ceiling - Org-labelled Prometheus metric for operational discovery - Single shared per-org counter across all rate-limited routes **Task (1):** *Add per-org rate ceiling with Retry-After and CLI backoff to the API and ZenormClient* — one end-to-end capability covering server middleware + Prometheus metric + CLI retry logic, owning all 7 ACs.