Inside Zopyr’s Zero-Postgres Hot Path: Sub-25ms Failover & Distributed Circuit Breaking
How we engineered our inference gateway to authenticate keys, reserve nano-USD holds, evaluate sliding-window circuit breakers, and fail over across upstream providers without touching Postgres.
1. Why Database Queries Do Not Belong on the Inference Path
When an application streams tokens from an LLM, every millisecond before the first token arrives is felt directly by the end user. Traditional API gateways query a relational database to validate the API key, check wallet balance, load model pricing, and write a request log row before forwarding the request.
In Zopyr, the steady-state inference path makes zero Postgres calls. Catalog snapshots, organization pricing rules, and encrypted provider configurations are synchronized into in-memory caches via Redis Pub/Sub, while rate limits and credit reservations execute as single-round-trip Lua scripts in Redis.
2. Sliding-Window Circuit Breakers & Pre-First-Byte Failover
Upstream AI providers experience transient rate limits (HTTP 429) and regional capacity spikes (HTTP 502/503) daily. Zopyr maintains a per-endpoint sliding 60-second window in Redis that trips open after 5 consecutive failures or a 50% error rate across 20 requests.
If an upstream fails or times out before the first SSE chunk reaches the caller, Zopyr transparently retries the next healthy endpoint or fallback model in the plan—without billing the failed attempt.
{
"model": "deepseek/deepseek-v4.1-flash",
"models": ["google/gemini-3.8-flash", "qwen/qwen-3.8-max-prime"],
"provider": {
"sort": "latency",
"allow_fallbacks": true
},
"messages": [{ "role": "user", "content": "Generate a SQL migration." }]
}3. Exact Nano-USD Holds & Asynchronous Double-Entry Ledger
Floating-point numbers introduce rounding drift when billing fractions of a cent per token. Zopyr represents all monetary values as 64-bit integer nano-USD (1 USD = 1,000,000,000 nanos).
Before calling an upstream provider, an atomic Lua script reserves the worst-case token budget against the organization’s spendable balance. The moment the stream completes—or if the client disconnects mid-stream—the hold is settled to the exact tokens generated and an idempotent UsageEvent is enqueued for double-entry ledger posting.