Routing a JSONL batch in under 50ms

A JSONL file lands at POST /v1/quotes/model. About 50ms later, the client has an estimate, a per-lane cost breakdown, and a small explanation of why we picked the lane we picked. The customer thinks of this as a single request. Internally it's a tiny pipeline — parse, validate, shortlist, score, split, persist, serialize — and the whole thing has to run inside one Cloudflare Worker invocation without ever touching the dispatch queue. This post walks the hot path step by step. We're focused on the quoting path here, not the dispatch path. Quoting is read-mostly and synchronous. The expensive work — calling the provider's Batch API, polling for status, downloading the artifact, stitching results — happens later, after batch creation with quote_id. Keeping quoting fast is what lets an agent loop fire 50 quotes a minute against /v1/quotes/model without spending its budget on round trips. The hot path From the worker's perspective, the request handler is a six-step pipeline. The shape is roughly this: Each step is small on purpose. The parser is a streaming reader, not a JSON.parse over a buffered string — at 5,000 rows we don't want to allocate a 10MB string in V8 just to throw it away. Validation runs inline on each parsed row so we can fail fast on the first bad custom_id instead of finishing the parse and then discovering trouble. Shortlisting is a Map lookup. Scoring is a tight inner loop over a list that's almost never longer than six lanes per model. Splitting is a group-by. Persistence is a single D1 prepared statement. Where the time actually goes We instrument every step with performance.now() deltas and ship them to a Durable Object that holds a rolling 60-second histogram. The p50 breakdown for a typical 1,200-row mixed-model JSONL looks like this: Two honest caveats. The p99 number isn't a hot-path regression — it's almost entirely D1 write tail latency, which we've measured spike to ~120ms during a maintenance window once a week. We've considered moving the quote write off the request path by returning a quote_id before the write completes, but the next public hop is POST /v1/batches with that returned quote_id. Batch creation depends on the persisted quote snapshot when it reserves credits and dispatches, so the quote write stays on the quote request path. The other caveat: cold starts add 40-80ms the first time a Worker isolate sees a request, but on Cloudflare's network we get warm isolates for the vast majority of traffic. We tried. The write rate per quote — six instrumentation points, each a INSERT — moved D1 from ~15ms median to ~30ms, doubling our total quote time. The Durable Object with an in-memory ring buffer plus a periodic flush is ~0.3ms per data point and survives DO restarts via storage snapshots. Decisions made up front A lot of what makes the hot path fast is what isn't on the hot path. Three things in particular. The model catalog is an in-memory map, populated at module init from a versioned KV namespace. It's about 8KB and covers ~140 model slugs. The lookup is a Map.get — single-digit microseconds. The cost is paid once per isolate, on first invocation, as part of cold start. We rebuild the catalog on a webhook from our admin tool whenever a lane goes into or out of rotation, and we ship the new isolate via a normal Cloudflare deploy when the schema changes. Token estimation is a heuristic, not a tokenizer call. Calling a real BPE tokenizer per row would be 0.5-2ms each, which on a 5,000-row batch turns into multi-second quote latency. Instead we estimate from the request shape: count the JSON string characters in the prompt and message fields, apply a per-model coefficient we calibrated against real provider receipts, and clip to a sensible band. The estimate is wrong by 3-7% on average and we know it; the actual settlement at receipt time corrects it, and we hold a small headroom in the credit reservation precisely so the customer never gets surprised. Lane health comes from a snapshot, not f…