Lessons from routing 100M tokens

Last week we crossed 100M routed tokens. That's not a lot in the grand scheme of inference, but it's enough volume that the routing engine has met most of its real-world failure modes, and we've gotten to make corrective changes to it under load. Here are five lessons from the production logs — each one a thing we genuinely didn't predict, and each one of which changed how the engine works today. 1. Mixed-model JSONL is the norm, not the exception When we designed the splitter, we assumed mixed-model JSONL would be a minority case — maybe 5-10% of incoming jobs. We sized the per-lane control-plane fee around that assumption and gave splitting a deliberately spartan UI in the dashboard. Then we started looking at the data. The pattern is consistent across customer segments. Document pipelines run a cheap classifier (Haiku, gpt-4o-mini) against every row, then send the ones that scored above some threshold to a stronger model for extraction. Enrichment jobs use a small model for the long tail and a frontier model for the hard cases. Eval runs mix a reference model and a candidate. Once you actually let customers send mixed JSONL, they will, because it's how production work was always shaped — they just couldn't express it that way before. What we changed: we made splitting a first-class path, not a fallback. The quote response now line-items the per-lane breakdown by default, the dashboard renders split jobs as a single batch with sub-lane status, and the webhook payload includes a lanes[] array so a customer's reconciliation script doesn't have to infer the split from the line items. 2. Latency flips routing even when prices are identical For models available on multiple lanes — Claude on Anthropic direct vs. Claude on Bedrock, for instance — the published prices are often pennies apart. Our first version of the scorer treated them as effectively tied and tie-broke on a static lane preference. That was wrong, and the bug was visible in the SLA reports before it was visible to us. The Bedrock lane was 2.3x slower at p95 completion, mostly because of regional queue depth, and had a 4x higher error rate during one provider incident. At identical price, the engine should clearly route to the direct lane. The old scorer didn't, because it only consulted latency as a tiebreaker below a price threshold of $0.0001 per row — and identical published rates plus a small public-provider discount put both lanes just barely above that threshold for some token counts and below it for others. Routing was effectively non-deterministic in the tied case. What we changed: latency and recent error rate are now part of the primary score, not a tiebreaker. The score formula is (cost × 1.0) + (p95_completion_minutes × cost_per_minute_penalty) + (error_rate × penalty). The penalty coefficients are tuned per workflow — a workflow that cares about completion time pays more attention to latency, a workflow that cares about absolute floor cost weights cost more. The default profile prefers the direct lane in the example above by a margin large enough that no token count flips it. 3. Webhook redelivery dominates the post-completion cost We deliver a webhook when a batch lands. For a $0.30 batch that's a thirty-cent operation; for a $40 batch, it's still one delivery. The variable cost we hadn't sized correctly was the redelivery path. A customer endpoint that returns a 503 once, or that doesn't respond inside our 10-second timeout, triggers an exponential-backoff redelivery sequence: 30s, 2m, 10m, 1h, 6h, 24h. On low-cost batches with intermittently slow customer endpoints, the redelivery work — Worker time, R2 reads of the cached payload, DO state updates — was approaching the cost of the original dispatch. If your webhook endpoint is slow but correct, our redelivery costs you nothing — we de-dupe by event id on our side. If your endpoint is fast but non-idempotent, a single delivery that times out on your end and succeeds on a redelivery can produce dupli…