Why we built Batchrouter

Every major model provider sells the same inference at two prices. The sync price — what you pay when you call /v1/chat/completions and wait for tokens. And the batch price — what you pay when you're willing to wait up to 24 hours. The batch price is usually 50% lower. Sometimes more. This isn't a secret tier, it isn't gated, and it isn't hard to find — it's right there in OpenAI's and Anthropic's pricing pages. And yet, when we talk to teams burning real money on LLMs — classification queues, nightly enrichment, eval runs, document pipelines, agent backfills — almost none of them are using batch. They know the discount exists. They've usually tried it once. They went back to sync. We started Batchrouter because that gap is the most expensive habit in modern AI infra, and the reason it persists isn't laziness. It's that the 50% you save almost never survives contact with the operational work it takes to actually claim it. The hidden tax of async A sync call is one line of code. A batch job is a project. You have to assemble JSONL in a format the provider accepts, upload it to their storage, poll a job endpoint for state transitions, handle the partial-failure case where 12 of 4218 lines errored, download results, parse them, reconcile them with your original input by custom_id, and only then do you get to use the output. Multiply that by two providers if you want to route across OpenAI and Anthropic — because their batch APIs aren't the same shape, the error semantics differ, and the file formats diverge in small but expensive ways. Then there's billing. Each provider has its own credit balance, its own minimum top-up, its own invoice cycle. Reconciling a sub-cent batch line against a provider invoice that lands a week later is genuinely awful. We've watched competent teams give up on batch not because the savings weren't real, but because finance couldn't tell them, with any confidence, what a given batch had actually cost. That's the operational tax. And in our experience, it eats the entire 50% discount for any team smaller than a dedicated platform group. Most teams quietly stay on sync, pay double, and tell themselves it's worth it for the simplicity. They're not wrong — given those two options. Routing as the missing layer We think there's a third option: a router. Not a wrapper, not a proxy — a routing layer that owns lane selection, retries, settlement, and reconciliation, and exposes the OpenAI Batch API shape on top. You PUT a JSONL file. You GET a batch back. Same custom_id semantics, same status enum, same results format. Underneath, we pick the cheapest eligible lane that meets your SLA and privacy constraints, dispatch to the provider, retry on a different lane if the first one errors, and settle the cost back to your credit balance in micro-dollars. The reason this is interesting is that lane selection is genuinely a decision problem, not a lookup. Eligible depends on the model. Cheapest depends on the current provider rate plus our 5.9% service fee — and native batch lanes (OpenAI Batch, Anthropic Message Batch) carry an automatic 10% discount on the provider rate, which usually beats whatever the dynamic option would be. Capacity matters because OpenAI Batch has a 200MB-per-file ceiling and rate limits per org. Privacy matters because some workloads must not leave a specific region. Solving this once, well, in a dedicated control plane is dramatically less work than every customer reinventing it inside a queue worker. Our service fee is the smallest line item in any batch. The bigger number is what doesn't show up: the engineer-months you don't spend writing a job orchestrator, the on-call pages you don't take when a provider 5xxs, the reconciliation script you don't have to maintain when settlement is already micro-dollar accurate. One customer batch ID, many internal lanes Real production JSONL is rarely a single model. A nightly enrichment job might use gpt-4o-mini for the cheap rows and claude-3-5-sonnet…