llm-router

Why llm-router

Mon, 01 Jan 0001 00:00:00 +0000

Why llm-router

One narrow job — rewrite one alias into different upstream model ids, spread across providers, fail over — done by a single static Go binary instead of a control plane.

LiteLLM vs llm-router

	llm-router	LiteLLM
Licence	none published yet	MIT
Performance	one ~17 MB static binary; measured idle at 5.9 MB RSS, 0% CPU, binds instantly	litellm 1.104 proxy measured idle at 94.8 MB RSS across its two processes, ~25 s from launch to first bind; a Python dependency tree spanning boto3, azure-*, polars, rq, redis and cryptography
Security	constant-time key check, inbound credentials stripped, secrets in env, cli upstreams store nothing, no bodies in logs — details below	boots only with a `LITELLM_MASTER_KEY`; virtual keys, budgets and teams need a Postgres database — a second credential store to run and protect
Alias → several providers	per-target model ids, weights, weight-0 failover-only targets	`model_name` with multiple deployments, weights and fallbacks
Wire translation	OpenAI Chat ↔ Anthropic Messages in both directions, streams translated incrementally without buffering	serves the OpenAI format and converts to provider SDKs; passthrough routes for some providers
Usage accounting	a SQLite file, one row per request, queryable over `/usage` with no database server	spend tracking through its database or external callbacks
Control plane (keys, budgets, teams)	deliberately none; optional operational console and playground	a core feature

Both were measured idle on the same machine (macOS, arm64, one worker): llm-router 5.9 MB RSS, LiteLLM 94.8 MB — about 16× — and LiteLLM took 25 s to start serving against llm-router’s instant bind. CPU was 0% for both while idle. Under load the difference is structural: the Go relay copies bytes; the Python proxy parses and re-serialises.

Running

Mon, 01 Jan 0001 00:00:00 +0000

Running

Daemon mode

llm-router runs as a long-lived foreground server: it binds its listen address, serves requests until it receives SIGTERM or SIGINT, then drains in-flight streams for up to 20 seconds before exiting. There is no self-daemonizing flag on purpose — detaching, logging and restart-on-crash are owned by whatever supervises it (Docker, Kubernetes, systemd, launchd, tmux on a laptop). Kill signals and restarts are handled cleanly.

./llm-router -config router.yaml # -config defaults to ./router.yaml
./llm-router -listen 0.0.0.0:8787 # override the bind address
./llm-router -check # validate config, print resolved routes, exit

For systemd, use Restart=on-failure; the process expects to be supervised.

Pointing clients at it

Mon, 01 Jan 0001 00:00:00 +0000

Pointing clients at it

Any OpenRouter client. Change the base URL and nothing else:

export OPENAI_BASE_URL=http://127.0.0.1:8787/api/v1 # was https://openrouter.ai/api/v1

omp / pi. In ~/.omp/agent/models.yml, use discovery.type: litellm so context windows and pricing come from /model_group/info rather than a guess:

providers:
 my-router:
 baseUrl: http://127.0.0.1:8787/api/v1
 apiKey: ROUTER_KEY  # only if `apiKey:` is set in router.yaml
 authHeader: true
 api: openai-completions
 disableStrictTools: true
 discovery:
 type: litellm

Aliases then appear as my-router/anthropic/claude-opus-5.5 and my-router/anthropic/claude-sonnet-5.5:

Configuration

Mon, 01 Jan 0001 00:00:00 +0000

Configuration

router.yaml is the whole config, commented and ready to run. The keys worth knowing:

Key	Meaning
`listen`	Bind address. Loopback by default.
`apiKey`	Shared secret clients must present. `""` accepts any.
`ui.enabled`	Enable the embedded console at `/ui/`. Requires a nonempty `apiKey`.
`ui.secureCookies`	Use HTTPS cookies and origin checks behind a TLS-terminating proxy. Defaults to false.
`defaults.*`	Retry budget, timeouts, sticky affinity, breaker thresholds.
`store.path`	SQLite file for usage history. Empty keeps no history.
`store.queueSize`	Records waiting to be written before they are dropped.
`store.maxRows`	Records kept, oldest deleted first. `0` keeps everything.
`upstreams.<name>.baseUrl`	Provider root; the protocol’s path is appended.
`upstreams.<name>.apiKeyEnv`	Env var holding the key. Preferred over `apiKey`.
`upstreams.<name>.kind`	`http` (default), `cli` for Claude Code, or `codex` for ChatGPT subscription access.
`upstreams.<name>.mode`	Claude Code: `one-shot` (default) or `persistent`. Codex is always persistent.
`upstreams.<name>.toolMode`	Claude Code: `claude` or non-streaming `client`. Codex uses streaming-capable native client tools.
`upstreams.<name>.command`	Claude one-shot uses `{prompt}`; persistent may use `{model}`. Codex defaults to `codex app-server --listen stdio://` without placeholders.
`upstreams.<name>.cwd`	Working directory for a CLI process; Claude Code discovers project configuration here.
`upstreams.<name>.timeout`	Bounds one CLI turn. Defaults to `maxStreamDuration`.
`upstreams.<name>.maxSessions`	Maximum live persistent CLI processes. Defaults to `maxConcurrency`.
`upstreams.<name>.sessionIdleTimeout`	Closes inactive persistent CLI processes. Zero keeps them alive until shutdown.
`upstreams.<name>.authStyle`	`bearer` (the default) or `anthropic`.
`upstreams.<name>.headers`	Extra headers per upstream, overriding the client’s.
`upstreams.<name>.bodyDrop`	Top-level JSON fields removed before dispatch.
`upstreams.<name>.bodyPatch`	Top-level JSON fields forced.
`upstreams.<name>.maxConcurrency`	In-flight turn cap. At the limit: skipped, never queued.
`models.<alias>.api`	Wire protocol clients speak for this alias.
`models.<alias>.contextWindow`, `maxOutputTokens`, `cost`	Advertised to clients for budgeting; not enforced upstream.
`models.<alias>.targets[]`	Where the alias may be served: upstream, model id, weight.

Choosing a model version, provider and thinking mode

These are set per target — per provider route — because the same logical model needs different fields on each:

Usage history

Mon, 01 Jan 0001 00:00:00 +0000

Usage history

Point store.path at a file and the router keeps a row per served request: the alias, the upstream that answered, the tokens, the latency and the cost.

store:
 path: ~/.llm-router/usage.db
 queueSize: 1024
 maxRows: 130000 # about 10 MB, keeping the newest 130k requests

Query it through the router:

curl -H "Authorization: Bearer $ROUTER_KEY" \
 'http://127.0.0.1:8787/usage?group_by=alias&since=7d'

or open the file directly, even while the router runs (WAL is enabled):

Design notes

Mon, 01 Jan 0001 00:00:00 +0000

Design notes

Failover happens strictly before the first response byte is committed. The request body is buffered (bounded by maxBodyBytes) so it can be replayed against the next target. Once a status line is written, the response belongs to the client: a truncated or failed stream is logged and returned, never re-dispatched, because the client cannot un-see it.

Retried statuses. 429, 401, 402, 403, 5xx. Ordinary 4xx are the client’s bug and are returned unchanged instead of being fired at every provider. A 429 parks the upstream without counting against its health — it is busy, not broken.

Codex subscription

Mon, 01 Jan 0001 00:00:00 +0000

Codex subscription

kind: codex connects the router to the official Codex app-server over stdio. Codex owns ChatGPT sign-in, credential storage, and token refresh. The router never reads or forwards subscription tokens. This is separate from OpenAI API billing and from any Codex login stored by omp.

Install and connect

The adapter is verified against Codex 0.163.0. Install that release under the same OS account that runs the router, for example: