The router that doubled the bill

The chat on this site has two agents: the assistant, who covers career and contact, and the tutor, who explains the published articles and tutorials. Each has its own prompt, its own tools and its own boundaries. When the Anthropic invoice came in higher than my napkin math predicted, I audited the AI layer — and the problem was not the size of the prompts. It was the number of calls.

Every message paid for two calls

To decide which agent answers, the site used LangChain's RoutingChain: a model call that reads the question, compares it against a description of each agent, and returns a category name. Only after that answer did the question reach the chosen agent — the second call, the one the visitor actually sees.

The router ran on claude-haiku-4-5, Anthropic's cheapest model, and returned a single word. It looked harmless. But the cost of a call is not just its output: the routing prompt carried the chain's instructions plus both agent descriptions, somewhere between 400 and 700 input tokens on every message. In practice, routing accounted for 15% to 30% of the tokens in each interaction — and for half the requests.

There was also a cost that never shows up on an invoice: the calls were sequential. The answer only started streaming after the classification finished. Every message bought itself a second of silence.

The detail that bothered me most: the router ran on every message, including the fifth question of a conversation that had belonged to the tutor since the first. A classification whose result almost never changes, billed again, message after message.

The page is already a classifier

The information the router was trying to guess already existed for free: the URL. Someone reading a tutorial who opens the chat almost always wants to ask about the tutorial; someone on the about page wants to talk career and hiring. The site even had this function already — it picked the panel's initial agent:

def agent_for_path(path) when is_binary(path) do
  case Portfolio.Paths.resolve(path) do
    {:ok, _content} -> :tutor
    :error -> :secretary
  end
end

If the path points at published content, the tutor answers; every other page belongs to the assistant. Zero tokens, zero latency. The LLM router only existed for the rare case: the out-of-place question, like someone on the about page asking about a tutorial.

The rare case became one sentence in the prompt

For that case, the right answer was never a second model call — it was the agent itself saying this is not its area and pointing the way. That fits in one rule of the policy that already ships inside the prompt:

9. Do not name or summarize the site's articles or tutorials, and never say one
   does or does not exist. That area is not yours: point anyone asking to
   [/en/articles](/en/articles) and [/en/tutorials](/en/tutorials), explaining
   that the assistant on those pages answers questions about the published
   content.

The tutor carries the mirrored rule, pointing at the about page. The panel renders Markdown, so the link arrives clickable. An out-of-place question still costs one call — the same call any question costs — and the visitor gets an honest redirect instead of an invisible second charge.

One adjustment came with the change: a conversation can no longer follow the visitor across sections. The conversation id lives in the browser, and any page used to resume the same process. Resuming now checks the owner:

defp resume(socket, page_agent) do
  with true <- connected?(socket),
       id when is_binary(id) <- get_connect_params(socket)["chat_conversation_id"],
       ^page_agent <- AI.conversation_agent(id, page_agent),
       {:ok, turns} <- AI.transcript(id) do
    {id, Enum.map(turns, &entry(&1.role, &1.text))}
  else
    _fresh_start -> {nil, []}
  end
end

Note the ^page_agent pin: if the stored conversation belongs to the other agent, the with falls through to else and the visitor starts a fresh thread with the agent the page promises. The old history stays alive in the section where it was born.

The cache that never existed

The audit turned up a second, quieter problem. The model configuration requested prompt caching with a one-hour TTL — Anthropic stores the prompt prefix and bills reads at roughly a tenth of the input price. On paper, each agent's system prompt would be written once and re-read cheaply all day.

Except every model has a minimum cacheable prefix, and on claude-haiku-4-5 that minimum is 4,096 tokens. The site's largest prompt is around 1,400. Below the minimum the API returns no error: it silently ignores the request, and both cache_creation_input_tokens and cache_read_input_tokens stay at zero. The cache the configuration promised never wrote a byte.

The lesson cuts two ways. First: trust telemetry, not configuration — if cache_read_input_tokens is zero across repeated requests, the cache does not exist, whatever the code says. Second: the one-hour TTL bills writes at twice the input price, against 1.25x for the default five-minute TTL. If a prompt ever grows past the minimum, the cheaper default is already in place.

What it costs

The real trade sits in the second row: asking about a tutorial on the wrong page now earns a link instead of the answer — one extra click. For a portfolio site, that is a low price for cutting half the requests, 15% to 30% of the tokens, and a second of waiting from every message. The removal also took 480 lines of code with it, along with the tests that guarded them.

The practical takeaway travels beyond this site: before asking a model to decide something, look at what the context has already decided for free. The URL, the form, the session — there is almost always a free classifier standing right next to the paid call.