Optimized for the way AI agents actually consume tokens

Someone asked ChatGPT what above.dev is, and it came back with a line we liked enough to put on the homepage: an OpenAI-compatible API optimized for the way AI agents actually consume tokens. This post explains what that means in practice, because it is a specific set of engineering choices rather than a slogan.

Agents are not chat

A chat app sends a prompt, gets an answer, and the user reads it. An agent sends a prompt, gets a tool call, runs the tool, appends the result, and sends everything again. It does this dozens or hundreds of times per task, without a human in the loop between turns. That changes four things about how tokens get consumed, and each one shaped a decision in the gateway.

1. Most input is repeated, so cache pricing is the real price

By the fiftieth tool call, the system prompt and the whole earlier conversation have been sent fifty times. On real agent traffic through above.dev, 80 to 95% of input tokens are served from the provider's prompt cache. The list input price barely matters. The cache-read price is what you pay.

So we pass the upstream cache discount through unchanged and add a flat 10% on every token class. We never flatten cached and uncached tokens into one blended rate, because that would hide the number that decides your bill. We also choose upstreams by their cache pricing, not their headline pricing, which is why GLM runs on Fireworks rather than the official API. The arithmetic is in an earlier post.

2. Context grows all session, so limits must match the model

An agent's context only gets longer. Every tool result is appended and stays. A model that advertises a 1M window is useless if the gateway in front of it rejects requests at 300k. Our input limit is 1M tokens because the models support it, and the token estimator counts tool definitions and image parts, not just message text, so large agent requests are not refused on a bad guess.

3. Turns are bursty and unattended, so failures must be handled quietly

Nobody is watching the loop. A transient upstream error at turn 40 either gets retried or kills the task. The gateway retries once on 429 and 5xx responses before any bytes reach you, honouring the upstream's retry-after, and never bills the failed attempt. Requests that clients drop mid-stream are settled for what was actually generated rather than the full estimate. Rate limits are set for tool-call cadence, 300 requests per minute per key, rather than for a human typing.

Small compatibility gaps matter more here too. Modern SDKs send the developer role, which some upstreams reject with confusing errors. We normalise it for every provider, so an agent framework that works on one model works on all of them.

4. Cost is per task, so it has to be visible per request

Agents spend money in a loop you cannot see. Every above.dev response carries x-cost-usd with the exact cost of that request and x-input-cache-hit-rate with the fraction of input served from cache. Log those two headers and you know what a task cost and why. The dashboard shows the same per request, with the cache hit percentage in its own column.

What we deliberately left out

There is no smart routing that picks a model for you. You name the model and get that model, because an agent tuned against one model's tool-calling behaviour should not be silently moved to another. There are no subscriptions. Credit is bought in $10, $20 and $50 packs, never expires, and is consumed at the published rates. And there is no model we serve at a loss to win traffic: every rate is the upstream cost plus 10%, which is what makes the cache pass-through credible.

Try it on a real loop

The only honest test is your own agent on your own repo. There is $10 of free credit for GLM 5.3 Flash for new accounts, which at agent cache rates is a couple of weeks of use. Point your client at https://api.above.dev/v1, run a session, and read the cost headers.