Token Costs per Tenant: Why Rate Limiting Works Differently for LLM-SaaS

Classic SaaS rate limiting is fundamentally always the same exercise: count requests per minute per customer, return 429 on excess. The metric is stable because a request causes relatively predictable latency—a database API call takes time proportional to complexity, but the difference between a 10-row and 1,000-row result set is usually a matter of milliseconds, not orders of magnitude. The variance is foreseeable and small.

With an LLM product, that assumption breaks down completely. Here, tokens cost direct money—the provider literally bills per token the model processes or generates. A request with a short question and a request that ships a 200-page PDF as context are identical at the HTTP level—a POST to an endpoint—yet cost vastly different amounts. Cost variation depends radically on input and output size: a request with heavy context and a long answer costs many times more than a short request. Since output is typically more expensive than input per token, requests with thick RAG context or long responses are the real cost drivers. A tenant with such patterns costs multiples of another—completely independent of request count.

Anyone using "requests per minute" as a limit is therefore limiting the wrong thing. A tenant making many short requests gets artificially throttled, while a tenant making few but massive requests slips under the radar—incurring multiples of the cost. The metric that actually matters is tokens—and even that isn't straightforward to count.

Why Tokens Alone Don't Equal Costs—Three Complicating Factors

Tokens by themselves aren't costs; they only become costs through three factors that all interact:

Input and output tokens cost differently. With nearly every provider, output is significantly more expensive than input—sometimes by a factor of 3 to 5. A tenant requesting long, detailed answers (via system-prompt specifications or frequent summarization use cases) incurs noticeably different costs than one wanting short yes/no answers—at identical request count and similar input length.

Models within a provider cost differently, and in a multi-provider architecture (OpenAI, Anthropic, Gemini, plus possibly self-hosted open-weight models), this multiplies. A tenant with access to the priciest model can incur three or four times the costs of one with a cheaper model at identical usage patterns—and the price ratio between models shifts when providers adjust prices, which happens regularly in this market.

Caching shifts the cost basis without changing usage visibility. Prompt caching (available at the major provider APIs) charges repeated context blocks—like a long system prompt or recurring RAG documents—at a fraction of regular price once they hit the cache. For your own cost accounting, this means: the same tenant, same token count, might cost "full price" one day and "cache price" the next, depending on cache warmth. Anyone estimating costs roughly per tenant instead of capturing them exactly per call loses control over margins exactly here—when a tenant is most active and margins matter most.

The Reservation Problem: Only Learning What Something Cost After It's Done

The real architectural crux is temporal: with a classic request, you know costs roughly beforehand—with an LLM call, you only learn the actual token count of the response after generation completes. With streaming responses, results arrive token-by-token while the call is running.

This creates a timing problem anyone who's built a prepaid system knows: check balance only before the call, and a nearly-empty tenant can still start an expensive call (heavy context, high max_tokens) and end up negative. Check it only after, and the problem already happened before you noticed.

The clean solution is a reservation pattern, familiar from payments (same logic as credit-card pre-authorization at hotel booking):

async function callLlm(tenantId: string, request: LlmRequest) {
  // 1. Estimate upper bound: input tokens are known,
  //    output conservatively assumed to be max_tokens
  const estimatedMaxCost = estimateCost(request.inputTokens, request.maxTokens, request.model);

  // 2. Create reservation — balance immediately reduced by estimate,
  //    not after call completes
  const reservation = await creditLedger.reserve(tenantId, estimatedMaxCost);
  if (!reservation.ok) throw new InsufficientCreditsError();

  try {
    const response = await llmProvider.call(request);
    // 3. Replace reservation with actual costs (usually lower,
    //    since responses rarely max out max_tokens)
    await creditLedger.settle(reservation.id, actualCost(response.usage));
    return response;
  } catch (err) {
    // 4. On error: fully release reservation
    await creditLedger.release(reservation.id);
    throw err;
  }
}

The point often missed: step 3 almost always gives credits back, because actual responses rarely exhaust max_tokens. Without this settlement step, you reserve too pessimistically forever, and tenants complain about balance "disappearing" without visible consumption. With it, you get a ledger that exactly matches the sum of actual provider bills at month-end—something non-negotiable for your own margin arithmetic because otherwise your actual provider invoice and your internal bookkeeping diverge, and no one can explain the gap.

The Noisy Neighbor Effect at Provider Level

One aspect barely mentioned in rate-limiting discussions but the first real incident you'll face: LLM providers don't limit per end-customer but per your own API key—typically tokens-per-minute (TPM) and requests-per-minute (RPM) at account or organization level. This means all your tenants share a single global quota at the provider, regardless of how cleanly you partition costs internally.

Without your own countermeasures, a single tenant with a load spike (e.g., batch import with thousands of documents, all embedded and summarized) can exhaust the entire provider quota for a short time—and all other tenants get 429 errors from the provider, despite doing nothing wrong. This is the LLM variant of the classic noisy-neighbor problem, except the "neighborhood" isn't CPU or I/O, but a shared provider quota.

The countermeasure is a two-tier limit: an internal per-tenant sub-limit well below the global provider quota (e.g., token bucket per tenant, continuously refilling), plus your cost tracking. This prevents any single tenant from monopolizing the shared quota even if they technically have budget left. For multiple providers, it's additionally worth fallback logic: if Provider A's quota is exhausted, automatically switch to Provider B—provided model behavior is exchangeable enough for the use case, something you should test beforehand, not in an outage.

Credit System Instead of Request Counter

From the three previous points, an architectural decision nearly follows inevitably: instead of counting requests, an LLM-SaaS needs a credit system using tokens (not requests) as the base unit, with a conversion factor per model and direction (input/output). Tenants buy or receive monthly credits, each call consumes credits per actual token usage of the model in use—costlier models consume proportionally more credits per token, keeping price structure for the customer stable regardless of which model backend runs.

Crucial for implementation: tracking must happen at the individual API-call level, reading actual usage directly from the provider's response, not estimated or aggregated afterward. Providers supply actual token usage including cache hits as standard in responses—that number is truth, not an internal estimate. Every divergence between your estimate and provider truth should be exposed as a monitoring metric, because growing divergence usually signals a bug in cost logic, not just imprecision.

Conclusion

Rate limiting for LLM-SaaS isn't a variant of classic API rate-limiting, but its own problem with its own failure modes: tokens not requests as the unit, costs only knowable after the call, caching that shifts the cost basis, and a shared provider quota limiting across all tenants. Anyone ignoring this and just reusing "requests per minute" from their last project discovers the problem typically when the first provider bill is noticeably higher than the sum of internally-booked usage—at which point the gap is almost impossible to reconstruct.

Anyone starting instead with reservation patterns, true token-based ledgers, and per-tenant sub-limits solves three things simultaneously: cost accounting that matches provider invoices, fair access to shared quotas, and billing you can actually trust when a tenant asks about their charges.

---

Further reading: AWS – SaaS Tenant Isolation Strategies for general silo/pool/bridge vocabulary, which also translates to shared provider quotas.