Rate limiting for a conventional API typically protects against a single concern: too many requests overwhelming a backend service. AI applications need rate limiting applied at several distinct layers, each protecting against a different failure mode, since a single request-level limit doesn't adequately address the specific risks of expensive model calls, tool usage that can trigger real-world effects, and agent loops that can consume resources far beyond what a simple request count would suggest.
Request-level limiting protects against conventional abuse
The most basic layer, limiting the rate of incoming requests per identity or session, protects against the same conventional abuse patterns rate limiting has always addressed: automated scripting, credential stuffing adjacent probing, simple denial-of-service attempts. This layer should exist regardless of any AI-specific considerations, using the same established rate limiting patterns applied to any public or semi-public API.
Model call limiting protects against cost exposure specifically
Because model API calls often carry meaningful per-request cost, particularly for longer contexts or more capable models, a separate limit specifically on model call volume and, where the provider's billing model makes it relevant, on total token consumption per identity or session, protects against a cost exposure risk a simple request-count limit never fully captures. A single request triggering a very long context and a very long generated response can cost meaningfully more than many short, simple requests, and cost-aware limiting accounts for that difference instead of treating every request as equivalent.
Tool call limiting protects against real-world action risk
For any agent with tool access, limit the rate and volume of tool calls specifically, separate from the rate limit on requests or model calls generally. This connects directly to the action risk running throughout this content series: a limit on how many tool calls, especially calls to a specific high-impact tool, can happen within a given time window for a given identity bounds the real-world consequence of a compromised or looping agent, independent of how the request-level or model-call-level limits are configured.
Agent loop iteration limits protect against runaway autonomous behavior
For agents that can autonomously call tools repeatedly in pursuit of a goal, a hard limit on the number of iterations within a single task, enforced independently of the model's own judgment about when to stop, is a distinct and necessary control covered in agentic security guidance. This isn't the same as a rate limit measured in requests per minute. It's a bound on how many steps a single logical task can take before it gets forced to stop, regardless of how fast or slow those steps happen.
Set limits based on legitimate usage patterns, not arbitrary round numbers
Establish rate limits based on actual analysis of legitimate usage patterns for a given feature, rather than picking round numbers that feel reasonable with no data behind them. A limit set too low creates friction for legitimate users and generates support burden. A limit set too high provides little real protection. Reviewing actual usage data where it exists, or making a deliberate, documented estimate where historical data doesn't exist yet, produces more defensible and more effective limits than an arbitrary guess ever would.
Apply different limits for different risk tiers of action
Following the risk classification approach covered in governance guidance, apply stricter limits to higher-risk actions specifically: a tool that can issue refunds deserves a tighter per-identity rate limit than a tool that only retrieves publicly available information, rather than applying one uniform rate limit across every action a given agent can perform regardless of each action's actual risk.
Monitor for patterns suggesting deliberate limit evasion
Beyond simply enforcing limits, monitor for patterns suggesting an identity or a coordinated set of identities deliberately staying just under a rate limit threshold, distributing requests across many accounts or sessions specifically to avoid triggering any single limit. This connects to the broader abuse monitoring guidance covered elsewhere, and it takes looking at aggregate patterns across identities, not just whether any single identity's own limit got individually exceeded.
Where TELEON fits
TELEON's policy enforcement, applied through the gateway or supported middleware, can implement tool call rate limits and agent loop iteration bounds directly at the point of each tool request, providing a consistent enforcement layer across different agents and model providers using that deployment path.
Setting the request-level and model-call-level limits, and analyzing legitimate usage patterns to calibrate them appropriately, remains application-level and product decision work specific to each feature's actual usage.
A short checklist
- Apply conventional request-level rate limiting as a baseline layer.
- Apply a distinct limit on model call volume and cost exposure.
- Apply a distinct limit on tool call rate and volume, especially for
high-impact tools.
- Enforce a hard cap on agent loop iterations, independent of the
model's own judgment.
- Set limits based on actual usage pattern analysis, not arbitrary round
numbers.
- Apply stricter limits to higher-risk actions specifically.
- Monitor for coordinated patterns suggesting deliberate limit evasion.
Rate limiting an AI application well requires several distinct layers, requests, model calls, tool calls, and agent loop iterations, each protecting against a different failure mode that a single, generic limit would not adequately address on its own.