A model's response is not automatically safe just because your own system generated it. Whatever produced the response, a correct model call, a successful injection attack, an ordinary hallucination, output validation treats the result the same way it would treat any other untrusted input to whatever consumes it next: a browser, a database, a shell, an API, or another agent.
Why "our model generated it" is not a trust signal
Insecure output handling, treating generated content as trusted because it came from your own model, is a recurring cause of downstream vulnerabilities. If a model's output is inserted directly into an HTML page without encoding, an attacker who successfully injects a script-like instruction can achieve something functionally similar to a stored cross-site scripting attack. If output is passed into a shell command or a database query without proper handling, the same categories of vulnerability that plague conventional applications apply here, triggered through a different input mechanism.
The output's origin does not change what it needs from a security standpoint. What matters is where it is going next, and that destination's requirements should be applied regardless of whether the content came from a user, a database, or a language model.
Format validation catches malformed output
The first layer of output validation checks whether generated content matches the expected format for its destination: valid JSON with the expected fields for a tool call, safe characters for a database identifier, properly encoded content for HTML rendering. This layer catches malformed or unexpected structure, which can happen even without any adversarial intent, since models occasionally produce output that does not match the requested format.
Policy validation catches output that is well-formed and still wrong
The more important and more often skipped layer checks whether valid, well-formed output actually satisfies business rules. A generated refund request might be syntactically perfect JSON specifying an amount that exceeds an approved limit. A generated database query might be properly parameterized and still request access to a record outside the current user's authorization. Format validation would pass both of these; only policy validation catches them.
This distinction matters because teams that implement format validation often consider output validation "done," while the actual risk that led to a real-world consequence usually passes format checks easily and only fails a policy check that was never implemented.
Validate before execution, not after
Output validation needs to happen before a generated action executes or generated content reaches its destination, not as a post-hoc audit of what already happened. This sounds obvious, but some architectures log the generated output and validate it asynchronously, after the action has already taken effect, which defeats the purpose for any action that is difficult or impossible to reverse.
Validate output intended for other agents, not just for humans or systems
In multi-agent architectures, one agent's output becomes another agent's input, and that output deserves the same validation as output headed to a database or a rendered page. A downstream agent that treats an upstream agent's output as fully trusted, without validating that a requested action fits the expected task and scope, can propagate a manipulated instruction further into the system than a single-agent architecture would allow.
Design for destinations, not for the model
A useful practice is designing output validation around each specific destination's requirements, rather than trying to build one generic filter that catches everything a model might produce. HTML rendering needs encoding appropriate to HTML. A shell command destination needs allowlisting of permitted commands and arguments, ideally avoiding shell execution of model-generated strings entirely where possible. A database query destination needs parameterization, not string concatenation, no matter how the query string was constructed. Treating each destination on its own terms produces more effective validation than one generic pass applied uniformly.
Where TELEON fits
TELEON's policy enforcement, applied through the gateway or supported middleware, provides the policy-validation layer specifically for tool calls: checking whether a generated action, even if well-formed, actually satisfies configured limits and authorization rules before it executes.
Format validation appropriate to each specific destination, HTML encoding, query parameterization, shell command allowlisting, remains application-level work that has to be implemented for each integration, since it depends on the specific technology and destination involved.
A short checklist
- Treat generated output as untrusted input to whatever it reaches next,
regardless of its origin.
- Apply format validation appropriate to each specific destination.
- Apply policy validation that catches well-formed output violating
business rules.
- Validate before execution, not as a post-hoc review after an action
already took effect.
- Apply the same validation to output passed between agents as to output
reaching a human or a system.
- Avoid shell execution of model-generated strings where a safer
alternative exists.
Output validation is the layer that catches what happens after a model has already been influenced, whether by injection or by an ordinary mistake. Treating generated content as untrusted, and validating both its format and its compliance with business rules before it reaches anything sensitive, is what keeps a bad model decision from becoming a real-world consequence.