Pipeline¶

Every agentic system — whether a workflow or an agent — processes requests through a pipeline. The pipeline builds the context window that the LLM eventually sees.

Pipeline as a bloodstream¶

Each stage reads from a shared context, adds its results, and passes it forward. The final stage, context assembly, turns the accumulated data into the actual prompt. A good pipeline follows two rules:

Fail fast: cheap checks first, expensive LLM call last.
Single responsibility: each stage owns one concern.

Stages¶

Stage	Purpose	Example guard
Validation	Reject malformed, empty, or unsupported input.	One regex can save $2,500/month on a million requests.
Input security	Detect prompt injection, PII leakage, impersonation.	Run before any tool or LLM call.
Query enrichment	Add user profile, session, and environment context.	"I want coffee" becomes "User wants coffee; team meeting in 1 hour."
Memory / RAG retrieval	Pull user preferences and domain knowledge.	Filter results for quality and safety.
Content filtering	Verify retrieved data is clean and relevant.	Detect injection through documents.
Context assembly	Build the final prompt within the token budget.	Sliding window, summarization, or compaction.
Smart routing	Select model and reasoning mode.	Cheap model for simple queries; reasoning for complex ones.
Generation	The LLM call.	In an agent, this starts the loop.
Output security	Check response for PII, system-prompt leaks, off-topic content.	Cheap secondary LLM or rule-based filter.
Post-processing	Format, validate JSON, finalize.	Always validate returned JSON.

Kroki Kroki

Workflow vs agent pipelines¶

In a workflow pipeline, the sequence is linear. Some stages may have local loops — for example, output security sends a response back for rephrasing — but the developer decides where to loop.

In an agent pipeline, the generation stage contains a loop. The LLM decides whether to call another tool. The pre-loop stages (validation, security, enrichment) still run once; the post-loop stages (output security, formatting) run once at the end.

Kroki Kroki

Context assembly¶

Context assembly is the stage that turns accumulated data into a prompt. It has four inputs:

System prompt — role, boundaries, rules, format, example.
User query — enriched with context.
Memory — user preferences and history.
RAG results — domain documents.

All four compete for a token budget. Anthropic calls context engineering "the natural progression of prompt engineering": it is the practice of curating the smallest set of high-signal tokens that maximize the likelihood of the desired outcome [Anthropic, 2025]. Redis notes that models use information at the beginning and end of the input far better than information in the middle, so place decision-critical facts at the edges and give every section a hard token budget [Redis, 2026].

If the window is too large, use one of these strategies:

Sliding window: keep the last N messages, drop older ones.
Summarization: compress older turns into a summary.
Compaction: collapse the entire history into a new, dense context once.

For long-horizon tasks, prefer structured note-taking (agent writes persistent notes) and sub-agent architectures that return only condensed summaries, rather than dumping everything into the main context [Anthropic, 2025].

Kroki Kroki

KV cache and context mutations¶

During an agent loop, the context should be append-only. Every mutation in the middle invalidates the KV cache and forces the provider to reprocess the entire prefix. Manus calls KV-cache hit rate the single most important metric for a production agent: with Claude Sonnet, cached input tokens cost $0.30/MTok while uncached tokens cost $3/MTok, a 10× difference [Manus, 2025].

Common cache killers:

A timestamp injected into the system prompt.
Modifying a previous tool result.
Non-deterministic JSON key ordering.
Dynamically adding or removing tool definitions mid-iteration.

When the context grows beyond the budget, reassemble the whole window once rather than mutating it every iteration. Better yet, use just-in-time references (file paths, stored queries, URLs) that the agent loads on demand instead of pre-loading everything up front [Anthropic, 2025].

Security layers¶

Input and output security are not optional. Hossain's production guardrail stack runs validation, PII detection, injection classification, the LLM, an output filter, and a response gate as independent layers [Hossain, 2026]. Microsoft FIDES adds deterministic information-flow control: every content item carries integrity (trusted/untrusted) and confidentiality labels, and policies are enforced before a sensitive tool runs [Microsoft, 2025].

Layer	What it catches	Typical latency
Schema validation	Malformed input, bad parameters	<2 ms
PII redaction (inbound)	Credit cards, SSNs, emails pasted into prompts	15–80 ms
Prompt-injection classifier	`[SYSTEM OVERRIDE]`, hidden instructions in user input or retrieved docs	30–120 ms
Tool-output trust boundary	Untrusted data driving privileged tools	0 ms (policy check)
Output PII / policy filter	Hallucinated or leaked sensitive data	60–250 ms
Schema gate (tool args)	Invalid function-call arguments	<5 ms

Metrics per stage¶

Treat each stage as an observable component:

Stage	Metrics
Validation	Reject rate, latency, top rejection reasons
Input security	Injection score distribution, false-positive rate
Enrichment	Cache hit rate for user profile / session lookups
Retrieval	Recall@k, MRR, contaminated/unsafe result rate
Context assembly	Tokens per section, budget overflow events, KV-cache hit rate
Smart routing	Model distribution, cost per request class
Generation	Latency, token count, stop reason
Output security	PII leak rate, policy violation rate
Post-processing	JSON parse failure rate, formatting errors

Design principles¶

Fail fast first: validation and security are cheap; do them before any LLM call.
Single responsibility per stage: do not mix format validation with injection detection. They have different owners and failure modes.
Configurable depth: start with validation, security, generation, and post-processing. Add enrichment, RAG, content filtering, and smart routing when data shows you need them.
Context is a budget: give system prompt, history, retrieved docs, and query each a hard token allocation; do not let one section silently crowd out the others.
Trust labels beat defensive prompts: deterministic provenance checks are stronger than asking the model to "ignore the above" [Microsoft, 2025].

Deeper dive: request pipeline patterns at scale¶

At high volume, the pipeline becomes a context-building funnel rather than a linear script. Each stage reads from a shared context object, appends its results, and passes the object forward without rewriting earlier state [Zalesov, 2026]. This append-only discipline is not just a cleanliness choice: mutating the prefix invalidates the KV cache and forces the provider to reprocess expensive tokens. The production consensus is therefore to push cheap, deterministic work as far left as possible — rate limiting, schema validation, bot detection, prompt-injection classification, and PII screening all run before the first model call [Zalesov, 2026].

Once the request is clean and enriched, the next failure mode is usually not the prompt but what the model is allowed to see. A production context pipeline rewrites short or pronoun-heavy queries, over-fetches with hybrid dense-and-sparse retrieval, reranks candidates for precision, deduplicates near-identical or contradictory chunks, and only then assembles the final window under an explicit token budget [Singh, 2026]. The assembly stage is the highest-leverage place to intervene: place decision-critical facts at the start or end of the window to exploit positional bias, give every section a hard token allocation, and treat "concatenate everything and hope" as the default failure mode [Singh, 2026].

For agents, the same stages apply but the loop changes the contract. Validation, security, and enrichment still run once before the loop; inside the loop, retrieval and memory become tool calls whose results are appended to the thread [Zalesov, 2026]. There is no fixed assemble-then-call step — the context grows incrementally — so production agents cap iterations, total tokens, and per-tool calls to prevent runaway cost and compounding hallucinations [Zalesov, 2026].

Common context-assembly failure modes and the stage that prevents them:

Failure mode	Preventive stage
Vague or pronoun-heavy query	Query rewriting / enrichment
Exact-match IDs lost in semantic retrieval	Hybrid retrieval
Contradictory or duplicate documents in the window	Reranking + deduplication
Critical fact buried in the middle	Position-aware context assembly
Silent history bloat crowding out retrieved docs	Hard per-section token budget
Re-running the full pipeline for near-identical requests	Reuse / semantic cache

References¶

Anthropic. "Building Effective AI Agents." Dec 2024. https://www.anthropic.com/engineering/building-effective-agents
Anthropic. "Effective Context Engineering for AI Agents." Sep 2025. https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents
Hossain, M. S. "AI Agent Guardrails in Production: Input Filtering, PII Redaction & Prompt Injection Defense." Mar 2026. https://mdsanwarhossain.me/blog-ai-agent-guardrails.html
Manus. "Context Engineering for AI Agents: Lessons from Building Manus." https://manus.im/en/blog/Context-Engineering-for-AI-Agents-Lessons-from-Building-Manus
Microsoft. "Agent Security with FIDES." Microsoft Learn, 2025. https://learn.microsoft.com/en-us/agent-framework/agents/security
Redis. "Context Assembly: Building the Prompt the Model Sees." Redis blog, Jul 2026. https://redis.io/blog/context-assembly-building-the-prompt-the-model-sees/
Singh, H. "Context Engineering Is Replacing Prompt Engineering: Building Production Context Pipelines for LLM Apps." DEV Community, Jul 2026. https://dev.to/hrsvd/context-engineering-is-replacing-prompt-engineering-building-production-context-pipelines-for-llm-14m8
Zalesov, A. "A Practical Memo on Building LLM Agents." May 2026. https://medium.com/@zallesov/a-practical-memo-on-building-llm-agents-cc5179cb2e50