Designing Enterprise Agentic Systems

Section 1

Agent Orchestration & Frameworks

Orchestration is the central nervous system of your system. It dictates how tasks are broken down, how tools are selected, and how agents collaborate.

🔄

ReAct Loop

Observe → Reason → Act → Observe. The LLM reasons about the state, calls a tool, then processes the result — cycling until the task is done.

🗺️

Workflows vs Agents

Anthropic recommends predictable workflows for 90% of tasks (routing, chaining, parallelisation) and reserving autonomous agents for truly open-ended problems.

📊

State Machines / DAGs

Production standard. Represent workflows as directed graphs — nodes are agents/functions, edges are conditional routing logic. Enables pause, resume, and inspection.

🏗️ Managed Agents & Infrastructure

Moving from a local script to production requires a lifecycle manager, not just a deployed model.

State Persistence

Save the agent's full graph state to a database (e.g., Postgres) at every node transition so it survives server restarts and can be inspected.

Human-in-the-Loop (HITL)

Pause execution and route high-stakes actions to a human approval queue before resuming — e.g., "Can I execute this DROP TABLE command?"

Enterprise Architecture Overview

Hover each node for details.

👤 User / Client

🔒 API Gateway / Auth

🧠 Orchestrator LLM

⚡ Router
small model

🗄️ State DB
Postgres

🙋 HITL Queue
approval gate

🤖 Sub-Agent A
specialist

🔧 MCP Tools
GitHub/Slack/…

📚 RAG / Vector DB
long-term memory

Interactive: ReAct Loop Simulator

Step through the agent's reasoning cycle.

💬 User Query Received

🧠 LLM: Reason (inner monologue)

🔧 Emit Tool Call (JSON)

⚙️ Tool Executes (API / DB / code)

👁️ Observe Result → Loop or Done?

✅ Final Response to User

—

Section 2

Memory & Context Management

Context is your most precious resource. How you fill it, trim it, and cache it determines your agent's cost, speed, and quality.

⚡

Short-Term Memory

The active context window — conversation history, recent tool outputs. Limited by token budget.

Optimise: sliding window truncation or LLM-based summarisation

💾

Long-Term Memory

Episodic and semantic memory stored in vector databases, retrieved on demand via RAG.

Optimise: Parent-Child chunking, HyDE query rewriting

⚙️

Prompt Cache

Anthropic and OpenAI both support KV-cache reuse for static prefixes — system prompts, rules, large documents.

Saves up to 90% on input token costs & cuts TTFT

💡 Token-Aware Routing

Use cheap, fast models (Claude Haiku, GPT-4o-mini) for intent classification and routing. Reserve expensive frontier models (Claude Sonnet, GPT-4o) only for complex reasoning steps that truly require it. This alone can cut inference costs by 60–80%.

Context Size vs Latency & Accuracy

Larger context windows improve recall but cost exponentially more time and money. RAG + prompt caching targets the sweet spot.

Token Economics: Three Approaches Compared

❌ Naive Full Context

Input tokens: ~80,000

Est. cost: ~$0.24 / call

TTFT: 8–14 s

Accuracy: Suffers from "lost-in-the-middle"

Not recommended at scale

⚡ RAG Only

Input tokens: ~6,000

Est. cost: ~$0.018 / call

TTFT: 2–4 s

Accuracy: High on specific queries

Good — but cold start latency

✅ RAG + Prompt Cache

Input tokens: ~6,000 (+cached prefix)

Est. cost: ~$0.004 / call

TTFT: 0.8–1.5 s

Accuracy: High + consistent system context

Production standard

Section 3

Reliability & Latency

Agents failing is expected and recoverable. Agents acting destructively is not. Both require deliberate architectural design.

⚡ Latency Reduction Tactics

1

Streaming (SSE)

Stream token-by-token. Users perceive Time-To-First-Token (TTFT), not total generation time. A fast TTFT with slow tail feels faster than batch delivery.

2

Parallel Tool Calling

LLM emits a JSON array of tool calls in one response. All tools execute concurrently. Only use when tools are independent — never if Tool B depends on Tool A's result.

3

Prompt Caching

Cache large static prefixes (system prompts, policy docs). Cached tokens are 10× cheaper and skip the encode step, dramatically reducing TTFT on repeated calls.

🛡️ Trustworthy Agent Design

Blast Radius

Define the maximum damage a misbehaving agent can cause. Scope write access via IAM roles — not just prompt instructions. A prompt that says "never delete" is not a security boundary.

Constitutional AI

Encode safety constraints in the system prompt. Useful — but insufficient alone. Prompt-level constraints are bypassable through prompt injection.

API / Tool Level Enforcement

The tool implementation itself must validate and refuse dangerous operations. The LLM calls the tool; the tool decides whether to execute. This is the real safety layer.

Interactive: Failover Cascade Simulator

See how a production agent degrades gracefully when its primary model fails.

1

Primary: Claude Sonnet (Anthropic)

Complex reasoning, 200k context, highest quality output

Standby

↓ timeout / rate-limit →

2

Fallback: GPT-4o (OpenAI)

Cross-provider redundancy eliminates single-vendor outage risk

Waiting

↓ timeout / rate-limit →

3

Degraded: Llama-3 8B (local / Ollama)

No external dependency — always available. Limited capability but keeps service up.

Waiting

Ready. Click to simulate an API timeout on the primary model.

Section 4

MCP & Advanced RAG

How your agent connects to the world determines its capabilities and its attack surface. Standardise your integrations and design your retrieval pipeline carefully.

🔗 Model Context Protocol (MCP)

Anthropic introduced MCP as an open standard to solve the fragmented tool-integration problem — every tool required a bespoke wrapper.

Before MCP

🔧 Custom GitHub wrapper → fragile, hard to maintain

🔧 Custom Slack wrapper → duplicated auth logic

🔧 Custom DB wrapper → different interface each time

N tools = N custom integrations = N security audit surfaces

With MCP

🟢 GitHub MCP Server — standardised protocol

🟢 Slack MCP Server — same client interface

🟢 Database MCP Server — unified auth model

1 client protocol × N servers = decoupled, auditable, swappable

📚 Advanced RAG Design Patterns

Pattern 1

Query Rewriting / HyDE

A vague user query like "how do agents handle memory?" embeds poorly. HyDE (Hypothetical Document Embeddings) first asks the LLM to write an ideal answer paragraph, then embeds that to search the vector DB. The hypothetical document is much closer in embedding space to the real answer.

User query → LLM writes hypothetical answer → embed → vector search → retrieve real docs → LLM answers

Pattern 2

Parent-Child Retrieval

Embed small, precise chunks (128 tokens) for high-accuracy similarity matching. When a chunk is retrieved, return its full parent document (1024+ tokens) to the LLM. Solves the tradeoff between embedding precision and answer completeness.

Small chunks: precise matching, no context leakage

Large parent: LLM gets full surrounding context

Pattern 3

Graph RAG

Build a knowledge graph over your document corpus — entities as nodes, relationships as edges. Enables multi-hop reasoning: "How does X relate to Y?" Standard vector search returns similar documents; Graph RAG traverses connected concepts. Best for complex domains where relationships between entities matter.

Trade-off: Significantly higher ingestion cost and infrastructure complexity. Justifiable only when relational queries are common.

Section 5

Case Studies & Interview Template

Two real-world architectures and a structured framework for designing any agentic system on a whiteboard.

Real-World Architecture Comparisons

Core Philosophy

"The LLM is an OS." The quality of your data pipeline — not your model — determines your agent's quality. Clean data beats a smarter model.

Primary Risk Guarded Against

Garbage-in-garbage-out. The system invests heavily in preprocessing (PDF → clean Markdown) before the LLM ever sees the data.

Data Pipeline Flow

PDFs / Web / Notes

→

Clean Markdown Conversion

→

Chunking & Embedding

→

Vector DB

→

RAG → LLM

System Design Interview Template

0 / 6 covered

Check off each pillar as you walk through a whiteboard design. State is saved automatically.

1. Scope & Blast Radius

Read-only or write? IAM constraints, HITL approval gates, maximum reversible action scope.

2. State Machine / DAG

Draw Entry Point → Router → Specialist Agents. State persisted to DB at every node for resume/inspection.

3. Data Ingestion (RAG / MCP)

MCP for tool connections. Define chunk size, embedding model, RAG pattern (HyDE / Parent-Child / Graph).

4. The Agent Loop

ReAct with <thinking> phases. Define max steps before forced human review.

5. Evaluation Harness

Dockerised CI/CD harness, golden dataset, pass@1 / pass@k metrics. Never ship without evals.

6. Bottlenecks (Latency / Cost / Reliability)

Streaming + parallel tools (latency). Prompt caching (cost). Failover cascade (reliability).

Section 6

Knowledge Check

Answer questions in your own words. An AI evaluator will score your answer 1–5 and give detailed feedback on what you got right and what to strengthen.

Note: This feature requires the Python server to be running (python server.py with ANTHROPIC_API_KEY set).

Progress 0 / 19 answered

—

Avg Score

0

🔥 Streak

— —

Click "Challenge Me" to receive a question.

Section 7 · Live

Latest in Agentic AI

Daily-fetched research summaries from arXiv and top sources, verified by Claude for accuracy and curated into a practitioner synthesis.

Last updated: —

Update Pipeline

① Fetch RSS → ② Date + Keyword Filter → ③ Claude Summarize → ④ Claude Verify → ⑤ Claude Synthesize → ⑥ Coverage Check → ⑦ Publish

⏰ Scheduling Daily Auto-Updates

Add a cron job to run the fetcher automatically every morning:

# Run daily at 8:00 AM — add to crontab (crontab -e)
0 8 * * * cd /Users/avocado21/Documents/github/AgenticAgents && ANTHROPIC_API_KEY=sk-ant-... python3 fetch_updates.py >> fetch.log 2>&1

Or run manually any time: python3 fetch_updates.py — use --dry-run to preview without writing.

🤖 Agentic Systems