Building and Orchestrating Agentic AI Workflows

From static RAG to AI agents

A RAG pipeline is a fixed graph: embed the question, retrieve context, generate an answer. It is predictable, cheap to reason about, and completely unable to decide that a question needs three lookups, a calculation, and a write to a CRM.

An agentic workflow replaces that fixed graph with a controlled loop: the model observes state, chooses a tool, the system executes it, and the result feeds the next step until a stop condition fires. The intelligence moves from the prompt into the orchestration layer — which is exactly where your engineering effort should go.

Use agents when at least one of these is true:

• The number of steps depends on the input and cannot be known up front.
• The task requires acting on external systems, not just reading from them.
• Multiple retrieval or computation strategies exist and one must be selected.
• Work spans minutes or hours and must survive restarts.

If none of them hold, keep the deterministic pipeline. A hard-coded chain that always works beats an agent that is right 85% of the time.

Anatomy of an agentic workflow

1. Intent router. Classify the request before spending a single agent turn. Simple lookups bypass the loop entirely.
2. Planner. Produce an explicit, inspectable plan — a list of steps with tools and arguments — rather than letting the model improvise turn by turn.
3. Executor. Run each step against real tools with timeouts, retries, and idempotency keys.
4. Critic. Validate the result against the goal and either accept, retry with feedback, or escalate to a human.
5. Memory. Persist state so a run can be paused, resumed, and replayed.

Bound every loop: a maximum turn count, a token budget, and a wall-clock deadline. An unbounded agent is not autonomous, it is an unmonitored spend.

Tool calling that survives production

• Design tools like a public API. Narrow scope, strict JSON schemas, descriptive names, and documentation written for the model as the caller.
• Prefer few coarse tools over many fine ones. Selection accuracy falls off sharply once a catalogue passes roughly a dozen options; group related actions behind one tool with a mode parameter.
• Validate arguments server-side. Treat every tool call as untrusted input — schema-validate, authorize against the caller's tenant and permissions, then execute.
• Make writes idempotent. Retries are inevitable; a deterministic request key prevents duplicate emails, tickets, and CRM records.
• Return structured, compact results. Dumping a 20 KB API response into context poisons the next turn. Summarize, truncate, and keep a reference for a follow-up fetch.
• Gate side effects. Read tools run freely; write tools require an explicit confirmation step or a human approval for anything irreversible.

Multi-agent coordination

Multi-agent is an organizational choice, not a capability upgrade. Split into separate agents only when roles need different tools, different models, or different permissions — otherwise one agent with a good tool catalogue is cheaper and easier to debug.

• Supervisor pattern. One orchestrator decomposes the task and delegates to specialists. Predictable and easy to trace; the default I reach for.
• Parallel fan-out. Independent subtasks (research several accounts, summarize several documents) run concurrently and are merged by a reducer step. The main latency win in agentic systems.
• Handoff chains. A specialist transfers the conversation to another with an explicit summary payload. Keep the payload typed — free-text handoffs lose critical detail.

Avoid free-form agent-to-agent chatter. Every message costs tokens and latency, and debates between models converge slowly and confidently on wrong answers.

State, memory, and long-running tasks

• Persist run state in a real database keyed by run ID — plan, completed steps, tool outputs, and cursor. Conversation history in RAM is not state.
• Run long tasks on a queue with checkpoints between steps so a crash resumes rather than restarts.
• Separate working memory (this run) from long-term memory (user preferences, prior decisions), and retrieve long-term memory the same way you retrieve documents.
• Compact aggressively: summarize completed steps into a short ledger instead of replaying every tool payload each turn.

Guardrails and failure handling

• Permissions at the tool boundary. The agent inherits the user's scope; never a service-level account that can cross tenants.
• Treat retrieved content as data, not instructions. Prompt injection arrives through documents, emails, and web pages — a tool call originating from retrieved text must never bypass authorization.
• Fail loudly and recoverably. Tool errors return to the loop as structured feedback the model can act on; repeated failure escalates instead of looping.
• Trace everything. Every turn, tool call, argument set, result, and token count against one run ID. Agentic systems are undebuggable without replay.

Evaluating agentic workflows

Answer quality is no longer the metric — task completion is. Build a suite of real tasks with known-correct outcomes and score:

• Task success rate — did the end state match the goal, regardless of the path taken.
• Tool selection accuracy — right tool, right arguments, at the right step.
• Efficiency — turns, tokens, and wall-clock time per completed task.
• Safety — unauthorized or unintended side effects, which should be zero and gated in CI.
• Refusal quality — for tasks the agent cannot do, escalation beats improvisation.

Run the suite on every change to prompts, tool schemas, or models. Agentic regressions are silent: the system still returns something confident, it just stops finishing the job.

Case study: agentic features in Sally

Sally is the meeting-intelligence suite I build at Liquid Technologies. Its conversational layer started as pure RAG over meeting transcripts and became agentic once users stopped asking "what was said?" and started asking "who owns this, and put it in the CRM."

An intent router decides whether a question needs retrieval at all — metadata questions such as attendance or duration answer from Postgres without touching the vector store, which is both cheaper and faster. Questions that do need context go through hybrid retrieval scoped to the tenant at query time. Action requests enter a bounded tool loop with typed, permission-checked tools for creating follow-ups, syncing to the CRM, and scheduling. Bot orchestration for joining calls runs as its own queue-backed workflow with checkpoints, so a dropped connection resumes instead of losing the meeting.

The pattern that made it reliable was boring: route before you reason, keep the tool catalogue small and typed, validate every argument server-side, and trace every run. The model choice mattered far less than the orchestration around it.

For the retrieval foundation these agents sit on, see my guide on building production-grade RAG systems.