RAG (retrieval-augmented generation) became the default: dump documents into a vector store and ask. It helps. It also misleads.
Documents ≠ operations
A procedure manual describes the norm. Operational memory is what the team actually does: exceptions, special customers, ticket states, “do not touch this on Fridays”. That memory lives in ERP, CRM, chats and heads — not in the onboarding PDF.
If your agent only reads the wiki, it answers like a first-day intern.
What RAG gives you (and what it does not)
Gives you: textual context, policies, FAQs, code snippets or contracts.
Does not give you alone: current order status, user permissions, incident history, or yesterday’s supervisor decision.
For that you need tools and state: APIs, queries, and sometimes a store of living facts — not just embeddings.
A useful mental model
| Layer | Question it answers |
|---|---|
| RAG / KB | “What does the documentation say?” |
| State / tools | “What is happening right now?” |
| Episode memory | “What did we do in this case?” |
| Policy | “Are we allowed to?” |
Decent agents combine all four. Demos stop at the first.
How to start without overengineering
- List the 10 questions that actually matter to the business.
- Mark which ones need text vs live data.
- Wire tools only where needed; elsewhere, clean RAG with good chunks and citations.
- Measure failures: hallucination vs stale data vs denied permission — different problems.
RAG remains useful. It just stops being the whole architecture the moment the agent must operate, not only explain.