Rag

Ai Agents

Gen Ai

LLM

AI Agents Don’t Need Vector Search Anymore: Inside the Agentic Search Stack Replacing RAG in 2026

22 min readMay 12, 2026

In May 2025, Anthropic did something that should have been a bigger deal than it was: they took vector search out of Claude Code. The embedding pipeline, the local vector database, the chunking heuristics, gone. In their place, grep. According to Claude Code's creator Boris Cherny, the result "outperformed everything. By a lot." A year later, the question isn't whether AI agents need vector databases, it's why anyone thought they did. Cursor hired the engineers behind the decision. Windsurf, Cline, Devin, and Sourcegraph Amp dropped vectors for tool-driven search. Anthropic's own multi-agent research system, built on the same idea, beat a single Claude Opus 4 by 90.2% on internal evals. An Amazon Science paper at AAAI 2026 measured agentic keyword search at 94.5% of RAG faithfulness with zero vector store. Search-R1 trained the retrieval policy with reinforcement learning and beat RAG by 24% relative. The vector database hasn't died. It's been demoted from the default to the fallback.

*The foundational building block of every modern AI agent, per Anthropic: an LLM augmented with* ***retrieval, tools, and memory****. The agent-as-retriever pattern collapses the first two, retrieval becomes a tool call, and the agent itself drives the search. Source:* *Anthropic Research, Building Effective Agents*.

1. Introduction: The Quote That Killed Vector RAG for Code

On the Latent Space podcast in May 2025, Boris Cherny, creator of Claude Code, said something that quietly broke the RAG consensus[1]:

“Early versions of Claude Code used RAG + a local vector db, but we found pretty quickly that agentic search generally works better. It is also simpler and doesn’t have the same issues around security, privacy, staleness, and reliability.”

And later in the same conversation:

“It outperformed everything. By a lot, and this was surprising.”

That admission would have been a footnote in 2024. In 2026 it’s the canonical example of an architectural shift the entire frontier-coding industry has now copied, and one that Anthropic has since formalized into a named pattern.

In their September 2025 engineering post “Effective context engineering for AI agents,”[2] Anthropic gave the approach a name: just-in-time context loading. The idea is that “rather than pre-processing all relevant data, agents maintain lightweight identifiers (file paths, stored queries, web links, etc.) and use these references to dynamically load data into context at runtime using tools.” Pre-built embeddings are replaced by an agent that retrieves on demand, the same way a human reads a codebase, not the way a search engine indexes one.

By 2026, the pattern is everywhere. Claude Code, Cursor, Windsurf (and Devin via Cognition’s acquisition), Cline, Sourcegraph Amp, and a growing list of agentic systems no longer index their target corpora into a vector database. Instead, they expose retrieval as a set of tools, and let the LLM decide what to call, when, and how often. The pattern has multiple names now: agent-as-retriever, agentic search, vectorless RAG, just-in-time context loading, or simply tool-use retrieval.

This isn’t a vibes-based trend. It’s measurable:

Anthropic’s official guidance now tells developers: “start with agentic search and only add semantic search if you need it”[2][3].
Anthropic’s own multi-agent research system (Opus 4 lead + Sonnet 4 subagents, all using agentic search) outperformed single-agent Claude Opus 4 by 90.2% on internal research evals, cutting research time by up to 90% on complex queries, at the cost of roughly 15× more tokens[4].
Amazon Science published “Keyword search is all you need” (AAAI 2026, arXiv:2602.23368) showing tool-use agents reach 94.5% of RAG faithfulness, 88.0% of context recall, and 91.5% of answer correctness using only rga and pdfgrep[5].
Search-R1 (arXiv:2503.09516) trained the retrieval policy itself with reinforcement learning and beat RAG by 24% relative on a 7-dataset average (Qwen2.5–7B)[6].
Cursor hired Boris Cherny and Cat Wu, the leads behind Claude Code’s agentic search, directly from Anthropic in July 2025[7].
Windsurf shipped SWE-grep, a specialized retrieval model achieving 10× faster context retrieval than generic agentic search[8].
Chroma released Context-1, a 20B agentic-search model with ~10× faster inference and ~25× lower cost than frontier models on retrieval tasks[9].

This post unpacks the agent-as-retriever pattern in 2026: the just-in-time loading paradigm, the architectures (Claude Code, Cline, Probe, Search-R1), the benchmarks that justify them, and how to slot the pattern into a real stack alongside vectors, graphs, and long context.

2. Background: Why Vector RAG Broke for Code (and Then for More)

To understand the shift, look at the original baseline. SWE-bench, the canonical benchmark of real GitHub issues, was released in October 2023 with a simple RAG baseline: chunk the codebase, embed, retrieve top-k, generate a patch. It scored 1.96%[10]. The first agent system, SWE-agent, jumped to 12.47% by replacing retrieval with tools (open_file, scroll_down, edit_lines). By 2026 the SWE-bench Verified leaderboard is dominated by agentic systems above 80%, and none of the top entries rely on vector retrieval over the target repo.

Why does vector RAG fail so hard on code specifically?

Semantic similarity ≠ relevance. “Most similar embedding” is a poor proxy for “the function that breaks when I change this line.” Code has explicit structural relationships, imports, type definitions, call graphs, that flat embeddings flatten.
Identifiers are the search. When you ask “where is processPayment defined?", you want an exact match. Vector search introduces false positives (handlePayment) and false negatives (the real definition out-ranked by a similar-looking comment).
The index is always wrong. Code drifts. Every commit invalidates part of the index. Continuous re-indexing is expensive and never quite catches up, the same problem GraphRAG faced before LazyGraphRAG.
The index is a liability. A vector index of proprietary code is a copy of that code, sitting on some other infrastructure, often with weaker access controls than the source repo.
Single-shot is brittle. Top-k gets one chance. Miss the right file once and the model confidently generates wrong code.

The Amazon Science paper generalized this beyond code, testing six datasets including FinanceBench, BlockchainSolana, Llama2Paper, and HistoryOfAlexnet. On FinanceBench, the agentic keyword-search approach actually beat traditional RAG by 6 percentage points on answer correctness (30.40% vs 24.24%)[5]. The failure modes of “chunk and embed” turn out to be general, not code-specific.

3. The Anthropic Decision: Four Reasons to Replace RAG with grep

Cherny and Cat Wu have been unusually explicit about why they pulled vector search out of Claude Code. From the podcast[1] and follow-up interviews, the reasoning collapses to four points:

3.1 Accuracy (the surprise)

The team expected agentic search to be worse than RAG and were prepared to accept some quality loss for operational simplicity. Instead it outperformed. Cherny’s exact words: “It outperformed everything. By a lot, and this was surprising.” The mechanism: an LLM driving grep iteratively can refine its query, look at adjacent files, follow imports, and self-correct. A single embedding lookup cannot.

3.2 Freshness

An agent reading the filesystem reflects the current state of the repo. There is no “index lag.” Edit a file, ask Claude Code about it 100ms later, and it reads the new bytes. A vector index would be stale until the next re-embedding pass.

3.3 Security & Privacy

From Cherny: “There’s this whole indexing step that you have to do for RAG… there’s security issues because this index has to live somewhere… it’s just a lot of liability for a company to do that.” Enterprise customers in particular do not want a separate embedded copy of their proprietary code in someone else’s infrastructure.

3.4 Reliability

Fewer components, fewer failures. A grep-based retriever has no embedding model to drift, no vector DB to fall over, no re-indexing pipeline to lag, no chunking strategy to tune. ripgrep just works. So does find. So does cat.

Cat Wu summarized the deployment story: “Claude is really good at agentic search. You can get to the same accuracy level with agentic search and it’s just a much cleaner deployment story.”

4. The Just-in-Time Context Loading Paradigm

Anthropic’s framing: prompt engineering treats writing prompts as a discrete task, while context engineering is an iterative process of curating what enters the context window. Just-in-time context loading is the retrieval half of that curation. Source: *Anthropic, Effective context engineering for AI agents*.

Anthropic’s “Effective context engineering for AI agents”[2] draws a sharp line between two retrieval philosophies:

Pre-inference retrieval. The traditional RAG approach: embed everything up front, store vectors, look up at query time, stuff the top-k into the prompt. Everything the model might need has to be predicted and indexed in advance.
Just-in-time loading. Agents “maintain lightweight identifiers (file paths, stored queries, web links, etc.) and use these references to dynamically load data into context at runtime using tools.” Nothing is pre-loaded; the agent fetches what it needs, when it needs it.

The Anthropic post calls out Claude Code as the canonical example: “CLAUDE.md files are naively dropped into context up front, while primitives like glob and grep allow it to navigate its environment and retrieve files just-in-time.”

The deeper insight is what this does to the shape of the context window. In pre-inference RAG, you spend tokens on chunks you guessed would be relevant. In just-in-time loading, you spend tokens only on chunks the agent decided were relevant, and you skip everything else. The principle Anthropic articulates is “finding the smallest set of high-signal tokens that maximize the likelihood of desired outcomes.”

This is why the pattern doesn’t just survive token-cost criticism, it inverts it. Naive agent loops use more tokens than vector RAG. But just-in-time loading combined with subagent context isolation (more on this below) can spend fewer tokens than RAG by avoiding low-signal chunks entirely. A frequently cited internal Anthropic figure: Claude Code’s tool lazy loading, not loading tool definitions until they’re needed, reduces context usage by ~95%[11].

5. The Agent-as-Retriever Architecture, In Detail

What does the pattern actually look like? In Claude Code (and the Claude Agent SDK more broadly), retrieval is exposed as a small, deliberately boring set of tools[3][12]:

Glob, file-path pattern matching. Near-zero token cost.
Grep, regex content search backed by ripgrep (and as of v2.1.117, April 2026, by embedded ugrep + bfs invoked via Bash on macOS/Linux native builds[13]).
Read, full or partial file contents into context.
Bash, fallback shell, for the long tail (tail, head, jq, git log, find with predicates).
Explore subagent, a dispatched read-only agent (Haiku 4.5 by default) with its own context window, used for parallel codebase exploration.

A reverse-engineering study of Claude Code’s TypeScript source (arXiv:2604.14228) documents the broader system: 54 built-in tools (19 unconditional, 35 feature-gated), and only 1.6% of the codebase is AI decision logic; the remaining 98.4% is operational infrastructure, context management, permissions, tool dispatch, compaction[14]. The decision layer is small. The retrieval-and-context-management layer is enormous.

5.1 The Five-Layer Compaction Pipeline

Because just-in-time loading still fills the context window eventually, Claude Code runs a five-layer compaction pipeline when approaching the 200K-token limit[14]:

Budget reduction, drop the least relevant content first.
Snip, remove redundant tool-call output.
Microcompact, summarize individual long messages.
Context collapse, fold older turns into a shorter recap.
Auto-compact, final summarization pass when nothing else fits.

This is the necessary complement to just-in-time loading: once you’ve loaded only what you need, you also need to gracefully forget what you no longer need. Anthropic describes compaction as the first lever of long-horizon context engineering, “taking a conversation nearing the context window limit, summarizing its contents, and reinitiating a new context window with the summary”[2].

5.2 The Control Loop

plan -> glob/grep -> read candidates -> refine query
      -> repeat (or spawn subagent) -> compact -> answer

This is the same shape as agentic RAG (Self-RAG, CRAG, A-RAG), with one critical difference: there is no pre-built index between the agent and the bytes. The “retriever” is whichever shell tool the agent chose to call.

6. The Proof: “Keyword Search Is All You Need” (Amazon, AAAI 2026)

The most rigorous public benchmark of agent-as-retriever vs. vector RAG is Subramanian et al.’s “Keyword search is all you need: Achieving RAG-Level Performance Without Vector Databases Using Agentic Tool Use”[5], at AAAI 2026. Same LLM (Claude 3 Sonnet, 200K context, temperature 0.001), same six datasets, same evaluation harness. The only difference is the retriever, Amazon Bedrock Knowledge Base with Titan Text Embeddings V2 on one side, a ReAct agent calling pdfmetadata, rga, and pdfgrep on the other.

*The Amazon Science paper’s contrast: vector RAG pipeline (top) vs. agentic keyword search (bottom). Three Linux command-line tools,* `pdfmetadata.sh`, `rga`*, and* `pdfgrep`*, inside a ReAct loop. No embedding model. No vector store. Source:* *Subramanian et al., AAAI 2026, Fig. 1*.

Headline numbers across all datasets:

Faithfulness: agent 0.81 vs RAG 0.86 → 94.52% attainment
Context Recall: agent 0.68 vs RAG 0.77 → 88.05% attainment
Answer Correctness: agent 0.59 vs RAG 0.65 → 91.48% attainment

And the outliers tell a deeper story. FinanceBench, long, table-heavy financial filings, flipped the order: the keyword-search agent scored 30.40% answer correctness vs RAG’s 24.24%. BlockchainSolana, technical documentation with precise term-matching requirements, hit 99.97% attainment.

Spider plot for the LLM Survey paper dataset: agentic keyword search (orange) vs. vector RAG (blue). The two retrieval strategies trace nearly the same shape, without a vector database in sight. Source: *Subramanian et al., 2026, Fig. 2*.

The paper’s punchline: “Vector databases are not essential for high-quality retrieval performance. Agentic approaches using simple keyword search tools are a viable alternative for many applications.”

7. Search-R1: When the Retrieval Policy Is Trained, Not Prompted

The Amazon paper shows that a prompted agent matches RAG with just keyword tools. Search-R1 (Jin et al., arXiv:2503.09516)[6] goes one step further: train the retrieval policy itself with reinforcement learning, and the agent doesn’t just match RAG, it beats it.

*Search-R1’s training loop with PPO and GRPO: the model emits* `<search>` tokens mid-reasoning, retrieved documents are inserted but masked from the policy loss, and an outcome-based reward shapes when and what to query. Once retrieval is a tool call, it becomes a learnable policy. Source: *Jin et al., arXiv:2503.09516, Fig. 1*.

The setup: extend an R1-style reasoning model with the ability to emit <search>query</search> calls mid-reasoning. The model learns, via outcome-based RL (using veRL + RAGEN), when to search, what to query, and when it has enough. Token masking during retrieval keeps training stable.

The results on seven QA datasets (NQ, TriviaQA, PopQA, HotpotQA, 2WikiMultiHopQA, Musique, Bamboogle) with Qwen2.5–7B-Base:

Search-R1 average EM: 0.431
RAG baseline average EM: 0.304
SFT average: 0.207 · R1 without retrieval: 0.276 · Rejection sampling: 0.348
Per-dataset Search-R1 scores: NQ 0.480, TriviaQA 0.638, PopQA 0.457, HotpotQA 0.433, 2WikiMultiHopQA 0.382, Musique 0.196, Bamboogle 0.432

That’s a 24% average relative improvement over RAG for Qwen2.5–7B and 20% for the 3B model. CoSearch and Agentic-RAG-R1 have since pushed those numbers further (CoSearch hits avg F1 0.568 with a 7B agent, +6.6% relative over Search-R1)[15].

The architectural takeaway is bigger than the benchmark. Once retrieval is a tool call, it’s a learnable policy. You can fine-tune the agent to be a better retriever using the same RL machinery that gave us reasoning models. You cannot do this to a frozen embedding model.

8. The Spectrum: Five Flavors of Agent-as-Retriever in 2026

By 2026, the pattern has fragmented into at least five distinct architectural flavors. Each has a poster product.

8.1 Pure Agentic (Claude Code, Devin)

No persistent index. Glob, Grep, Read, Bash, Explore subagent. Devin (Cognition) follows the same pattern. The bet: an LLM driving ripgrep in a loop beats any frozen embedding model on a codebase that changes every commit.

8.2 Hybrid Lexical + Semantic (Cursor, Sourcegraph Amp)

Cursor’s docs describe both modes side by side[16]: Instant Grep for exact symbols, semantic search for conceptual queries, agent picks based on query shape. Cursor’s cited internal research: +12.5% accuracy from combining semantic with grep. Sourcegraph’s Amp (the agentic evolution of Cody, 2025–2026) layers a similar agent atop Sourcegraph’s longstanding code graph[17]. Hybrid is the path most enterprise tools converge on.

8.3 Structural / AST-Aware (Cline, Probe, ast-grep)

Pure grep is lexical. Pure embeddings are semantic. There’s a third path: structural search. Tools like ast-grep and Probe parse code with tree-sitter and let agents search for syntactic patterns, not strings.

Cline’s open-source approach[18] is the cleanest production example: a three-tier retrieval stack.

Tier 1: ripgrep-based content search with output caps.
Tier 2: fuzzy file/folder search via fzf with custom scoring.
Tier 3: tree-sitter AST extraction for multi-language definition discovery.

The agent orchestrates all three through a plan-and-act loop, weighting currently open files higher. The Cline paper reports the system maintains structural code awareness while keeping retrieval token utilization around 17.5% per turn, far below pure-agentic baselines.

Probe takes the same idea and ships it as a single binary: “One Probe call captures what takes other tools 10+ agentic loops, deeper, cleaner, and far less noise.”[19] It returns complete functions, classes, or structs rather than text chunks that break mid-function. Critically, it argues that embeddings exist to solve a vocabulary-mismatch problem that LLM-driven agents already handle: the agent translates user intent into precise boolean queries (AND, OR, +required, -excluded, "exact phrases", ext:rs, lang:python), and Probe returns complete AST blocks in milliseconds. No embedding model, no index, no setup.

*ast-grep matches syntactic patterns instead of strings: tree-sitter parses the file into an AST, and the agent searches and rewrites by structure. The “find every* `fetch().then(...)` *and convert to* `await`*" class of query that grep can't reliably express. Source:* *ast-grep.github.io*.

8.4 Specialized Retrieval Models (Windsurf SWE-grep, Chroma Context-1)

Train a small model that’s specifically a retrieval agent. Windsurf’s Wave 13 release in early 2026 shipped SWE-grep and SWE-grep-mini for fast context gathering inside Cascade[8]. They run 8 parallel tool calls per turn across 4 turns, hitting 10× faster retrieval than generic agentic search. Chroma’s Context-1, a 20B agentic-search model released March 2026[9], hits ~10× faster inference and ~25× lower cost than general-purpose frontier models on the same multi-hop tasks.

8.5 RL-Trained Retrieval Policies (Search-R1, CoSearch, Agentic-RAG-R1)

Section 7 above. The agent learns when and what to retrieve via reinforcement learning, beating prompted-agent baselines by double digits. This is the cleanest theoretical case for why agent-as-retriever wins long-term: it’s a learnable system, not a fixed pipeline.

The five flavors share an architectural premise: the agent owns retrieval. They differ only in what’s behind the tools the agent calls and how those tools are produced.

9. The Multi-Agent Multiplier: Anthropic’s Own Proof Point

The clearest non-coding validation comes from Anthropic’s own engineering team. In their post “How we built our multi-agent research system”[4], they describe an orchestrator-worker architecture for the Research feature in claude.ai:

Anthropic’s multi-agent research system: a lead agent (Claude Opus 4) decomposes the query and spawns 3–5 subagents (Claude Sonnet 4) in parallel. Each subagent runs its own agentic-search loop with 3+ tools per turn; only condensed findings return to the lead. The full system outperformed single-agent Opus 4 by ***90.2%*** *on Anthropic’s internal research eval. Source:* *Anthropic Engineering*.

A Lead Researcher (Claude Opus 4) analyzes the query, builds a plan, and spawns 3–5 subagents.
Each subagent (Claude Sonnet 4) runs its own agentic search loop in parallel, calling 3+ tools per turn.
Subagents return only condensed findings; their full tool-call traces stay quarantined.
The lead synthesizes, optionally spawning another wave.
The lead persists its plan to Memory so context survives the 200K-token limit.

The headline result: the multi-agent system beat single-agent Claude Opus 4 by 90.2% on Anthropic’s internal research eval. Parallel tool calling “cut research time by up to 90%” on complex queries. The cost: ~15× more tokens than standard chat. Anthropic explicitly notes that token usage alone explains about 80% of performance variance on browsing-heavy evals, when the task value justifies the spend, throwing more agentic-search tokens at a problem buys you better answers almost linearly.

An important caveat from the same post: multi-agent works for breadth-first tasks (research, “find all the X across many sources”) and is less effective for tightly interdependent tasks like coding, where serial dependencies dominate.

10. Tool Design Principles: The Reason This Works

Agent-as-retriever lives or dies on tool design. Anthropic’s context-engineering post[2] lays out the principles bluntly:

Tools should be “self-contained, robust to error, and extremely clear with respect to their intended use.”
Input parameters must be “descriptive, unambiguous, and play to the inherent strengths of the model.”
Avoid bloated tool sets with overlapping functionality. Each tool should have a single, clear purpose.
The acid test: “If a human engineer can’t definitively say which tool should be used in a given situation, an AI agent can’t be expected to do better.”

This is why the Claude Code tool surface is so small. Glob does one thing. Grep does one thing. Read does one thing. Bash does everything else, with explicit permission gates. The model never has to choose between find_file_by_name, search_file_by_path, and locate_file. There's only one tool for each shape of question.

11. MCP: The Standardization Layer

Model Context Protocol’s core idea: turn every data source, filesystems, databases, observability, ticketing, SaaS apps, into a uniform set of tools and resources the agent can call. The agent becomes the retriever; MCP makes “what to retrieve” interoperable across hosts. Source: *Anthropic, Introducing the Model Context Protocol*.

If agent-as-retriever is the pattern, Model Context Protocol (MCP) is the protocol that turns it from a Claude Code idiosyncrasy into an ecosystem default[20].

MCP, introduced by Anthropic in November 2024 and broadly adopted across IDEs and AI applications through 2025–2026, is a JSON-RPC 2.0 protocol that lets any LLM-driven host (Claude Code, Cursor, VS Code, Claude Desktop) connect to any “MCP server”, a program that exposes tools, resources, and prompts to the host.

MCP Host (AI app)         MCP Servers
┌──────────────────┐      ┌─────────────────────┐
│  MCP Client 1    │◄────►│ Filesystem (local)  │
│  MCP Client 2    │◄────►│ Database (local)    │
│  MCP Client 3    │◄────►│ Sentry (remote)     │
│  MCP Client 4    │◄────►│ Postgres (remote)   │
└──────────────────┘      └─────────────────────┘
     (Claude Code,           (each server exposes
      Cursor, etc.)           tools + resources)

The official MCP filesystem server is the cleanest production example of agent-as-retriever in action. It exposes a curated tool surface, read_file, write_file, list_directory, search_files, get_file_info, under an explicit allowlist of directories. The agent, not the protocol, decides which to call.

The implication: any MCP-aware host becomes an agent-as-retriever system over anything with a filesystem-ish shape. Sentry exposes incidents. Postgres exposes tables. Filesystem servers expose repos. The agent treats them identically: discover, search, read, refine.

Once retrieval is a tool call, every data source becomes a candidate retriever, without anyone building a vector index for it.

12. The Counter-Argument: When Agent-as-Retriever Is the Wrong Tool

The pattern is not a free lunch. Real critiques worth taking seriously:

12.1 Token Cost

Anthropic’s own data is candid: their multi-agent research system uses 15× more tokens than chat[4]. The Milvus team published a pointed critique titled “Why I’m Against Claude Code’s Grep-Only Retrieval” arguing the iterative grep loop costs dramatically more per query than a precomputed lookup[21]. Industry estimates: 5–30× more tokens per task than chatbot interactions, complex agent loops at $0.02-$0.10 per query vs. fractions of a cent for vanilla RAG. Prompt caching and tool lazy loading close much of the gap, but not all.

12.2 Latency

5–10 tool calls per query takes seconds, not milliseconds. For interactive coding this is fine. For sub-second user-facing chat, it isn’t. SWE-grep and Context-1 exist precisely to push that latency down.

12.3 Massive Corpora

Grep over a 10-million-file monorepo is not free. ripgrep and parallel traversal help, the Explore subagent pattern lets you fan out, but at petabyte scale a precomputed index still wins. The hybrid answer: use agentic search inside a smaller, agent-selected slice of a larger indexed corpus.

12.4 Genuinely Semantic Queries

“What does this codebase say about retry policies?” is harder for grep than embeddings, the answer might be spread across files that never use the word “retry” (think backoff, requeue, circuit_breaker). Agent-as-retriever's answer is to issue multiple queries and synthesize. Probe and AST-aware tools answer this by understanding structure. The hybrid answer (Cursor) keeps a light semantic layer for the synonyms.

12.5 Tightly Interdependent Tasks

Anthropic’s own caveat: multi-agent research helps breadth-first questions but hurts deeply serial tasks like coding[4]. The shape of the work matters.

12.6 Determinism and Caching

A vector lookup is deterministic and cheap to cache. An agent loop isn’t. Production teams are converging on RAGAS, BenchmarkQED, and SWE-bench Verified as the eval scaffolding catches up, but evaluation, regression testing, and SLA enforcement are harder than for static retrievers.

13. Comparing the Architectures

Vector RAG vs. Agentic RAG vs. Agent-as-Retriever vs. Hybrid (Cursor-style) vs. Structural (Cline/Probe), property by property:

Best for. Vector RAG: stable FAQ over text. Agentic RAG: multi-hop QA. Agent-as-Retriever: code, evolving corpora, exact-match queries. Hybrid: large codebases with mixed query types. Structural: precise, structure-aware code queries.
Pre-built index? Vector: yes (vectors). Agentic RAG: yes. Agent-as-Retriever: no. Hybrid: small one. Structural: no (AST parsed on demand).
Freshness. Vector: stale until re-embed. Agentic RAG: same. Agent-as-Retriever: real-time. Hybrid: hybrid. Structural: real-time.
Security surface. Vector: index lives somewhere. Agentic RAG: same. Agent-as-Retriever: filesystem ACLs only. Hybrid: smaller surface. Structural: filesystem ACLs only.
Per-query cost. Vector: ~$0.001. Agentic RAG: $0.02–0.10. Agent-as-Retriever: $0.01–0.05. Hybrid: middle. Structural: low (one structured call replaces many grep calls).
Latency. Vector: ~100ms. Agentic RAG: 2–10s. Agent-as-Retriever: 1–10s. Hybrid: 1–5s. Structural: 100ms-1s.
Accuracy on code (SWE-bench class). Vector: 1.96% baseline. Agentic RAG: ~12–30%. Agent-as-Retriever: 50–80%+. Hybrid: similar or higher. Structural: high (returns complete units).
Robustness to drift. Vector: poor. Agentic RAG: poor. Agent-as-Retriever: excellent. Hybrid: good. Structural: excellent.
Semantic-synonym handling. Vector: good. Agentic RAG: good. Agent-as-Retriever: weak (mitigated by iteration). Hybrid: good. Structural: weak unless agent reasons.
Scale ceiling. Vector: billions of docs. Agentic RAG: same. Agent-as-Retriever: millions of files comfortably. Hybrid: best of both. Structural: hundreds of thousands of files.
Trainable? Vector: only the embedding model. Agentic RAG: the agent (Self-RAG, A-RAG). Agent-as-Retriever: yes (Search-R1, CoSearch). Hybrid: yes. Structural: query DSL, not learned.

14. When to Use Which (a 2026 Decision Framework)

What production teams are actually shipping in 2026:

Code over private repos → Agent-as-retriever (pure or hybrid). Claude Code, Devin, Cursor have all converged here.
Large enterprise monorepo with cross-service queries → Hybrid (Cursor, Sourcegraph Amp) or structural (Probe).
Evolving corpora (logs, dashboards, CRM, ticket trackers) → Agent-as-retriever via MCP servers. Freshness > peak recall.
Stable factual knowledge bases (product docs, FAQ, glossaries) → Vector RAG with a reranker. Don’t over-engineer.
Long, layout-heavy PDFs (financial filings, scientific papers, contracts) → ColPali / ColQwen2 (visual retrieval) or agent-as-retriever with pdfgrep, both work.
Global “themes across the corpus” questions → LazyGraphRAG. Agent-as-retriever struggles when no single query surfaces the answer.
Multi-hop reasoning + dynamic search → Search-R1-style RL-trained retrieval policies, or an agentic RAG framework.
Latency-critical chat → Specialized retrieval model (SWE-grep, Context-1) or a hybrid with vector fallback.
Breadth-first research → Multi-agent + agentic search. 90.2% lift, 15× tokens. Math: when task value > token cost, multi-agent wins.
Tightly serial tasks (coding a single feature end-to-end) → Single agent with strong context engineering. Don’t pay multi-agent overhead.
Strict data-residency / compliance environments → Agent-as-retriever is structurally easier to approve, no external index, no embeddings leaving the host.

15. The Road Ahead: Where the Pattern Is Going

What to watch through the rest of 2026 and into 2027:

RL-trained retrieval becomes standard. Search-R1 will look like the first generation. Expect every frontier reasoning model in 2026 to ship with native, RL-trained search/tool-call ability.
Specialized retrieval models proliferate. SWE-grep, Context-1, plus several more in the next 12 months. A 1–3B parameter “retrieval-LLM” class is forming.
MCP everywhere. By late 2026, “expose your data via MCP” will be a more common architectural decision than “build a vector index.”
Structural search converges with agentic. Probe-style AST tools become first-class MCP servers; agents call them instead of stringing together grep + read.
Hybrid as the default. The “pure agentic” vs “pure vector” debate loses to “small vector layer + lots of tools”, Cursor’s path. Hybrid is strictly more expressive.
Agent-as-retriever for the agent’s own memory. Letta, MemGPT, and Mem0 already treat memory as a retrieval target. With MCP, an agent’s history becomes another MCP server.
Better caching. Prompt caching closed a big chunk of the token-cost gap. Expect KV-cache reuse across agent turns to push it further.
Evaluation catches up. RAGAS, BenchmarkQED, GraphRAG-Bench, SWE-bench Verified. 2026 is the year teams stop arguing about which retrieval style is “better” and start measuring on their own data.

16. Conclusion

For three years, the dominant assumption in RAG was that retrieval is a system upstream of the LLM, chunks, embeddings, top-k, prompt. The agent-as-retriever pattern, and its formalization as just-in-time context loading, invert that: retrieval is a behavior of the LLM, expressed through tool calls. The agent decides what to look for, when to look again, when to stop, and how to combine what it finds. The “retriever” is whichever shell command, MCP server, AST query, or RL-trained search policy fit the situation best.

The evidence isn’t speculative. Claude Code shipped it and watched it outperform RAG. Anthropic’s own multi-agent research system showed a 90.2% lift over a single Opus 4 using the same pattern. Cursor’s response was to hire the people who built it. Amazon measured it at 94.5% of RAG’s faithfulness with zero vector database. Search-R1 trained the policy with RL and beat RAG by 24% relative. Windsurf and Chroma trained specialized retrieval models. The pattern is now copied, benchmarked, productized, standardized via MCP, and trained end-to-end.

None of this means vector search is dead. It means vector search is no longer the default. In 2026, the default is: give the agent the tools, design them carefully, let it retrieve just-in-time. Add embeddings where the workload genuinely demands them, semantic generalization, very large stable corpora, sub-second chat. Skip them otherwise.

The teams that internalize this shift first will run leaner stacks, cleaner deployments, faster-iterating products. The teams that don’t will spend 2026 maintaining vector indexes for problems that didn’t need them.

References & Further Reading

Latent Space podcast, Claude Code: Anthropic’s Agent in Your Terminal (Boris Cherny & Cat Wu interview, May 2025).
Anthropic, Effective context engineering for AI agents, defines just-in-time context loading and tool design principles.
Anthropic / Claude, Building Agents with the Claude Agent SDK, official “start with agentic search” guidance.
Anthropic, How we built our multi-agent research system,90.2% lift, 15× tokens, parallel subagent architecture.
Subramanian et al., Keyword Search Is All You Need: Achieving RAG-Level Performance Without Vector Databases Using Agentic Tool Use, arXiv:2602.23368 (AAAI 2026); Amazon Science page.
Jin et al., Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning, arXiv:2503.09516.
Tigerdata, Why Cursor Is About to Ditch Vector Search (and You Should Too), context on the July 2025 Cherny/Wu hires.
Windsurf, Windsurf 2.0: Agent Command Center and Devin in Windsurf; Vibecoding, Windsurf Review (2026): SWE-1.5, Codemaps, Cascade, SWE-grep specs.
MarkTechPost, Chroma Releases Context-1: A 20B Agentic Search Model (March 2026).
SWE-bench, SWE-bench original results, RAG baseline 1.96%, SWE-agent 12.47%.
Morph LLM, Context Engineering: Why More Tokens Makes Agents Worse, Claude Code tool lazy loading (~95% context reduction).
Vadim’s blog, Claude Code Doesn’t Index Your Codebase. Here’s What It Does Instead.
AI Free API, Claude Code Tool Search Explained: Glob, Grep, Read & More (2026), v2.1.117 transition to ugrep + bfs.
Dive into Claude Code: The Design Space of Today’s and Future AI Agent Systems, arXiv:2604.14228,54 built-in tools, five-layer compaction pipeline, 1.6%/98.4% code-split.
CoSearch, CoSearch: Joint Training of Reasoning and Document Ranking via Reinforcement Learning for Agentic Search; Agentic-RAG-R1.
Cursor Docs, Semantic & Agentic Search.
Sourcegraph, Amp (formerly Cody), agentic coding atop the Sourcegraph code graph.
Cline, How code retrieval is implemented in Cline (three-tier: ripgrep + fzf + tree-sitter).
Probe Labs, Probe: AI-friendly semantic code search engine, AST-aware structural search.
Model Context Protocol, Architecture overview, official MCP specification.
Milvus, Why I’m Against Claude Code’s Grep-Only Retrieval, the counter-argument.
Agentic Patterns, Agentic Search Over Vector Embeddings, pattern catalog entry.
MindStudio, Why Cursor, Claude Code, and Devin Use grep, Not Vectors.
Lushbinary, RAG Production Guide 2026, per-query cost figures.
Shuttle, Launching an Army of Haiku 4.5 Agents, Haiku 4.5 cost ratios.

Connect

If you found this useful, follow me on Medium for more deep dives on LLM architecture, retrieval systems, and AI infrastructure. Building something with agentic search, MCP, or vectorless RAG? I’d love to hear what you’re shipping.

Email: abdullahramzan120@gmail.com
GitHub: github.com/buzzgrewal