Agentic RAG, Graph-R1, MetaRAG and Friends on One Search Engine: What Actually Helps in 2026?#

RAG papers each bring their own retriever, corpus split, generator and metric, so their tables don't compose. The practical question is narrower: if my search engine is fixed, which way of asking it questions works best? This post benchmarks nine retrieval strategies published between 2024 and 2026 - from a single hybrid query to RL-trained search agents and knowledge hypergraphs - with everything except the retrieval strategy held constant: one OpenSearch node, one embedding model, one generator, one judge, and the same 1,000 questions over five datasets.

TL;DR - Summary and Recommendations#

Method Selection Guide#

Key Takeaways#

1. Turning on the generator's reasoning moved accuracy more than any retrieval architecture. On the same 200 multi-hop questions, gpt-5.4-mini with reasoning_effort="medium" lifted naive RAG from 56 % to 78 % and the agentic loops from ~66 % to 85-86 %. Reasoning also made the agents search more (A-RAG: 2.1 → 6.0 searches per question).

2. With reasoning off, nine quite different systems land within a few points of each other. On the multi-hop sets MetaRAG (72.5 %), LightRAG (71.3 %) and plain hybrid search + a cross-encoder (71.2 %) share the top, with the agents (68 %) and one-shot hybrid search (67 %) right behind - differences mostly inside the confidence intervals.

3. The cheapest strong system is boring: OpenSearch hybrid query (BM25 + k-NN, RRF) → local bge-reranker-v2-m3 → one LLM call. $1.26 per 1,000 questions. LightRAG matches it at 14× the per-query cost plus a ~$20 graph build.

4. Graph-R1 is only as good as its RL-trained policy, which is not released. With a prompted policy its think/query loop asks one question and stops (93 % of 2Wiki questions), landing at 43 % on multi-hop vs 67 % for naive RAG.

5. MetaRAG is a reasoning prosthesis: best on 2Wiki with reasoning off (79 %), nearly useless once the model reasons.

6. An RL-trained 7B (Search-R1) on a MacBook reaches 64 % on multi-hop, 4 pp behind gpt-5.4-mini naive RAG on the same questions (tied on MuSiQue), at $0 API cost and 1-2 minutes per question.

7. Check your benchmark first: on GraphRAG-Bench Medical the LLM alone scores 90 %.

Test Environment and Methodology#

Everything except the retrieval strategy is held constant:

• Search engine: one OpenSearch 2.19 node - BM25, Lucene HNSW vectors, the native hybrid query with an RRF search pipeline, phrase search with highlighting
• Embeddings: bge-base-en-v1.5, running locally
• Generator: gpt-5.4-mini (reasoning_effort="none" for the main runs)
• Judge: gpt-4.1-mini - a different model family, using A-RAG's correct/incorrect rubric
• Datasets: the unified datasets released with A-RAG - MuSiQue, HotpotQA, 2WikiMultiHopQA (from the LinearRAG setup), plus GraphRAG-Bench Medical and Novel - each with its own closed corpus, 200 sampled questions each

Where a paper's contribution is how to ask for data - agentic RAG, A-RAG, MetaRAG, Search-R1 - its tools are backed by the same OpenSearch indexes. Only the graph methods build a second index, because that is their idea.

The Contenders#

AgenticRAG-R1, which I wanted to include, turned out to be a GRPO training framework without a released checkpoint; Search-R1 represents that family because its weights are public.

Results (Reasoning Off)#

LLM-judged accuracy (%) per dataset (the heatmap also includes graph_r1_fc, the Graph-R1 function-calling ablation from Finding 3). "Multi-hop avg" averages MuSiQue, HotpotQA and 2Wiki.

95 % bootstrap CIs are roughly ±6-7 pp per cell (the full table with CIs is in the repository README).

Finding 1: The Generator's Reasoning Is the Biggest Lever#

Config: six methods, first 100 MuSiQue + 100 2Wiki questions, reasoning_effort "none" vs "medium"

The main runs use reasoning_effort="none" for every method: gpt-5.4-mini refuses function tools combined with reasoning on Chat Completions, and one setting keeps the methods comparable. This experiment re-runs six methods with "medium" through the Responses API.

Analysis#

The agents stop making answer-type mistakes. With reasoning off, asked "Which film has the director who died earlier, X or Y?", they often search correctly and then answer with the director's name.

They explore more. A-RAG triples its searches. A simple one-tool agent loop goes from 70 % to 94 % on 2Wiki.

Reasoning is not free. A-RAG goes from $7 to $20 per 1,000 questions. Naive RAG with reasoning ($3.4) is the cost-efficient point if you can live with 78 %.

Finding 2: MetaRAG Is a Reasoning Prosthesis#

MetaRAG checks its own answers: an independent "expert" answers too; if the two disagree, a critic classifies the error ("the question asks whose residence, not which building") and a planner rewrites the query.

With reasoning off, this is the best method on 2Wiki (79 % vs 65.5 % naive), at 4-6 LLM calls per question. With reasoning on, it adds almost nothing (68.5 → 74 %) and ends below naive RAG with reasoning (78 %): the model now does that checking itself.

Finding 3: Graphs - LightRAG Is Solid but Heavy, Graph-R1 Needs Its Policy#

LightRAG is in the top cluster (71.3 % multi-hop) and has the best context recall of any method (82.5 % of short answers appear in its context), but its default context is ~23k tokens per question: $17.4 per 1,000 questions, 14× naive RAG, plus ~$20 of gpt-4o-mini to build graphs for the five corpora. A reranker gets the same accuracy for the price of naive RAG.

Graph-R1 builds a hypergraph of "knowledge segments" (n-ary facts as sentences, $8.4 for all five corpora) and lets an agent query it. Its authors train that agent with GRPO; without the weights the policy here is the prompted generator, and it collapses: one query for 93 % of 2Wiki questions, 42.7 % on multi-hop.

An ablation gives the same hypergraph to the function-calling loop of agentic_rag: 3× the queries and +17 pp (59.7 %), still below plain passage retrieval, because rewritten facts drop details that raw passages keep (context recall 59 % vs 65-83 %). Reasoning helps it most of all (+28 pp), but even then it issues about one query. Graph-R1's index is only as good as the policy that explores it, which is exactly what its own RL ablation says.

Finding 4: An RL-Trained 7B on a Laptop#

Config: Search-R1 (Qwen2.5-7B trained with PPO on NQ + HotpotQA), Q4_K_M GGUF on llama.cpp / Metal, same OpenSearch hybrid retriever through its <search> tags, top-3 passages per search, greedy decoding

Analysis#

It searches ~3 times per question (2.6-3.2) and its contexts contain the answer 73.7 % of the time, the same as the frontier-model agentic_rag loop (73.5 %): the RL-trained policy decides when and what to search as well as a much larger prompted model; it loses on reading.

HotpotQA, which it was trained on, is its best dataset. It is the only method with zero API cost, and the slowest: 70-100 s median per question on an M3 that was also running other jobs.

Finding 5: Benchmark Hygiene#

• Medical (GraphRAG-Bench) is saturated: closed book 90 %, every method 84-92 %. It measures medical knowledge, not retrieval.
• MuSiQue is the discriminating dataset: closed book 14 %, best non-reasoning method 58 %, best reasoning one 81 %.
• The LLM judge is lenient on near misses ("early 1980s" accepted for "1981"). EM / F1 / Contain are reported next to it for the short-answer sets; they reorder methods inside the top cluster but agree on who is in it (reranked hybrid, MetaRAG, LightRAG) and on the bottom (closed book, prompted Graph-R1).

Engineering Notes (Things That Cost Hours)#

1. gpt-5.4-mini + function tools + reasoning is rejected on /v1/chat/completions for any reasoning_effort other than "none". Use /v1/responses and replay the encrypted reasoning items between tool turns (include=["reasoning.encrypted_content"], store=False).

2. Embedding speed on Apple Silicon. Qwen3-Embedding-0.6B (A-RAG's choice) embedded ~5 passages/s on an M3 via MPS; bge-base-en-v1.5 ~50/s.

3. LightRAG indexing slows quadratically with its default file storages, because it commits every store after every document. Insert many corpus chunks per LightRAG document, micro-batch its embedding calls, and check for FAILED documents afterwards: they are not retried, and an interrupted build can leave one that cannot be repaired.

4. OpenSearch's hybrid query + score-ranker-processor (RRF) pipeline is a few lines of JSON and was the most valuable single component in the study.

Limitations#

• One generator family and one run per configuration; at n = 200, differences under ~5 pp are noise.
• Graph-R1 and AgenticRAG-R1 are evaluated without their RL-trained policies (none released).
• Search-R1 runs a 4-bit checkpoint on 100 questions per multi-hop dataset, with this benchmark's retriever instead of the E5 + Wikipedia setup it was trained against, and is not evaluated on the long-answer sets.
• MetaRAG's T5 helper models are replaced by gpt-5.4-nano; LightRAG's MuSiQue graph misses 4.7 % of the corpus.

Reproducing These Results#

All method implementations, run scripts, raw per-question results and figures are available in the accompanying repository. Total API spend for everything in this post: ≈ $80.

# 1. search engine
docker compose up -d                       # OpenSearch on :9200
 
# 2. python env + data
uv sync
python scripts/download_data.py
export OPENAI_API_KEY=...
 
# 3. indexes (embeddings run locally; graph builds call gpt-4o-mini)
uv run ragbench index
uv run ragbench index --method graph_r1
uv run ragbench index --method lightrag
 
# 4. run + report (search_r1 needs a local llama-server, see the README)
uv run ragbench run --methods closed_book naive_rag rerank_rag metarag agentic_rag a_rag graph_r1 lightrag search_r1 -n 200
uv run ragbench report                     # -> results/summary.md, results/*.csv, results/figures/*.png

Runs are resumable (one JSONL line per question in results/runs/<dataset>/<method>.jsonl) and every LLM response is cached, so re-running costs nothing. The raw JSONL files contain the prediction, every metric, token usage and the full retrieval trace for each question - far more detail than the tables here.

If you run into questions or issues while reproducing these results, please open an issue on GitHub.