strata

Strata

A deep research agent over arXiv abstracts. It answers a question by planning - decompose, retrieve per sub-question, reflect on coverage, search again if needed, then write a report in which every claim carries a citation to a specific retrieved passage, and every citation is validated against the passages that were actually retrieved.

The claim this repo is built to test: a planning loop beats single-shot RAG, and there is an eval harness here that says by how much. SingleShotRAG lives in the same module as the agent, uses the same retriever, the same prompt and the same citation validator. The planning loop is the only variable.

No LangChain, no LlamaIndex, no agent framework. The loop is ~150 lines of strata/agent.py.

question
   |
   +-- plan ------------------> 3-6 sub-questions            1 LLM call
   |
   +-- for each sub-question:
   |      BM25 + dense -> RRF -> rerank -> k passages        0 LLM calls
   |      synthesize -> cited sub-answer + sufficiency       1 LLM call
   |
   +-- reflect ---------------> sufficient? follow-ups?      1 LLM call
   |      |                                                  (loop, bounded)
   |      +--> not sufficient: reformulate, retrieve again
   |
   +-- write report ----------> cited report                 1 LLM call
   |
   +-- validate citations ----> strip and count markers      0 LLM calls
                                that resolve to nothing

Quickstart

python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

# No API key needed: runs the whole loop against the offline fixture client.
python -m strata ask "How is retrieval evaluated without reference summaries?" --offline
pytest -q

# Live, against OpenRouter.
export OPENROUTER_API_KEY=sk-or-...       # or put it in .env
python -m strata ask "How is retrieval evaluated without reference summaries?"
python -m strata ask "..." --baseline     # the single-shot comparison
python -m strata ask "..." --rerank lexical   # auto | lexical | cross-encoder | llm | none
python -m strata eval --limit 5           # benchmark both systems

data/corpus.jsonl (841 abstracts, 1.2 MB) is committed, so the repo works with no network. To rebuild it: python -m strata ingest --per-category 300.

Layout

strata/llm.py          OpenRouter client + OfflineLLMClient, retries, usage
strata/corpus.py       Document/Chunk records, chunker, JSONL storage
strata/ingest.py       arXiv API fetch, cached to data/raw/
strata/retrieval.py    BM25, dense index, RRF, rerankers, Retriever.search
strata/agent.py        StrataAgent, SingleShotRAG, citation validation
strata/trace.py        per-run JSON trace: every call, retrieval, token, dollar
strata/prompts/        plan / synthesize / reflect / report, as files
strata/eval/           questions.jsonl, metrics.py, judge_prompt.txt, run_eval.py
tests/                 57 tests, all pass with no API key
site/                  static results page, no build step

Retrieval

Retriever.search(query, k) -> list[Passage]. Each Passage carries the chunk id, the source document id, the text, a score, and which retrievers surfaced it.

from strata.corpus import load_corpus
from strata.retrieval import Retriever

retriever = Retriever.from_documents(load_corpus())
for passage in retriever.search("reciprocal rank fusion", k=5):
    print(passage.score, passage.sources, passage.doc_id, passage.title)

BM25 is written out in strata/retrieval.py, not imported:

score(q, d) = sum_t IDF(t) * f(t,d)*(k1+1) / (f(t,d) + k1*(1 - b + b*|d|/avgdl))
IDF(t)      = ln(1 + (N - n(t) + 0.5) / (n(t) + 0.5))

k1=1.5 sets term-frequency saturation; b=0.75 sets length normalisation. Postings are numpy arrays, so scoring a query touches only the documents that contain its terms. tests/test_bm25.py recomputes a score by hand from the formula rather than snapshotting the implementation's own output.

Dense defaults to sentence-transformers/all-MiniLM-L6-v2 when that package is installed, and otherwise to a pure-numpy TF-IDF + truncated-SVD (LSA) embedder built from the corpus itself. No embeddings API is called. Vectors are cached to data/embeddings.npy behind a fingerprint of the backend and the corpus text, so a corpus change invalidates the cache instead of silently serving stale vectors. Which backend ran is in retriever.describe() and in every trace.

Fusion is Reciprocal Rank Fusion, RRF(d) = sum_r w_r / (k + rank_r(d)), with k and the per-retriever weights (alpha) exposed on the Retriever. Fusing ranks rather than scores avoids inventing a normalisation between unbounded BM25 sums and bounded cosines.

Rerank is one interface with three implementations: LexicalReranker (default, no dependencies), CrossEncoderReranker (cross-encoder/ms-marco-MiniLM-L-6-v2, used automatically when available) and LLMReranker (pointwise, through OpenRouter, off by default because it costs a call per candidate).

Citation validation

The report's citation markers are not trusted. validate_citations resolves every [P<n>] against the passages that were actually retrieved, strips the ones that do not resolve, leaves the sentence in place uncited, and reports the rate:

cleaned, report = validate_citations(text, mapping)
report.resolved              # ["P1", "P3"]
report.unresolved            # ["P9"]  -> stripped from `cleaned`
report.hallucination_rate    # unresolvable marker occurrences / all occurrences

A fabricated marker is worse than an uncited sentence: it looks checkable and is not. The same function runs in the agent, in the baseline and in the eval, so the number means the same thing in all three.

Bounds

The loop is bounded twice, and both checks happen before a call is made: max_iterations (default 2: the initial plan plus at most one round of follow-ups) and budget_usd (default 0.25 per question). Reflection that cannot be parsed is treated as "sufficient" rather than retried, so a malformed reply cannot spend the budget in a loop. Every run writes runs/<timestamp>.json with each LLM call, each retrieval with its per-retriever ranks, timings, tokens and cost. runs/example-offline-run.json is committed as a worked example.

Evaluation

strata/eval/questions.jsonl holds 30 hand-written questions, each derived from a specific paper in the corpus and labelled with relevant_doc_ids and key_points.

python -m strata eval              # writes results/eval_results.json + RESULTS.md
python -m strata eval --offline --limit 5

metric	what it asks
recall@5, MRR	did retrieval surface the gold paper, and how early
evidence recall	did it surface it anywhere in the run, ignoring rank
cited recall	did the gold paper survive into the report's citations
citation faithfulness	does the cited passage support the claim (LLM judge)
key-point coverage	fraction of labelled key points present in the answer
hallucinated-citation rate	share of markers that resolve to nothing
cost/query, latency	what the planning loop costs to get its gains

Current committed results are from an offline run. Retrieval metrics below are real; every metric downstream of text generation is a stub value and is marked pending until the suite is run with a key.

metric	single-shot RAG	Strata
recall@5	1.000	0.967
MRR	0.940	0.888
evidence recall	1.000	1.000
cited recall	0.967	0.933
citation faithfulness	pending	pending
key-point coverage	pending	pending
hallucinated-citation rate	pending	pending
cost per query	pending	pending
latency per query	pending	pending

The retrieval result is worth stating plainly rather than hiding: on this question set single-shot retrieval is already at ceiling, because each question was written from one paper and carries that paper's distinctive terms. The planning loop cannot improve a recall of 1.000, and its extra sub-question retrievals slightly dilute the ranked list, which is what the lower MRR shows. Any advantage the loop has here has to come from coverage and faithfulness of the written answer, and that is exactly the half of the table that needs a key. A harder question set - multi-paper comparisons, questions that do not quote their source paper - is the honest next step, and is listed in the limitations.

Limitations

• Abstracts, not full texts. Methods sections, tables and appendices are not in the corpus, so any question whose answer lives in a results table is unanswerable here, and the agent will confidently retrieve the abstract that looks closest.
• 841 documents from three categories, skewed to recent submissions. A question outside cs.AI/cs.CL/cs.LG retrieves the nearest neighbours in a corpus that does not contain the answer.
• The eval question set is single-gold: one labelled paper per question, written from that paper. It measures "can retrieval find the source" well and "can the agent synthesise across papers" not at all.
• The LLM judge for faithfulness is itself a language model with the same failure modes it is auditing. It is a signal, not ground truth, and the rubric it applies is in strata/eval/judge_prompt.txt so its bias is at least legible.
• Key-point coverage is lexical overlap at a 0.6 threshold: a correct answer in different words is undercounted.
• English only. No temporal reasoning: "recent" and "state of the art" are resolved against whatever is in the corpus, not against the calendar.
• The default dense backend is LSA, not a trained sentence encoder, unless sentence-transformers is installed. It is weaker on paraphrase.

Configuration

variable	meaning
OPENROUTER_API_KEY	required for live runs; read from the env or .env
STRATA_MODEL	OpenRouter model id, default anthropic/claude-3.5-sonnet

---

Strata, 2026.