LitReview: literature reviews where every claim points to a checked quote
A literature review agent whose citations are verified against the papers' full text.
- 2.6×the recall of Asta Paper Finder on RealScholarQuery
- 0fabricated citations across evaluation runs
- 89%test coverage across 265 tests
Overview
Describe a research question in plain words. LitReview turns it into weighted criteria you approve, searches Semantic Scholar, OpenAlex and arXiv, screens every candidate, reads the full text of the strongest papers and writes the review. Every citation points to a verbatim quote that code has found in the paper.
Researchers miss papers because the same idea goes by different names, and general-purpose AI tools invent references. LitReview is a fixed LangGraph pipeline with two stops for a person: approve the criteria, with a time and cost estimate, before any money is spent on search, and approve the paper set before anything is written.
Agents only work where their output can be checked. A full-text reader quotes evidence for each criterion; code confirms each quote exists verbatim in the paper and downgrades the verdict when it doesn't, and the final paper set is chosen by code from those verdicts. A second stage audits a paper's GitHub repository for reproducibility (missing files, dead data links, unpinned dependencies) with file and line evidence, without writing or running the code.
I lead the team: I set the architecture, own the search pipeline, and redesigned the system for production on AWS: FastAPI with idempotent paid endpoints, a SQLite run queue and a separate worker, stop and resume from checkpoints, per-account credits and OpenTelemetry traces.
Recall against human-labelled answers: 0.326 on RealScholarQuery and 0.197 on SPARBench, against 0.127 and 0.134 for the Asta Paper Finder baseline. Zero fabricated citations in every run.
Highlights
- Search fans out into nine aspect queries across Semantic Scholar, OpenAlex and arXiv, expands one hop through citations, and runs a second round written by an exploration agent from what is still missing.
- 92.6–95.1% of the reader's quotes match the paper verbatim; the rest are caught by code and never used as evidence.
- Report sentences fully supported by a quote rose from 0.222 to 0.853 on the RealScholarQuery dev set after the support check.
- 265 tests at 89% coverage. Paid requests carry an Idempotency-Key, so a retry never pays twice.