74 of 100 Research Papers Became Working Agents. The Tools Came From the Tutorials, Not the Code
By Surya Pratap
September 27, 2026
9 min read
AI & Technology
September 27, 2026
9 min read
Built from what runsHover to exploreMost attempts to make a codebase usable by an agent start with the code: point the model at the repository and let it work out what to call. A paper published in Nature this month does something narrower, and its results suggest the narrower approach is the one to copy.
Paper2Agent was published in Nature on 16 September 2026 by Jiacheng Miao, Joe R. Davis, Yaohui Zhang, Jonathan K. Pritchard and James Zou. The Nature article is behind a paywall, so the published figures here come from two summaries by AI Weekly that agree on the numbers I use. The pipeline details and the baseline comparison come from the authors' 2025 preprint, which is an earlier version of the work, and I say which is which. The code is open source. The reading from section 3 onward is mine.
Paper2Agent takes a research paper and its code and produces a Model Context Protocol (MCP) server: a set of tools any MCP-capable assistant can call, each one applying a method from the paper to new data.
The preprint describes six steps: find the codebase, set up its environment, find its tutorials, run them, turn each worked example into a tool, and assemble the tools into a server. Four sub-agents split the work: one manages the environment, one scans for tutorials, one turns tutorials into functions, and one writes and runs tests.
The part that matters is the testing. Each extracted tool is tested against the tutorial's own output, in a loop of writing tests, running them, diagnosing failures and fixing. The preprint's bar is that a tool must reproduce the original results. A tool that keeps failing is left out of the server.
It does not wrap everything in the repository. It wraps what demonstrably works, and drops the rest.
From the Nature version, as summarised:
The preprint, published a year earlier, compared an AlphaGenome agent with two alternatives on 15 tutorial queries and 15 new ones. The paper agent answered all 30 correctly. Claude pointed directly at the repository answered 9 of 15 tutorial queries and 12 of 15 new ones. Biomni, a general biomedical agent, answered 6 of 15 and 9 of 15. The paper agent was also faster, by between 1.8 and 4.6 times depending on the comparison.
Read the baseline with its sample size
Fifteen queries per condition is small, and it comes from the preprint, not the published version. The published figures for the same case study are slightly different. I use the comparison for its direction — a tested tool layer beat raw repository access — not for its exact size.
An agent pointed at a raw repository has to work out, on every query, which function matters, which arguments are valid, what order things run in, and which of the three similar-looking helpers is the one the authors actually used. It can be good at this and still be wrong often, because the repository does not say which paths are real.
A tutorial does. It is the author's own statement of this is how the method is meant to be used, with inputs, the call sequence and the expected output. Turning tutorials into tools gives the agent a small set of operations that already have a known-good answer. Testing each tool against that answer turns a guess into something checked.
This is the same pattern that makes small, well-defined tools beat large, general ones in production agents: fewer choices, each one verified. What Paper2Agent adds is a way to produce that layer automatically, from material most projects already have.
The summaries do not explain the failures, and I will not guess at them paper by paper. The preprint's own limitations section is general but clear: papers with incomplete, poorly documented or error-prone codebases cannot be reliably converted.
Put that next to how the pipeline works and it describes a test your own product would face. If your examples do not run from a clean environment, there is nothing to extract. If they run but nobody wrote down the expected output, there is nothing to test against. If the only documentation is an API reference, the agent is back to guessing which calls matter.
So the practical question for a founder is not "should we build an MCP server?" but:
If someone ran this pipeline on your public repository tomorrow, how many working, tested tools would come out?
Make every tutorial runnable from clean. One command to set up the environment, no hidden state, no "then configure your credentials somehow." A tutorial that only works on the author's laptop is not an example anyone can learn from, human or agent.
Write down the expected output. A worked example without its result is a suggestion. With its result, it is a test. That single change is what lets any automated pipeline — or your own CI — tell a working tool from a plausible one.
Build your MCP server from examples, not from the reference. Exposing every endpoint gives an agent a large surface to guess across. Exposing the ten tasks your tutorials show, each tested, gives it a small surface that works. You can add breadth later.
Test the tools, not just the server. A server that starts is not a server that works. Keep a test per tool that checks it still reproduces its tutorial's output, and run it on every release, because your API will change and the tools will quietly drift.
The cost argument is also worth noting. On the published figures, one case study took about 45 minutes and $14 to convert, and queries cost about $0.20. Whatever your product is, a first version of an agent interface built this way is not a quarter-long project.
The same lesson applies from the other side. If your agent needs to use a library, an internal service or a vendor's SDK, do not hand it the repository and hope. Find the worked examples, turn each one into a narrow tool, test each tool against the example's result, and give the agent only those. The preprint's baseline is a small test, but it points the same way as most production experience: an agent choosing among a few tools that work does better than one choosing among everything that exists.
I have not read the full Nature paper. It is behind a paywall. The published figures come from two summaries by the same outlet, AI Weekly, which agree on every number I use. Coverage is not fully consistent on one further AlphaGenome figure, 82.7%, which one summary gives as accuracy on open-ended researcher questions and another describes as a baseline, so I have left it out. The pipeline description and the baseline comparison come from the 2025 preprint, which is an earlier version and reports slightly different numbers.
This is computational biology. Its tutorials and code follow shared conventions. Whether the same success rate holds for general software, where examples are often less tidy, is untested.
Tutorial-derived queries favour tutorial-derived tools. A tool built from a tutorial will do well on questions drawn from that tutorial. The 300-question benchmark and the preprint's new queries are the better evidence, and even those were written by the researchers.
The failure analysis is missing. Twenty-six of 100 papers did not convert, and the summaries do not say why. My section 4 reading uses the preprint's general limitations, not a breakdown of those 26 papers.
The baseline is small. Fifteen queries per condition, from the preprint. It shows a direction, not a precise gap.
Paper2Agent's headline is that research papers can become agents. For founders, the more useful result is how: not by wrapping a codebase, but by extracting the worked examples that run, testing each against its known output, and discarding the tools that fail.
That turns documentation into something closer to an interface specification. The projects that converted were the ones whose examples could be run and checked. The ones that did not are a fair picture of what happens to a product whose tutorials drifted out of date.
Run your own tutorials from a clean environment this week, and write down what each one is supposed to output. That is most of the work of making your product usable by agents.
Sources: Jiacheng Miao, Joe R. Davis, Yaohui Zhang, Jonathan K. Pritchard and James Zou, "Reimagining research papers as interactive and reliable AI agents", Nature, 16 September 2026 — published figures as summarised by AI Weekly in "Paper2Agent turns research papers into working AI agents" and "Paper2Agent converts research papers into callable AI agents": the 22 tools, 45 minutes and $14, the 98.7 ± 1.3% on tutorial queries, the 74 of 100 papers, 593 tools, 91.2 ± 1.6% on 300 questions, $0.20 and 1.6 minutes per query. The same authors' preprint, "Paper2Agent: Reimagining Research Papers As Interactive and Reliable AI Agents", arXiv:2509.06917, v2 of 16 October 2025 — the six-step pipeline, the four sub-agents, the testing loop and exclusion of failing tools, the 15-plus-15 query comparison with Claude with repository access and Biomni, the run-time ratios, and the stated limitations. Code: Paper2Agent on GitHub, MIT licence. The reading in sections 3 to 6 is mine. For what MCP is changing this year, see MCP was built for a human at a laptop. For why an agent reading everything costs more than you think, see every file your agent reads stays on the bill.
IdeaToMVP Academy
4-week live cohort for founders. Learn to ship AI agents, scope MVPs, and automate your business — taught by the same team that writes these guides.
