Theorizer uses AI to turn papers into theories

Theorizer is an AI system that analyzes dozens of scientific papers and turns them into structured theories with laws, limits, and traceable evidence. The tool, introduced by the Allen Institute for AI, aims to help researchers understand a field in minutes instead of months.

It does not just summarize papers. It tries to detect patterns that recur across multiple studies and express them as claims that can be tested.

From thousands of results to possible laws

In science, experiments produce concrete results, but theories connect them. Kepler’s laws, for example, condensed centuries of observations about planetary motion into a few rules.

Theorizer tries to do something similar with the scientific literature. Given a request such as "create theories about memory in language models," the system looks for relevant research, extracts its findings, and proposes shared regularities.

Each theory is organized into three parts:

• Law: a qualitative or quantitative relationship, such as one variable increasing when another does.
• Scope: the conditions under which the law should hold, including limits and exceptions.
• Evidence: specific findings extracted from papers, with references to the research supporting them.

In practice, each theory usually contains one or two laws. The tool also assigns it a name and description to place it within the field being studied.

How Theorizer works

The process has three stages. First, the system turns the user’s query into a literature search and gathers up to 100 relevant papers. It combines PaperFinder, Semantic Scholar, and a tool that converts PDFs into text, including when they require optical character recognition.

If documents are missing, it reviews the references in the papers it found and adds those that appear most useful. The result is a research corpus that serves as the basis for building the theories.

Next, Theorizer creates an extraction template adapted to the question. For a query about memory in language models, for example, it might record the task type, the memory mechanism used, and the performance difference with and without memory.

Finally, another model combines the data, generates theories, and reviews them. It also estimates whether a law is novel and discards those that are too similar to claims already known.

Theories backed by papers work better

The team compared two methods: one based only on what the model already knew and another that incorporated evidence retrieved from the literature. The second was more expensive, approximately seven times more, but produced more specific, better-supported, and more plausible theories.

To check whether the laws could anticipate later results, the team generated predictions and compared them with research published afterward. The analysis included 2,983 laws, 4,554 papers, and 16,713 evaluations between a law and an article.

In the precision-focused generation, both methods achieved high precision, around 0.88 to 0.90. This means that most of the predictions that could be checked were supported by later studies.

The difference appeared in coverage: theories supported by literature reached a recall of 0.51, compared with 0.45 for the method based only on the model’s knowledge. Recall measures how many claims find later evidence that allows them to be evaluated.

The advantage was greater when the goal was to look for new ideas. In that case, precision rose from 0.34 to 0.61 and recall from 0.04 to 0.16 when the literature was added.

It is not an automatic truth detector

Theorizer still produces hypotheses, not scientific conclusions. Its results can be incomplete or misleading, and the system itself can find research testing approximately 51% of the theories generated in the precision-focused scenario.

There are other important limits:

• Scientific literature tends to publish more positive results than negative ones.
• The system depends on papers being available through open access.
• A query takes approximately 15 to 30 minutes, although several parts can run in parallel.
• It currently works best in areas with extensive digital documentation, such as artificial intelligence and natural language processing.

The tool is released with its code, interface, API, and the prompts used in the process. It also includes a dataset of approximately 3,000 theories, created from 13,744 papers and 100 queries about AI and language topics.

For you, the change is not that AI will replace the researcher who validates a theory. It is that AI can become a first layer of orientation: locating patterns, pointing out connections between studies, and suggesting what deserves more rigorous testing. The next step will be to measure whether those suggestions withstand new experiments and whether they help uncover relationships researchers have not yet formulated.