Can an AI agent discover alpha without fooling itself?

Empirical asset-pricing research has always had a human at the center of it — someone forming a hypothesis, writing the code, running the backtest, and deciding what counts as a real result. This paper describes what happens once that whole loop is handed to an LLM agent, and it is less interested in whether the agent finds a good trading signal than in a harder question: how would you even know if it did?

A new paradigm, and a name for it

The authors call it Agentic Empirical Asset Pricing, or AEAP — systems in which an LLM agent autonomously generates hypotheses, synthesizes them into executable code, and statistically validates the result, end to end. This is not a hypothetical near-future scenario. The paper notes that in the last two years, at least a dozen systems have already implemented some version of a hypothesize-code-backtest-refine loop for mining formulaic factors. What has been missing is a rigorous way to evaluate them.

That gap is the paper's real target. Existing practice backtests the output of a discovery system — the factor or the trade it settled on — and stops there. The authors argue that is the wrong unit of analysis. An agent capable of generating and silently discarding thousands of hypotheses can make a mediocre discovery process look brilliant just by cherry-picking its best result and presenting only that. The backtest of the final factor tells you almost nothing about whether the system that produced it is any good.

Grading the researcher, not just the research

So the paper proposes evaluating the discovery system itself, and contributes three concrete pieces toward doing that: a reference architecture for what an AEAP system's components should look like, a rigorous evaluation standard for judging discovered factors, and — the more novel piece — a method for out-of-sample backtesting of the discovery process. Rather than freezing the system after one run, this rolls it forward through time and asks whether it reliably produces good factors across many independent attempts, not just once.

As a concrete instance of that architecture, the authors evaluate a system called SEADS against five reimplemented baseline discovery systems, all run on two U.S. equity panels. The headline finding is almost anti-climactic, and that is the point: no single metric ranks the systems consistently. One system looks best on statistical significance, another on economic magnitude, another on stability across re-executions — and the rankings shuffle depending on which axis you privilege.

No single metric ranks the discovery systems consistently — evaluation has to happen on multiple axes at once.

Why one number was never going to be enough

That instability is itself informative. It confirms that judging an autonomous discovery system by a single backtested Sharpe ratio is a category error — you are measuring one output of a stochastic process and mistaking it for a description of the process. The separate rolling re-execution test asks the complementary question directly: is the discovery process itself reliable, producing usefully similar quality of factor run after run, or did it get lucky once? A system that occasionally stumbles onto a great factor but cannot repeat the trick is a very different thing from one that reliably finds good, if unspectacular, factors — and a single backtest cannot tell the two apart.

Honest caveats

This is explicitly foundational, methodological work, and the authors are unusually candid about it: alongside the architecture and evaluation standard, they report negative findings and open limitations that surface further evaluation pitfalls for anyone building the next AEAP system. The evaluation is confined to two U.S. equity panels and one concrete implementation (SEADS) compared against reimplemented, not necessarily state-of-the-art, baselines. This is a preprint, and the paper is explicit that it is laying foundations rather than delivering a finished, universally applicable benchmark.

Why it matters

Once agents can generate and test factors far faster than any human team, the standard multiple-testing problem in quantitative finance — the risk that enough attempts will eventually produce a result that looks significant by chance — gets dramatically larger, not smaller. A discovery system that quietly runs ten thousand backtests and shows you the best five is a well-known way to manufacture an illusion of alpha, and it is exactly the failure mode this paper is built to catch. The practical implication for anyone building or buying an autonomous research platform is a shift in what to ask for: not "show me the backtest of the factor you found," but "show me the discovery process rerun ten times, and show me how you know it isn't lucky." That is a much harder thing to fake — and a much more useful thing to have.