Project Results

I am excited to share this report we just published on the results from our pilot study and their implications.

Reviewers in our study were new to the expected utility forecasting process and gave generally positive reactions. In their feedback, reviewers said that they appreciated how the framing of the questions prompted them to think about the proposals in a different way and pushed them to ground their assessments with quantitative forecasts. The focus on just three review criteria–probability of success, scientific impact, and social impact–was seen as a strength because it simplified the process, disentangled feasibility from impact, and eliminated biased metrics. Overall, reviewers found this new approach interesting and worth investigating further. We encourage grantmakers, both public and private, and policymakers to reach out to our team to learn more and receive assistance in implementing this approach.

About This Project

Scientific grant review consists of experts assessing the approach and impact of a given proposal. The process is thus an implicit exercise in forecasting – reviewers are betting on which projects are likely to accomplish their goals and produce valuable outcomes.

Explicit and grounded forecasts could produce better funding decisions, but this approach has yet to be widely tested. We aim to launch a pilot forecasting challenge to build understanding of forecasting in scientific review.

Ask the Scientists

What is the context of this research?

Merit review of scientific proposals is ubiquitous across the federal R&D enterprise. But the review process is arguably too conservative, focusing on feasibility over potential impact and limiting the uptake of new research models. Focused research organizations (FROs) – standalone, time-bound, non-profit organizations designed to tackle projects not incentivized by academia or industry – are a relatively untested model with high potential value. FROs aim to produce public goods: processes, tools, and datasets that unlock new methods and accelerate the pace of research.

Federal funders are tentatively curious about this model, but are often wary of (a) the likelihood of proposals achieving their goals and (b) the value of funding one large project rather than several small projects.

What is the significance of this project?

Forecasting can be a valuable tool for assessing the risk-reward profile of novel research models. Because of their bounded time span and technical milestones, FROs are particularly well-suited to forecasting.

The intersection of FROs and forecasting has the potential to yield more than the sum of its parts. By piloting this approach to review, we can (a) generate quantitative predictions of the feasibility and potential impact of real proposals, which funders can use to inform decision-making; (b) provide actionable insight to proposal authors regarding their approach; and (c) obtain metascientific understanding of how forecasting does – or does not – work in a scientific review context.

Follow-up research could entail a formal study comparing forecasting to traditional peer review.

What are the goals of the project?

FAS and Metaculus will run a pilot forecasting tournament on five high-quality FRO proposals sourced by partners at Convergent Research. The forecasting tournament will ask participants to make quantitative forecasts in response to three questions on the probability of success, the social impact, and the scientific impact, of achieving the technical milestones in each proposal. We will recruit and compensate subject matter experts and professional forecasters to participate in the tournament, which will be hosted on Metaculus' platform. After the tournament, we will use the forecasts to calculate the expected value of achieving each technical milestone and categorize the proposals based on their risk-reward profile. The results will be published in a public report.