How does ForecastBench work?

ForecastBench is a dynamic, continuously-updated benchmark designed to measure the accuracy of ML systems on a constantly changing set of forecasting questions.

We evaluate LLMs by regularly asking them to make probabilistic forecasts about future events, thereby creating a contamination-free benchmark.

We use two types of binary prediction questions:

Dataset questions are automatically generated from real-world time series (ACLED , DBnomics , FRED , Yahoo! Finance , and Wikipedia ) using pre-specified templates. Each dataset question generates multiple forecasts at different time horizons, since we ask the same question with 8 different resolution dates, ranging from 7 days to 10 years out.
Market questions are drawn from leading prediction platforms: Manifold , Metaculus , Polymarket , and Rand Forecasting Initiative .

ForecastBench operates as a fully automated, dynamic system. New forecasting rounds occur every two weeks, with each round generating 500 questions split evenly between market and dataset questions. The leaderboard is updated nightly as new data becomes available and market questions resolve over time, allowing us to continuously track forecasting performance.

To construct the performance ranking, we evaluate forecasters separately on market questions and dataset questions. The overall ranking combines these scores, equally weighting performance by question type. As a result, the overall ranking provides a comprehensive assessment of forecasting ability across both structured time-series data (dataset questions) and real-world events (market questions).

Blog

For a high-level overview of ForecastBench, including motivation, key design decisions, and early results, see our introductory blog post on the Forecasting Research Institute Substack .

Team

ForecastBench is developed and maintained by the Forecasting Research Institute , a nonprofit research organization dedicated to advancing the science, practice, and use of forecasting. The ForecastBench team is committed to open science and we publicly provide our code, datasets (where licensing permits), and methodology to support reproducible research. For correspondence, please contact forecastbench@forecastingresearch.org.