Updated Jan. 24, 2025
Ranking | Organization | Model | Dataset Score (N=421) | Market Score (resolved) (N=48) | Market Score (unresolved) (N=28) | Market Score (overall) (N=76) | Overall Resolved Score (N=469) | Overall Score (N=497) | Overall Score 95% CI | Pairwise p-value comparing to No. 1 (bootstrapped) | Pct. more accurate than No. 1 | Pct. Imputed |
---|---|---|---|---|---|---|---|---|---|---|---|---|
1 | ForecastBench | Superforecaster median forecast | 0.117 | 0.078 | 0.066 | 0.074 | 0.098 | 0.096 | [0.075, 0.116] | 0% | 0% | |
2 | ForecastBench | Public median forecast | 0.152 | 0.119 | 0.036 | 0.089 | 0.136 | 0.120 | [0.1, 0.141] | <0.001 | 22% | 0% |
3 | Anthropic | Claude-3-5-Sonnet-20240620 (scratchpad with freeze values) | 0.138 | 0.144 | 0.042 | 0.106 | 0.141 | 0.122 | [0.099, 0.145] | <0.001 | 31% | 0% |
4 | Anthropic | Claude-3-5-Sonnet-20240620 (scratchpad with news with freeze values) | 0.142 | 0.148 | 0.049 | 0.112 | 0.145 | 0.127 | [0.104, 0.149] | <0.001 | 29% | 0% |
5 | OpenAI | GPT-4-Turbo-2024-04-09 (zero shot with freeze values) | 0.162 | 0.127 | 0.038 | 0.094 | 0.144 | 0.128 | [0.105, 0.151] | <0.001 | 31% | 0% |
6 | Anthropic | Claude-3-5-Sonnet-20240620 (zero shot with freeze values) | 0.145 | 0.160 | 0.042 | 0.117 | 0.152 | 0.131 | [0.103, 0.159] | <0.001 | 30% | 0% |
7 | OpenAI | GPT-4 (zero shot with freeze values) | 0.168 | 0.127 | 0.041 | 0.096 | 0.147 | 0.132 | [0.109, 0.154] | <0.001 | 30% | 0% |
8 | OpenAI | GPT-4o (scratchpad with news with freeze values) | 0.162 | 0.127 | 0.065 | 0.104 | 0.144 | 0.133 | [0.112, 0.153] | <0.001 | 25% | 0% |
9 | Anthropic | Claude-3-5-Sonnet-20240620 (scratchpad) | 0.138 | 0.177 | 0.056 | 0.132 | 0.157 | 0.135 | [0.113, 0.157] | <0.001 | 28% | 0% |
10 | OpenAI | GPT-4o (scratchpad with freeze values) | 0.161 | 0.144 | 0.057 | 0.112 | 0.153 | 0.136 | [0.115, 0.158] | <0.001 | 27% | 0% |
11 | Anthropic | Claude-3-5-Sonnet-20240620 (scratchpad with news) | 0.142 | 0.171 | 0.077 | 0.136 | 0.156 | 0.139 | [0.117, 0.161] | <0.001 | 26% | 0% |
12 | Anthropic | Claude-3-Opus-20240229 (zero shot with freeze values) | 0.163 | 0.153 | 0.048 | 0.115 | 0.158 | 0.139 | [0.114, 0.164] | <0.001 | 24% | 0% |
13 | Anthropic | Claude-3-5-Sonnet-20240620 (superforecaster with news 3) | 0.158 | 0.158 | 0.060 | 0.122 | 0.158 | 0.140 | [0.119, 0.16] | <0.001 | 25% | 4% |
14 | Anthropic | Claude-3-5-Sonnet-20240620 (superforecaster with news 1) | 0.150 | 0.173 | 0.067 | 0.134 | 0.162 | 0.142 | [0.12, 0.165] | <0.001 | 25% | 0% |
15 | OpenAI | GPT-4o (scratchpad) | 0.161 | 0.156 | 0.069 | 0.124 | 0.159 | 0.142 | [0.125, 0.16] | <0.001 | 25% | 0% |
16 | OpenAI | GPT-4o (scratchpad with news) | 0.162 | 0.159 | 0.066 | 0.125 | 0.160 | 0.143 | [0.122, 0.165] | <0.001 | 22% | 0% |
17 | Anthropic | Claude-3-Opus-20240229 (scratchpad with freeze values) | 0.160 | 0.152 | 0.087 | 0.128 | 0.156 | 0.144 | [0.125, 0.163] | <0.001 | 22% | 0% |
18 | Mistral AI | Mistral-Large-Latest (zero shot with freeze values) | 0.172 | 0.155 | 0.048 | 0.116 | 0.164 | 0.144 | [0.12, 0.168] | <0.001 | 23% | 0% |
19 | Gemini-1.5-Pro (scratchpad with news with freeze values) | 0.163 | 0.166 | 0.065 | 0.129 | 0.164 | 0.146 | [0.126, 0.165] | <0.001 | 23% | 1% | |
20 | Gemini-1.5-Pro (scratchpad) | 0.163 | 0.169 | 0.065 | 0.131 | 0.166 | 0.147 | [0.129, 0.164] | <0.001 | 23% | 1% | |
21 | Mistral AI | Mistral-Large-Latest (scratchpad with freeze values) | 0.161 | 0.157 | 0.090 | 0.132 | 0.159 | 0.147 | [0.129, 0.165] | <0.001 | 22% | 0% |
22 | OpenAI | GPT-4-Turbo-2024-04-09 (zero shot) | 0.162 | 0.169 | 0.079 | 0.136 | 0.165 | 0.149 | [0.129, 0.169] | <0.001 | 23% | 0% |
23 | OpenAI | GPT-4-Turbo-2024-04-09 (scratchpad with freeze values) | 0.176 | 0.148 | 0.078 | 0.123 | 0.162 | 0.149 | [0.126, 0.173] | <0.001 | 26% | 0% |
24 | OpenAI | GPT-4 (scratchpad with freeze values) | 0.174 | 0.163 | 0.062 | 0.126 | 0.168 | 0.150 | [0.129, 0.17] | <0.001 | 22% | 1% |
25 | Gemini-1.5-Pro (scratchpad with freeze values) | 0.163 | 0.166 | 0.088 | 0.137 | 0.164 | 0.150 | [0.131, 0.168] | <0.001 | 22% | 0% | |
26 | Anthropic | Claude-3-5-Sonnet-20240620 (zero shot) | 0.145 | 0.189 | 0.095 | 0.155 | 0.167 | 0.150 | [0.123, 0.177] | <0.001 | 24% | 0% |
27 | Gemini-1.5-Pro (scratchpad with news) | 0.163 | 0.168 | 0.093 | 0.141 | 0.165 | 0.152 | [0.133, 0.17] | <0.001 | 22% | 0% | |
28 | Anthropic | Claude-3-Opus-20240229 (superforecaster with news 1) | 0.157 | 0.195 | 0.062 | 0.146 | 0.176 | 0.152 | [0.13, 0.173] | <0.001 | 22% | 1% |
29 | OpenAI | GPT-4-Turbo-2024-04-09 (scratchpad with news with freeze values) | 0.178 | 0.131 | 0.121 | 0.127 | 0.154 | 0.152 | [0.13, 0.175] | <0.001 | 25% | 1% |
30 | OpenAI | GPT-4o (zero shot with freeze values) | 0.198 | 0.149 | 0.037 | 0.108 | 0.173 | 0.153 | [0.128, 0.178] | <0.001 | 26% | 0% |
31 | Meta | Llama-3-70b-Chat-Hf (scratchpad with freeze values) | 0.191 | 0.145 | 0.065 | 0.115 | 0.168 | 0.153 | [0.135, 0.171] | <0.001 | 25% | 0% |
32 | Mistral AI | Mixtral-8x22B-Instruct-V0.1 (scratchpad with freeze values) | 0.181 | 0.163 | 0.063 | 0.126 | 0.172 | 0.154 | [0.136, 0.171] | <0.001 | 21% | 0% |
33 | OpenAI | GPT-4-Turbo-2024-04-09 (scratchpad) | 0.176 | 0.167 | 0.072 | 0.132 | 0.171 | 0.154 | [0.137, 0.171] | <0.001 | 23% | 0% |
34 | Anthropic | Claude-3-Opus-20240229 (scratchpad) | 0.160 | 0.188 | 0.079 | 0.148 | 0.174 | 0.154 | [0.136, 0.173] | <0.001 | 21% | 1% |
35 | Gemini-1.5-Pro (zero shot with freeze values) | 0.185 | 0.165 | 0.053 | 0.124 | 0.175 | 0.154 | [0.127, 0.181] | <0.001 | 25% | 0% | |
36 | OpenAI | GPT-4 (scratchpad) | 0.174 | 0.168 | 0.082 | 0.136 | 0.171 | 0.155 | [0.139, 0.171] | <0.001 | 18% | 1% |
37 | Qwen | Qwen1.5-110B-Chat (zero shot with freeze values) | 0.197 | 0.148 | 0.057 | 0.115 | 0.173 | 0.156 | [0.133, 0.178] | <0.001 | 20% | 0% |
38 | Anthropic | Claude-2.1 (scratchpad with freeze values) | 0.217 | 0.109 | 0.071 | 0.095 | 0.163 | 0.156 | [0.138, 0.174] | <0.001 | 24% | 23% |
39 | Gemini-1.5-Flash (zero shot with freeze values) | 0.190 | 0.165 | 0.056 | 0.125 | 0.178 | 0.157 | [0.129, 0.186] | <0.001 | 22% | 0% | |
40 | Mistral AI | Mixtral-8x22B-Instruct-V0.1 (zero shot with freeze values) | 0.185 | 0.156 | 0.085 | 0.130 | 0.171 | 0.158 | [0.131, 0.184] | <0.001 | 24% | 0% |
41 | Meta | Llama-3-70b-Chat-Hf (zero shot with freeze values) | 0.194 | 0.163 | 0.055 | 0.123 | 0.179 | 0.159 | [0.133, 0.185] | <0.001 | 25% | 0% |
42 | OpenAI | GPT-4-Turbo-2024-04-09 (scratchpad with news) | 0.178 | 0.177 | 0.084 | 0.143 | 0.178 | 0.161 | [0.139, 0.182] | <0.001 | 24% | 0% |
43 | ForecastBench | Imputed Forecaster | 0.250 | 0.094 | 0.035 | 0.072 | 0.172 | 0.161 | [0.142, 0.18] | <0.001 | 25% | 100% |
44 | Qwen | Qwen1.5-110B-Chat (scratchpad with freeze values) | 0.183 | 0.177 | 0.077 | 0.140 | 0.180 | 0.162 | [0.144, 0.18] | <0.001 | 18% | 0% |
45 | Anthropic | Claude-2.1 (scratchpad) | 0.217 | 0.112 | 0.102 | 0.108 | 0.164 | 0.163 | [0.143, 0.182] | <0.001 | 22% | 24% |
46 | Qwen | Qwen1.5-110B-Chat (scratchpad with news with freeze values) | 0.179 | 0.182 | 0.086 | 0.147 | 0.180 | 0.163 | [0.143, 0.182] | <0.001 | 22% | 0% |
47 | Anthropic | Claude-2.1 (zero shot with freeze values) | 0.221 | 0.147 | 0.038 | 0.107 | 0.184 | 0.164 | [0.141, 0.187] | <0.001 | 26% | 1% |
48 | Mistral AI | Mistral-Large-Latest (scratchpad) | 0.161 | 0.194 | 0.122 | 0.167 | 0.178 | 0.164 | [0.146, 0.183] | <0.001 | 22% | 0% |
49 | Anthropic | Claude-3-5-Sonnet-20240620 (scratchpad with SECOND news) | 0.197 | 0.158 | 0.086 | 0.132 | 0.178 | 0.165 | [0.144, 0.185] | <0.001 | 19% | 0% |
50 | Anthropic | Claude-3-Opus-20240229 (zero shot) | 0.163 | 0.217 | 0.080 | 0.166 | 0.190 | 0.165 | [0.139, 0.19] | <0.001 | 18% | 0% |
51 | OpenAI | GPT-4o (superforecaster with news 3) | 0.198 | 0.172 | 0.067 | 0.133 | 0.185 | 0.166 | [0.145, 0.186] | <0.001 | 21% | 8% |
52 | OpenAI | GPT-4 (zero shot) | 0.168 | 0.191 | 0.118 | 0.164 | 0.179 | 0.166 | [0.147, 0.184] | <0.001 | 22% | 0% |
53 | Gemini-1.5-Pro (superforecaster with news 3) | 0.183 | 0.175 | 0.105 | 0.149 | 0.179 | 0.166 | [0.146, 0.186] | <0.001 | 22% | 0% | |
54 | Mistral AI | Mixtral-8x22B-Instruct-V0.1 (scratchpad with news with freeze values) | 0.201 | 0.171 | 0.068 | 0.133 | 0.186 | 0.167 | [0.148, 0.185] | <0.001 | 23% | 0% |
55 | Mistral AI | Mixtral-8x22B-Instruct-V0.1 (scratchpad with news) | 0.201 | 0.171 | 0.074 | 0.135 | 0.186 | 0.168 | [0.151, 0.185] | <0.001 | 22% | 0% |
56 | OpenAI | GPT-4o (zero shot) | 0.198 | 0.168 | 0.088 | 0.139 | 0.183 | 0.168 | [0.147, 0.19] | <0.001 | 21% | 0% |
57 | Qwen | Qwen1.5-110B-Chat (scratchpad with news) | 0.179 | 0.194 | 0.100 | 0.159 | 0.186 | 0.169 | [0.15, 0.189] | <0.001 | 21% | 0% |
58 | Mistral AI | Mixtral-8x22B-Instruct-V0.1 (scratchpad) | 0.181 | 0.186 | 0.109 | 0.158 | 0.183 | 0.169 | [0.152, 0.187] | <0.001 | 20% | 0% |
59 | Qwen | Qwen1.5-110B-Chat (scratchpad) | 0.183 | 0.196 | 0.090 | 0.157 | 0.190 | 0.170 | [0.152, 0.189] | <0.001 | 19% | 0% |
60 | Anthropic | Claude-3-Opus-20240229 (superforecaster with news 3) | 0.177 | 0.201 | 0.105 | 0.166 | 0.189 | 0.171 | [0.149, 0.193] | <0.001 | 22% | 8% |
61 | Anthropic | Claude-3-Opus-20240229 (scratchpad with news with freeze values) | 0.183 | 0.196 | 0.098 | 0.160 | 0.189 | 0.171 | [0.15, 0.192] | <0.001 | 23% | 0% |
62 | Anthropic | Claude-3-5-Sonnet-20240620 (superforecaster with news 2) | 0.194 | 0.188 | 0.083 | 0.150 | 0.191 | 0.172 | [0.149, 0.195] | <0.001 | 22% | 0% |
63 | Meta | Llama-3-8b-Chat-Hf (zero shot with freeze values) | 0.205 | 0.166 | 0.092 | 0.139 | 0.186 | 0.172 | [0.146, 0.198] | <0.001 | 24% | 0% |
64 | OpenAI | GPT-4-Turbo-2024-04-09 (superforecaster with news 3) | 0.199 | 0.174 | 0.100 | 0.146 | 0.186 | 0.173 | [0.151, 0.194] | <0.001 | 20% | 13% |
65 | Mistral AI | Mixtral-8x7B-Instruct-V0.1 (scratchpad) | 0.205 | 0.154 | 0.126 | 0.143 | 0.179 | 0.174 | [0.154, 0.195] | <0.001 | 27% | 15% |
66 | Anthropic | Claude-3-Opus-20240229 (scratchpad with news) | 0.183 | 0.201 | 0.107 | 0.166 | 0.192 | 0.174 | [0.153, 0.196] | <0.001 | 22% | 0% |
67 | Gemini-1.5-Flash (scratchpad with freeze values) | 0.214 | 0.163 | 0.087 | 0.135 | 0.189 | 0.175 | [0.152, 0.197] | <0.001 | 18% | 0% | |
68 | Mistral AI | Mistral-Large-Latest (zero shot) | 0.172 | 0.194 | 0.149 | 0.178 | 0.183 | 0.175 | [0.149, 0.201] | <0.001 | 20% | 0% |
69 | Anthropic | Claude-3-Opus-20240229 (superforecaster with news 2) | 0.175 | 0.205 | 0.133 | 0.179 | 0.190 | 0.177 | [0.152, 0.201] | <0.001 | 22% | 0% |
70 | Meta | Llama-3-8b-Chat-Hf (scratchpad with freeze values) | 0.225 | 0.157 | 0.077 | 0.128 | 0.191 | 0.177 | [0.161, 0.192] | <0.001 | 22% | 0% |
71 | Qwen | Qwen1.5-110B-Chat (superforecaster with news 1) | 0.203 | 0.196 | 0.076 | 0.152 | 0.199 | 0.177 | [0.153, 0.202] | <0.001 | 22% | 0% |
72 | Gemini-1.5-Pro (zero shot) | 0.185 | 0.207 | 0.107 | 0.170 | 0.196 | 0.178 | [0.15, 0.206] | <0.001 | 22% | 0% | |
73 | Mistral AI | Mistral-Large-Latest (scratchpad with news with freeze values) | 0.201 | 0.175 | 0.120 | 0.155 | 0.188 | 0.178 | [0.157, 0.199] | <0.001 | 22% | 0% |
74 | Mistral AI | Mixtral-8x7B-Instruct-V0.1 (zero shot with freeze values) | 0.225 | 0.180 | 0.046 | 0.131 | 0.203 | 0.178 | [0.148, 0.208] | <0.001 | 29% | 0% |
75 | Mistral AI | Mixtral-8x22B-Instruct-V0.1 (superforecaster with news 3) | 0.214 | 0.158 | 0.116 | 0.143 | 0.186 | 0.178 | [0.161, 0.195] | <0.001 | 16% | 20% |
76 | Qwen | Qwen1.5-110B-Chat (zero shot) | 0.197 | 0.193 | 0.106 | 0.161 | 0.195 | 0.179 | [0.159, 0.198] | <0.001 | 15% | 1% |
77 | Anthropic | Claude-2.1 (scratchpad with news with freeze values) | 0.214 | 0.173 | 0.096 | 0.145 | 0.193 | 0.179 | [0.158, 0.201] | <0.001 | 21% | 4% |
78 | OpenAI | GPT-4o (superforecaster with news 1) | 0.203 | 0.199 | 0.084 | 0.157 | 0.201 | 0.180 | [0.153, 0.206] | <0.001 | 25% | 0% |
79 | Gemini-1.5-Flash (scratchpad with news with freeze values) | 0.221 | 0.158 | 0.108 | 0.140 | 0.189 | 0.180 | [0.157, 0.204] | <0.001 | 21% | 0% | |
80 | OpenAI | GPT-4o (scratchpad with SECOND news) | 0.236 | 0.154 | 0.076 | 0.125 | 0.195 | 0.181 | [0.161, 0.2] | <0.001 | 17% | 5% |
81 | Meta | Llama-3-8b-Chat-Hf (zero shot) | 0.205 | 0.191 | 0.097 | 0.156 | 0.198 | 0.181 | [0.154, 0.207] | <0.001 | 24% | 0% |
82 | Mistral AI | Mixtral-8x22B-Instruct-V0.1 (superforecaster with news 1) | 0.212 | 0.194 | 0.087 | 0.155 | 0.203 | 0.183 | [0.158, 0.208] | <0.001 | 21% | 0% |
83 | ForecastBench | LLM Crowd (gpt-4o, claude-3.5-sonnet, gemini-1.5-pro) with news | 0.237 | 0.162 | 0.076 | 0.130 | 0.199 | 0.184 | [0.166, 0.201] | <0.001 | 15% | 43% |
84 | ForecastBench | LLM Crowd (gpt-4o, claude-3.5-sonnet, gemini-1.5-pro) with news | 0.237 | 0.162 | 0.076 | 0.130 | 0.200 | 0.184 | [0.166, 0.202] | <0.001 | 16% | 43% |
85 | ForecastBench | LLM Crowd (gpt-4o, claude-3.5-sonnet, gemini-1.5-pro) with news | 0.237 | 0.163 | 0.075 | 0.130 | 0.200 | 0.184 | [0.165, 0.202] | <0.001 | 17% | 43% |
86 | Anthropic | Claude-2.1 (scratchpad with news) | 0.214 | 0.176 | 0.115 | 0.154 | 0.195 | 0.184 | [0.161, 0.207] | <0.001 | 22% | 9% |
87 | Gemini-1.5-Pro (superforecaster with news 1) | 0.202 | 0.219 | 0.079 | 0.167 | 0.210 | 0.184 | [0.16, 0.209] | <0.001 | 22% | 0% | |
88 | OpenAI | GPT-4-Turbo-2024-04-09 (superforecaster with news 1) | 0.195 | 0.205 | 0.121 | 0.174 | 0.200 | 0.185 | [0.16, 0.21] | <0.001 | 20% | 0% |
89 | Meta | Llama-3-70b-Chat-Hf (zero shot) | 0.194 | 0.211 | 0.115 | 0.176 | 0.203 | 0.185 | [0.162, 0.208] | <0.001 | 21% | 0% |
90 | Mistral AI | Mistral-Large-Latest (scratchpad with news) | 0.201 | 0.205 | 0.113 | 0.171 | 0.203 | 0.186 | [0.165, 0.207] | <0.001 | 21% | 0% |
91 | OpenAI | GPT-4o (superforecaster with news 2) | 0.229 | 0.165 | 0.109 | 0.144 | 0.197 | 0.187 | [0.162, 0.212] | <0.001 | 23% | 1% |
92 | Meta | Llama-3-70b-Chat-Hf (scratchpad) | 0.191 | 0.218 | 0.124 | 0.183 | 0.205 | 0.187 | [0.171, 0.203] | <0.001 | 24% | 0% |
93 | Gemini-1.5-Flash (scratchpad) | 0.214 | 0.190 | 0.112 | 0.161 | 0.202 | 0.187 | [0.167, 0.208] | <0.001 | 17% | 0% | |
94 | Mistral AI | Mixtral-8x22B-Instruct-V0.1 (zero shot) | 0.185 | 0.220 | 0.139 | 0.190 | 0.202 | 0.188 | [0.161, 0.214] | <0.001 | 19% | 0% |
95 | Gemini-1.5-Flash (zero shot) | 0.190 | 0.224 | 0.127 | 0.188 | 0.207 | 0.189 | [0.162, 0.217] | <0.001 | 18% | 0% | |
96 | Gemini-1.5-Flash (scratchpad with news) | 0.221 | 0.179 | 0.124 | 0.159 | 0.200 | 0.190 | [0.168, 0.212] | <0.001 | 20% | 0% | |
97 | Mistral AI | Mistral-Large-Latest (superforecaster with news 1) | 0.212 | 0.203 | 0.107 | 0.168 | 0.208 | 0.190 | [0.165, 0.215] | <0.001 | 21% | 0% |
98 | Qwen | Qwen1.5-110B-Chat (superforecaster with news 3) | 0.212 | 0.205 | 0.106 | 0.169 | 0.209 | 0.191 | [0.17, 0.211] | <0.001 | 21% | 7% |
99 | Mistral AI | Mixtral-8x7B-Instruct-V0.1 (zero shot) | 0.225 | 0.185 | 0.118 | 0.161 | 0.205 | 0.193 | [0.164, 0.222] | <0.001 | 21% | 0% |
100 | Mistral AI | Mixtral-8x7B-Instruct-V0.1 (superforecaster with news 2) | 0.252 | 0.154 | 0.105 | 0.136 | 0.203 | 0.194 | [0.169, 0.218] | <0.001 | 27% | 22% |
101 | Anthropic | Claude-2.1 (zero shot) | 0.221 | 0.211 | 0.098 | 0.169 | 0.216 | 0.195 | [0.175, 0.216] | <0.001 | 19% | 0% |
102 | OpenAI | GPT-4-Turbo-2024-04-09 (superforecaster with news 2) | 0.218 | 0.203 | 0.123 | 0.173 | 0.210 | 0.195 | [0.17, 0.221] | <0.001 | 25% | 2% |
103 | Meta | Llama-3-8b-Chat-Hf (scratchpad) | 0.225 | 0.222 | 0.079 | 0.169 | 0.224 | 0.197 | [0.18, 0.215] | <0.001 | 22% | 0% |
104 | Mistral AI | Mixtral-8x22B-Instruct-V0.1 (superforecaster with news 2) | 0.235 | 0.189 | 0.108 | 0.159 | 0.212 | 0.197 | [0.179, 0.216] | <0.001 | 23% | 2% |
105 | Mistral AI | Mistral-Large-Latest (superforecaster with news 3) | 0.231 | 0.187 | 0.125 | 0.164 | 0.209 | 0.198 | [0.177, 0.219] | <0.001 | 21% | 9% |
106 | Anthropic | Claude-2.1 (superforecaster with news 3) | 0.227 | 0.190 | 0.132 | 0.169 | 0.208 | 0.198 | [0.175, 0.22] | <0.001 | 23% | 6% |
107 | Mistral AI | Mistral-Large-Latest (superforecaster with news 2) | 0.210 | 0.218 | 0.133 | 0.187 | 0.214 | 0.198 | [0.173, 0.223] | <0.001 | 21% | 1% |
108 | Mistral AI | Mixtral-8x7B-Instruct-V0.1 (superforecaster with news 1) | 0.247 | 0.186 | 0.089 | 0.150 | 0.217 | 0.199 | [0.173, 0.224] | <0.001 | 25% | 15% |
109 | Mistral AI | Mixtral-8x7B-Instruct-V0.1 (scratchpad with freeze values) | 0.205 | 0.222 | 0.162 | 0.200 | 0.213 | 0.202 | [0.173, 0.231] | <0.001 | 24% | 11% |
110 | Gemini-1.5-Pro (superforecaster with news 2) | 0.220 | 0.211 | 0.141 | 0.185 | 0.215 | 0.203 | [0.174, 0.231] | <0.001 | 21% | 0% | |
111 | Qwen | Qwen1.5-110B-Chat (superforecaster with news 2) | 0.225 | 0.229 | 0.102 | 0.182 | 0.227 | 0.204 | [0.182, 0.225] | <0.001 | 22% | 4% |
112 | Meta | Llama-2-70b-Chat-Hf (zero shot with freeze values) | 0.234 | 0.226 | 0.089 | 0.176 | 0.230 | 0.205 | [0.177, 0.232] | <0.001 | 24% | 1% |
113 | Gemini-1.5-Flash (superforecaster with news 3) | 0.237 | 0.205 | 0.122 | 0.174 | 0.221 | 0.205 | [0.182, 0.229] | <0.001 | 19% | 11% | |
114 | Meta | Llama-2-70b-Chat-Hf (scratchpad with freeze values) | 0.227 | 0.232 | 0.105 | 0.185 | 0.229 | 0.206 | [0.19, 0.222] | <0.001 | 22% | 0% |
115 | Gemini-1.5-Flash (superforecaster with news 2) | 0.228 | 0.213 | 0.144 | 0.187 | 0.220 | 0.208 | [0.182, 0.233] | <0.001 | 20% | 0% | |
116 | Mistral AI | Mixtral-8x7B-Instruct-V0.1 (scratchpad with news with freeze values) | 0.295 | 0.123 | 0.125 | 0.123 | 0.209 | 0.209 | [0.19, 0.229] | <0.001 | 23% | 13% |
117 | Anthropic | Claude-3-Haiku-20240307 (superforecaster with news 2) | 0.233 | 0.214 | 0.140 | 0.186 | 0.223 | 0.209 | [0.191, 0.228] | <0.001 | 20% | 0% |
118 | Anthropic | Claude-3-Haiku-20240307 (scratchpad with freeze values) | 0.238 | 0.227 | 0.104 | 0.182 | 0.232 | 0.210 | [0.191, 0.228] | <0.001 | 21% | 0% |
119 | Mistral AI | Mixtral-8x7B-Instruct-V0.1 (superforecaster with news 3) | 0.250 | 0.176 | 0.164 | 0.172 | 0.213 | 0.211 | [0.189, 0.233] | <0.001 | 23% | 14% |
120 | Gemini-1.5-Flash (superforecaster with news 1) | 0.232 | 0.239 | 0.114 | 0.193 | 0.235 | 0.212 | [0.184, 0.24] | <0.001 | 22% | 1% | |
121 | Anthropic | Claude-3-Haiku-20240307 (scratchpad) | 0.238 | 0.224 | 0.125 | 0.187 | 0.231 | 0.213 | [0.195, 0.23] | <0.001 | 22% | 0% |
122 | Anthropic | Claude-3-Haiku-20240307 (zero shot with freeze values) | 0.301 | 0.153 | 0.086 | 0.128 | 0.227 | 0.214 | [0.196, 0.233] | <0.001 | 19% | 1% |
123 | Meta | Llama-2-70b-Chat-Hf (scratchpad) | 0.227 | 0.252 | 0.145 | 0.212 | 0.239 | 0.219 | [0.201, 0.238] | <0.001 | 22% | 1% |
124 | Anthropic | Claude-2.1 (superforecaster with news 2) | 0.241 | 0.228 | 0.150 | 0.199 | 0.235 | 0.220 | [0.195, 0.246] | <0.001 | 23% | 11% |
125 | Anthropic | Claude-2.1 (superforecaster with news 1) | 0.260 | 0.230 | 0.099 | 0.182 | 0.245 | 0.221 | [0.197, 0.245] | <0.001 | 23% | 4% |
126 | OpenAI | GPT-3.5-Turbo-0125 (scratchpad with freeze values) | 0.258 | 0.235 | 0.116 | 0.192 | 0.247 | 0.225 | [0.205, 0.244] | <0.001 | 21% | 0% |
127 | Anthropic | Claude-3-Haiku-20240307 (scratchpad with news with freeze values) | 0.277 | 0.212 | 0.112 | 0.175 | 0.245 | 0.226 | [0.209, 0.244] | <0.001 | 21% | 0% |
128 | Anthropic | Claude-3-Haiku-20240307 (scratchpad with news) | 0.277 | 0.208 | 0.122 | 0.176 | 0.242 | 0.227 | [0.21, 0.244] | <0.001 | 21% | 0% |
129 | Mistral AI | Mixtral-8x7B-Instruct-V0.1 (scratchpad with news) | 0.295 | 0.186 | 0.115 | 0.160 | 0.241 | 0.227 | [0.203, 0.252] | <0.001 | 22% | 14% |
130 | ForecastBench | Always 0.5 | 0.250 | 0.250 | 0.134 | 0.207 | 0.250 | 0.229 | [0.22, 0.237] | <0.001 | 16% | 0% |
131 | OpenAI | GPT-3.5-Turbo-0125 (scratchpad) | 0.258 | 0.247 | 0.133 | 0.205 | 0.252 | 0.232 | [0.212, 0.251] | <0.001 | 20% | 0% |
132 | Anthropic | Claude-3-Haiku-20240307 (zero shot) | 0.301 | 0.198 | 0.115 | 0.167 | 0.250 | 0.234 | [0.216, 0.252] | <0.001 | 18% | 1% |
133 | Anthropic | Claude-3-Haiku-20240307 (superforecaster with news 3) | 0.266 | 0.237 | 0.160 | 0.209 | 0.252 | 0.238 | [0.219, 0.256] | <0.001 | 18% | 26% |
134 | Meta | Llama-2-70b-Chat-Hf (zero shot) | 0.234 | 0.317 | 0.140 | 0.252 | 0.275 | 0.243 | [0.215, 0.27] | <0.001 | 23% | 1% |
135 | Anthropic | Claude-3-Haiku-20240307 (superforecaster with news 1) | 0.286 | 0.294 | 0.120 | 0.230 | 0.290 | 0.258 | [0.232, 0.284] | <0.001 | 22% | 0% |
136 | OpenAI | GPT-3.5-Turbo-0125 (zero shot with freeze values) | 0.423 | 0.150 | 0.089 | 0.128 | 0.286 | 0.275 | [0.247, 0.304] | <0.001 | 29% | 0% |
137 | ForecastBench | Always 0 | 0.328 | 0.292 | 0.200 | 0.258 | 0.310 | 0.293 | [0.241, 0.344] | <0.001 | 41% | 0% |
138 | ForecastBench | Random Uniform | 0.357 | 0.300 | 0.201 | 0.263 | 0.328 | 0.310 | [0.274, 0.346] | <0.001 | 22% | 0% |
139 | OpenAI | GPT-3.5-Turbo-0125 (zero shot) | 0.423 | 0.247 | 0.189 | 0.225 | 0.335 | 0.324 | [0.293, 0.355] | <0.001 | 21% | 0% |
140 | ForecastBench | Always 1 | 0.672 | 0.708 | 0.568 | 0.657 | 0.690 | 0.664 | [0.61, 0.719] | <0.001 | 26% | 0% |