A few typos: Q25 starts with a surplus "24." Q26 and Q29 say 2025 where you want 2026. Q38 has the hyphen in Aaron Taylor-Johnson's name in the wrong place. And the summary of the post says "A prediction contest for 2024" where you want 2025.
Also there's a duplicate of Q1 between Q21 and Q22, but I'm not sure if you can remove that without affecting the results so far.
Hoping to beat last year's score of 0.183 and last year's ranking of fourth. My basic assumption was that things will be pretty much the same and change is unlikely to be radical. We'll see whether that holds this year.
Maybe I'm missing something here, but if you're doing this by mean Brier Score, why shouldn't I just forecast on 1 question that seems particularly likely/unlikely (say, 'Will Trump be assassinated?' at 1%), and then sit back and get an extremely low mean Brier? Probably you should do it by adding up the difference between a participant's Brier and the median Brier on that question for each question.
Also, just as a note, your claim that Brier will necessarily result in you being incentivised to give your true belief is not necessarily true in this context. Consider a one question tournament where the winner gets £100k. The one question is: 'Will Keir Starmer be PM at EOY 2025?' I think the true probability is 90%. But I should clearly put 100% here - if I put 100% and he remains PM, I'm necessarily the (joint) winner. If I put anything other than 100%, I'll lose to anyone who put 100% (unless he isn't PM, in which case I'll lose to someone who went lower than me).
I think your second paragraph doesn't hold because there are 45 questions rather than just 1. In the 1-question case, if you believe the true probability is 90%, then you put 100% and you have a 90% shot at winning the prize and a 10% chance of winning nothing, so that's worth doing. But if there are lots of questions, then the equivalent of "putting 100% and he remains PM" is "putting 100% for *all* the items that do end up happening and putting 0% for *all* the items that don't". If you can do that, then you're beyond a superforecaster and deserve the win. In reality, I think you would score better on expectation if you hedge a bit by putting your true estimate for every question.
The point I'm making here is that OP is referring to the Brier Score being a strictly proper scoring rule, which means that you maximise your score in expectation by giving your true probability estimate. However, this does not mean that in every context you're incentivised to give your true probability estimate, and tournaments are an example of where extremising can give you an advantage.
Yes, I agree there are ways of increasing your chance of winning, at the cost of lowering your expected core, by extremising in certain ways. E.g. if you usually came 5th - 10th, but there were three questions you thought were correllated about (say) Reform, that were each 20% likely to happen (and thought most people would think similarly), you could just put 90% for all of them. In this case, 4 times out of 5 you'd do worse than you would have done, but 1 time out of 5 you would do better - maybe enough to get you to 1st place!
I think this sort of tactic is (a) unavoidable under any score system; (b) requires a fair bit of skill to do well and could easily backfire; and (c) carries its own price in terms of more often getting a worse score; so I can live with it.
Re your first paragraph, I don't think doing very well on one question will give you an extremely good score overall. It's just one question out of 45, [EDIT: sorry, this part is wrong [and because Brier uses a mean *squared* error, your wrong predictions will harm your score *more* than your good predictions will help it.]]
Last year, I thought "extraterrestrial life discovered" was very unlikely and rated it 2%. So I got a good score for that question, but [EDIT: sorry, this part is wrong [it would take several good scores like that to outweigh a bad score on even *one* question.] Correction: [this is being averaged with all the bad scores for the other questions.]]
Maybe I'm misunderstanding the scoring system being used here but if you're just taking the mean Brier score among questions that you answered, if you put 1% and the event does not occur, your mean Brier score is 0.0001.
I’ve just come across this contest and would be interested in answering the questions myself since the competition is now closed (according to the Microsoft form) are the questions hosted anywhere
A few typos: Q25 starts with a surplus "24." Q26 and Q29 say 2025 where you want 2026. Q38 has the hyphen in Aaron Taylor-Johnson's name in the wrong place. And the summary of the post says "A prediction contest for 2024" where you want 2025.
Also there's a duplicate of Q1 between Q21 and Q22, but I'm not sure if you can remove that without affecting the results so far.
Thanks! Penalty of trying to finish this while having a specific train I need to catch. :-)
All now fixed, including the duplicate question.
The summary still says "A prediction contest for 2024" (the snippet below the post title on the edrith homepage and at the top of the post itself).
The Starmer is PM question is asked twice.
Question 26 should probably read 1 Jan 2026
Thanks - both fixed.
Hoping to beat last year's score of 0.183 and last year's ranking of fourth. My basic assumption was that things will be pretty much the same and change is unlikely to be radical. We'll see whether that holds this year.
Congratulations on last year! Those feel like pretty decent assumptions in general. :-)
I made the same basic assumption last year and it led me to underestimate Reform and the Lib Dems and thus lose marks.
Maybe I'm missing something here, but if you're doing this by mean Brier Score, why shouldn't I just forecast on 1 question that seems particularly likely/unlikely (say, 'Will Trump be assassinated?' at 1%), and then sit back and get an extremely low mean Brier? Probably you should do it by adding up the difference between a participant's Brier and the median Brier on that question for each question.
Also, just as a note, your claim that Brier will necessarily result in you being incentivised to give your true belief is not necessarily true in this context. Consider a one question tournament where the winner gets £100k. The one question is: 'Will Keir Starmer be PM at EOY 2025?' I think the true probability is 90%. But I should clearly put 100% here - if I put 100% and he remains PM, I'm necessarily the (joint) winner. If I put anything other than 100%, I'll lose to anyone who put 100% (unless he isn't PM, in which case I'll lose to someone who went lower than me).
I think your second paragraph doesn't hold because there are 45 questions rather than just 1. In the 1-question case, if you believe the true probability is 90%, then you put 100% and you have a 90% shot at winning the prize and a 10% chance of winning nothing, so that's worth doing. But if there are lots of questions, then the equivalent of "putting 100% and he remains PM" is "putting 100% for *all* the items that do end up happening and putting 0% for *all* the items that don't". If you can do that, then you're beyond a superforecaster and deserve the win. In reality, I think you would score better on expectation if you hedge a bit by putting your true estimate for every question.
The point I'm making here is that OP is referring to the Brier Score being a strictly proper scoring rule, which means that you maximise your score in expectation by giving your true probability estimate. However, this does not mean that in every context you're incentivised to give your true probability estimate, and tournaments are an example of where extremising can give you an advantage.
Yes, I agree there are ways of increasing your chance of winning, at the cost of lowering your expected core, by extremising in certain ways. E.g. if you usually came 5th - 10th, but there were three questions you thought were correllated about (say) Reform, that were each 20% likely to happen (and thought most people would think similarly), you could just put 90% for all of them. In this case, 4 times out of 5 you'd do worse than you would have done, but 1 time out of 5 you would do better - maybe enough to get you to 1st place!
I think this sort of tactic is (a) unavoidable under any score system; (b) requires a fair bit of skill to do well and could easily backfire; and (c) carries its own price in terms of more often getting a worse score; so I can live with it.
Some interesting stuff here on this: https://arxiv.org/pdf/2106.11248
Re your first paragraph, I don't think doing very well on one question will give you an extremely good score overall. It's just one question out of 45, [EDIT: sorry, this part is wrong [and because Brier uses a mean *squared* error, your wrong predictions will harm your score *more* than your good predictions will help it.]]
Last year, I thought "extraterrestrial life discovered" was very unlikely and rated it 2%. So I got a good score for that question, but [EDIT: sorry, this part is wrong [it would take several good scores like that to outweigh a bad score on even *one* question.] Correction: [this is being averaged with all the bad scores for the other questions.]]
Maybe I'm misunderstanding the scoring system being used here but if you're just taking the mean Brier score among questions that you answered, if you put 1% and the event does not occur, your mean Brier score is 0.0001.
No, the rubric says that for any questions you omit to answer, your prediction is taken as 50%.
Ah okay, I missed this entirely, my mistake!
I’ve just come across this contest and would be interested in answering the questions myself since the competition is now closed (according to the Microsoft form) are the questions hosted anywhere
I'll be putting up the questions in the next few days. :-)