Understanding prediction market reports


Prediction markets on Conjoint.ly are designed for product and concept testing through aggregating the audience’s predictions about future success or failure of product launches, modifications, and other decisions. However, respondents need to be trained to use the interface of predictions in the first instance. This guide covers how to read the outputs of both calibration and prediction questions.

Summary of calibration

The calibration accuracy score is an overall guide to how reliable the predictions made by participants will be.

  • A score of 65% and above suggests that the respondents were diligent, knowledgeable, and capable of using the survey to make good predictions for the future.
  • Low accuracy of calibration (50% to 65%) should make you cautious of relying on the predictions.
  • When responses to calibration questions were inaccurate on average (below 50% score), predictions should not be considered trustworthy.
Calibration accuracy
74%
Bets placed on correct answer

The percentage of skipping of calibration questions (i.e., choosing the "I don't know" option) should not exceed 20%: otherwise you are dealing with lazy respondents.

Skipping of calibration questions
4%
Calibration questions were skipped

Calibration questions

For each calibration question, we calculate the percentage of bets placed on correct responses. Scores over 65% are commonly sufficient to conclude respondents were "correct on average".

CQ1: Who won the 2017 Indian Premier League (IPL)?

The betting summary table contains the number of answers (i.e., respondents) who made a particular choice, the average number of points they bet, and the total number of points bet.

Number of answers Average points bet Total points bet
Correct answer: Mumbai Indians 87 68 5920
Incorrect answer: Rising Pune Supergiants 24 54 1285
Skipped question 5 N/A N/A
Total 116 62 7205

The consensus history chart plots the percentage of correct bets against the timeline of responses. Convergence of scores suggests that the summary is firm and that a market consensus has emerged.

For example, in this chart the first several responses were correct. Response 3333185 was wrong and moved consensus to 93% correctness. Response 3370385, on the other hand, was correct and moved consensus from 83 to 84% correctness.

Predictions and commentary

Each prediction will be displayed together with the confidence in prediction (in addition to a betting summary table and a consensus history chart).

In interpreting predictions, we recommend conservatism: If the decision at stake is product launch, only a strong prediction (over 80%) in support of the launch would be enough to support the launch.

Will an official Indian Premier League (IPL) Android app hit 1 million downloads on Google Play in 1 year after launch?
Yes, it will get 1M downloads
A strong prediction (81% of bets)

For each side of the argument, you will see respondents' rationale for making a particular prediction. Comments that were voted by other respondents as convincing will be displayed on top with a higher conviction score.

Comments in support of Yes, it will

Comment Conviction score
Becasue the Apple store app is very popular 90%
it will be 10 mil. People watch IPL!! 70%
i wuld download it myself 60%
because most people think so 30%
it think fans will follow IPL. it is a popular code 30%