iPredict

iPredict launched in September 2008. Since then, over 280 contracts have closed. How well did iPredict forecast these outcomes? This page summarises the prediction performance of closed contracts on the iPredict live site since launch.

Note: Data for the forecasting performance analysis on this page is for the period 12 June 2008 through 27 July 2009. Price data at the end of the page is for the period 12 June 2008 through 30 Setember 2009. The period 12 June 2008-8 September 2008 was a pre-launch trial period using a small group of testers.

Price as a Predictor

Is a price on iPredict a predictor of future events?

Figure 1 shows that price on iPredict is an excellent predictor of the likelihood or expected outcome of future events. This chart is produced by collecting up all observations of trades between twenty "buckets" of prices each 5 cents wide, then comparing this to the average ultimate closing price for those contracts (= frequency of occurrence). Figure 1 incorporates around 300,000 observations.

Figure 1: Price vs Outcome - iPredict Forecasting Accuracy

Figure 1 shows a strong relationship between price and a contract's closing price i.e. the event outcome. The strong relationship between price and outcome means price on iPredict can be interpreted as the likelihood or expected outcome of that event, given the available information at the time. Figure 1 follows an analysis originally done by Google for their prediction market in January 2009.

Performance of Individual Contracts

Binary Contracts

On average, do binary contracts correctly pick the contract that will be judged true, as determined by a price above $0.50, and contracts which will be judged false, as determined by a price below $0.50?

To test this, for each contract a daily price was taken at a fixed time every 24 hours between its opening and closing dates, for a total of 6,185 observations across 177 binary contracts. Of these, 5,419 (88%) were on the correct side of $0.5000. That is, the proportion of contracts that would go on to be judged true that were above $0.5000 plus the proportion of contracts below $0.5000 that would go on to be judged false represented 88% of all observations.

Our interpretation of this is that a naive level of confidence in a contract, that is, the level of confidence one might have without any additional information about the specific contract, is that about 7 times in 8 a contract price above (below) $0.5000 correctly indicates a contract will be judged true (false).

Indexed Contracts

The closing price of indexed contracts is tied to an observable, external variable - for example, an inflation forecast can be derived from the trading price of an indexed contract that pays $0.01 for each 0.1% of inflation reported for any time period. A problem for indexed contracts is that they attract so little trading: on average around 1/8th the trading of a binary contract. So do they forecast well?

Figure 2 records the performance of indexed contract prices for 26 indexed contracts over their life times. All contracts close at day 0, and variation along the horizontal axis varies time until contract's close. The greater the number of days before the indexed contract's contract close, the fewer the number of indexed contracts that are open (shown by the upwards-sloping red line), giving us fewer observations. Figure 4 records a reduction in absolute error as the closing dates for the indexed contracts approach.

Figure 2: Indexed Contract: Absolute Price Performance Against Closing Price

Figure 3 below displays error in nominal (not absolute) terms and shows that indexed contracts have a persistent upwards price bias that is generally close to, and 100 days out from contract's close actually is, significantly different to 0 at the 95% level. Liquidity constraints may explain the bias.

Figure 3: Nominal Average Error with 95% Confidence Intervals

The average closing price for indexed contracts is $0.4043, meaning it is relatively costly in liquidity to remove an upwards bias (shorting ties up $(1-P) in cash per contract, where P is current price; the closer price gets to $0 and the greater the liquidity cost for any given expected return).

Bundles Performance

How reliable are predictions from stock prices within iPredict bundles?

Across Bundles, Day Before Close

Bundles are collections of mutually exclusive contracts. For example, iPredict's first bundle was for the 2008 New Zealand election winner, the bundle comprised PM.NATIONAL (paid $1 if National led the post-election government), PM.LABOUR (Labour-led) and PM.OTHER (neither led). Table 1 shows collections of bundles grouped by category. Of the 32 bundles, 28 bundles correctly picked the outcome. Of the four missed, two are for Reserve Bank of Australia cash rates, one is for RBNZ Official Cash Rate, and one is for an election-related contract (asking how many seats would National lose in the 2008 election official results).

Table 1: Bundle Accuracy: Percentage Correct Contract In Bundle Is Highest-Priced Day Before Deciding Event

For Bundles, Any Time

Another approach to measuring accuracy is to poll contract bundles once per day each for their lifetime and to measure the proportion of polling which correctly selects the contract within each bundle that will go on to be judged true. This measure yields a total of 939 observations across the 32 bundles were recorded. Of these, 694 (73.9%) had correctly selected the contract that would go on to be closed as the highest priced contract in the bundle (generally, not always, meaning a closing price of $1).

Data to this point on the page is to 27 July 2009.

Source Data

Data in the following table is to 30 September 2009. Complete data set is available for download here.

Complete data set is available for download here. This page last updated 30 September 2009.