Andreas Gräfe

Prediction markets versus alternative methods. Empirical tests of accuracy and acceptability

Karlsruhe: Fakultät für Wirtschaftswissenschaften der Universität Karlsruhe(TH) 2009
[Volltext/pdf / 1.826 kb]   [Abstract]   [Contents]

INTRODUCTION

Prediction markets were quite popular already in the late 19th century. In analyzing historical markets that were operated for betting on the 15 U.S. presidential elections from 1884 and 1940, Rhode and Strumpf (2004, p.127) found that these markets “did a remarkable job forecasting elections in an era before scientific polling”. At certain times, trading activity in these markets was higher than in the stock exchanges on Wall Street, and, during some election campaigns, newspapers like the New York Times reported market prices on a nearly daily basis. Nonetheless, with increasing availability of other forms of gambling (like horse races) and the rise of opinion polls – which were not subject to moral concerns associated with gambling – presidential betting markets disappeared after 1940.

In recent years, there has been a resurgence of interest in prediction markets in the field of forecasting. In 1988, the Iowa Electronic Market (IEM)[1]  was launched to predict the outcome of the U.S. presidential elections of the same year. Its initial success, accompanied by the rise of the internet, ignited the interest of researchers. Since the mid-1990s, various studies have been published that demonstrated accuracy of prediction markets in areas beyond election forecasting, for example, for predicting sports events or business figures. In response to increasing academic interest, the Journal of Prediction Markets was initiated in 2007. In 2008, in an effort to transfer research findings to scholars and practitioners in the field, the Special Interest Group on Prediction Markets[2]  was launched on behalf of the International Institute of Forecasters.[3] 

The final boost in popularity for prediction markets can be traced back to two events. Ironically, the cancellation of the Policy Analysis Market (PAM)[4]  in 2003, which was covered by more than 600 media articles, initially made a broad public aware of prediction markets. Second, James Surowiecki’s bestselling book ‘The Wisdom of Crowds’[5], published in 2004, described prediction markets as one of many ways to harness collective intelligence. In the following year, prediction markets were listed on the Gartner Hype Cycle (Fenn & Linden 2005) and soon major business consultancies saw them as an emerging trend (Manyika et al. 2007).

Not surprisingly, prediction markets became increasingly appealing to companies. Since 2005, the media regularly reports on companies experimenting with prediction markets. For example, news articles published in BusinessWeek (King 2006) or IEEE Spectrum (Cherry 2007) named various companies (e.g. Microsoft, Yahoo, Intel, Eli Lilly or Nokia) that have experimented with prediction markets to improve their internal decision-making processes, for example by forecasting the success of new products, commodity prices, or sales figures. With almost 1,500 employees participating between 2005 and 2007 the largest known internal prediction market ran at Google (Cowgill et al. 2008). The Google markets aimed at predicting future demand (e.g. ‘number of Gmail users by the end of the quarter’), performance (e.g. ‘Google Talk quality rating’), company news (e.g. ‘Google’s Russia office to open by…’), or industry news (e.g. ‘Will Apple release an Intel-based Mac?’). For an overview of ongoing media coverage about the use of prediction markets, see the Special Interest Group on Prediction Markets.

The evidence from the literature suggests that prediction markets have potential to improve on forecasting. However, published case studies are few and they often draw on small samples. In addition, prediction markets have often been compared to weak benchmarks like individual polls, individual expert judgments, or naïve models (like random walk).

To date, it is not known of any major organization that has implemented prediction markets as an integral part of their forecasting activities. Although larger organizations, in particular technology and pharmaceutical companies, were the first to experiment with the approach, its use has not spread to other domains.[6]  As a result, mainstream awareness of the approach is still limited. Currently, prediction markets are undergoing the typical life (or hype) cycle of innovative technologies. As framed by Cain and Drakos (2008), prediction markets have overcome the ‘peak of inflated expectations’ and are entering the ‘through of disillusionment’: early adopters – scientists as well as practitioners – who had overestimated prediction markets’ accuracy and overall usefulness, are now to some extent disenchanted.

Notes

[1]  In 1988, the Henry B. Tippie College of Business at the University of Iowa launched the Iowa Electronic Markets (IEM), the first prediction market that facilitated participation over the internet (http://www.biz.uiowa.edu/iem/). With its first attempt to predict the outcome of the 1988 U.S. presidential election, the IEM provided more accurate forecasts than traditional opinion polls. And it did so for all consecutive elections. In analyzing 964 polls for the five presidential elections from 1988 to 2004, Berg et al. (2008a) found that the respective market forecasts were closer to the actual election results 74% of the time. This superior performance compared to individual polls was replicated for the 2008 election (Berg et al. 2008b). For further information see Section 2.3.1.

[2]  http://www.marketsforforecasting.com

[3]  http://www.forecasters.org

[4]  From 2001 to 2003, the Defense Advanced Research Project Agency (DARPA) of the U.S. government sponsored the FutureMAP project, also known as the Policy Analysis Market (PAM). The original goal of this project was to improve existing intelligence institutions by predicting military and political instability around the world, how the U.S. would affect such instabilities, and vice versa. Later, the focus was narrowed to predict five parameters for each of eight nations in the Middle East: military activity, political instability, economic growth, U.S. military activity, and U.S. financial involvement. In addition, traders should predict additional parameters like U.S. GDP growth, world trade, or total U.S. military casualties.
On July 28, 2003, shortly before the scheduled start of PAM on September 1, two Democratic Senators held a press conference accusing the U.S. Department of Defense to plan a ‘terror market’ for people to bet on terrorist events. The topic caught the interest of the media. During the next two days, 128 media articles were published and most of them cast PAM in an unfavorable light. Not surprisingly, PAM was rapidly terminated.
Later, Hanson (2007), who was involved in the project, conducted a statistical news analysis on more than 600 media articles that mentioned PAM. He found that the more informed articles favored PAM. Yet, the political decision to dismiss PAM was made and it is unlikely that it will be reversed anytime soon. See Hanson (2007) for a review of the origin and development of the project.

[5]  In presenting examples of situations in which averaged crowd opinions outperformed single experts, James Surowiecki’s 2004 book ‘The Wisdom of Crowds’ demonstrated the power of collective intelligence to a broad audience. Yet, the title of the book is misleading: crowds are not wise when acting together. The actual conclusion of the book is that the combined knowledge of many individual contains wisdom, when the individuals act independently. Surowiecki described prediction markets as one of many ways to harness crowd opinion and, thus, contributed largely to their awareness level.

[6]  In a talk at the Forecasting Summit (Graefe 2008d) – a conference for business forecasting practitioners – the audience of approximately 50 people was asked if they were using prediction markets within their companies. Nobody used prediction markets – and only a handful had even heard of them before.