Bigdata – The latest art in decision making – Part 3 - “Bigdata’s role in predicting the US Presidential Election 2012”

The mind-boggling reality of Exabytes(1 EB = 1000 Petabytes = 1 billion Gigabytes),Zettabytes (1 ZB = 1000 Exabytes = 1 billion Terrabytes)and, soon to come, Yottabytes (1 YB = 1000 Zettabytes = 1 trillion Terrabytes), is well beyond the grasp of our intuition. An Economist video reports that the quantity  of global data is forecast to be an staggering 7,910 Exabytes by 2015, over 60 times greater than 2005. Twitter alone generates over 230 million tweets each day, equivalent to 46 megabits of data per second.

In the near future, people will live in a world of sensors and software in which their “every move is instantly digitized and added to the flood of public data. This is where a statistician (also referred as a Quant) will use is capability to forecast almost everything you need. As my topic suggests, Bigdata has indeed changed the way of traditional decision making style.

I wish to share with you the results of the US Presidential Elections 2012, which many of you would have studied on the internet and how big data has influenced the prediction of the results. Nate Silver, a political blogger had a clean sweep of 50-50 states plus the District of Columbia in forecasting the results of the election. He is neither a pundit nor a former politician, yet he made it.

How did he do it?

With Bigdata in hands, Silver uses the polls of other firms as a data stream that he analyzes and models. His methodology is highly developed, but it boils down to a handful of things. First, he looks at all available polls, except those with patently flawed methodologies. Second, he weights these polls on factors affecting their accuracy. Third, he fits regression curves and trend lines that bring the various polls together. Finally, he runs simulations on key parameters in his models. From all of this, he calculates several results, including two crucial ones – a predicted outcome and the odds of that prediction coming true.

Bigdata redefines expertise
 
Gurus and gut-feel won’t cut it in a world of Big Data, not in politics or marketing. Big Data requires statisticians (quants) who can wrestle it to the ground. In a world flooded with data, good, solid quantitative analytics will be table stakes for success.