Rianna Herzlinger
In 2013, Google Flu Trend (GFT) became famous for seemingly predicting the 2012-13 flu season better than the CDC. They used data from users’ search patterns. They used an unsupervised model tasked with making the best matches among 50 million searches to fit 1152 clusters the model itself had to discover. However, aspects of the search engine itself diluted the outcomes. For example, the autosuggest function in Google led to altered user searches that the model didn’t account for. Further, it overfit to seasonal terms, picking up on unrelated terms like ‘high school basketball’ as related to the flu because it jumped in searches during a particular season. Although GFT has since updated their model.
However, after the hype died down, it became clear that GFT actually overpredicted flu cases by 50% each year (article). The spectacular claim that devolved into error led academics to coin the term “big data hubris” to describe Google’s failure. What this article describes as data hubris is essentially an implicit belief in the end of theory. GFT used no theoretical knowledge to sort its clusters or identify search inquiries with the flu. Instead, it used the brute force of massive amounts of data and a black box algorithm to categorize these searches as indicative/not indicative of the flu. The article critiques that big data is often seen as “a substitute for, rather than a supplement to, traditional data collection and analysis” and that “we are far from a place where we can supplant more traditional methods or theories,” (page 1).
With these criticisms, let us return to the thematic questions. First, what is the value of theoretical knowledge? This example further highlights this value because the bar is so low. For example, the theoretical knowledge at stake here is simply the understanding that ‘high school basketball’ searches do not bear on flu rates. We are far from the realm of protein sequencing or physics modeling. However, this elementary conclusion is missed by a model that processes data without understanding its context or sentiment. No theory can be generated from this model, even if it correctly predicted flu rates. It has no way to discern which of its search queries resulted in someone that matched the description of the flu and further in someone who actually had the flu. At best, it diagnoses at one step further away from symptomatology by focusing not on the symptoms themselves but an inquiry into the symptoms. This does not create the type of causality required for a good theory. Finally, the consequences of mistakes like GFT are stark and grave. GFT was proven inaccurate because of comparison to reputable and carefully collected data from the CDC. We had not transitioned into making decisions based on the GFT results, thus there were no severe negative consequences to this error. However, in a true post-theory world, there would be no CDC to compare the GFT to. In this case, we risk making important, costly, and live-altering medical decisions based on an algorithm we do not understand and cannot trust. These consequences ought to give pause to advocates of the end of theory.
This Case Study is part of a larger essay, Science, Data, and Knowledge: End of Theory and Understanding the World.
_______________
Lazer, David, et al. “The Parable of Google Flu: Traps in Big Data Analysis.” Policy Forum, 14 Mar. 2014, dash.harvard.edu/bitstream/handle/1/12016836/The Parable of Google Flu (WP-Final).pdf.
Poeter, Damon. “Big Data Hubris: How Google’s Flu Tracker Went Wrong.” PCMAG, PCMag, 14 Mar. 2014, www.pcmag.com/news/big-data-hubris-how-googles-flu-tracker-went-wrong.
