book.chapter The rise of data-driven decisions

Digital decision-making is rapidly evolving thanks to the rise of "super crunching." Rather than relying solely on human intuition and expertise, super crunchers analyze massive stores of data to predict outcomes and determine which choices have the greatest statistical likelihood of success. Across many fields, number-crunching is transforming real-world decision making. People have always been swayed by recommendations from others with similar tastes. In the past, this meant paying attention to critics' reviews before seeing a new film or buying a bottle of wine based on a newspaper columnist's opinion. Today, number crunching takes this to a new level of sophistication. With huge databases at their disposal, companies can now forecast individual preferences through detailed analysis of millions of other consumers' aggregated choices. This capability lies at the heart of super crunching. Some examples demonstrate super crunching in action. Many check iTunes' most downloaded songs to predict hits versus flops. Amazon's bestseller list serves a similar purpose. This exemplifies "wisdom of crowds" - large groups make better recommendations than individuals, with collective wisdom found through averaging. eHarmony offers data-driven matchmaking, first discerning users' personality types then matching with statistically compatible others based on 29 variables like emotional temperament and relationship skills. eHarmony claims its service generates 30,000 marriages annually between happier couples than those paired otherwise. Harrah's casinos use super crunching to determine a gambler's "pain point" - how much they can comfortably lose in one session yet still return. Customers' losses are tracked via swipe cards, combined with other data like income and age to calculate pain points. As customers approach their limits, "luck ambassadors" lure them away, retaining revenue versus overextending patrons. These examples only scratch the surface. More advanced systems are in development, but most super crunching relies on two key statistical techniques: Regressions - pioneered in 1877 by Francis Galton, Charles Darwin's cousin. Galton found sweet peas' size could be predicted from parent seeds, with offspring larger but regressing to the mean. Similar patterns occur in human height. With sufficient data, regressions to the mean can be precisely forecast, even quantifying the accuracy. Regressions now screen employees, as Walmart has found three personality traits better predict good fits than ability tests. Randomized trials - used since the 1940s, suggested by Ronald Fisher in 1925. Given large enough samples split into groups, traits evenly distribute. Different offers to each group reveal impacts by comparing results. Capital One, a top credit card issuer, ran 28,000 randomized test offers in 2006, dividing 600,000 prospects into groups of 100,000. Testing variables like interest rates and letter wording exposed optimal terms for maximizing responses. Super crunching requires: Vast data, gathered internally or via warehouses like Teradata, which handles major retailers' and airlines' data. Processing power, ever-improving in speed and affordability, to sift data. Tracking systems to compile and compare results. Tests only work if results are measured in context. Applications are limited only by imagination. Google AdWords makes ads compete, automatically favoring higher performing versions. Credit Indemnity's 50,000 direct mailings tested interest rates and a smiling woman's photo, finding images effectively cut rates. Capital One turns customer service calls into sales opportunities, routing callers based on past interactions to appropriate automated systems or retention specialists with tailored offers if profitable. As Ian Ayres notes, super crunching invades traditional expertise, changing decisions themselves. While conferring advantages, firms must become proactive adopters. Historical qualitative data from focus groups should be augmented with multivariate regressions and randomized trials. Companies must identify and fill information gaps via testing. In short, number crunching is changing choices - generally for the better.