Super crunchers

Super crunching techniques are infiltrating politics and medicine, where evidence-based approaches spark heated debates. These number-crunching methods even address a question long-pondered by researchers: What makes more accurate predictions - super crunchers or traditional experts?

Surprisingly, amid partisan disputes over basic facts, political parties largely agree that randomized policy trials bring value. Spectacular super crunching successes in public policy explain this consensus.

In 1993, the Labor Department's chief economist Larry Katz aimed to convince Congress that spending $2 billion on job search assistance for the unemployed would yield $4 billion in savings annually. Skeptical politicians demanded proof, so Katz launched trials in five states, assigning jobless workers to control groups receiving no aid or intervention groups getting help. The tests revealed workers with assistance found new jobs around a week faster and landed better-paying roles than the control groups. Armed with these findings, Labor persuaded Congress the savings would more than fund a $2 billion national expansion. Each dollar invested in assistance saved two dollars - a persuasive case.

Similarly, Mexico's 1997 Progresa program gives cash incentives for deprived families to keep children in school, requiring regular health clinic visits and 85 percent attendance. Over 24,000 households across 506 villages entered Progresa's program or a control group. Dramatically better health and education resulted for the program's families. Consequently, Progresa reached two million-plus families countrywide by 2002, and the World Bank now actively promotes comparable initiatives internationally. As the Bank's chief economist Paul Gertler observes, randomization "strips away roadblocks...If people make political decisions, they’ll do so despite the facts."

Super crunching refers to the use of data mining and statistical analysis to uncover insights and make predictions. It is on the rise today for several key reasons:

First, more information than ever is being created and stored digitally. From supermarket purchases to private emails, electronic records now exist for many aspects of daily life. In addition, scanning projects aim to digitize published content like books. For example, Google seeks to scan over 30 million texts. Second, governments are selling citizens' data more readily. Public and private entities aggregate and sell this information like credit reports and records. Retailers also have impressive data processing abilities and will sell customer data. Third, the internet combines databases seamlessly. Previously siloed, incompatible systems can now be integrated. Better data translation technologies also exist. Fourth, firms are equipping themselves to capture and merge digital data. Fifth, per Moore's Law, computer processing power rises while costs drop. Similarly, data storage is cheaper - a terabyte cost about $400 in 2007 versus millions for Yahoo to store daily. Sixth, scientists are developing neural networks that can sift data to make predictions. By processing massive combinations, these systems gain insights. Seventh, nanosensors embedded in products can trace usage. Paired with database advances, this offers opportunities.

As super crunching and data-driven decision making become more prevalent, many traditional roles are seeing their status decline. Rather than relying on human judgment, organizations are increasingly using expert systems and algorithms to guide decisions. Employees merely input data and follow the software's recommendations. This shift means changes must occur at the personal level. Ignoring these trends is futile - super crunching is here to stay. Instead of resisting, people should embrace data-driven decision making. This is the only way to adapt and stay competitive.

Loan officers once held significant influence, able to approve loans based on personal relationships. Today, banks use statistical models to automate lending decisions, greatly reducing loan officers' discretion and status. Employees simply enter applicant data for the algorithm to evaluate.

Similar trends are emerging in education. Previously, teachers developed creative lesson plans tailored to their classes. Now methods like direct instruction (DI) are gaining prominence. DI utilizes scripted lessons optimized for learning through data analysis. Studies confirm DI students significantly outperform peers taught via other approaches. Despite the evidence, many oppose DI for restricting teacher discretion. In response, 2001's No Child Left Behind law mandated using scientifically-validated teaching methods. Although DI meets these criteria, it has only captured 1% of the grade school market due to teacher lobbying against it. Debates around limiting instructor autonomy will likely continue.

The rise of statistical analysis and "super crunching" data will continue spreading throughout the commercial world given the immense competitive advantages it provides. Rather than resist this incoming tide of data-driven decision making, business leaders should embrace it. Becoming an active participant in this revolution by adopting basic statistical tools for your own use is smart strategy.

Specifically, familiarity with two quantitative techniques will prove helpful: the 2 standard deviation (2SD) rule and Bayes' theorem. The 2SD rule states there is a 95 percent chance that a normally distributed variable will fall within 2 standard deviations of the mean. This allows you to translate statistics into relatable numbers. For example, IQ scores average 100 with a standard deviation of 15, so the 2SD rule tells us 95 percent of people have IQs between 70 and 130.

Bayes' theorem enables combining multiple probabilities into a single figure. You simply multiply the initial probability by a "likelihood ratio" that inflates or deflates the original estimate, allowing you to update predictions as new information appears. For instance, a pregnant woman's risk of Down syndrome is calculated by combining three blood test predictors via Bayes' theorem into one overall probability.