Data versus Intuition Featured Pattern: P0762 April 2015
Abstracts in this Pattern:
In 2012, a team of researchers led by the Massachusetts Institute of Technology's (Cambridge, Massachusetts) Andrew McAfee and Erik Brynjolfsson conducted a study and found that the more companies described themselves as data driven, the better their financial and operational results tended to be. According to Drs. McAfee and Brynjolfsson, "Throughout the business world today, people rely too much on experience and intuition and not enough on data" (see SC-2012-12-05-089).
In recent years, the popularity of data-driven decision making has grown. As a result, demand has increased in Silicon Valley, California, for microeconomists who use detailed data analysis to solve problems. For example, Smarter Travel Media's (TripAdvisor; Newton, Massachusetts) Bryan Balin helped create an algorithm that uses data such as click speed and website-visit history to calculate the potential value of website visitors. And hiQ Labs (San Francisco, California) chief data scientist Genevieve Graves employs algorithms that analyze data to predict which workers might leave an employer and why they might do so.
The investment-finance industry's appetite for data continues to grow. Various hedge funds use data from Orbital Insight (Mountain View, California), which developed software that analyzes satellite images to make interferences about the condition of the construction industry. Investors also use data from Dataminr (New York, New York), which analyzes tweets about investment-related information on Twitter's (San Francisco, California) social network. And in agriculture, Monsanto (Creve Coeur, Missouri) is increasingly focused on helping its clients make data-driven decisions. Monsanto subsidiary The Climate Corporation (San Francisco, California) claims that its software has all US agricultural land mapped with soil and climate data to a 10-meter-by-10-meter resolution. The software can offer farmers detailed recommendations such as how much water and fertilizer they should use.
Although data-driven decision making has its limits (for example, for radical innovation), more organizational decisions likely could benefit from the systematic use of data. For example, US home builders have been steadily increasing the average size of family homes, but a few years ago, a study by University of California, Los Angeles (Los Angeles, California), archeologists and anthropologists found that much of the floor area in many houses remains unused because residents prefer to spend most of their time in certain rooms. These findings suggest that the use of data could enable the design of homes that have much less wasted space.
The Development of this Pattern
Various hedge funds use data from Orbital Insight, which developed software that analyzes satellite images to make interferences about the condition of the construction industry.
Monsanto subsidiary The Climate Corporation claims that its software has all US agricultural land mapped with soil and climate data to a 10-meter-by-10-meter resolution.
A study by University of California, Los Angeles, archeologists and anthropologists found that much of the floor area in many houses remains unused. This finding suggests that the use of data could enable the design of homes that have much less wasted space.
Data versus Intuition
Businesses are increasingly turning to data to drive their decision making.
- SoC034 — eScience (November 2003)
Scientists' race to develop the capabilities to handle data is creating a fortuitous spiral of increasing productivity. Computation- and network-intensive scientific techniques are creating practices and procedures that make up the new domain that many people call eScience.
- SoC041 — Automating Research (February 2004)
Novel developments mark a new level of automation for research processes. Automation will help scientists deal with infoglut and will enhance their ability to characterize and understand extremely complex systems.
- P0199 — Data-Mining Science (May 2011)
Advances in data mining are establishing a new science paradigm. Data-mining science applies to a wide range of application areas, but it requires some changes in approaching research.
- SoC614 — Big Science: Correlating the World (October 2012)
As the tools to collect and analyze large sets of data become more powerful, new relations and associations between variables will emerge. These relations and associations may help to explain the past and to predict trends, events, and behaviors more accurately on an ever-increasing scale.
- SoC632 — Big Data and Social Sciences (January 2013)
The growth of big data has been heralded as one of the most important developments in the recent history of science and business. In the social sciences, the ability to capture vast quantities of real-world or real-time data about human behavior seems to promise special rewards.
- SoC655 — Big Data, Big Concerns (May 2013)
Although big-data applications undoubtedly offer many benefits, organizations must watch out for potential pitfalls such as spurious analysis, lack of ethical frameworks, and privacy infringements.
- SoC747 — Small Steps in Big Data (September 2014)
Big data is making steady progress in energy forecasting, dynamic pricing, human resources, and other practical application areas.
- P0725 — Data Restrictions Limit Big Science (January 2015)
Big data can accelerate scientific progress—but only if scientists can access new data sources.