The practice of data alchemy, which uses continually refreshed, broadly gathered data to find correlations and draw actionable conclusions, has been evolving for several years. But it was only with the pressures of COVID-19, during the “flatten” phase of the pandemic, that this approach became widespread in business. Companies found themselves coping all at once with volatility in workplace dynamics (with the rise of remote working and the cutbacks in budgets and staff), customer engagement, political circumstances, public and private investment, travel and tourism, international supply chains, and health care.

One powerful example of the value of data alchemy is itself related to COVID-19—and to the Canadian health care risk assessment firm BlueDot, which foresaw the pandemic before just about anyone else. Founded in 2013 to alert governments and businesses to potential infectious-disease outbreaks, BlueDot continually gathered the most recent data from a wide variety of sources: news media in 67 languages, reports on animal and plant diseases, blog entries, airline-ticketing statistics, and social media. It processed all this data through machine-learning algorithms that picked up correlations and patterns that would otherwise be impossible to see.

When BlueDot’s algorithms indicated unusual patterns of activity in Wuhan, China, the system automatically brought them to the attention of the company’s epidemiologists. On December 31, 2019—a week before the World Health Organization’s first official report—the company published a warning about the COVID-19 virus and its potential impact. Throughout the year, the company continued to correctly predict the spread of the pandemic around the world, projecting its rapid growth in locations like India and Brazil, months before the actual cases and deaths began to climb.

The ability to build a business on this kind of prescience was seen as unusual in the past, attributed to human genius. Now, companies in a wide variety of industries can deploy a similar ability to see unusual correlations and act on them immediately. The practice of data alchemy is grounded in a new appreciation for what analytics can do, how to use it effectively, and—most importantly—how to embed it in day-to-day decision making.

Introducing Data Alchemy

Although the sweep of change may not yet be evident on a broad scale, it is visible within many companies as they adjust the way they apply analytics. This practice of data alchemy can be summed up in a simple equation with two main components that combine to produce positive decision outcomes, as illustrated in Exhibit 1.

The first element is the AI algorithm, whose logic does not differ from that of traditional use cases. An adequate “objective function”—the formal specification of the problem the algorithm is solving—is identified to optimize decision making. The algorithm may combine different software-engineering techniques, such as natural-language programming, to address the various types of raw data with which it is fed.

The second element is data. The enterprise gathers as much data for analysis as possible, regardless of its perceived quality or usefulness. It should not be structured in any particular way; rather, the data should be raw (analyzed without any advance selection or processing), broad (from a wide and ever-increasing variety of sources), and fresh (continually updated and expanded).

This approach to collecting data is critically important because it avoids human bias and preconception. Since no one knows where the weak signals of significant events and insights will come from, there is no such thing as useless data.

For example, around 2015, the Ben and Jerry’s division of Unilever set its AI-based marketing system to track any references to ice cream in popular culture or on the web. The search seemed dubiously far-ranging and unfocused, until the company found more than 50 popular songs with lyrics referring to “ice cream for breakfast.” Suddenly, here was an idea that nobody had recognized, with a built-in market. According to a report on the marketing website Campaign, Stan Sthanunathan, Unilever’s head of insights, indicated that Ben & Jerry’s used this insight to introduce its own breakfast lineup. He said, “Two years down the road and our competitors are now doing the same.”

Anticipating Risk and Opportunity

Already, the full data alchemy approach to decision making is becoming a way of life in many companies. These early adopters of data alchemy are unlikely to go back to their old ways, and others are likely to join them. The power to anticipate upcoming changes amid turbulence, and take advantage of opportunities that would otherwise be unseen, is too great to overlook.

If business leaders expected to keep up their competitiveness, and their cash flow, they would have to make strategic choices more swiftly and with greater granularity and accuracy. Data alchemy provided that means. By comparison, legacy methods of analytics—regardless of whether they use machine learning—are more like conventional gold mining. They simply cannot take place with the same level of granularity, or at any comparable speed, and they lack the predictive capabilities that business leaders are rapidly learning to take for granted.

Other businesses have also found, in the wake of the virus, that alchemical capabilities, including those that were begun several years earlier, have given them remarkable levels of resilience. Consider, for example, the predictive analytics system that British Airways built with the Alan Turing Institute (the UK’s national data science organization). When COVID-19 struck, the airline was already prepared for one of the most sudden existential crises faced by any industry in history. Not only did airlines lose 75% of their passengers overnight, with no clear idea of when customers would return, but the old methods of scheduling flights rapidly became obsolete.

Travelers now adjust their plans constantly, borders open and close, and the rules for quarantines and social distancing keep shifting without warning. Through “dynamic forecasting,” as the Turing Institute calls its form of data alchemy, British Airways can reroute its flights and adjust its ticket prices, according not only to customers’ existing travel requests but to the probabilities of newsworthy events, responses to previous price changes, and a wide variety of other proprietary and publicly available data.

Similar forms of data alchemy have helped many other industries as well. Insurance companies have used these methods to assess the coverage risks for small and midsize enterprises (SMEs); after COVID-19 struck, insurers’ past performance metrics were no longer accurate indicators. Luxury brands have used scenarios based on data alchemy methods to manage their global inventories, taking into account such seemingly unrelated factors as online searches conducted in East Asia. Logistics firms have used data alchemy to develop higher flexibility in supply chains, an essential quality in a world where threats of tariffs and trade barriers are far more common. In all these cases, and many more, the algorithms routinely recognized and acted upon risks and opportunities that human decision makers would have missed.

Beyond AI-Enabled Gold Mining

In fairly stable and predictable business environments, gold mining was adequate even with its inherent limitations. Company decision makers knew what information they were looking for and could plan for the time needed to find and refine the data. Because the range of potential strategic paths forward was constrained, business leaders didn’t worry about what they weren’t looking for with data gold mining. Rather, they chose to analyze the nuggets of information that seemed most relevant to the critical problems at hand.

But now, in this more turbulent era, many companies have begun to use AI and big data in order to optimize their gold-mining processes. When introducing a new product, for example, they might conduct market research and customer sentiment analyses to build forecasts, with machine learning in place to refine the forecasts.

Because it takes time to select sources, data gold mining produces results more slowly than data alchemy. (See Exhibit 2.) Moreover, gold mining processes a limited selection of data that is inherently prone to bias, owing to the way it is selected. This gives gold mining a relatively low predictive capability.

Combining AI and the gold-mining approach addresses these concerns to some extent. For example, when the Mars global confectionary business partnered with the Tmall Innovation Center (the market research division of the Chinese e-tailer Tmall), Mars used an AI-enabled approach to analyze purchased data from Tmall’s 500 million users in China, intending to reduce the risk of product development failures. An analysis of historical snack purchase information alerted Mars to trends in local consumer preferences, so the company brought the Spicy Snickers candy bar to market. It was an immediate hit. But even this success represented a somewhat limited improvement compared with the greater opportunities that the company might have found with a broader sweep of data—if it had gone beyond gold mining to a full implementation of data alchemy.

Even in their use of AI, most business decision makers have relied on the gold-mining approach. But in the COVID-19 era of increased volatility, gold mining is less and less viable. Simply put, decision making based on experience—relying on past data (such as human-led promotional campaigns or machine learning trained on a well-defined earlier data set)—is now obsolete. Companies are better off practicing data alchemy and, considering the level of uncertainty and unpredictability of today’s world, alchemy is likely to be a perennial practice in many companies.

In China, for example, Ant Financial (a subsidiary of Alibaba) opted for an alchemy approach in granting loans to Chinese SMEs. The company developed an algorithm that provides a decision for each routine loan application, including the interest rate to offer. The algorithm continually refines itself, relying on a steady stream of fresh data, with billions of activity inputs replenished from more than 3,000 sources of activity throughout the Alibaba system, including popular e-commerce platforms like Taobao. The percentage of defaults is less than 1%—lower than a firm would obtain with a gold-mining paradigm—and will presumably drop even further over time as the accuracy of the algorithm continues to improve.

Ant Financial has also adopted data alchemy as the core logic of its business model and the source of its competitive advantage. By promoting its 3-1-0 model directly to customers (3 minutes to fill in the application, 1 second to get an answer, and no time after that to receive funds, with zero human intervention), the company has reached the point where it holds loans for more than 50% of all Chinese SMEs.

Had Ant gone with a traditional gold-mining approach, it would have based its loan-granting decisions on analysis (by humans, software, or machine learning) of the past credentials and forecasted business outlooks for each applicant. This analysis would not be reliable, because of the rate and magnitude of change in business operations today, particularly in relation to COVID-19. Past performance is not a predictor of future financial situations.

The practice of data alchemy, though it may seem unfamiliar at first, is accessible and scalable. It is not just the province of digital natives. On the contrary, a growing number of traditional companies are making the shift and updating their decision-making processes accordingly.

For example, traditional insurance companies increasingly rely on data alchemy when underwriting policies. Why? Because—as highlighted by the pandemic—the traditional analyses of potential business losses, using selected data such as past years’ financial results, are inadequate. Alchemy has enabled insurers to make extensive changes in their internal decision-making processes: for example, they can streamline multiple decision layers into a more rapid, seamless process. It also changes their externally facing procedures, such as the handling of customer interactions.

A Stronger Role for People

When AI is introduced, concerns often arise regarding the implications for people. Exhibit 3 depicts the function of human decision makers in the data alchemy equation. People play four critical roles: building the data ecosystem, designing the algorithm and the process around it, selectively monitoring the output for bias and other anomalies, and managing exceptions flagged by the system.

Data Architects. People play a fundamental role in building the digital ecosystem to gather as much data as possible. They set up an architecture that ensures a constant flow of fresh data from the widest possible array of sources. These include a variety of external wellsprings: publicly accessible data from internet searches, purchased-access databases and information marketplaces, and proprietary records shared by other companies within a data-sharing ecosystem. Some companies also have internal sources, resident within a firm’s own data lake and analyzed together with external data.

Algorithm and Process Designers. The skills of AI specialists, already highly sought-after in the labor market, are particularly relevant to data alchemy. The design of the algorithm is key to success. And this is only one part of a broader decision-making process that must be entirely reinvented with AI at its core. Only skilled human experts, with technical and business capabilities at their disposal, have the knowledge and creativity to design and implement an alchemical system.

Sentinels. The companies benefiting the most from AI are those where humans and machines learn from one another. That was the primary conclusion of Expanding AI’s Impact with Organizational Learning, the 2020 annual study by the BCG-MIT Artificial Intelligence Global Executive Study and Research Project. Hence the need for sentinels: data scientists who continually pay attention to algorithms in order to investigate suspected biases and make adjustments when required. For example, sentinels may detect cases where the algorithm consistently produces outputs that have a low level of accuracy or where it lacks insight about the appropriate context.

One such situation occurred when a European apparel company used data alchemy to plan its January sales campaign. The algorithm didn’t discount Christmas-themed sweaters because they had been very popular during the previous month. People on the marketing team understood the context that the algorithm did not: the holiday market is only temporary. Once humans noticed this problem, they rapidly adjusted the algorithm to account for this context in all future holiday sales.

Exception Managers. Some executives are concerned that data alchemy will force them to relinquish control over their decisions. In fact, the opposite is true. Data alchemy frees up their attention for the decisions that require in-depth consideration. The algorithm makes all routine choices, but whenever it flags something (such as a low level of confidence in an outcome or a decision with major strategic implications), human managers step in and set the course, using their expertise and intuition.

For example, in underwriting policies, an insurance company’s algorithm worked mostly autonomously. It would flag cases for escalation only when the margin of error rose above a preset threshold or the amount of money at stake exceeded an agreed-upon financial value. For the human underwriter, the benefits are clear: less time on routine but necessary busywork for simple claims and more time to investigate the complex, challenging cases and consider their strategic implications.

Becoming an Alchemist

In the years to come, the level of uncertainty and unpredictability in the business environment is unlikely to decrease. Data alchemy will not be a short-lived trend. Because it enables faster, more granular, and more accurate decisions, it is bound to lead to permanent changes in decision-making mechanisms—both for incumbent companies and for digital natives.

So where should you start? Across your organization, identify important decisions where the level of accuracy has recently dropped, owing to the unprecedented speed and magnitude of the change you have been facing. This is a key indicator that a shift to data alchemy would be valuable. If you also sense that you don’t have all the elements in hand to make a sound decision anymore, this is an additional red flag. Bear in mind: if you identify this need, your competitors will likely do so as well. And because data alchemy can become a source of competitive advantage, it’s urgent that you act before others do.

The BCG Henderson Institute is Boston Consulting Group’s strategy think tank, dedicated to exploring and developing valuable new insights from business, technology, and science by embracing the powerful technology of ideas. The Institute engages leaders in provocative discussion and experimentation to expand the boundaries of business theory and practice and to translate innovative ideas from within and beyond business. For more ideas and inspiration from the Institute, please visit our website and follow us on LinkedIn and X (formerly Twitter).