The Self-Tuning Enterprise

Wouldn’t it be nice if an algorithm could tell you when to develop a new business model or whether to enter a new market?

We’d be lying if we said that such an algorithm exists. It doesn’t, and we don’t imagine a time in the foreseeable future when algorithms (or other forms of artificial intelligence) will be able to answer such difficult strategic questions. But we do believe that something almost as interesting is emerging: a way for organizations to apply algorithmic principles to make frequent, calibrated adjustments to their business models, resource allocation processes, and structures—without direction from the top.

That’s a provocative claim, but it’s based on actual developments we’ve observed at Internet companies like Google, Netflix, Amazon, and Alibaba. These enterprises have become extraordinarily good at automatically retooling their offerings for millions of individual customers, leveraging real-time data on their behavior. Those constant updates are, in fact, driven by algorithms, but the processes and technologies underlying the algorithms aren’t magic: it’s possible to pull them apart, see how they operate, and use that know-how in other settings. And that’s just what some of those same companies have started to do.

In this article we’ll look first at how self-tuning algorithms are able to learn and adjust so effectively in complex, dynamic environments. Then we’ll examine how some organizations are applying self-tuning across their enterprises, using the Chinese e-commerce giant Alibaba as a case example.

Dynamic Markets Demand Dynamic Management

Before we dig into those algorithms, let’s consider why many companies need a new approach to strategy and organization.

In recent years, technology has introduced unprecedented change and uncertainty to markets. It’s no surprise that CEOs talk so much about agility and adaptation. Less remarked upon is the uneven effect of those forces, which also makes markets more diverse than ever. Established, stable businesses in the developed world must be managed alongside young, unpredictable ones in the developing world, and fast-evolving tech-based businesses alongside slow-moving cash cows. To deal with this diversity, companies need to tailor their approach to strategy and execution to each environment in which they operate. And because of the increasing pace of change, they need to constantly “retune” this collage of approaches. But it’s simply not feasible to manage all these shifts using traditional top-down, deliberative decision making. The ability to adjust needs to be woven into the fabric of the enterprise. (See the sidebar “Crafting Strategy in Disruptive Times.”)

And that’s where self-tuning comes in. Self-tuning is related to the concepts of agility (rapid adjustment), adaptation (learning through trial and error), and ambidexterity (balancing exploration and exploitation). Self-tuning algorithms incorporate elements of all three—but in a self-directed fashion.

The algorithms in the recommendation systems at Amazon and Netflix are iconic examples. To be effective, they have to strike the optimal balance between exploiting known preferences and exploring different items that have the potential to surface new preferences. Serve up too many safe bets, and users will get bored, and the companies will forgo the opportunity to collect data that will inform new recommendations. Serve up too many exploratory options, and users may become dissatisfied and lose faith in the recommendations. To manage this trade-off dynamically, the recommendation engines constantly update their suggestions, drawing on an evolving understanding of users.

These algorithms operate on the basis of three interlinked learning loops.

Experiment: discover what works. First, this means generating new options. A growing library of choices is vital to a system’s success. Second, it means testing options economically: providing recommendations that are based on knowledge of content and customer behavior, but with a degree of randomness or stretch to avoid getting stuck in a rut. Third, it means amplifying what works. The systems track click rates, purchases, and ratings to learn more about personal preferences and then use that information to improve future recommendations.

Modulate: adjust how and how much you experiment. Self-tuning algorithms learn not just from trial and error but also by adapting the rate of experimentation to the environment. In other words, the experimentation machine modifies itself as it proceeds. With new customers, higher rates of experimentation are necessary to unearth what they do and don’t like. This can be scaled back as the algorithms learn more about them. However, experimentation should never drop to zero, because all users should experience a degree of exploration and surprise.

Shape: influence preferences. Much of the delight of recommendation engines comes from discovering products and content you would not have otherwise found. Being directed to a new category or product both reveals and shapes what a user finds interesting. It plays the same role that advertising does in traditional marketing—not only reinforcing existing preferences but also creating new ones.

Critically, these three loops are all executed in a self-directed manner, without any human decisions. No analyst has to handcraft recommendations, individually interpret user feedback, adjust the exploration rate manually, or deliberate about the best new options to test out on users. And that allows self-tuning systems to operate at very high speeds.

Reinventing the Enterprise

As the companies pioneering self-tuning algorithms grow and mature, they increasingly face the challenge of running versus reinventing themselves—and not just in the marketing department. No surprise, then, that some are introducing new managerial practices that extend self-tuning principles across the entire enterprise.

To understand how this works, think of the enterprise as a nested set of strategic processes. At the highest level, the vision articulates the direction and ambition of the firm as a whole. As a means to achieving the vision, a company deploys business models and strategies that bring together capabilities and assets to create advantageous positions. And it uses organizational structure, information systems, and culture to facilitate the effective operation of those business models and strategies.

In the vast majority of organizations, the vision and the business model are fixed axes around which the entire enterprise revolves. They are often worked out by company founders and, once proved successful, rarely altered. Consequently, the structure, systems, processes, and culture that support them also remain static for long periods. Experimentation and innovation focus mostly on product or service offerings within the existing model, as the company leans on its established recipe for success in other areas.

The self-tuning enterprise, in contrast, takes an evolutionary approach at all levels. The vision, business model, and supporting components are regularly calibrated to the changing environment by applying the three learning loops. The organization is no longer viewed as a fixed means of transmitting intentions from above but, rather, as a network that shifts and develops in response to external feedback. To see what this means in practice, let’s look at Alibaba.

Started in 1999, the Alibaba Group initially focused on building a B2B website for small Chinese manufacturers. But in the years since, it has expanded its portfolio in many directions. Today, the group spans ten businesses and has approximately 27,000 employees and more than $8 billion in revenues. In China’s fast-changing e-commerce market, it could not have achieved that level of success without constantly retuning the enterprise at all levels. The way Alibaba went about this suggests a number of guidelines for other organizations.

Keep resetting the vision. When Alibaba began operations, Internet penetration in China was less than 1 percent. While most expected that figure to grow, it was difficult to predict the nature and shape of that growth. So Alibaba took an experimental approach: At any given time, its vision would be the best working assumption about the future. As the market evolved, the company’s leaders reevaluated the vision, checking their hypotheses against reality and revising them as appropriate.

In the early years, Alibaba’s goal was to be “an e-commerce company serving China’s small exporting companies.” This led to an initial focus on Alibaba.com, which created a platform for international sales. However, when the market changed, so did the vision. As Chinese domestic consumption exploded, Alibaba saw an opportunity to expand its offering to consumers. Accordingly, it launched the online marketplace Taobao in 2003. Soon Alibaba realized that Chinese consumers needed more than just a site for buying and selling goods. They needed greater confidence in Internet business—for example, to be sure that online payments were safe. So in 2004, Alibaba created Alipay, an online payment service. By providing both an escrow service and a merchant-rating system, Alipay introduced the ingredients for transparency and trust, which sped up the penetration of e-commerce in China. Ultimately, this led Alibaba to change its vision again, in 2008, to fostering “the development of an e-commerce ecosystem in China.” It started to offer more infrastructure services, such as a cloud-computing platform, microfinancing, and a smart-logistics platform. More recently, Alibaba recalibrated that vision in response to the rapid convergence of digital and physical channels. Deliberately dropping the “e” from e-commerce, its current vision statement reads simply, “We aim to build the future infrastructure of commerce.” By regularly retuning its vision, Alibaba has been able not only to respond quickly and effectively to new market realities but also to shape the way consumers and businesses interact.

Experiment with business models. Alibaba could not have built a portfolio of companies that spanned virtually the entire digital spectrum without making a commitment to business model experimentation from very early on. However, when the firm first ventured beyond its core B2B e-commerce platform to launch Taobao, the decision was hotly debated within the company, as it involved going head-to-head with an apparently almighty eBay. To minimize distraction at the budding B2B business, Taobao was set up as an independent company, with a separate office (or in this case, apartment) and separate funding (a 50/50 joint venture between Alibaba and SoftBank).

At each juncture in its evolution, Alibaba continued to generate new business-model options, letting them run as separate units. After testing them, it would scale up the most promising ones and close down or reabsorb those that were less promising. In 2006, for example, spotting two new trends, Alibaba decided to launch two units. To tap the growing B2C market, it began building Taobao Mall, a platform for established brands to reach Chinese consumers, which eventually became Tmall and is a major part of the group portfolio today. To catch the software-as-a-service wave, it started Alisoft, which probably entered the market too early. Alisoft could not find a killer app that generated enough customers. The business was shut down in 2009. 

Another driver of Alibaba’s success has been the ability to modulate its rate of business model experimentation to fit the circumstances. For example, within just four years of launch, Taobao had captured more than 80 percent of the digital Chinese consumer market, and by 2011 it had become a national phenomenon. Many companies would have taken this leadership position as validation and focused on optimizing the successful model. Instead, Alibaba saw the sustained rapid growth of China’s online population and the increasing sophistication of consumers and retailers as signals of great uncertainty in the marketplace and a risk to the current model.

Again there was heated debate within the company about which direction to take and which model to build. Instead of relying on a top-down decision, Alibaba chose to place multiple bets and let the market pick the winners. In 2011 the company split the very successful Taobao into three independent businesses. Each took a different view of the future of e-commerce in China. Taobao focused on consumer-to-consumer transactions, Tmall on business-to-consumer transactions, and Etao, a new unit, on product search. Although the outcome might have been the dominance of any one of the models, Alibaba actually succeeded in creating two successful mass-market businesses (Tmall, which has a 60 percent market share in the fiercely competitive B2C market, and Taobao, the market leader in C2C) and one strong niche-market model (Etao).

Increasing experimentation at the height of success runs contrary to established managerial wisdom, but for Alibaba it was necessary to avoid rigidity and create options. Recalibrating how and how much to experiment was fundamental to its ability to capitalize on nascent market trends.

Focus on seizing and shaping strategic opportunities, not on executing plans. In volatile environments, plans can quickly become out-of-date. In Alibaba’s case, rapid advances in technology, shifting consumer expectations in China and beyond, and regulatory uncertainty made it difficult to predict the future. To deal with this situation, Alibaba adopted a continuous process of “replanning.” Rather than meticulously executing a fixed, detailed blueprint, the company keeps revising its strategy and tactics as circumstances change.

Alibaba does have a regular planning cycle, in which business unit leaders and the executive management team iterate on plans in the fourth quarter of each year. However, it’s understood that this is only a starting point. Whenever a unit leader sees a significant market change or a new opportunity, he or she can initiate a “co-creation” process, in which employees, including senior business leaders and lead implementers, develop new directions for the business directly with customers.

ALIBABA'S EVOLUTION

At Alibaba co-creation involves four steps. The first is establishing common ground: identifying signals of change (based on data from the market and insights from customers or staff) and ensuring that the right people are present and set up to work together. This typically happens at a full-day working session. The second step is getting to know the customer. Now participants explore directly with customers their evolving needs or pain points and brainstorm potential solutions. The third step entails developing an action plan based on the outcome of customer discussions. An action plan must identify a leader who can champion the opportunity, the supporting team (or teams) that will put the ideas into motion, and the mechanisms that will enable the work to get done. The final step is gathering regular customer feedback as the plan is implemented, which can, in turn, trigger further iterations.

The co-creation process highlights the self-directed nature of self-tuning enterprises. Alibaba’s business units can initiate co-creation sessions whenever they see a relevant market stimulus, without any central mandate or oversight. And although the process now follows a successful pattern, each co-creation initiative is tailored to the situation at hand. By creating a forum for regular exchange with customers, Alibaba is able to evolve synchronously with the market. The approach leaves more room for innovations to bubble up from the market, as opposed to being pushed down from the top of the organization. Leadership in effect stops managing something that is better left to a market-driven mechanism.

Get good at adapting the organization. At Alibaba, maintaining organizational flexibility is an area of intense focus. A few lessons stand out from its experience—for instance, that it’s critical to build an expectation of change into the culture from the outset. “Embrace change” has been one of the company’s six core values since its early years. Its founder and chairman, Jack Ma, regularly emphasizes this theme with employees, investors, and customers. “In the information era,” he says, “change is the best equilibrium. No single organization structure is perfect and can solve all problems.” That mind-set has become a pillar of Alibaba’s recruiting. The company evaluates potential hires not only on their technical skill but also on a demonstrated ability to thrive under conditions of rapid change.

Another important lesson is that change must be actively pursued, not merely tolerated. In traditional companies, organizational change is often executed through infrequent, large-scale initiatives. An enterprise that regularly retunes itself, in contrast, limits the need for such risky, one-shot transformations.

Consider a new program Alibaba experimented with in 2012, which rotated its top 22 managers across its broad portfolio of businesses. There was some concern about disrupting the continuity of operations, but the program proved successful, partly because managers were required to institutionalize and transfer knowledge. In addition to enhancing the skills of top talent, the program showcased the leadership’s commitment to flexibility. Judged a success, the program continues: a portion of the senior leadership is rotated every year.

Build systems that support fluidity and feedback. In its early years, Alibaba used a leading enterprise-resource-planning system to manage information flows and resources. But over time it became clear that ERP was straitjacketing rather than facilitating change. It was designed for a traditional stable structure with clear unitary reporting lines, and adjusting it to reflect an evolving structure was taxing. The company needed something more dynamic.

An Alibaba team set out to create a better alternative by expanding the functionality of the company’s employee portal. As an initial step, the team built functionality that allowed employees to nominate themselves for promotions. This gave them more control over their own career trajectories. Next, the team added a Web-based interface through which HR executives could update organizational structures and links to all employee records easily at the same time. This helped internal HR systems and processes keep up as operational teams frequently realigned themselves to market needs. Then the team created a more flexible goal-setting interface that improved the performance review system. Instead of static annual goals, employees can set goals with different time frames for different projects and align their objectives with those of colleagues outside their units. It’s also easier now to give feedback to coworkers through the interface.

While Internet natives such as Alibaba have been the pioneers of the self-tuning enterprise, the emerging lessons are relevant for a broad array of companies, in both digital and more traditional industries. Digital pioneers will need to reinvent themselves in the face of unrelenting technological change, but more-traditional industries will face complex, dynamic environments, too. Technologies such as two-sided marketplaces, which help both to create and to exploit such environments, are spreading well beyond the digital sector. By borrowing from the playbook of current experimenters, companies can find a way to keep up with unpredictable markets—and even get ahead of the curve.

A version of this article appeared in the June 2015 issue of Harvard Business Review. It is republished here with permission.