Attack of the Algorithms: Value Chain Disruption in Commodity Trading

As digital technologies become more powerful and prevalent, they continue to transform commodity trading’s value chain. They are also changing what it takes for traders to succeed.

In this article, the second of a series on the impact of digitalization on commodity trading, we detail how each part of the ­value chain, from the securitization of assets to the closeout of traded positions, is being deconstructed and reshaped. We also identify signals that managers should look for when assessing the potential impact of ­digital forces on their value chains.

The Irreversible Force of Digitalization

Traditionally, commodity traders have created value by using three primary tools to build dynamic advantage: access to supe­rior market information, control of critical assets, and superior trading capabilitdies, which stem, in part, from strong IT systems and agile and entrepreneurial traders and trading teams. As we discussed in our first article in this series, rapidly advancing digital technologies are pushing commodity markets increasingly toward hyperliquidity and putting pressure on all three of these tools. (See “Hyperliquidity: A Gathering Storm for Commodity Traders,” BCG article, November 2016.)

Information is the glue that holds value chains together, and it used to be proprietary, at least to some degree, in commodity trading. (See “The Deconstruction of ­Value Chains,” BCG article, September 1998.) Increasingly, however, information is becoming available broadly, in real time and at lower cost, reducing the competitive advantage that incumbents used to gain through access to superior information.

Similarly, control of critical assets, such as refineries and power plants, is becoming a less effective source of advantage as these assets have become securitized and thus able to be traded digitally on financial markets. Securitization of assets—including solar power, residential demand response, and oil storage—affords nontraditional traders many of the advantages of asset ownership but with greater flexibility and much less operational risk and required investment.

The value of established superior trading capabilities, in turn, is under pressure from the increasing sophistication and deployment of algorithm-based trading systems. The amount and variety of information available on subjects as diverse as weather and refinery performance is exploding. ­Algorithms can absorb this information much faster and analyze it far more thoroughly than even the smartest, most experienced team of human traders.

These forces, coupled with commodity markets’ ongoing progression toward hyper­liquidity, are fundamentally—and irreversibly—transforming commodity trading and its sources of competitive advantage.

Disruption Across the Value Chain

To understand the effects of these forces on commodity trading’s traditional ­business model, it helps to examine the three main stages of the industry’s value chain: pretrade, trade, and posttrade. (See the ­exhibit.)

Pretrade

The first stage of the value chain consists of two elements: securitization and investment decision support.

Securitization. This can take one of two forms: securitization of assets or secur­itization of contracts (also known as ­origination).

The securitization of assets can involve both aggregation (for example, the pooling of solar panels to create a virtual power plant) and disaggregation (for ­instance, the division of financial ownership of a ­facility for exporting liquefied natural gas [LNG] or storing oil). Securitization is ­highly ­vulnerable to digital forces, since both ­aggregation and disaggregation can be ­performed automatically on digital ­platforms: instead of each deal being ­negotiated by commercial departments and commodity traders, financial products can be grouped, regrouped, and traded ­instantly. This changes market dynamics by enabling more players, including small and nontraditional ones, to gain ­access to assets and to do so with very little capital, which can partly erode the ­advantage of owning and operating an asset.

The securitization of contracts involves ­creating standardized products from ­large-scale, nonstandard contractual ­agreements between two parties. Examples include agreements regarding the long-term ­off-take of LNG and structured ­investment products, including those based on energy consumption patterns. Many traders in less-­developed commodity ­markets have created a business based on the ­securitization of contracts. Their ­business model will face growing pressure as commodity markets become increas­ingly ­developed and as more short-term ­markets emerge, offering greater ­liquidity, price transparency, and ability to hedge risk. That evolution is already evident in a ­number of markets, including ­European gas.

In addition to the risks for traders, ­however, there will also be opportunities. A trading firm could, for example, build an algorithm-based platform that gives price quotations on structured products in near real time and without the need for human traders. Such a platform would appeal to sellers, buyers, and intermediaries.

Investment Decision Support. This refers to the analytical activities associated with preparing a trading strategy. It includes price formation analysis, fundamental supply-demand modeling, and the quantification of market sentiment and customer preferences. These activities are changing drastically as new quantities and types of data allow market players to understand supply and demand patterns and market sentiment much more thoroughly.

Examples abound. Gaining access to data related to commodity pricing, for instance, no longer requires tapping the broker middle­man; the data can be obtained via any number of channels. Raw data on the movement of global vessels that transport commodities is abundant and readily available. Fuel retailers can better ­understand demand patterns by installing ­micro­sensors at fuel pumps. Traders ­focused on crops can gain real-time data on crop maturation through smartphone ­videos recorded while driving by the fields.

Commodity traders also now have immediate access to new types of data that can help them understand future supply-­demand dynamics. This includes data mined from social media and online news sources, which make data on market ­sentiment, weather conditions, accidents, and other ­relevant variables available in real time and often in advance of official news releases. Indeed, the principle ­c­hallenge for traders now is not how to retrieve such information but rather how to organize and analyze it. An emerging set of busi­nesses is focused on gathering data from multiple sources, structuring it, and feeding it ­directly to trading systems. ­RavenPack, for example, follows news from ­various sources and organizes it on the basis of impact and novelty; Quandl, a data-­collection service, gathers and organizes data on prices and fundamentals, sparing traders the task of having to extract, transform, and load the data themselves.

The value of this growing abundance of information is being enhanced significantly by learning algorithms. The volume of data is simply too big for human-built analytical models to mine fully and with sufficient speed. Today’s learning systems, in contrast, can quickly crunch the numbers and readily identify patterns invisible to human ­traders. Machine learning, for example, can classify data, cluster principal components, and create decision-making frameworks based on data and algorithms ­independently, without need of human guidance.

Trade

The second stage of commodity trading’s value chain is divided into three parts: ­position generation, portfolio management, and execution.

Position Generation. This is the process of deciding which positions to take—what to buy and sell, with related decisions on quality, quantity, location, and timing. Digital technologies are transforming position generation by allowing automated decision making that is faster and more precise, and by driving down costs, including financing costs. Consider gas storage, for example. Optimizing a gas-storage position against forward prices is time-­consuming and laborious work because a company must constantly readjust its position as the market moves. Indeed, large players often deploy teams of analysts to optimize their storage strategy, yet the companies still move relatively slowly and inefficiently relative to the market. Today’s digital technologies significantly improve both the speed and the economics of executing this strategy. At BCG, for example, we built a pilot optimization algorithm that allowed hourly, rather than daily, reoptimization and automatically readjusted itself in response to the market’s rapidly changing dynamics. The result was an increase in storage value of 8% to 10%. Real-time optimization could yield even greater benefits. Further, the automation of data and information feeds would allow companies to manage such a model with a relatively small support team.

We see similar opportunities in physical production. Oil refineries, for example, typically deploy an optimization model that helps calculate the best inputs of different grades of crude given existing demand and refinery configuration. Refineries have generally run this model weekly or daily; some players have run it a few times a day. Given the rapid evolution of digital technologies, we can easily imagine intelligent systems that permit real-time analysis of the technical and commercial variables, providing managers a minute-by-minute understanding of the optimum crude blend and product output. (Making this work, however, would require changing the refinery supply team’s operating model and ensuring very tight integration of data flows.) A current example of such capabilities in physical production management can be found in power generation, where data on supply and demand can be fed automatically to algorithms that execute trades to exploit any imbalances.

Portfolio Management. Managing a portfolio involves combining, prioritizing, and sanctioning trading opportunities on the basis of an assessment of the portfolio’s value and risk. Traditional portfolio management systems typically use streamlined return distribution assumptions to calculate outcomes—but this results in overly simplified conclusions. Integrated intelligent trading systems, in contrast, allow traders to perform much more realistic, path-dependent risk assessments as well as sophisticated scenario analyses. Performing such instant and comprehensive portfolio assessments requires massive computational capacity, which is now available from cloud providers and is thus available to all companies. What’s more, computing capacity itself is being traded today, adding another dimension to the trading-optimization possibilities and illustrating the general trend in markets toward commoditization.

Execution. This is the process of selling or buying commodities, whether physical or financial. Traditionally, trading companies’ transactions would be executed by brokers who worked in banks. Increasingly, how­ever, machines are able to perform execution directly in the marketplace and to do so faster, more cheaply, and more accu­rately. They are also gaining the ability to do so in a more sophisticated manner that prevents other players from detecting the company’s intentions.

Consider, for example, an oil producer looking to hedge its risk exposure to 100,000 barrels of Brent crude oil. In the past, the company would ask its broker to execute the hedge; the broker would divide the hedge into blocks to smoothen the impact on the market and start to execute the hedge over the course of the trading day. Today, however, algorithm-based programs would sense this strategy and trade against it. As a result, any liquidity evident in the market at a given point in time might not be there when a company wants to execute the strategy—the bid-ask spread and the order book are very dynamic.

Intelligent trading systems can make a company’s execution less obvious and ­optimize the trades so that they have only a minimal effect on the market price. Such systems identify the best block size, which is often randomized, and the ideal timing for placing the hedge (for instance, when there is a high amount of liquidity in the market). These systems can also react to the behavior of other market players and do so dynamically and in near real time. A human trader could not possibly replicate this process, which explains why the ­technology is already in widespread use in such areas as foreign exchange and ­equities trading. Use of this technology is ­growing particularly rapidly in European power markets, where algorithms are ­taking over execution activities related to short-term trading. Traders that have ­begun to explore the possibilities here have been impressed by the results. To ­capitalize fully on the potential these ­systems afford, however, traders will need to make changes to their governance ­policies, which will present a large challenge to ­incumbents.

Posttrade

This is the tail end of the trading process, where the deal is settled, documented, and reported. Adoption of digital technologies is already widespread here, driven largely by trading companies’ desire to minimize costs by replacing manual work with straight-through processing. Some trading firms have also digitalized their back-office processes to improve the quality of their posttrade reports.

The use of digital technologies is also often a practical response to the increased reporting requirements driven by regulation, such as the Dodd-Frank Act, the European Market Infrastructure Regulation, and the Regulation on Wholesale Energy Market Integrity and Transparency. Regulation can also spawn opportunities. Traders that can carefully analyze trade repositories, for example, could achieve a better understanding of market dynamics.

In sum, digitalization is affecting all parts of the commodity-trading value chain. It is changing the way that information flows through it and, essentially, deconstructing the chain in the process. This progression will vary in speed and degree by market, however. How can managers determine where their market stands?

Change Signals

Digitalization of commodity trading, and the associated deconstruction of the trading value chain, can happen across all commodity asset classes, from milk to iron ore. Not every asset class will be equally exposed to digital forces, however. Market players can find valuable clues regarding the prospects for their asset class by examining product or service signals and market signals.

Product or Service Signals. These are changes in a product or service itself or in the ability to assess it. The easier a product or service is to assess, the more ripe the market is for disruption by digital forces. The first signal to look for is whether the product or service is becoming increasingly homogeneous, either because of greater enforcement of standards or more widespread assimilation of technology. Increased homogeneity as a result of these forces means that machines will play a greater role in establishing the value for a particular product or service, opening the door to digital execution of that part of the value chain. An example of this progression can be found in global transportation. As the industry has increased its use of standardized containers, capacity has become increasingly well suited to trading in a digital marketplace (though barriers such as local route duopolies might mitigate this potential).

Increasing transparency of heterogeneous characteristics—such as product quality and specifications, time and duration, and location—can also promote the digitalization of products and services because it facilitates the valuation of those characteristics. Crude oil, for example, has hundreds of qualities, and calculating the value of each can be difficult. As oil merchants and refineries improve their system capabilities, and oil producers quote crude specifications more precisely, market participants can perform netback calculations instantly and more accurately assess the value of each shipment from each refinery. This allows for rapid and automatic arbitrage among different types of crude. That, in turn, makes crude oil increasingly susceptible to digital disruption, potentially crowding out the large teams of traders and petro­chemical specialists who currently perform technical analysis on each shipment. The market for iron ore, while less mature, is heading in the same direction because market data suppliers have started offering netback analyses for different qualities of ore and steel, a first step ­toward advanced transparency.

Market Signals. These are changes in the way market participants interact when buying and selling goods, including changes in who those participants are. Companies should be looking at elements such as the capabilities of new entrants, the volume of data available to support ­decision making, and the potential for exchange-based trading.

For instance, the advent of new—typically niche—market entrants that trade in data or analytics, or conduct activities on the basis of digital capabilities, could be indications of a fundamental change in the value chain. The growing presence of hedge funds in many energy and minerals commodity classes is an example of a nontraditional player deploying a mix of capabilities that differ from those of traditional merchant traders, signaling that the winning recipe in these markets could be changing.

In many asset classes, new sources of data are changing the dynamics of supply and demand that inform decision making. For this second market signal, companies should be looking at whether new, comprehensive sources of data are becoming available or whether it is possible to link separate data sets. While the emergence of large, raw data sources does not, in itself, lead to the digitalization of trading, it does provide fuel to digital forces, assuming that they are already at play in the asset class.

The third important market signal to look for is changes in the nature of transactions themselves, including differences in the prevalence and nature of exchange-based buying and selling. Growth in the volume of trading conducted through exchanges, or greater investment in digital infrastructure, often indicates the potential for a significant increase in the role that digital forces will play in a particular segment. Electronic, high-volume, over-the-counter trading may push commodities toward expanded digitalization in the same way that exchange-based trading does, because it brings faster-paced trading, greater transparency, and wider ­access to market participants.

Knowing how to discern the relevant ­product or service and market signals can help trading companies recognize and ­adjust to digital disruption, whether active or pending, of the value chain. This can ­ultimately mean the difference between a successful transition into this new competitive environment or a downward spiral toward obsolescence.

As digital forces push commodity markets toward hyperliquidity, they are creating profound and lasting changes across the commodity trading value chain. In the next article in this series, we will discuss the strategic implications of these changes, including both the threats and the opportunities, for incumbents.