Digitization is reshaping commodity trading at an accelerating pace, and players are investing substantial resources to develop critical capabilities. Many companies, though, are moving ahead without a clear view of everything it will take to translate this investment into bottom-line results.

Capturing digitization’s full potential requires more than just a series of isolated pilot projects or the establishment of a dedicated digital division. It demands that companies develop an entirely new business model, one that starts with a strong commitment from the top of the organization and spans governance, organizational structure, information technology, and other considerations.

This is the fourth article in our series on the impact of digitization on commodity trading. The first three articles covered the gathering storm of hyperliquidity, how algorithms are disrupting the value chain, and the $70 billion prize in commodity trading.

Digitized Commodity Trading’s Development to Date

Many players started down the digitization path, albeit unwittingly, in the early 2010s (or before); they launched efforts to streamline back-office operations by standardizing contracts, introducing straight-through processing, and eliminating middle men (mostly brokers and banks) and replacing them with computer screens that allowed traders to interface directly with counterparties. These moves were not digitization of trading per se but rather cost-cutting measures aimed at staying competitive. The initiatives did, however, establish a foundation for a more pronounced digitization of the companies’ trading businesses.

The first commodity traders to experiment with systematic trading across the full commodities chain were a few experiential power and gas utilities exploring parametric trading opportunities in physical intraday markets. These players typically armed mathematicians with Matlab or Python, gave them predefined parameters, and let them trade small positions with limited risk exposure. Most of these efforts ultimately failed, partly because the companies lacked a technical understanding of the algorithms and the underlying technology but mainly because of limited support from company leadership, which often viewed the activities as curiosities rather than the beginnings of a structural shift in trading. Some commodity traders also started using algorithms to optimize the execution of hedging strategies—inspired by the success of investment banks in the equity-trading arena—but not with the primary aim of maximizing profits.

Events really started to unfold in quick succession from 2015 through 2017, with executives at commodity-trading businesses seemingly realizing the immense possibilities of digitization and subsequently pushing it down through their organizations. This led to the launch of numerous pilots across different commodities:

• On the power and gas front, exchanges such as Nord Pool opened their application programming interfaces (APIs) to clients, and several independent service providers registered with exchanges to offer algorithmic trade execution, which spared clients the trouble of building their own algorithms or testing them with the exchanges. Simultaneously, more-advanced players started developing increasingly complex augmentation algorithms—for example, ones that use machine learning to enhance forecasting of price volatility. Many players ultimately let their organizations freely build their own applications, often resulting in an abundance of applications and data sources but little coordination among them.
• On the oil front, things evolved more slowly. This happened partly because of the high trading margins afforded by the “super contango” of 2015 and 2016 (an extreme distortion in the conventional relationship between spot and futures prices, in which the latter were higher than the former), a situation that muted the perceived urgency among traders to invest heavily in digitization. In cases where players did invest, they frequently did so in relatively straightforward applications, such as the execution of hedging strategies. Although many traders also contemplated developing applications that would enable or support fundamental analysis using a data-driven approach, these discussions were often dismissed over worries about excessive complexity. But a number of oil traders ultimately set up smaller teams that were focused on accelerating the realization of digital opportunities.
• On the soft-commodities front, the market split in two. Some companies, such as Cargill, made large investments in data and data science. The majority of companies, however, did nothing. As in the oil and power and gas arenas, most players closed the era having launched many pilots, a good number of which had limited success and were ultimately abandoned or largely forgotten. Very few companies developed a systematic approach to the digitization of their business, one that would make a real difference in the company’s performance.

Since the beginning of 2018, however, the evolution of digitization among commodity traders has started to take a different turn. Rather than thinking in terms of pilots, most players are now talking about data and infrastructure. (We hear things like “Data is the new oil.”) Although this zeal and shift in focus will undoubtedly prove valuable, we would argue that, to truly make a difference in their profits, trading companies must first take a few steps back and make sure that they’re covering all their bases.

Going Forward

From this point forward, the arc that digitized commodity trading takes as it evolves will depend on two factors: traders’ ambition regarding each commodity, combined with the actions of disruptors; and the readiness of the respective trading markets for digitization.

The first factor is strongly influenced by the general profitability of trading the commodity, a consideration that is becoming increasingly important in a trading world characterized by rising competition.

The second factor is a reflection largely of the physical and trading characteristics of each commodity. Here, we consider three variables in particular: the heterogeneity of the commodity, meaning the degree to which the commodity’s physical characteristics vary; the quality and sophistication of the infrastructure supporting the trading (including the availability of electronic trading and exchange- and platform-sponsored APIs); and the “fairness” of, or transparency of information about, the market.

And these variables are interrelated: commodities that are homogenous generally have advanced trading infrastructures and a high level of information transparency. These characteristics allow for business models that are more centered on digitization. The power and gas industry is one such market and hence is at the forefront of the evolution of digitized trading; in contrast, the physical-oil market has been slow to embrace digitization owing to a combination of factors, including the market’s special terms on deals, limits on the size of parcels traded, a dated approach to bilateral-deal confirmation, and the fact that physical oil has hundreds of characteristics, such as sulfur content and viscosity. Agriculture seems to be a special case—trading, which is governed by two-sided transactions and involves long lead times, is often inefficient and could benefit materially from digitization. But numerous complexities, particularly those related to supply chains, have stymied moves by traders in that direction.

Regardless of how quickly the trading of individual commodities becomes digitized, the directional trend is clear. We see the evolution toward the Holy Grail of full digitization proceeding in four stages. (See Exhibit 1.)

The first is the relatively simple deployment of electronic platforms and straight-through processing of commodity derivatives, steps that will enhance back-office efficiency and cut direct costs. This step boosts the company’s cost competitiveness and sets the digitization process in motion.

The second stage is the adoption of automation based on parametric algorithms (that is, algorithms that are based on human-crafted rules). This enhances front-office efficiency and effectiveness by eliminating manual tasks (such as trade splitting), optimizing the timing of trades, and reducing shadow slippage (or undetected changes in the spread between bid and ask prices that occur between a trade’s inception and completion). It can also enable further automation of middle- and back-office activities, such as reporting, first-line controls, and nomination. (Some traders have attempted to deploy blockchain-based approaches to facilitate such efforts, with varying degrees of success to date.) This stage is aimed at increasing speed and efficiency and is easily tailored to compliance and risk management needs, as the rules are set by humans and the computer follows them rigorously.

The third stage is the first really transformative one: algorithms that can glean insights from vast volumes of data begin to support traders. Here, the company is no longer relying on humans to establish direct parameters or rules for the algorithms; instead, it lets the computer derive insights by itself (though the trader remains responsible for the ultimate execution decisions). This is a step toward replacing human intelligence. The highest degree of this stage is where the computer actually makes the decision (although a trader remains in the middle of the process, acting as a “qualifier”—for example, vetting the algorithm’s recommendation). In the most extreme cases observed so far, companies have employed e-gamers—because they have quick reaction times—to physically press buttons on the basis of computer guidance.

The fourth stage takes the trader entirely out of the equation and digitizes the value chain from end to end, automating everything from strategy development to back-office reconciliation. This stage involves a complete overhaul of the operating model. It is a move from a front-middle-back-office structure led by a commodity desk to a fully functional one, with almost no commodity-based siloes and a heavy emphasis on data sourcing and algorithm development.

How far will trading companies go in this evolution? After discussing the subject with most major players, we believe the consensus is that the industry will likely reach stage 3.5, on average. Intraday trading of power is already well on its way to reaching stage 4, while trading of exotic crudes might only get to an early stage 3. During an era of progressive digitization, the days of big bets, as we know them, will be over. Trading will increasingly become a function of a greater number of smaller bets, which will change the risk profile of commodity trading significantly and necessitate a new business model.

The Building Blocks of a Successful Business Model

These days, trading companies are excited about digitization—but few businesses have developed a pragmatic approach to maximizing the returns on their digitization journey. Until fairly recently, as noted, the majority of companies took a scattershot approach, characterized by multiple pilots across the organization but only limited control or big-picture logic behind it all. In most cases, this resulted in problems, including a lack of scale in the initiatives due to missing infrastructure, data, and organizational support; inefficiency, such as duplication of effort, in the pilot development, reflecting insufficient visibility for initiatives across the organization and the absence of a single guiding hand; and serious compliance and infrastructure troubles.

The right approach, in contrast, is to build the digital business model layer by layer. Seven critical building blocks should underpin this effort. (See Exhibit 2.)

Leadership and Vision

The foundation of any transformational change in an organization’s direction is leadership and vision from management. From our experience, we have found that commodity-trading executives too often consider digitization interesting and perhaps worthy of experimentation—but fail to visualize the full potential or commit the necessary resources. This can lead to participation in low-yield initiatives and what we call mushrooming, in which dozens—perhaps hundreds—of pilots are launched, most of which ultimately show limited returns or have poor scalability.

To establish a proper foundation for digitization, leadership must set the direction and lead the change management effort. We believe in a “think big, start small, scale fast” approach. This starts with the definition of an overall vision, one that spells out such things as which initiatives are important, what the key enablers are, and how the campaign will ultimately change the way the company works. After defining its vision, the company can start with a few (typically one to three) key initiatives. But it must execute them properly. Once these have been established, the organization can scale them up and begin to add more initiatives to the pipeline. Proceeding in this manner allows the company to learn its new way of working gradually and see immediate results while maintaining a holistic focus and continuing to give proper attention to its overall business.

Governance

Getting governance right is critical—as power and gas traders can attest, given the experience they had moving quickly down the digitization path without the proper organization and guidelines. The digital campaign, centered on complex, data-driven decisions made by sophisticated algorithms, presents a host of demands that are far removed from those of the traditional business, which is staffed by traders adhering to the industry’s classic operating model and mandates.

The governance challenge becomes particularly acute when the two sides clash. Governance that can effectively span the two worlds requires carefully crafted risk frameworks, incentives, and profit-sharing models, as well as properly designed organizational and trading book structures. If the company has both an “algofactory” and traditional trading desks, who owns the risk and reward associated with the algorithms if the desks deploy them? Many solutions to such challenges exist, but they must be tailored to the vision and needs of each player.

Organization and Capabilities

Commodity trading’s classic organizational structure is vertical and siloed by commodity. Digitized trading puts fundamental stresses on that structure. A vital component of an optimized digital trading setup is scale—in platforms, data, and capabilities. To enable this, companies must organize their platforms and capabilities horizontally across desks. This structure contrasts sharply with the heavy verticalization of costs and risk in traditional setups. Players headed down the digitization path often try to combat or delay the necessary change in organizational setup by adopting one of several matrix structures. But eventually, as a company gets closer to stage 3 or 4, it needs to consider—or reconsider—adopting a horizontal structure.

Another element that changes as a trading company embraces digitization is the required human capabilities. Traders who possess a commercial mindset and quick wits as well as analysts who have a strong understanding of industry fundamentals continue to be valuable assets. But companies also increasingly need data engineers, who can manage data pipelines end to end; mathematicians, who can translate complex strategies into equations; and data scientists, who can find patterns in the data. These people and capabilities must be acquired and nurtured.

Data Management

Large players have traditionally built their business models around superior access to information. Today, we are seeing a massive democratization of information availability due to third-party data providers and aggregators. As a result, virtually all traders now have access to microwave radar, frequency measurements, and other sophisticated metrics that allow them to develop a relatively thorough situational awareness of commodities in a time frame that was once possible for only the largest global players.

Hence, the challenge has shifted from data access to data management. In the future, the ability to manage—collect, clean, analyze, store, and distribute across the organization—all this data optimally will be an increasingly crucial source of competitive advantage. Developing this capability will be a challenge for all traders, especially those whose data management has historically been siloed by commodity or, in the worst cases, not organized at all.

Infrastructure

Traditionally, developing an efficient core IT infrastructure has been the critical technology hurdle for most trading companies. Although this remains the case for players at stage 2 of the digitization journey, those aspiring to move to stages 3 and 4 will likely need to augment this with an effective systematic trading infrastructure. The best systematic hedge funds have created extremely efficient back-testing, data storage, and computation infrastructure, enabling advanced data collection, processing, analytics, and order placement. Some trading companies have attempted to access these capabilities via the cloud. But many of these players have ultimately chosen to develop an in-house infrastructure solution, as cloud services have proved too slow and costly.

Research

The traditional analytical flow supporting commodity trading is grounded in fundamental market analysis supplemented by shorter-term studies (typically either continual updates of existing models or opportunistic examination of big events). When a trading company progresses toward stages 3 and 4, its research requirements change. Rather than trying to identify a few significant events in the market, the game becomes systematically finding more, smaller opportunities.

This change in emphasis has major implications for the company’s research efforts. New and different talent is needed—namely data scientists and mathematicians. The organization also needs critical capabilities in back-testing, validation, approvals, and so forth, as well as a constant review and updating of existing strategies. Further, the company must establish a pipeline of new research topics to investigate and develop because the cycle of launching new trading strategies will become shorter and shorter.

Execution

The final key building block is algorithm-backed execution of the company’s vision. This can be divided into three components: algorithms that execute trades on behalf of traders, the use of algorithms to support traders in decision making, and fully independent algorithms.

Algorithms that execute trades are nothing new—several independent software vendors and specialists provide this service to traders already. Current execution algorithms are highly sophisticated, able to execute trades on the basis of volume-weighted average prices and beyond.

Algorithms can support traders’ decision making by providing direct guidance to the trader or by performing some of the trader’s tasks, such as hedge execution or delta hedging. Using algorithms in this manner requires strict governance, however, to ensure an appropriate balance of risk and reward as well as proper compliance.

Deploying fully independent algorithms effectively and safely requires transforming the company’s operating model into a systematic trading setup, one in which everything—from data feeds to processing, analytics, and execution—is effectively one big pipeline. This necessitates a complete overhaul of the company’s governance, IT, and organizational structure.

The digitization of commodity trading is marching forward rapidly. Most trading executives recognize this, as well as the implications for future competitive advantage—and are eager to position their company optimally by adapting their business model for this new environment. As you navigate this path, keep a few key things in mind.

First, you need to define a clear vision. Cut all but one to three of the company’s current digital initiatives, keeping only those that connect most directly to that vision. In parallel, define the roadmap. Do you aspire to be a good stage 2 player or go all the way to stage 4?

Second, you must set up the appropriate governance to support the new model. And expect resistance. Establishing clear guidelines regarding testing, risk, compliance, responsibilities, and structures is a fundamental change to traditional governance and culture.

Third, it’s critical that you identify the people and infrastructure your company requires. But do not go on a spending spree. Do not hire data scientists for the sake of it; do not invest in NoSQL databases if you do not need them. Focus on your core value creation scheme and communicate that to the organization.

Digital commodity trading is a brave new world. But by making smart moves and adopting the right business model, companies can mitigate the risks and capture the upside.