Code Wars: The All-Industry Competition for Software Talent

Few technologies have created more value than software. Almost every industry today depends on software in some way, and for countless companies, it plays an essential role in designing products, analyzing data, running facilities, and managing customers—to name just a few functions. Software talent, as a result, has become critical to success in most industries today.

Consider some examples. Ford Motor’s leaders have said that the company’s most critical positions are in software and systems engineering, pitting the company directly against its technology partners in a competition for software talent. ServiceNow, a cloud-based IT services provider, is headquartered in Silicon Valley and maintains a major presence in its first home of San Diego because, in the words of CEO Frank Slootman, “These days, talent doesn’t move much anymore, and employers need to set up shop where the concentrations of talent are.” The Chevy Volt runs on 10 million lines of code—2 million more than the F-35 fighter jet. This level of software sophistication is far more the rule than the exception these days. The Boston Consulting Group’s most recent report on value creation in the technology, media, and telecommunications industries highlights the critical role of software. (See The 2013 TMT Value Creators Report: The Great Software Transformation, BCG report, December 2013.)

In new industries and old, software has become a critical determinant of success. The big questions that company executives of all kinds need to be asking themselves are, “Who is writing all that code?” and, more to the point, “Who is writing it for our company, and how do we make sure that we maintain access to the talent we need?”

To get a handle on the extent of the challenges that companies face in an age when, as venture capitalist Marc Andreessen puts it, “software is eating the world,” BCG undertook a major study of the demand for, and supply of, software talent in the United States. The project included both top-down and bottom-up assessments. The first assessment used data from the U.S. Bureau of Labor Statistics (BLS), among other sources. The second included analyses of the employment landscape from well-known research firms and employment-related companies and networks, as well as our own proprietary survey of 1,000 executives who are familiar with software development in a wide variety of high-tech and non-high-tech industries.

This report presents our findings and suggests the steps that companies need to take to make sure that they are well positioned to win in the increasingly intense and complex battle for software talent.

A Growing Imbalance

Large imbalances in the demand for, and supply of, software talent exist today. They are the result of multiple factors, including the sheer volume of code, growing demand and mounting technical requirements for code and for technologies that are tailored to specific applications, the rising importance of intellectual property (IP) at all kinds of companies, and increasing complexity in the way code is used.

More Code. Several laws of software growth are ascribed to Nathan Myhrvold, a former chief technology officer at Microsoft, and they all arise from a common underlying point: software will expand to fill the space available to it. The rise of the industrial Internet and machine-to-machine connectivity, the increased prevalence of cloud computing, and consumers’ seemingly insatiable desire for mobile connectivity have led to more and more pervasive software usage in recent years—and new sources of demand will keep driving usage in the future. While software was once something that the IT guys dealt with, today it is integral to myriad business functions. New software-intensive technologies—additive manufacturing (also called 3-D printing) the Internet of Things, and robotics, for example—are invented seemingly every day, but older technologies, such as payroll automation, hang around, too. They still work fine; they can be tweaked and improved. The lines of code keep multiplying.

More Differentiation. Software, once just a part of a company’s value proposition, is turning into the key differentiator. Companies in industries from automobiles to textiles, and from utilities to apparel, are using software to enhance product design, cut costs, reduce development time, and improve speed and agility. Enhancing customer “stickiness” with customer-relationship-management programs, for example, is a software-driven function. Software programs help determine how to offer the right products and services at the right time and how to create new business models, converting a product into a service, or a sale into a subscription. All the hype, and indeed the value, associated with big data is possible only because of the underlying software technology. Netflix committed $100 million to produce its hit series House of Cards for reasons that included insights gained through algorithmic programming and from the Apache Cassandra database about the company’s customers, the original British series of the same name, Kevin Spacey (the U.S. star of the series), and David Fincher (the first U.S. director of the series).

More Strategic Advantage Through IP. As BCG has recently pointed out, smart companies are increasingly using IP, including software, as a means of establishing competitive advantage in the marketplace. (See The Most Innovative Companies 2013, BCG report, September 2013.) Protecting IP rights—that is, maintaining exclusive ownership of a product or process—is also an important defensive strategy. The recent avalanche of high-profile patent cases and patent sales, mainly in the technology and telecommunications sectors but also in others, has made it clear that innovation depends, in part, on owning an idea. Strong innovators are more than twice as likely as their weaker counterparts to consider IP criteria when deciding which new product ideas to push forward. They are more likely to use IP as a source of competitive advantage.

And, Ultimately, More Complexity. Software applications bring many benefits, but they also add complexity and challenges for management. Software often needs to be tailored to individual functions and uses. Cloud computing and big data are transforming companies. Market leaders, many in traditional industries, such as manufacturing, consumer products, and financial services, are investing heavily in technical talent. Can others afford not to follow suit? Building new capabilities requires many kinds of software expertise. Interoperability—making both new and old functions work together—adds another layer of software-rich complexity. Daily headlines on security and security breaches underscore another major challenge for companies holding valuable digital data.

The Increasing Demand for Talent

These trends present companies, especially those in nontech industries that nevertheless need tech talent, with multiple challenges. If such companies cannot attract and retain the software expertise they need with current recruiting and retention efforts—and many cannot—they risk falling behind in technological capability and product development. New and in-demand technologies are harder to recruit for but are still necessary to win, further exacerbating the threat of stagnation. Maintaining legacy code will become increasingly difficult because skills gaps are widening and many software developers are retiring. Yet the use of legacy language programs shows little sign of abating.

Companies also face the challenge of determining the skills they need for their strategy and product development. They need to build the capability, if they have not already done so, to translate corporate and product strategies into resource plans, specifically with respect to software talent. The product strategy, for example, defines the technology and software architecture needed to keep new products moving through the development pipeline, including the required mix of software languages and frameworks (reusable platforms for developing software applications, products, and solutions). The resource plan is the resulting roadmap for getting the right talent in the right places at the right times to enable the software work required to drive the product strategy.

None of this is remotely simple or easy in a highly competitive and complex environment for talent. It all presents entirely new challenges to wide swaths of the economy. How does an industrial-goods company find the talent it needs, for example, when management lacks the expertise to assess the skills required and when potential hires are concentrated in places where the company has no presence, such as Silicon Valley or India?

There were more than 1 million software developers employed in the United States in 2012, according to BLS. The median pay for those developers was $93,250—nice work if you could get it, and get it you almost certainly could: demand for software engineers in the U.S. outpaced supply by some 35,000 positions. Demand is expected to grow at more than 20 percent per year through 2022—a sixfold increase.

Fast-rising demand is one problem, but from a management standpoint, the issue is far more complex. The types of demand for software talent vary greatly depending on the applications, language skills, and frameworks that companies are looking for and the corporate roles they are seeking to fill.

Applications. In terms of types of applications, the largest shortfalls are in security, enterprise applications, and systems-networking and storage engineers. The security field presents an especially daunting challenge: some 200,000 software security positions in the U.S. are currently unfilled. Meanwhile, there is a surplus of nearly 200,000 employees working with software architecture in the data and database layers and in data management (See Exhibit 1.)

Within security, there is a particularly high shortage of experts in network security (Web and enterprise firewall, for example). Rapid growth in the popularity of cloud computing and mobile connectivity has created a host of new IT security concerns. The emerging practice of bringing your own device to the workplace requires additional end-point security layers, such as code signaling and code protection mechanisms. Agile software development places even more stress on security analysis, requiring robust source-code management. It’s one thing to be able to fix a product’s problem with a software patch distributed over the air, as Tesla Motors has done; it’s another to be sure that the transmission of the code—and the receiving cars’ onboard computers—cannot be hacked or otherwise interfered with.

These challenges are emblematic of the kind that all sorts of companies face in keeping up with demand for software expertise, and nowhere are those challenges as acute as in software security. In April 2014, a security-testing company discovered the Heartbleed bug, a vulnerability in OpenSSL—the technology used by up to two-thirds of the world’s websites, including many that store passwords, personal files, bank details, and even Social Security numbers. The discovery illustrates how pervasive network security issues have become and how exposed the world’s data might be without the right talent to address the risks.

Languages and Frameworks. Hundreds of software languages and frameworks are in use today, and proliferation is increasing as new software technologies are developed to meet more specialized needs. Languages and frameworks vary substantially in complexity and function. Languages range from those that are low level, such as C and Embedded C++, which are written to operate certain types of systems or applications, to high-level languages, such as Java and Python, which are designed for specific purposes and to solve problems. New languages and frameworks are tailored for individual applications. For example, Embedded C++, a derivative of C++, is designed to address specific shortcomings of C++ with respect to embedded systems—reducing the “bloat” on compiled code significantly and allowing the code to run efficiently on limited memory and 32-bit systems. Some languages, such as FORTRAN and COBOL, have been around for decades and are still in widespread use. Others are only a few years old but power some of the most prolific applications in use today.

Object-oriented programming languages are more prevalent than others right now. Java, C++, and C# are most popular for coding, though Java is widely used for browser applications as well. Perl, PHP, and Python are most used for scripting, while PL/SQL, XQuery, and JQuery are widely employed for database programming. HTML and C are the most popular functional languages. Many developers can work in multiple languages and frameworks: on average, software developers in our survey knew six languages at an “intermediate” or “above skill” level and three at an “expert” level. But the demand for such talent in multiple areas keeps growing faster than the supply. Approximately half the software developers today view skills in three areas as the most important: SQL, Java, and JavaScript. Another third include skills in .NET, C++, and Oracle as well. SQL, HTML, C++, and C# skills are in oversupply; the largest talent shortages are in C, Objective-C, JQuery, and Visual Basic. Another 25 or so languages are also still widely used. (See Exhibit 2.)

Disparities in language skills lead to further problems. Many companies still use the longtime programming stalwarts COBOL and FORTRAN. As of two years ago, one major bank was running 100,000 COBOL programs aggregating some 3.4 million lines of code. What’s more, a Computerworld survey at about the same time found that more than half the companies surveyed were still developing new COBOL programs. Older developers, who work in such legacy languages, are reaching retirement age. (COBOL and FORTRAN predate the formation of the Rolling Stones; about one-third to one-half of COBOL and FORTRAN programmers are at least 50 years old.) Younger developers work primarily in newer languages, creating an impending skills gap for many businesses. At the same time, languages and frameworks such as Apache Hadoop and Objective-C are still in their relative infancy. Finding developers who are both fluent in these languages and sufficiently experienced to fill senior positions can be tough. Most Hadoop and Objective-C programmers, for example, are too young to have acquired significant management experience. (See Exhibit 3.)

Complicating matters further, the supply of generic-software developers and software program managers outpaces the demand right now, but there is a shortfall for product managers, quality assurance testers, and architects.

Managing Supply

Ensuring an adequate supply of the right kind of software talent will be a challenge for most companies in both the near term and the longer term. Managing demand imbalances will, of course, be one issue. But anticipating future needs in a fast-changing field is also difficult and is one reason why we undertook this research, which we plan to update periodically.

Part of the supply problem is that universities are struggling to update their curricula quickly enough to keep up with the rapid pace of change. Graduates entering the workforce are not prepared to fill gaps in skills and talent. Training and skills development, by definition, take place in advance of employment; some lead time is inherently essential. Geographic concentrations of resources creates additional difficulties for employers in less populous areas. And, of course, the most talented human resources are attracted to high-tech companies and startups. Silicon Valley is rife with stories of the need for top talent. Nine out of ten MIT graduates with computer science degrees in 2012 went to work for high-tech companies or start-ups.

Companies will want to make sure that they are getting the most out of the talent they have on board and that they are able to recruit. In the near term especially, their best bet may be making the most of the talent they already have.

Training and retraining—with a focus on languages, frameworks, platforms, and applications where supply is short—will rise in importance. Cross-training employees who may already know the fundamentals of another language can be a short-term way to fill gaps in key capabilities. Keeping software skills honed is likely to be a continuing need over the long term, although the nature and focus of the training programs should evolve over time to meet changing programming needs.

Providing attractive career paths for software talent will be another priority. Companies that don’t address this need will find themselves increasingly short of talent, especially for more senior and management-level positions. Intensifying training and telescoping career-trajectory paths (not unlike the military’s approach to battlefield promotions) will help minimize gaps in more senior positions (albeit at the sacrifice of real-world experience). There will also be a rising premium on productivity. Smart companies will develop processes that take full advantage of the benefits that software can bring them. Companies with poor processes will be doubly punished—first by failing to make the most of existing resources and then by losing much needed skills when frustrated talent decides to move on.

Securing the Talent You Need

Fast-rising demand, an increasingly complex mix of programming skills, and intense competition from the information and communications technology sector mean that companies in traditional industries are unlikely to solve their software talent needs without a well-planned strategy to guide acquisition, development, and retention—as well as the commitment of significant resources. Companies need to plan for, and manage, software talent through a combination of approaches.

Strategic Skills Planning. Companies need to develop a strategic view of the technologies required to realize their corporate and product strategies and translate those requirements into human resource capabilities and skill sets. Software talent should be regarded as a precious resource that is essential to realizing a broader corporate strategy. Managements must plan for the life cycle of their talent, including mapping patterns of normal attrition, determining when people with certain skills will be leaving their workforce, projecting future hiring needs, and developing programs to attract and retain new people. This undertaking must be fully integrated with overall strategic-planning efforts and monitored and updated on a continual basis. The difficulty of the challenge is compounded by the supply-demand imbalance in many areas and the fact that software skills are easily transferred across multiple industries.

Some companies will want to experiment with new models for key functions to make the most of available talent. Manufacturers are already using community collaboration models in the product design and development stages, tapping into external and internal open-source and crowd-sourced solutions to solve design challenges. New methodologies, such as Agile, are increasing productivity for certain types of software development. Companies of all kinds should consider collaborative techniques to drive innovation and to attract young engineering and design talent familiar with rapid, iterative, and open approaches. It is important both to hire these new employees with appropriate lead times and to plan career paths for them—including in technical and management tracks. Companies need to provide ongoing career training and integrate technical capabilities into the broader capabilities of the business.

Increased Use of the Global Talent Pool. Talent is a global commodity—sometimes. It is akin to carbon, which can take the form of ordinary coal or of precious diamonds. Specific skills can be highly valued and demand a premium. Companies will need to access remote centers of capability, especially when they need certain hard-to-find skills or a particular capability for a limited period of time. Multinational companies already have resources spread around the world. Automakers, for example, have been investing in more integrated product-development IT systems that will enable global engineering collaboration, virtual simulation of engineering designs, and greater reuse and sharing of components and designs.

Many companies in other industries have pursued using remote talent as a way to lower development costs—an approach that will become a more important means of gaining efficient access to necessary skills. India is a well-known source for software talent; China and Russia have also been making names for themselves as centers for outsourced software expertise. But companies should take note before looking overseas: recruiting international talent requires developing the management practices and infrastructure necessary to manage remote workforces.

New Employee Value Propositions. Competing with companies in the tech sector is tough, so if you can’t beat them, try joining them. Companies may be able to increase their attractiveness to software developers by establishing new employee value propositions and, in some instances, by rethinking their branding. Jobs interviewees at tech companies look for four distinct elements in a strong community: mentorship, professional networks, magnet companies, and visibility. Companies with significant software-development needs should consider developing in-house software organizations located in established centers of tech activity. Major manufacturers are setting up shop in Silicon Valley precisely because it puts them in the heart of the world’s leading high-tech community. Wal-Mart has opened two e-commerce offices there and is using e-commerce-company recruitment tactics to attract talent. Comcast recently expanded its technical presence in Silicon Valley as well—to ensure access to the best engineering and software talent.

Increased Use of Third-Party Talent. Companies will need to determine which layers of the software stack they want to develop in-house and which layers can be programmed by others. This determination will likely be made on the basis of how essential to the business various tasks might be. Companies may find themselves increasingly relying on key suppliers for product differentiation; and those that construct the most effective collaboration models, thereby encouraging and rewarding supplier investment in R&D, can build a long-term advantage. Companies will also need to think about codevelopment and joint-venture models to capture innovation from external sources. All of this raises potential issues of IP ownership and control, which will likely require companies to develop new IP and risk-management models to govern nontraditional, but increasingly important, relationships.

Emerging Recruitment Tools and Collaboration Models. Coming full circle, recruiters are now using big-data tools in the quest to hire top software talent. Workforce sciences, a growing field of human resource analysis and planning, uses empirical data to map future needs. Several new tools—such as Entelo and Gild—are increasingly used to predict programmer performance with proprietary algorithms. Some companies are exploring new models for collaborative development, especially for less sensitive—but no less complex—software modules. Working with local university resources, for example, gives them the added advantage of being able to get an up-close look at the skills of high-performing students prior to graduation. These models also demonstrate to students that there is often “cool” work to be done in nontech companies.

Software increases every company’s potential in all aspects of business: product design, manufacturing, supply chain management, and customer relationship management. It also heightens complexity—nowhere more so than with respect to talent. The Industrial Revolution elevated the importance of skilled labor. The advent of the information age put a premium on educated workforces. Globalization allowed companies to access talent wherever it is based, including in emerging markets. As the digital age extends its impact further into all industries and sectors, the significance of software talent with high-level skills developed through advanced education and training becomes a defining factor of success. Since every company these days truly is a software company, those that understand the implications of this reality—and organize their management teams and HR operations to address it—will have a head start in building a sustainable advantage.

Acknowledgments

The authors are grateful to the following BCG partners and colleagues for their assistance in the preparation of this research and report: Patrick Forth, Grant Freeland, John Fuller, Jaison Justin, Bensey Schnip, Michael Sherman, and Niranjan Vijayaragavan.