For traditional information services providers, generative AI (GenAI) represents an unprecedented opportunity to drive sustainable advantage. By scaling solutions that leverage their vast data and content assets, companies can help their customers harness the power of GenAI while generating new revenue for themselves. Leading players across financial information, credit, publishing, and other info services are already beginning to see the benefit of GenAI at scale.

But for the vast majority of companies, realizing the full potential of GenAI has proved challenging. According to a recent BCG cross-industry survey, 66% of leaders are ambivalent or dissatisfied with their progress on AI and GenAI. This has also been the case with information service providers, many of whom have created proofs of concept only to find that developing a customer-facing product is more costly and resource-intensive than anticipated. Issues pertaining to accuracy of GenAI output, organizational capability and capacity, systems integration, and data preparation are among those standing in the way of successful scaling.

Providers that address these challenges and scale GenAI successfully will have access to a rapidly growing market. According to BCG analysis, the current market opportunity for GenAI financial services information products alone is around $5 billion to $6 billion and is projected to grow to $15 billion by 2026.

The Promise of GenAI at Scale

To date, GenAI has gained wide use as a way for companies to enhance internal productivity, and some are already using it to drive end-to-end transformation. But for companies that provide information products—such as financial, business, and sustainability data—GenAI can be a tool to enhance the productivity of end-customers. This new revenue stream can help to justify early investments into the nascent technology.

While most traditional info service providers understand the value of this opportunity, few have managed to scale GenAI successfully. To learn why, in early 2024 we surveyed 100 professionals from providers who had experience scaling GenAI solutions. (See “About the Survey.”)

More than half of the survey respondents said that building GenAI products has been more time-intensive and costly than anticipated. Four challenges were cited most frequently: accuracy requirements, organizational capability or capacity, systems integration, and data preparation and management. (See Exhibit 1.)

Overcoming Critical Scaling Barriers

Through our work with clients, we have gleaned various ways to address the top four obstacles to successfully scaling GenAI.

Secret #1: To meet accuracy requirements, set an accuracy threshold and work iteratively towards that goal.

Accuracy requirements for GenAI output were the most frequently cited barrier to scaling up solutions. Yet while there are many ways to make an off-the-shelf GenAI model more accurate, determining which techniques to implement is not straightforward.

A key issue is that accuracy improvements often come at the expense of latency and cost. Complicating matters, changes to any part of the solution may have ripple effects elsewhere. Sometimes these effects can be positive—for instance, simply injecting the word “please” in a prompt can significantly improve the accuracy of model output. But often a change, such as adding an extra space, can reduce accuracy. And since GenAI solutions can provide different results each time, the outcome of any changes, large or small, can be difficult to predict.

Given these constraints, setting a clear accuracy threshold is a critical first step for avoiding overinvestments of time and resources. For a case where GenAI will be used to help with report writing, 80% accuracy (as good as a typical human writer) is likely good enough. This type of solution may be achievable with a lower-cost model (such as Chat GPT-3.5 instead of GPT-4). But for use cases where near 100% accuracy is required—such as contract review or credit scoring—predictive or deterministic AI models may need to be used alongside GenAI, at least for now.

Once the threshold has been determined, the next step is to apply techniques that may help meet this goal. We recommend starting with the simplest techniques, such as prompt engineering and context injection, even though many developers are not familiar with them. (See Exhibit 2.) If these techniques do not achieve the threshold, organizations should consider approaches that are more complicated and expensive, such as fine-tuning or training a new model. A “human in the loop” process to review output can also boost accuracy.

The key here is iteration, finding the right combination of changes to magnify the power of the entire solution and hit the accuracy threshold while limiting negative impacts to speed and budgets.

Secret #2: To manage organizational capability and capacity, prioritize use cases carefully.

Few organizations have the capability or capacity to develop every GenAI product they’d like to build. According to BCG’s recent cross-sector, C-suite survey, 62% of the executives who expressed dissatisfaction with their organization’s progress on AI and GenAI cited a shortage of talent and skills.

To make the most of the GenAI opportunity, therefore, it’s important for companies to determine which use cases to focus on. Resource requirements should be driven by three considerations that focus on value, readiness, and complexity. To scale a GenAI product successfully, companies should be able to answer “yes” to the following questions:

• Value potential: Are the benefits of this solution enough to drive significant usage and adoption?
• Organizational readiness: Is our team ready, willing, and able to build this solution? Does the team have leadership support?
• Complexity of the approach: Is the proposed approach something we can achieve and maintain?

Companies must fight the urge to prioritize based on only one or two of these dimensions. Focus only on value potential, and a company may take on more than it is ready for in terms of technical complexity. Focus only on organizational readiness, and a company may build products that don’t have strong enough market demand. A company that focuses only on complexity may conceptualize solutions that are valuable and feasible but could lack the necessary leadership support to execute.

Best-in-class companies use a balanced scorecard across these dimensions to focus their energy and resources in the right places. Once companies have a few early successes, they are usually able to scale the skills and capabilities they have built within their organization—and build increasingly sophisticated solutions as well.

For example, a financial information services provider built a retrieval-augmented generation (RAG) solution to help answer free-text questions about company financial performance. It took the company nine months to build this product, and only half that time to build subsequent solutions that leveraged a similar approach.

Secret #3: To optimize systems integration, aim for interoperability and adaptability.

It’s easy to lose sight of the fact that a GenAI solution requires many components besides the GenAI model. The front end, data, enterprise core systems, infrastructure, and security all must be developed to operate optimally, both now and in the future. Companies should start developing these other components early on, focusing on two main considerations.

Interoperability. To support high-quality, secure data sharing, the components of a particular GenAI solution need to be interoperable with the organization’s existing tech stack and with its other GenAI products. This helps ensure that different solutions can “talk” to each other, enabling cohesive workflows.

Adaptability. Given the fast pace of today’s tech cycles, it’s critical to ensure that the solution components can be updated cost-effectively at scale. Companies, therefore, should build components with standardized APIs and services that can be easily swapped out for upgrades when they become available. A standard solution reference architecture can then be used to guide the conformance to enterprise architecture principles and standards.

Often, companies will need to make substantive changes to a solution. For instance, when using an off-the-shelf model, it may be necessary to “remodel” or switch to a newer version of a foundation model when it comes available. That’s what many users of Chat GPT-3.5 did when GPT-4 came out. In other cases, companies may need to rebuild their solution completely, such as when new technology or techniques emerge or a strategic shift necessitates a different approach (e.g., moving from a RAG solution to a fine-tuned model). Although these changes go beyond what we would typically consider “application maintenance,” companies should treat them as part of the ongoing maintenance cycle for GenAI products.

Secret #4: To ensure robust data preparation and management, master these new techniques and mitigate new risks.

To build GenAI products that meaningfully leverage a company’s data assets, organizations need data readiness. Any data, whether structured or unstructured, needs to be clean, machine readable, and protected from risks of exposure. This means deploying preparation and management techniques specific to GenAI solutions.

Our survey of information services providers demonstrated that data readiness increases with scaling experience: two-thirds of the respondents who are experienced in scaling GenAI products report high data readiness. By contrast, only one-third of the participants who lack scaling experience said they are data-ready. (See Exhibit 3.)