Everyone Loves The Idea Of AI, But Not The Reality

AI is sold as a 3-slide presentation.

Slide 1: AI will do everything! It has incredible potential, and it’s easy.

Slide 2: ???

Slide 3: Massive competitive advantages, profits, efficiency, and productivity gains!!!

Business leaders love the idea of AI, Slides 1 and 3. It’s not until they are knee-deep in Slide 2 that reality kicks in. Prototypes don’t make it to production because the unit economics don’t work for most use cases. Building functional prototypes is simple. Most of the costs and complexity lie in turning them into reliable products.

In this article, I’ll address some lessons learned from the last two years of Generative AI adoption. The perception of how this technology cycle would develop is very different from the reality on the ground today.

The LLM Arms Race Is High-Cost & Low ROI

The AI foundational model arms race might be unsustainable even for companies like OpenAI. The startup booked $3.7 billion in revenue but still lost $5 billion this year. The AI field is increasingly crowded, so winning with the most advanced model takes a massive investment.

Meta, OpenAI, Google, and Salesforce released new LLMs and AI products in September alone. Some models only spend a few weeks as the best-in-class before another release dethrones them. This table provides a sense of how broad and rapidly evolving the landscape has become.

Most businesses don’t plan to compete with the largest AI tech companies. They shouldn’t plan to compete with industry peers for LLM dominance either. LLMs like GPT and Gemini cost millions of dollars for just the computing resources required to train them. It takes deep technical expertise to support multiple training phases.

There’s no way to justify continuously investing millions for short-lived competitive advantages. Unfortunately, some companies have chosen the expensive road to no ROI. Models are increasingly commoditized. At Dreamforce, Marc Benioff said, “Companies thought they’d buy access to a private instance and integrate an LLM, and it’ll transform their business, and that’s happened exactly 0 times.”

SaaS companies like Salesforce have transformed into AI-SaaS hybrids that integrate AI’s benefits into familiar applications. Building large foundational models internally isn’t practical for most businesses, and integrating open-source models into DIY solutions isn’t pragmatic. The problem is almost always unit economics.

Small models make sense for businesses to develop for a broad range of use cases. Large models need multiple monetization opportunities, or costs scale faster than returns. If a business sells an AI product to its customers and can scale rapidly enough, there’s significant ROI. Even then, building a competitive advantage based on the model is the wrong approach. Data provides a more sustainable path forward.

Data Is The Biggest Hurdle & Advantage

Foundational model training requires more data than most businesses have, but internal data creates value in new ways. Datasets contain 360 views of customers and some domain expertise required to deliver value to those customers. Relying on an LLM’s internal knowledge alone doesn’t take advantage of the strengths trapped in the business’s data.

AI platforms are built with more than just foundational models. LLMs provide functionality that no other technology can. Still, multiple technologies are necessary to turn a model into a product. Internal data plays a huge part in AI platforms.

Even companies that invest heavily in AI eventually pivot back to the data. Google realized that $60 million invested in computing delivered less than investing the same amount in higher-quality data. Reddit’s dataset makes Google’s GenAI search results more recent and relevant. For OpenAI, partnering with a range of companies like the Associated Press and the Atlantic gives them access to well-written, vetted content.

Most businesses are making similar pivots to data after realizing there’s a lot of data engineering technical debt hiding in Slide 2. Steve Hammond, EVP and GM of Salesforce’s Marketing Cloud, discussed how he often sees silos and walls between teams or processes translate into data silos and barriers to data access. If people can’t get access to the data they need, models don’t have a chance.

Sankar Chinnathambi, CIO of Driscoll’s, said his company used to get its supply numbers directly from farms. No, not through APIs, over the telephone. Every business has similar layers of data dysfunction that slow maturity and delivery.

They need time and resources to solve foundational data engineering and modeling challenges. Adding data modeling for LLMs to that workload is a stretch, to put it nicely. AI is data, so no path to AI avoids addressing the business’s data.

Reliability AI Is A Two-Sided Challenge

A single, powerful model generalizing across tasks and serving diverse business needs is the end goal for startups like OpenAI. So far, this approach has proven unworkable in practice. LLMs are highly capable but struggle with self-regulation. Like an overeager employee who takes on too much, they will attempt to handle any task but often fail to deliver consistent, reliable results.

Fine-tuning LLMs with internal data has also led to underwhelming outcomes. RAG replaced earlier methods. It offers improvements, particularly in reducing hallucinations, but still falls short of providing the level of control necessary for business and customer-critical applications. More robust guardrails are required to ensure the models behave predictably and accurately.

Salesforce’s Agentforce development environment is an excellent example of how much work goes into guardrails and validation.

• The UI allows users to specify exactly which data sources and APIs an agent can leverage.

• It supports defining the tasks that are in and out of bounds for agents to perform.

• It allows users to define those steps in natural language vs. coded logic.

• The builder has tools for testing scenarios and scaling those tests to dozens or hundreds of scenarios.

• Explainability and auditing features are part of the testing workflows.

• Users can monitor agents’ responses using tools they currently use to monitor people doing the same tasks.

The Agentforce platform wouldn’t be possible without the digital tools, data, and existing workflows that have been part of the Salesforce platform for years. What businesses didn’t realize going into the GenAI boom was how much work goes into putting the pieces around LLMs to make them reliable enough to support business and customer use cases.

Models must be built for reliability, but the platform they run on must be designed to enforce reliability requirements. AI platforms must balance flexibility with control and oversight, which is a massive challenge. An AI platform strategy is the only viable approach to generating immediate returns from AI.

Business leaders must have an acceleration strategy and partners who help the business rapidly mature their AI platforms. Building the accelerator pieces internally defeats the purpose.

Why Is Slide 2 Overlooked?

Ketan Karkhanis, EVP and GM of Salesforce’s Sales Cloud, said that every AI conversation he has with customers is now an ROI conversation. Business leaders realize they don’t need more AI initiatives. They need a higher success rate. It’s 2024, and over half of AI initiatives still fail to deliver value. Some studies find that as few as 1 in 5 succeed.

Studies show that AI initiatives succeed when “technical staff understands the project purpose and domain context.” But…

“Stakeholders often misunderstand or miscommunicate what problem needs to be solved using AI. The organization focuses more on using the latest and greatest technology than on solving real problems for their intended users.”

In 2022, an Accenture survey found that companies that generate value from AI prioritized hiring people who connected AI with business outcomes vs. technology builders. Builders are critical, but businesses need more builders who align what gets built with what the business needs.

Slide 2 rarely gets filled in because businesses are still coming to terms with what should be there. The hype cycle was driven by AI products being crowned a success before they were delivered. Use cases were called feasible before they were implemented. Marc Benioff said that so much about AI isn’t true, and AI platform vendors have disappointed their customers.

AI initiatives can take 14 to 17 months to reach breakeven, and many executives are still waiting to see if the business made the right investments. Benioff talked about customers switching to Salesforce after their first round of AI copilot experiments didn’t deliver Slide 3. Even companies that thought they had Slide 2 figured out are changing direction.

The most significant challenge has been starting with the business and bending AI to fit the way the business operates. I pressed Ketan Karkhanis on how he’s achieved that alignment. “You must see it from one perspective {the customer’s} to align all sides…strategic alignment around a single outcome.”

The most difficult reality check of the last 2 years has been that customers don’t buy AI, and C-level leaders don’t fund technology. They pay for outcomes.