In the commodities industry, operations move at high volumes and with thin margins. Decisions are only as good as the data and processes behind them, with small errors upstream introducing risk that can develop into significant financial losses downstream.
Many firms are looking into minimizing or even eliminating manual operations processes to improve data accuracy and throughput, but they face a fundamental question.
It’s a critical business decision with long-lasting implications. Building offers control and customization; buying offers speed and market-tested solutions.
Understanding both sides of the argument is critical before committing resources.
Some commodity firms have their own in-house development team. Others may consider commissioning custom development. The emergence of familiar, highly adaptable generative AI systems and open-source software has made this approach feel more plausible.
Developing internally can produce a system designed exactly for a company’s processes. For organizations with unusual requirements or highly differentiated ways of working, customization at this level may feel essential.
Some firms see software as more than a tool for internal use. Octopus Energy’s Kraken platform, for example, began as an in-house system but grew into a commercial venture in its own right.[1] For companies with ambitions to monetize their technology, building offers that possibility.
Owning the development cycle means a company can decide which features are prioritized and when. This level of autonomy can be appealing, particularly for businesses that prefer not to rely on external vendors.
These are valid considerations. In some cases, particularly for very large firms with resources to spare, they can add up to a stronger case for an in-house build. However, they must be weighed against the longer-term demands of maintaining such systems. Google’s seminal paper, Hidden Technical Debt in Machine Learning Systems, notes that using generic packages often results in an extensive need for “glue code” – supporting code necessary for moving data into and out of general-purpose packages.[2]
Maintenance is arguably more demanding with AI systems, where the technical challenges of adapting open-source and generative platforms can be easy to underestimate.[3] The authors of Google’s paper contend that adapting generic software hinders improvements by making it harder to take advantage of domain-specific properties or achieve domain-specific goals.
Operations intelligence is not simply a matter of applying AI to your available data.
Building this internally requires multiple specialist teams, not just data scientists, and that’s just the start.
Aligning an organization’s vision and resources can be an insurmountable challenge. In an article about why so many data science projects fail, MIT Sloan Management Review notes that in many organizations there remains a consistent disconnect between data scientists and the executive decision makers they support.[4]
Many firms assume that because they have strong AI talent, they can extend that expertise into operations. But an effective operations intelligence platform is not just an AI model. It must unify automations for data inputs (encompassing documents and data in multiple formats, frequently unstructured or semi-structured) to create the normalized data foundation that makes workflow, analytical, agentic, and interactive capabilities possible.
This goes well beyond any AI model. Whereas in-house IT organizations may struggle to unify these pieces, vendors specialize in weaving these components together into a cohesive product.
Internal AI teams, however capable, often lack the application development, data management, and workflow design expertise to deliver a market-honed standard consistently.[5]
"Vendors don’t just bring technology—they bring the accumulated expertise of solving these problems across multiple firms. That perspective is impossible to replicate in-house."
- Marc Lefebvre, CTO & Co-Founder, ClearDox
AI systems require ongoing retraining, tuning and monitoring as data changes. That means sustained investment not only in development but also in MLOps and governance. Talent risk compounds the issue: if a core AI or development team leaves (a very real risk at present) the organization may lose the ability to update or even understand its own models.
Consider what happened for major investment banks in the early 2000s, where custom-built was the order of the day. Many ended up with hundreds of applications. These became so cumbersome and expensive to manage that many banks had no option but to go through a disruptive change in strategy.[6] In turn, this accelerated adoption of then-new public cloud systems, extending and consolidating low-maintenance vendor infrastructures.
Another, more recent MIT report found that 95 percent of generative AI pilots at companies are failing, with internal teams often spending months or years building training data, refining models, and integrating them into applications before seeing results.[7]
“When it comes to AI in commodities, speed is everything. Buying a proven solution means you start realizing value immediately—while internal builds often stall for months or even years. The firms winning today are the ones that partner with specialized vendors, not the ones reinventing the wheel.”
- Rick Nelson, CEO, ClearDox
Everest Group, a research firm, reported that leading vendors “are also constantly investing in expanding the library of pre-built models and [out-of-the-box] packaged solutions, especially for industry-specific use cases and document types.”
ClearDox, for example, offers pre-built digitizers covering hundreds of counterparties. Each new customer benefits from this accumulated experience. Internal builds would need to create every digitizer, extractor, and workflow from scratch.[9]
For most firms, AI expertise is better applied to areas where it creates true competitive advantage — such as trading strategies or asset optimization — rather than duplicating commodity workflows that specialist vendors already solve.
The decision is ultimately strategic. For a handful of firms intent on spinning out a new software business, building can make sense. But for organizations whose primary focus is trading and operations, the economics and risk profile of in-house development—particularly in AI-heavy areas where innovation is advancing at breakneck speed—are often unfavorable.
Vendors bring not just technology, but also the accumulated experience of working across multiple firms and contexts. That perspective is difficult to replicate internally.
If the answer to most of these is “no,” then building is unlikely to offer a competitive edge.
–John Montgomery, Corporate Vice President, Azure AI (via CIO Magazine)
Proven solutions free internal teams from reinventing what already exists, while positioning firms to modernize faster and lead in the AI era.
In an industry where efficiency and agility determine competitiveness, the stronger argument is for buy over build.
Get the Report: "Beyond the Hype: AI in Energy and Commodities"
[1] Kahn, Yusuf. “The $10 Billion AI Startup That Thinks It Is an Energy Company.” The Wall Street Journal, May 2025
[2] Scully et al. “Hidden Technical Debt in Machine Learning Systems.” Google, Inc, December 2015
[3] “AI Risk Management Framework.” National Institute of Standards and Technology, January 2023
[4] Joshi et al. “Why So Many Data Science Projects Fail to Deliver.” MIT Sloan Management Review, March 2021
[5] Salama et al. “Practitioners guide to MLOps: A framework for continuous delivery and automation of machine learning.” Google, Inc, May 2021.
[6] Kot et al. “How banks can achieve next-generation legacy modernization.” McKinsey & Company, April 2021
[7] Estrada, Sheryl. “MIT report: 95% of generative AI pilots at companies are failing.” Fortune, August 2025.
[8] Eriksson, Viktor. “MIT study: 95% of corporate genAI projects fall short of success.” Computerworld, August 2025
[9] “Everest Group Intelligent Document Processing (IDP) Products PEAK Matrix Assessment 2024.” Everest Group, April 2024
[10] Sher, Robert. “When Should Your Company Develop Its Own Software?” Harvard Business Review, December 2021