Let’s be honest: Does your company have all business-relevant information available at the push of a button? Or is it also stuck in various data silos, largely unconnected – the ERP here, the CRM there, plus Excel spreadsheets on personal drives and strategy documents somewhere in the cloud?
If you’re nodding right now, you’re in good company. I regularly speak with CEOs and finance leaders, and the picture is almost always the same: The data would be there. But bringing it together to answer a specific question takes days – if anyone can do it at all.
Why This Is Becoming a Problem Right Now
The days when companies could rely on stable markets and predictable developments are over. Inflation, geopolitical tensions, disrupted supply chains, a labor market in flux – all of this demands a new discipline: Decisions must not only be good, they must be good fast.
Traditional business intelligence has a proven answer to this: dashboards, KPIs, monthly reports. But let’s be honest – these tools hit their limits as soon as questions get more complex. What happens to our margin if we switch suppliers? How does a price increase affect different customer segments? What scenarios emerge if the euro keeps falling?
Questions like these need more than static charts. They need a real conversation with your own data.
The Temptation: An AI Sparring Partner for Your Decisions
This is exactly where generative AI gets really exciting. The idea is compelling: An intelligent assistant that knows your company’s numbers, understands connections, and lets you explore strategic options – anytime, without scheduling, without someone having to build an analysis first.
“How did our top 10 customers develop last quarter?” “What if we reduced the product portfolio by 20%?” “Compare our cost structure with last year and show me the biggest outliers.”
A dialogue like this would democratize business intelligence. Not just the controller with their Excel expertise would have access to insights – every decision-maker could query the data themselves. I still find this idea fascinating.
The Problem: When AI Hallucinates, It Gets Really Expensive
But – and this is a big but – here’s the crux. Large Language Models are impressive at generating plausible-sounding answers. They’re considerably less reliable at delivering factually correct answers. Especially when it comes to concrete numbers.
An AI that misremembers a date in a creative text? Annoying, but manageable. An AI that invents a revenue figure or miscalculates a margin during a business decision? That can really hurt. The danger multiplies because the answers are so damn convincing. We humans tend to trust a confidently delivered statement – even when it comes from a statistical language model.
I say this from experience: A naive integration of ChatGPT with company data is a risk, not progress. Anyone who sees it differently has either been lucky or hasn’t noticed yet.
The Technical Challenge: Connecting Three Worlds
The solution lies in a well-thought-out architecture that intelligently brings together three different data sources:
Structured data via SQL: The hard facts – revenues, costs, quantities, customer histories – typically reside in relational databases. Here, the AI must not guess but query precisely. The system must generate SQL queries, execute them, and correctly interpret the results. No room for creativity.
Unstructured data via RAG: Beyond the numbers, there’s context – strategy papers, market analyses, internal guidelines, meeting notes. These documents can be accessed through Retrieval Augmented Generation: The system searches for relevant text passages and provides them to the language model as context.
The model’s world knowledge: Finally, the LLM brings its own knowledge – about industries, economic relationships, best practices. This knowledge is valuable for interpretation, but dangerous when mixed with concrete company figures.
The art lies in cleanly separating these three sources and making transparent where each piece of information comes from.
The Solution: Everything into the Context Window
Modern LLMs offer context windows of 100,000 tokens and more. This opens up an elegant architectural approach: Instead of letting the model guess which data might be relevant, we proactively load all needed information into the context.
A well-designed system works in several steps: It analyzes the user’s question and identifies relevant data sources. Then it executes the necessary SQL queries. In parallel, it searches the document base via RAG. And finally, the LLM receives all this information served up together – with clear labeling of sources.
The language model thus becomes an interpreter and communicator, not a fact generator. It can explain numbers, reveal connections, ask follow-up questions, discuss options for action – but it doesn’t invent data, because the real data is already in the context.
Transparency as a Design Principle
Such a system must build transparency into its DNA. Every statement about concrete numbers should cite its source. The user must be able to trace: Does this come from the database? Was it quoted from a document? Or is it an assessment by the model?
This transparency isn’t just a technical feature – it’s the prerequisite for trust. Anyone basing business decisions on AI-supported analyses must know what they’re relying on.
The Path Forward
Business intelligence with AI is neither utopia nor hype – it’s an architecture challenge. The technology is mature, the models are powerful, the interfaces exist. What many companies lack is a thoughtful approach that leverages the strengths of LLMs without falling prey to their weaknesses.
The future belongs to systems that intelligently connect structured databases, document knowledge, and language models – while always making transparent what is fact and what is interpretation. Companies that find this balance gain more than just another analytics tool. They gain a real sparring partner for better decisions in difficult times.
And yes – that’s exactly what we’re working on.
