Stop Bolting Artificial Intelligence onto Business Intelligence: A Layer-by-Layer Case for AI Across the Lakehouse

Most organisations bolt Artificial Intelligence (AI) onto the dashboard and call it done. That's AI applied at the surface, not built into the foundation. It may look impressive on the front-end, but it adds no real productivity where it matters most, in the pipelines and modelling underneath. Here's the case for building AI into every layer of the Lakehouse.

There's a script many follow with AI and data right now: craft the data pipelines and other complex back-end work pipelines with a few specialists, then, at the very end, bolt a natural-language chatbot onto the gold-layer so business users can “ask their data questions”.

“Ship it. We’ve got AI!”

We think it’s time to rip up the script.

Not because the chatbot is a bad idea. It's genuinely a good one. The problem is that putting AI only at the last layer misses where most of the time, cost and risk actually live. The hard work in a modern data platform isn't the final dashboard. It's everything underneath it: making sense of an unfamiliar source, building the tables and pipelines, mapping messy inputs to a model the business trusts and fixing things when they drift. Leave all of that as manual, specialist work and then perch an AI assistant on top, and you haven't built an AI-enabled platform. You've simply wrapped AI around the least valuable part of the process.

Where The Real Effort Goes

An AI assistant is only ever as good as the data beneath it, so the layers below the dashboard are where the outcome is really decided. None of this is a knock on data engineering. Done well, with the right frameworks, patterns and practices, hand-built pipelines are entirely trustworthy. Good engineering gets this right every day.

The challenge is the fact that the work underneath is genuinely complex: unfamiliar sources of data, intricate lifecycle rules and strict target models that leave no room for interpretation. That complexity has two main impacts: mistakes are easier to let creep in, so more effort goes on catching and correcting them. And, even when nothing goes wrong, getting it right simply takes time, and it leans on a small number of scarce specialists.

So, the real bottleneck is not whether the last layer can be trusted, but how long, how much effort, and how many specialists it takes to earn that trust in the first place. Which is exactly where AI, applied across the layers rather than bolted on at the end, changes the economics.

AI at Every Layer, Not Just The Last

So, what does it look like to put AI into the foundations rather than on top of them? On Databricks, a typical deployment is a medallion architecture, which already gives us the applicable layers: Bronze for raw, Silver for cleaned and conformed, Gold for business-ready and Business Intelligence (BI) for serving. The shift is to bring AI (in our case Databricks Genie) to work at not only BI, but also at bronze, silver and gold, with a human in the loop at every key step.

We've been proving this out on a deliberately hard problem: complex interest-rate swap trade data, arriving in dense, deeply nested Financial products Markup Language (FpML) that the engineers were meeting for the first time, that had to map cleanly into a fixed industry model - the International Swaps and Derivatives Association (ISDA) Common Domain Model (CDM). This came down to four moves across the stack.

Understand fast: At the Bronze layer, instead of a specialist spending days spelunking through XML, Genie profiles the source, its element hierarchy, data types and repeating groups, and proposes a relational model to land it into. The unfamiliar becomes legible in minutes.

Build fast: Still in Bronze, but also in Silver, Genie generates the Delta tables, the transformations and a parameterised loader, and orchestrates the flow. The engineer directs; the assistant does the heavy lifting. Data quality fixes, merges and conformation are decided in plain-English exchanges between Genie and the human in the loop, not hand-cut code.

Map fast: Bronze and Silver are source-aligned; Gold must be business aligned. This is where the real value lies, and where Genie maps the Silver data to an industry target model, such as ISDA CDM, recommends the required transformations, and helps resolve gaps through further Genie and human-in-the-loop exchanges. Whether the target model is pre-defined or must first be discovered, determining and implementing the complex transformations is normally the slowest part of the job. Here, it isn’t.

Fix fast: Finally, Genie unit-tests the Gold output back against the original source, checking row counts, reconciliations and referential integrity, so problems surface as a report, not as a surprise in production three weeks later.

Only after all of that do we reach the layer everyone else starts at: a Genie Space over the governed Gold data, where a risk analyst can ask, in plain English, “what's our net fixed-rate position by currency?” And get a governed, consistent answer, with no Structured Query Language (SQL) and no ticket to a specialist. This is also where the semantic layer lives: the business definitions, rules and trusted examples we set once on the Genie Space, so a term means the same thing every time anyone asks. Same AI, opposite end of the pipe. That's the difference between AI as a veneer and AI as an operating principle.

What Keeps it Trustworthy

Speed alone would just build a fragile thing faster. Three things make this trustworthy rather than reckless.

1. Skills

Rather than one enormous prompt, each capability is packaged as a self-contained, reusable skill: a folder holding the instructions, the runnable code and a worked example of what ‘good’ looks like. One skill does one job, whether that's profiling Bronze, mapping Silver to the CDM or running the Gold tests. They're versioned, portable and consistent, so the same task gives the same result no matter who runs it. The capability becomes the reusable asset, not the individual who happens to know how it's done.

2. The human in the loop

At every layer, Genie proposes and a data engineer or business expert decides. The AI compresses the effort; but the judgement and decisions remain with our human.

3. Governance built in, not bolted on

Shared definitions, a semantic layer and documented instructions mean numbers don't drift between teams, and the consumption layer answers are the same answers every time.

There is a cost dimension to this as well, and it rewards using a disciplined framework: letting the tools retrieve and execute, letting skills support the proven workflows and sharing only the context that is genuinely relevant rather than everything at once. If this is done properly, AI applied across the stack is not simply faster than the fully manual alternative it replaces; it also means controlled and predictable costs. Essentially, the very practices that make it quick are the ones that keep it economical.

Beyond The Obvious

The obvious move is to add an AI chat box to your BI and declare victory. The less obvious (and far more valuable) move is to bring AI into every layer of the stack, so it is doing meaningful work from raw data through to the business question, on foundations solid enough to trust the answers, within predictable cost boundaries.

We're presenting this pattern in full, including this worked example, as a talk at the Databricks Data + AI Summit Sydney World Tour stop. If you're on-site, come find the session, or stop by the Synechron booth afterwards to talk through where AI fits in your own Lakehouse.