AI agents can build your BI. Here's why they get the numbers wrong.

Ask Claude to build you a revenue dashboard and it will. It will read your schema, write the SQL, draw the charts and explain them, and the result will look finished. Some of the numbers will also be wrong. They won't be obviously wrong. They'll be wrong in a way that survives a meeting: revenue double-counted through a join, refunds forgotten, a timezone off by a day.

That is the quiet problem with "agents can do BI now". They can. The question is whether you can trust what they produce, and the research is unusually consistent about the answer.

What the benchmarks say

• dbt Labs, 2026. Text-to-SQL accuracy has roughly doubled since 2023, and it still trails a semantic layer. GPT-5.3-Codex reached 84.1% writing SQL against raw tables and 100% answering through the semantic layer; Claude Sonnet 4.6 went from 90.0% to 98.2%. (dbt Labs benchmark)
• Snowflake, Cortex Analyst. On Snowflake's own set of real BI questions, GPT-4o writing SQL in one prompt reached 51%. An agent answering through a semantic model reached 90%+. (Snowflake engineering)
• data.world AI Lab. GPT-4 answering questions over an insurance company's enterprise schema got 16% right. With a semantic representation of the same data, it got 54%. (benchmark report)

The studies span different years, models and vendors, and they show the same pattern. Accuracy depends less on the model than on the layer it queries.

The finding that matters most

The most useful observation in the dbt study is about the kind of failure rather than the percentage.

When text-to-SQL fails, it returns a confident wrong number. When a semantic layer fails, it returns "I can't answer that".

A wrong number that looks right is the most expensive output an analytics system can produce. It goes into the board deck and moves budget, and nobody re-checks it because nothing told them to. A refusal costs you a follow-up question.

If your agents are going to answer questions about revenue, the most important property of the system is whether it knows when it's wrong.

Why raw tables trip agents up

A schema contains columns, and the business logic lives somewhere else. Everything that makes "revenue" mean your revenue sits outside it:

• which order statuses count, and which are test orders;
• whether refunds are netted, and on which date;
• which currency, and at which rate;
• which join is safe, and which one fans out and doubles the total;
• what "active customer" means this quarter.

An agent reading a schema has to guess all of that every time, from names like amount and status_v2. It will often guess well. Sometimes it will guess differently from the agent next to it, and then two teams have two numbers.

What a semantic layer actually does

A semantic layer is where those decisions are written down once, as code:

• Dimensions and measures such as revenue, spend, cpc and active_customers, defined over specific tables with their filters and joins.
• Deterministic compilation. The agent picks measures and dimensions, and the layer writes the SQL. The agent never authors the join that gets revenue wrong.
• Scope. Anything outside the model is out of scope, so the honest answer is a refusal.

Every consumer reads the same definition: the app, the scheduled report, the agent in Slack, the CFO's spreadsheet. The board deck and the dashboard can't drift apart because they come from the same place.

Why one more layer isn't enough

Cube, the dbt Semantic Layer and the semantic models inside Snowflake and Databricks all assume you already run everything underneath: event collection, connectors, a warehouse, transformations, orchestration and quality checks. For a mid-size company that is five to nine vendors before the semantic layer is even installed, and meaning gets lost at every hand-off between them.

We're building SegmentStream Cloud to close that gap. Once agents are how people ask questions, BI, the warehouse and the pipelines between them stop being separate products. They become one thing: your company's context. That means the data, the numbers defined once, and who may see what, in one place every agent and app reads, in one repo your agent can work in. Switch agents and the context stays. The semantic layer sits in the middle of it instead of being bolted on.

What this means for how you build

1. Let agents build on defined metrics. Agents are excellent at writing the pipeline, the model and the app. Have them write metric definitions too, and review those like any other code.
2. Point agents at the layer instead of the tables. Give them an endpoint that answers from metrics, with an audit trail, rather than warehouse credentials.
3. Prefer refusals to guesses. Treat "I can't answer that" as a normal, visible outcome. It tells you a metric is missing.
4. Check the data under the metric. A perfectly defined metric over stale data is still wrong. Freshness and quality checks belong next to the definitions, where agents can read them.

Agents can build your company's analytics now. The infrastructure underneath decides whether you can believe the results.

On SegmentStream Cloud, measures are defined on the entities of the ontology and served to every agent as MCP tools. The internal tools and dashboards page shows how teams build on it.

Want it on your data? Start a pilot: our engineers build the first version in two weeks, at a fixed price. Self-serve opens to the waitlist this fall.