The most dangerous result in analytics is often not a failed query. It is a query that runs cleanly, returns a precise number, produces a polished chart—and answers the wrong question.
Suppose an operator asks, “How many new customers did we acquire this month?” The system finds a customer table, counts distinct IDs by created_at, and returns 12,486. The query is reproducible and syntactically correct. But Sales defines a new customer as a first-time payer, while Finance excludes refunded transactions and internal test accounts. The query counted new customer records, not the metric that will drive the budget decision.
The SQL is not broken. The mistake happened before SQL.
Correctness is a chain, not a switch
A data conclusion has to pass at least four layers:
- Syntactic correctness: the database can parse and execute the query.
- Data correctness: sources, joins, grain, time, and quality state match the task.
- Semantic correctness: the computation uses the organization’s valid business definition.
- Interpretive correctness: the claim does not exceed the evidence, and its limits are disclosed.
Most Text-to-SQL scores measure the first layer and part of the second. Decisions are often lost in the last two.
Five ways to be precisely wrong
1. One term names different events
A “customer” may mean a registered user, a verified company, a first-time payer, or the legal entity that signed a contract. “Revenue” may refer to bookings, cash collected, recognized revenue, or net revenue after refunds. Similar column names do not establish equivalent business meaning.
When multiple definitions are valid, silently choosing one turns a guess into an apparent fact. The agent should retrieve an authoritative definition and, when necessary, show the candidate interpretations and their numerical impact.
2. The time window is misaligned
“This month” can mean a calendar month, the last 30 days, or a finance period. User time zones may shift the boundary again. Comparisons also depend on complete periods, working days, and holiday effects. A harmless-looking CURRENT_DATE - 30 can change the management story.
3. A join changes the grain
An order table has one row per order; an item table has one row per SKU. Joining the two and summing order value duplicates multi-item orders. The query runs and spot checks may look plausible. Declaring the target grain and testing row counts and unique keys before and after the join makes the error visible.
4. Status rules live in old SQL
Rules for cancellations, refunds, test accounts, and risk-controlled transactions are often absent from field descriptions. They live in a dashboard expression or a query maintained by an analyst. Giving an agent only the schema is like giving it a dictionary without the organization’s accounting policy.
5. The definition changed but the answer did not
Metrics evolve. If documentation, dashboards, semantic models, and historical queries do not move together, one question can produce several “official” answers. Definition drift requires owners, effective dates, and deprecation states; a model should not resolve it by selecting the newest-looking paragraph.
Why more schema in the prompt is not enough
Spider 2.0 moves Text-to-SQL closer to enterprise work: 632 workflow problems, databases that often contain more than a thousand columns, and tasks that require metadata, dialect documentation, and project code. Its initial results showed that strong performance on classic benchmarks did not transfer directly to this environment.
EntSQL, released in 2026, isolates the next problem. Even when the schema is known, a query may depend on private metrics, reporting conventions, and organizational rules. Enterprise knowledge is not simply an annotated schema; the system must resolve priority, version, scope, and conflict.
The ICLR 2026 benchmark BIRD-INTERACT brings clarification, knowledge retrieval, and execution recovery back into the task. Real analytics is not a one-shot attempt to guess a reference query. It is an interaction that reduces uncertainty.
Turn meaning into an executable metric contract
Before a business definition enters production, it should become a machine-readable, versioned contract.
| Contract field | Question it must answer |
|---|---|
| Name and business definition | What does this metric describe—and exclude? |
| Formula and components | What are the numerator, denominator, deduplication key, and derivations? |
| Entity and grain | What does each row or result represent? |
| Default filters | How are cancellations, refunds, tests, and anomalies handled? |
| Time semantics | Which event time, load time, time zone, and completeness rules apply? |
| Version and effective period | Which version governs this query, and when was the old one retired? |
| Owner | Who resolves a conflict? |
A semantic layer should do more than map friendly names to columns. It should compile this contract into a stable query interface. The agent calls the contract instead of inventing the metric again for each question.
A validation flow for data agents
Before execution: bind the question to a definition
- Identify the business entity, time range, decision, and comparison baseline.
- Retrieve the definition from a certified metric catalog or semantic layer and record its version.
- Clarify when more than one interpretation is legitimate; model confidence is not business approval.
During execution: protect grain and quality
- Declare the key and expected grain of every intermediate result.
- Check row counts, unique keys, and duplication before and after joins.
- Validate freshness, partition completeness, nulls, and unexpected shifts.
- Execute with the caller’s identity so the policy engine—not the prompt—controls visibility.
After execution: verify the claim, not just the result set
- Compare magnitude and direction with a certified dashboard or verified query.
- Test alternative definitions to see whether the conclusion is sensitive to semantics.
- Preserve SQL, parameters, metric version, data timestamp, and validation results.
- If a conflict cannot be resolved, withhold a high-confidence conclusion and name the open question.
Evaluate answers, not only SQL
Execution accuracy can reward queries that happen to return the same result, and it says little about clarification or governance. An enterprise evaluation set should also measure:
- Definition selection: did the system choose the right version and scope?
- Clarification quality: did it ask the smallest set of questions that could change the answer?
- Grain and join safety: did it detect duplicate aggregation and entity mismatch?
- Paraphrase consistency: does the calculation remain stable when the question is reworded?
- Evidence completeness: can the answer be traced to SQL, sources, definition version, and quality checks?
- Drift detection latency: how quickly is an obsolete answer blocked or marked after a definition changes?
A trustworthy data agent is not one that always returns a number. It is one that knows the conditions under which a number has earned the right to become an answer.