After surveying many Data Agent products, a team beginning implementation still faces five basic questions: What work should the system own? Which data should it use? How may it execute? How will we prove the result? Who remains accountable when it fails?
Those questions are more durable than a vendor list. They connect definition, architecture, autonomy, evaluation, and rollout.
The diagram below maps those questions onto an operating path: how a task enters the system, how semantics constrain planning and execution, and how evidence, approval, read-back, and feedback close the loop.
1. Decide which work the agent will own
Business Q&A, end-to-end analysis, automated data science, heterogeneous research, and data engineering are different routes. Q&A needs stable semantics and read-only access. Open analysis needs a multi-step evidence chain. Data engineering adds change approval and rollback.
Do not start with a universal agent. Choose one workflow and state its inputs, artifact, and failure impact. The route answers what the system does; L0–L5 answers how responsibility is divided. They are different axes.
2. Correct answers begin with correct evidence
Executable SQL proves that a program ran. A Data Agent must first select authoritative data inside the user’s access scope, inspect schema and watermarks, and bind metric, entity, time, grain, and join rules.
A catalog describes what exists. A semantic layer states what it means and how it is calculated. A model can retrieve and organize that knowledge; it should not invent revenue, customer, or refund policy. Without versioned semantics, every answer is a fresh guess at enterprise rules.
3. Seven layers serve one task
A complete path is:
- Access and routing find relevant, authorized, sufficient evidence.
- Semantics bind definitions and relationships.
- Planning and orchestration turn an objective into a recoverable state machine.
- Controlled execution validate, authorize, and run queries, code, and tools.
- Verification and evaluation check source, semantics, program, analysis, and delivery.
- Governance and security span identity, policy, cache, artifacts, and actions.
- Delivery and action connect claims, evidence, artifacts, approval, and feedback.
These layers do not require seven products or microservices. They are responsibility boundaries. A team must know where each control is enforced and where failure returns.
4. More autonomy transfers more responsibility
At L1 the agent drafts and a person executes. At L2 the agent invokes tools while a person orchestrates. Proto-L3 plans within a bounded objective while a person supervises. Each step transfers initiation, planning, execution, or verification authority and therefore demands stronger governance and recovery.
One product can operate at different levels by workflow. Read-only Q&A may be L2 while production change stays L1. Attach a level to workflow × environment × permission, not to a brand.
5. Public benchmarks shortlist; internal tasks decide
Text-to-SQL, code analysis, heterogeneous data, and report benchmarks measure different contracts. Record data version, oracle inputs, model and tools, retries, evaluator, cost, and date when reading a leaderboard.
The release decision needs an enterprise task set. Add ambiguous metrics, late data, similarly named test tables, unauthorized requests, tool failures, and unanswerable cases to normal tasks. Compare completion, evidence support, human correction, p95 latency, and total cost, while treating unsafe access and writes as independent blockers.
Convert product cases into design questions
The lasting value of vendor cases is not the feature list but the engineering prompt:
- Internal Data Agents show that discovery and institutional context may be harder than SQL generation.
- Skills show how analytical practice can become a reusable, reviewable work protocol.
- Cloud products demonstrate reuse of data, semantics, identity, and runtime control planes.
- Read-only designs separate query authority from business-action authority.
- Multi-agent systems add value only when duties, context, or permissions require separation.
Turn each lesson into a POC question rather than copying the architecture.
A pilot task card
| Field | Example: weekly net-revenue review |
|---|---|
| User and decision | Regional leader; choose the first channel to investigate |
| Objective | Compare this week with the previous four and explain contributions |
| Metric | Net receipts v3; refunds on occurrence date; Asia/Shanghai |
| Data | Authorized production funnel and payment models; complete through Sunday |
| Tools | Read-only SQL, bounded Python, charts; no external send or write |
| Validation | Independent total recomputation; contributions sum to overall change within tolerance |
| Delivery | One-page brief, result table, definition, limits, and draft ticket |
| Stop conditions | Incomplete data, definition conflict, insufficient access, or exhausted budget |
| Owners | Business owner, metric owner, and system owner |
Run 20–50 real cases under this card in sandbox and shadow mode before limited production. Writes and external communication need separate approval and recovery; they do not inherit authorization from a read-only POC.
The final principle
A Data Agent is not a more conversational query interface or the sum of models, tools, and agent roles. It is a system for transferring data work to software while preserving business meaning, evidence, and accountability.
Make one task trustworthy before transferring more work. It is slower than a universal demo and much closer to a durable product.