Does installing 15 Skills give you a reliable Data Agent?
One essential step remains: turning “what it can do” into “how to prove it has done the job well.”
This article introduces no new official Skill. It presents an approach to product development synthesized from the source material discussed in the previous 9 installments. The task protocols, acceptance chain, and failure checklist below are editorial frameworks, rather than a unified specification defined by the source.
Skill count does not measure system maturity
Choose workflows according to the task: gathering context, defining metrics, diagnosing changes, checking quality, organizing calculations, presenting findings, or publishing.
You do not need to run every workflow sequentially for every task, and 15 documents do not represent 15 independent Agents. The source’s position is consistent: there is one primary Skill, supporting Skills join as needed, and the primary Skill determines what counts as completion.
The handoffs matter most. What evidence does one step pass to the next? Where should work stop if a source is missing? Who checks the completion criteria?
These three questions correspond to the protocol, work chain, and acceptance chain described below.
Write a compact protocol for each task
At a minimum, define six elements:
| Element | Question it must answer |
|---|---|
| Inputs | What essential information is needed to make the question sufficiently clear? |
| Required sources | Which system or document is authoritative for this answer? |
| Allowed methods | What may be done, and what is prohibited? |
| Deliverable | What form will the output take, and who will read it? |
| Acceptance criteria | What evidence demonstrates that the task is complete? |
| Failure branches | What happens when sources are missing, definitions conflict, or evidence is insufficient? |
Consider a task to “explain a change in retention”:
- Inputs must specify the time window, population, and product scope. Ask first if any is missing.
- Pause when a required source is unavailable, rather than substituting a related but different definition.
- The output must go beyond “it fell by 3%” to include definitions, evidence, alternative explanations, and limitations.
- Acceptance requires reproducible key numbers and an explicit comparison baseline.
Once all six elements are defined, task completion no longer rests on the model’s own assessment.
Design the work chain and acceptance chain together
Analysis forms a judgment. Validation checks whether that judgment goes beyond the evidence.
The two chains advance in parallel. In the work chain, Metric Diagnostics develops an explanation; in the acceptance chain, Validate Data checks the methods, numbers, and wording behind it.
High-risk conclusions require recalculating numbers, reconciling definitions, and checking comparison baselines and the limits of the claims. Asking the model to polish the language again only makes the wording more confident; it does not strengthen the evidence.
Reusable semantic knowledge is worth preserving, but the temporary context of a particular analysis need not all be written into the semantic layer. Keep maintenance actions separate from task analysis. Otherwise, one-off context can contaminate the semantic layer and reduce its reliability.
Test real behavior with six failure scenarios
This is the most practical step: design concrete failure scenarios instead of asking an Agent whether it knows what to do.
| Failure scenario | Test design | Expected behavior |
|---|---|---|
| Missing authoritative source | Supply only weak substitute data | Stop and explain which source is needed |
| Join inflation | Aggregate directly after a many-to-many join | Detect double counting and correct the grain |
| Aggregating ratios | Average conversion rates across groups directly | Recalculate the overall rate from numerators and denominators |
| Incomplete comparison period | Omit one day or one channel from the current period | Qualify the conclusion or complete the data first |
| Correlation alone | Provide a business event that coincides in time | Label it as a hypothesis and avoid a causal claim |
| Published snapshot | Make a page accessible while its data is stale | Explain that it is a snapshot and state its cutoff time |
Every test needs expected behavior and an inspectable deliverable. Do not score an answer solely because it “sounds professional.” An impression of expertise is precisely what models are especially good at imitating.
Start implementation with one task
The source’s recommendation applies equally well to engineering: begin with one frequent, clearly bounded scenario, and retain examples of success, qualified delivery, and blocked execution.
Keep all three kinds of example:
- Success: the evidence is complete and the conclusion can be recalculated.
- Qualified delivery: the conclusion holds within explicit limits, and recipients know where they can use it.
- Blocked execution: the system correctly declines to proceed and explains what is missing.
The third category is easiest to overlook, yet it is particularly revealing of system maturity. An Agent that never blocks usually never checks.
Build up definitions, tools, and acceptance checks gradually, then broaden coverage. The semantic layer and acceptance cases accumulate value: every failure scenario resolved removes one class of error from future tasks.
One simple question about maturity
If you could ask only one question to determine whether the system is reliable, ask:
Will it stop when the evidence is insufficient?
A system that can stop first needs a definition of sufficient evidence. That requirement drives the creation of protocols, a work chain, and an acceptance chain. A system that never stops passes its uncertainty on to the reader, however smooth its demonstration may be.
Looking back at the series
The previous 9 installments examined routing, context, metrics, diagnostics, market estimation, quality and validation, calculation and presentation, dashboards and reports, and publishing. Each addressed a local problem. This installment connects them into a chain whose completion can be verified.
A reliable Data Agent knows when the evidence is sufficient, when more evidence is needed, and how to communicate that distinction clearly.
Reference: OpenAI Data Analytics Skills public reference repository