Takeaway: if you give an agent definitions, check each one against the data, because the agent may not.
In one BIRD question, three models returned the correct count when given correct definitions. When I swapped two definitions, every run returned the wrong count without warning.
This is a preliminary experiment. The wider research direction remains exploratory, and this result does not establish how often the failure occurs.
The question
A semantic layer supplies definitions of database codes, metrics and business terms. In this pilot, it is a short text file included in the agent's prompt. If one definition is wrong, does the agent notice? Can an automatic check catch the mistake before the answer reaches the user?
What I tested
I used the financial database from BIRD, a text-to-SQL benchmark [1]. Its account.frequency column records how often a bank issues statements. I renamed the codes to F1, F2 and F3 so their wording would not reveal their meaning.
- No layer: the agent receives no definitions for the renamed codes.
- Correct layer: F1 means issuance after each transaction; F3 means weekly issuance.
- Swapped layer: the definitions of F1 and F3 are exchanged.
Q89 asks how many accounts in East Bohemia receive a statement after each transaction. The correct answer is 13. Using the swapped definition produces 22.
The full pilot comprised 81 runs: three questions, three layer conditions, three models and three repeats. Only Q89 depends on the changed definition. Q90 and Q100 were intended as controls.
What happened on Q89
The table gives the number of runs that returned the correct count of 13.
| Model | No layer | Correct layer | Swapped layer |
|---|---|---|---|
| gpt-oss-120b | 0 of 3 | 3 of 3 | 0 of 3 |
| Kimi K3.1 Pro | 1 of 3 | 3 of 3 | 0 of 3 |
| GPT-6.1 Sol | 0 of 3 | 3 of 3 | 0 of 3 |
All nine runs with swapped definitions made one query and answered 22, without a warning or a further check. All nine with correct definitions also made one query and answered 13. Without a layer, the runs made between zero and 15 queries; only one answered correctly.
What this does and does not show
- The pilot demonstrates one silent failure case. Nine repeats across three models do not replace testing more affected questions.
- The neutral codes make the agent depend on the layer. This isolates the definition error, but does not establish behaviour with readable codes or other sources of meaning.
- Both control questions failed in every condition. Q90 was ambiguous without BIRD's hint, and Q100 required seven values in an exact format. They do not provide a useful check that unrelated answers remain correct.
- BIRD's documentation and answer keys can contain errors. I excluded Q92 and Q97 because their reference queries contradicted the supplied descriptions. Further questions need manual checks before use.
What I want to investigate
Which checks catch wrong, missing, conflicting or outdated definitions, and at what cost? A check that a code exists cannot establish what it means. I want to compare checks against the data, an independent written source and constructed test data, with more affected questions and usable controls.
This extends my work on what makes an AI-generated answer reliable: the scope of the data, the evidence behind the answer and the definitions used to compute it.
References
- J. Li et al., "Can LLM already serve as a database interface? A BIg bench for large-scale database grounded text-to-SQLs," in Proc. NeurIPS 2023, Datasets and Benchmarks Track, 2023. arXiv:2305.03111.
