Takeaway: the model never touches your database. Your code runs every query, so put the safeguards, such as read-only access and row limits, in that code, not in the prompt.
Lately, some of us write less SQL by hand, mostly due to the rapid advancement of AI, especially in tools that can read our databases and query them for us. You type a question into a chat box, such as "Which region sold the most last month?", and an agent writes the query, runs it and gives you the answer. Agents now use our databases on their own, too. On Neon, a Postgres service, agents created 0.1% of new databases in October 2023 and 80% in October 2025 [1]. When building a data agent, I wanted to know: what happens between the question and the answer?
It is easy to picture the model connecting to the database and running the query itself; however, that's not what happens. The model only reads text and writes text, and "ordinary" code around it does the rest. This post takes one real question through a data agent (that code and the model working together) that I built, from the chat box to the answer. Each step shows the exact text that one part sends to the next. Together, the steps show how the model knows which tables exist, who runs the SQL, and why the model is needed at all if a tool runs the query.
The Setup: One Table, One Question, One Tool
The database is a SQLite file, kenya.sqlite, with one table, ke_table. Each row is one person, with four columns: id, name, region and status. The status column holds a one-letter code: R means resident and V means visitor. That meaning is not stored anywhere in the database. The rows hold only the letters, and the schema describes the column only as status TEXT.
The table holds seven made-up rows:
| id | name | region | status |
|---|---|---|---|
| 1 | Amina | Nairobi | R |
| 2 | Brian | Nairobi | R |
| 3 | Chege | Nairobi | V |
| 4 | Daudi | Coast | R |
| 5 | Esther | Coast | R |
| 6 | Faith | Coast | R |
| 7 | Grace | Nairobi | V |
The user's question/prompt is: "Find the region with the most people in Kenya." Nairobi has four people and Coast has three, so the correct answer is Nairobi, with 4. This question does not need the status column. The last section changes one word in the question, and then the column matters.
The agent has one tool, run_sql(query). A tool is a function that the model can ask the surrounding code to run. This one runs a single read-only SQLite query and returns up to 50 rows.
Every prompt, request and reply in this post comes from a real run of this agent, with seed 2. The code and the saved runs are on GitHub. The selected model is the open-weight gpt-oss:120b [2], running on Ollama's servers, and the code reaches it through Ollama's OpenAI-compatible API [3].
Who Talks to Whom
There are five parts in one answer:
- The user, through a chat UI: a web page or an app.
- The server, which receives the question, loads the schema and builds the prompt.
- The agent code, which holds the agent loop, the tool and the client that calls the model.
- The LLM provider, here Ollama's servers. They turn the request into the model's input, run the model and turn its output back into JSON.
- The database, here one SQLite file.
The server and the agent code together are the backend. In this agent, the whole backend is two Python files: harness.py and llm.py.
Figure 1: The parts and the path. The numbers are the steps in the sections below.
Figure 1 numbers the steps:
- Step 0a, at start-up: the backend reads the schema from the database.
- Step 1: the question arrives.
- Step 2: the backend builds the prompt.
- Step 3: the backend sends the request to the provider.
- Step 4: the provider turns the request into tokens.
- Step 5: the model writes a tool call.
- Step 6: the provider turns the model's text back into JSON.
- Step 7: the backend runs the query.
- Steps 8 and 9: the backend sends the rows back, and the model writes the answer.
In this run, the question needs two calls to the model. The first call writes the SQL, and the second reads the rows and writes the answer. Figure 2 shows the same steps in time order.
Figure 2: One question over time. Only the backend talks to the database.
Before the First Question
Two things exist before anyone types a question: the schema and the tool.
Step 0a: Loading the Schema
At start-up, get_schema() asks SQLite for the text of each table:
SELECT sql FROM sqlite_master WHERE type = 'table';
-- returns: CREATE TABLE ke_table (id INTEGER, name TEXT, region TEXT, status TEXT)
When a table is created, SQLite saves the CREATE TABLE ... statement as text in its schema table, sqlite_master [4]. The backend only reads that text and later pastes it into the prompt.
Note: Why CREATE TABLE, and not DESCRIBE? Currently, SQLite has no DESCRIBE command. MySQL has one [5], and PostgreSQL lists columns as rows in information_schema.columns [6]. The CREATE TABLE form is also what many text-to-SQL methods give the model. In an early comparison of prompt formats, Codex did best on Spider, a standard text-to-SQL benchmark, when each table came as its CREATE TABLE statement plus three sample rows [7]. DAIL-SQL also writes the schema this way, and it reached 86.6% execution accuracy on Spider. In other words, its queries returned the correct rows for 86.6% of the questions [8].
Caveat: The backend reads the schema once, at start-up. If a column changes later, the prompt keeps showing the old column until the backend restarts.
The Tool
The tool lives in llm.py and has two parts. The first is a description in JSON: a name, one sentence and the shape of the input.
RUN_SQL = {
"name": "run_sql",
"description": "Run a read-only SQLite query on the database.",
"parameters": {
"type": "object",
"properties": {"query": {"type": "string"}},
"required": ["query"],
},
}
The second is the Python function run_sql(), which does the actual work. The model only ever sees the JSON. It never sees the function.
This constant is the whole tool registry: one tool and one description. The description does not name ke_table or say what the data is about. The model learns that from the schema in the prompt.
Step 1: The Question Arrives
In a chat app, the user types the question, and the browser sends it to the server:
POST /chat
{"message": "Find the region with the most people in Kenya."}
The request handler reads the text. This agent has no chat UI, so the question comes from the command line instead: harness.py takes it as an argument. Everything after this point is the same.
Step 2: Building the Prompt
The prompt is the text that the model reads. build_prompt() puts it together by joining its parts with blank lines:
def build_prompt(schema, layer_text, question):
parts = [TASK, f"Database schema:\n```sql\n{schema}\n```"]
if layer_text:
parts.append(f"Domain definitions:\n{layer_text}")
parts += [f"Question:\n{question}", ANSWER_FORMAT]
return "\n\n".join(parts)
The function also has a slot for a file of definitions, layer_text. This run has no such file, so the slot stays empty. The last section fills it.
Figure 3: Step 2. The task and the answer format are fixed text in the code. The schema comes from step 0a and the question from step 1.
The backend sends this whole string as one user message. It writes no system message.
This is how the model knows that ke_table exists: the backend pasted its CREATE TABLE text into the prompt. In this design, that is the only way it knows. Another design gives the model a list_tables tool, so that it learns the table names from a tool result instead.
Steps 3 to 6: What Happens at the Provider
This is the part that most of us never see. The backend sends JSON and gets JSON back, but the model in between reads neither.
Figure 4: Steps 3 to 6. The backend sends JSON, the model reads tokens, and the backend gets JSON back.
Step 3: Sending the Request
chat() in llm.py adds the model name, the settings and the tool. Then it sends one HTTPS request to Ollama's OpenAI-compatible endpoint [3], through the OpenAI Python SDK:
POST https://ollama.com/v1/chat/completions
Authorization: Bearer <OLLAMA_API_KEY>
{"model": "gpt-oss:120b",
"messages": [{"role": "user", "content": "<the prompt from step 2>"}],
"seed": 2, "temperature": 1.0, "top_p": 1.0, "reasoning_effort": "medium",
"tools": [{"type": "function", "function": {"name": "run_sql",
"description": "Run a read-only SQLite query on the database.",
"parameters": {"type": "object", "properties": {"query": {"type": "string"}},
"required": ["query"]}}}]}
That is everything the provider gets: the model name, the settings, the messages and the tool description. The provider has no connection to the database.
Step 4: The Chat Template and the Tokenizer
A model reads one long sequence of tokens, which are small numbered pieces of text. So the provider first lays out the request as one text, with a fixed layout called a chat template. Function definitions are "injected into the system message in a syntax the model has been trained on" [9]. For the gpt-oss models, that layout is the published harmony format [10]. The API never shows this text, so I rebuilt it with OpenAI's harmony library [11]:
<|start|>system<|message|>You are ChatGPT, a large language model trained by OpenAI. …
Reasoning: medium …<|end|>
<|start|>developer<|message|># Tools … // Run a read-only SQLite query on the database.
type run_sql = (_: { query: string, }) => any; …<|end|>
<|start|>user<|message|>Answer the question below … Question:
Find the region with the most people in Kenya. …<|end|><|start|>assistant
Two things stand out. First, the provider adds a system message that the backend never wrote. Second, the tool description becomes a few lines of TypeScript-like text in a developer message. So the model learns about the tool the same way it learns about the table: by reading text.
The tokenizer then turns each piece of text into a number. In the question, "Find" is token 11437, " region" is 7967 and " Kenya" is 38396. Ollama counted 200 prompt tokens for this call, but the rebuild has 188. The provider's real template therefore differs from the rebuild in places that are not visible from outside, so the rebuild is close, not exact.
Step 5: The Model Writes a Tool Call
gpt-oss writes in channels, which are labelled parts of its output. It first writes its reasoning in the analysis channel. Ollama returns this reasoning as a separate field, so below it is put back into the harmony format:
<|start|>assistant<|channel|>analysis<|message|>We have a table ke_table with columns id, name, region, status. Probably each row is a person? We need region with the most people in Kenya. … So we need to count per region, order descending, limit 1. Using SQL: SELECT region, COUNT(*) AS cnt FROM ke_table GROUP BY region ORDER BY cnt DESC LIMIT 1; … We need to query. Let's run.<|end|>
Then it writes a tool call. It does this because it was trained on many examples in this format:
<|start|>assistant<|channel|>commentary to=functions.run_sql <|constrain|>json<|message|>
{"query":"SELECT region, COUNT(*) AS cnt FROM ke_table GROUP BY region ORDER BY cnt DESC LIMIT 1;"}<|call|>
<|call|> is a stop token [10]. When the model writes it, the model stops.
Each part of that SQL comes from a different place:
ke_tableandregioncome from the schema text in the prompt.COUNT(*),GROUP BYandORDER BY … DESC LIMIT 1come from what the model learnt about SQL in training.- "Probably each row is a person?" is a guess from the column names. The guess is right, but nothing in the prompt says it.
This is why the model is still needed, even though a tool runs the query. The model writes the SQL, and the tool only runs it. No other part of the system can turn an English question and a schema into a query. It also shows what "using a tool" means for a model. The model writes text in a pattern it was trained on: the tool's name, then the arguments as JSON. The model itself runs nothing.
Step 6: Back to JSON
The output parser turns the model's text back into JSON fields, and Ollama returns HTTP 200:
{"id": "chatcmpl-486", "model": "gpt-oss:120b",
"choices": [{"message": {"role": "assistant", "content": "",
"reasoning": "We have a table ke_table with columns id, name, region, status. …",
"tool_calls": [{"id": "call_v6i1lbkz", "type": "function", "function": {"name": "run_sql",
"arguments": "{\"query\":\"SELECT region, COUNT(*) AS cnt FROM ke_table GROUP BY region ORDER BY cnt DESC LIMIT 1;\"}"}}]},
"finish_reason": "tool_calls"}],
"usage": {"prompt_tokens": 200, "completion_tokens": 147}}
The API returns arguments as a string that contains a JSON object, not as the object itself. The backend must decode that string with json.loads() before it can read the query. The model can also write invalid JSON, so the backend must handle a string that does not decode. handle_call() in step 7 does both.
Step 7: Running the Query
So, who sends the SQL to the database? The backend does, not the model. OpenAI's guide says that when the model returns a function call, the application runs it and sends back the result [9]. Anthropic's documentation says the same for client tools: they run in the application [12].
handle_call() reads the query from the arguments and passes it to run_sql():
def handle_call(arguments, calls_used):
try:
query = json.loads(arguments)["query"]
except (json.JSONDecodeError, KeyError, TypeError) as e:
return arguments, f"ERROR: invalid tool arguments ({e})"
if calls_used > MAX_TOOL_CALLS:
return query, "ERROR: tool call limit reached"
return query, run_sql(query)
One thing is missing here: the tool's name. The code never reads it. With only one tool, every call goes to run_sql(). A system with many tools needs a dispatcher, which looks up the name, calls the matching function and handles a name that does not exist.
run_sql() has a simple guard: the query must start with SELECT, WITH or PRAGMA. The function opens the database read-only, stops any query after 10 seconds and keeps the first 50 rows:
{"columns": ["region", "cnt"], "rows": [["Nairobi", 4]], "row_count": 1}
The guard only checks the first word of the query, not what the query does.
Steps 8 and 9: The Loop and the Answer
Step 8: The Agent Loop
The agent loop is the code that calls the model, runs the tools it asks for and calls the model again. Here, the loop adds the result to the list of messages as a tool message, linked to the call by its id, and then calls the model again. This is the loop from harness.py:
def run_chat(client, cfg, prompt, seed):
messages = [{"role": "user", "content": prompt}]
calls_used = 0
while True:
response = chat(client, cfg, messages, seed=seed, tools=True)
message = response.choices[0].message
messages.append(message.model_dump(exclude_none=True))
calls = []
for call in message.tool_calls or []:
calls_used += 1
query, result = handle_call(call.function.arguments, calls_used)
calls.append(call.id)
messages.append({"role": "tool", "tool_call_id": call.id, "content": result})
if not calls:
return "answered", message.content or ""
if calls_used >= MAX_TOOL_CALLS:
return "step_limit", message.content or ""
Figure 5: The agent loop. Each call sends the whole history again.
The whole decision comes down to one test: does the reply contain tool calls? If it does not, the reply is the answer. I also set a limit of 15 tool calls, MAX_TOOL_CALLS, so that the loop cannot run forever. The limit is a setting in this agent, not a rule of the API.
The second request goes to the same URL, with three messages in messages:
[{"role": "user", "content": "<the prompt from step 2>"},
{"role": "assistant", "content": "",
"reasoning": "We have a table ke_table … Probably each row is a person? …",
"tool_calls": [{"id": "call_v6i1lbkz", "type": "function",
"function": {"name": "run_sql",
"arguments": "{\"query\":\"SELECT region, COUNT(*) AS cnt FROM ke_table GROUP BY region ORDER BY cnt DESC LIMIT 1;\"}"}}]},
{"role": "tool", "tool_call_id": "call_v6i1lbkz",
"content": "{\"columns\": [\"region\", \"cnt\"], \"rows\": [[\"Nairobi\", 4]], \"row_count\": 1}"}]
A chat completions request carries the whole conversation, because the API does not keep earlier turns [13]. So the backend sends everything again, including the model's reasoning from the first call. Ollama counted 381 prompt tokens for this call, against 200 for the first one.
The model cannot see the rows until the backend runs its query and sends them back, so the answer needs a second call. The model decides how many queries it needs. I ran the same question with seeds 1 to 5. In four runs, the model wrote one query. With seed 1, it first checked which values status holds, then made two more queries, so it needed four model calls. Each round goes through steps 3 to 8 again.
Step 9: The Answer
The model writes its answer, and Ollama returns finish_reason: "stop", with no tool call:
The region with the most people in Kenya is **Nairobi**, with 4 entries in the table.
```sql
SELECT region, COUNT(*) AS cnt
FROM ke_table
GROUP BY region
ORDER BY cnt DESC
LIMIT 1;
```
The SQL block is there because the last line of the prompt asks for it. extract_final_sql() takes the last sql block, and harness.py runs it again, so the rows can be checked against the answer. In a chat app, this text is what the user sees.
When the Schema Is Not Enough: Adding a Semantic Layer
Change one word in the question, and the status column starts to matter: Find the region with the most residents in Kenya.
The correct answer is now Coast, with 3 residents (Daudi, Esther and Faith). The model cannot learn that from the database, which holds only the letters. Their meaning lives in the heads of the people who built it, or in their documents.
A semantic layer is where that meaning is written down, for people and tools to read. The dbt Semantic Layer, for example, keeps metric definitions in one central place, so that every tool that uses the data uses the same definitions [14]. This agent's version is much smaller: one markdown file. For this example, the file has one line:
ke_table.status: 'R' = resident, 'V' = visitor.
Step 0b: Loading the Definitions
At start-up, read_layer() reads the file and returns its SHA-256 hash. harness.py saves the hash in the trace, so each run records which version of the file it used. Then build_prompt() fills the empty slot from step 2. It pastes the line under "Domain definitions:", between the schema and the question.
Figure 6: The same flow, with a semantic layer. Only the prompt changes. Every other step is the same.
Nothing else in the flow changes. The definitions reach the model the same way as the schema: as text in the prompt. In the trace, the model's reasoning now starts with "find region with most residents (status='R')", and the SQL filters on it:
SELECT region, COUNT(*) AS resident_count FROM ke_table
WHERE status = 'R' GROUP BY region ORDER BY resident_count DESC LIMIT 1;
SQLite returns [["Coast", 3]], and the answer reads: "The region with the most residents in Kenya is Coast, which has 3 residents." The prompt grew from 200 to 219 tokens.
What happens when the definitions themselves are wrong is the subject of the next post.
Not Every Step Runs in Your Code
In this agent, every step above runs in the backend, because I wrote the loop against a chat completions endpoint. Many systems hand parts of the loop to the provider or to a framework. Then some steps leave your backend, and your code can no longer see, check or log them in the same way:
- Server tools move step 7 to the provider. Anthropic's server tools, such as
web_search,web_fetchandcode_execution, run on Anthropic's infrastructure, and the application gets the results without running anything [15]. Your code cannot put a guard like the one in step 7 in front of those calls. - With remote MCP servers, the provider makes the calls. OpenAI's Responses API can take a remote MCP server as a tool. The API gets the server's list of tools and makes the calls itself, and your code sees them only as items in the response [16]. Anthropic's MCP connector does the same from the Messages API, with no MCP client in your code [17]. With OpenAI's version, the
require_approvalsetting is where a checkpoint stays [16]. - Provider-held conversation state changes step 8. With
previous_response_idin OpenAI's Responses API, the provider keeps the earlier turns, so your code no longer sends them again. You still pay for them: OpenAI bills all earlier input tokens in the chain as input tokens [13]. The full request is also no longer in your own logs, unless you save it. - With an agent framework, step 2 is not yours. The framework builds the prompt and runs the loop, so the text that the model reads is the framework's template plus your inputs. The only way to see that text is to log the final request that leaves your process.
The model does not change in any of these cases. It still reads text and writes a tool call. What changes is who runs the tool, who keeps the history and who can check the text before it reaches the model.
What This Is Really About
Between the question and the answer, the model does one job: it turns text into a query, and rows into an answer. Everything else is ordinary code that someone wrote and can read.
Every piece of knowledge that the model uses arrives as text that the backend chose to send: the schema, the tool description and, with a semantic layer, the definitions. So when a data agent gives an answer that nobody expected, the first place to look is the text it was given, before the model.
If you build data agents, take one question through your own system once, and note which steps run in your code and which do not.
Building a data agent too? I am happy to compare notes. Drop a comment or send me a DM.
References
[1] Databricks, "State of AI Agents," Databricks report, Jan. 2026. https://www.databricks.com/resources/ebook/state-of-ai-agents
[2] OpenAI, "gpt-oss-120b & gpt-oss-20b model card," arXiv:2508.10925, 2025. https://arxiv.org/abs/2508.10925
[3] Ollama, "OpenAI compatibility," Ollama Documentation. https://docs.ollama.com/api/openai-compatibility
[4] SQLite, "The schema table," SQLite Documentation. https://www.sqlite.org/schematab.html
[5] Oracle, "15.8.1 DESCRIBE statement," MySQL 8.4 Reference Manual. https://dev.mysql.com/doc/refman/8.4/en/describe.html
[6] PostgreSQL Global Development Group, "columns," PostgreSQL Documentation. https://www.postgresql.org/docs/current/infoschema-columns.html
[7] N. Rajkumar, R. Li, and D. Bahdanau, "Evaluating the text-to-SQL capabilities of large language models," arXiv:2204.00498, 2022. https://arxiv.org/abs/2204.00498
[8] D. Gao, H. Wang, Y. Li, X. Sun, Y. Qian, B. Ding, and J. Zhou, "Text-to-SQL empowered by large language models: A benchmark evaluation," Proc. VLDB Endow., vol. 17, no. 5, pp. 1132–1145, 2024. https://www.vldb.org/pvldb/vol17/p1132-gao.pdf
[9] OpenAI, "Function calling," OpenAI API Documentation. https://platform.openai.com/docs/guides/function-calling
[10] OpenAI, "OpenAI harmony response format," OpenAI Cookbook. https://developers.openai.com/cookbook/articles/openai-harmony
[11] OpenAI, "harmony: Renderer for the harmony response format," GitHub. https://github.com/openai/harmony
[12] Anthropic, "Tool use with Claude," Claude Documentation. https://docs.claude.com/en/docs/agents-and-tools/tool-use/overview
[13] OpenAI, "Conversation state," OpenAI API Documentation. https://developers.openai.com/api/docs/guides/conversation-state
[14] dbt Labs, "dbt Semantic Layer," dbt Developer Hub. https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl
[15] Anthropic, "How tool use works," Claude Platform Documentation. https://platform.claude.com/docs/en/agents-and-tools/tool-use/how-tool-use-works
[16] OpenAI, "MCP and connectors," OpenAI API Documentation. https://developers.openai.com/api/docs/guides/tools-connectors-mcp
[17] Anthropic, "Tool reference," Claude Platform Documentation. https://platform.claude.com/docs/en/agents-and-tools/tool-use/tool-reference
