Can an AI Agent Find Its Own Data? I Spent a Night Finding Out

I started this experiment because recent work gives surprisingly different answers to what sounds like a simple question:

Is finding the right data still a serious problem for AI agents?

LakeQA suggests that it is. The benchmark puts frontier models over a 9.5 TB data lake containing roughly 40 million documents and 1,007 tasks, yet GPT-5.2 reaches only 18.37% exact match and the best model reported in the paper, Claude Sonnet 4.5, reaches 32.87% [1]. Follow-up work from the same group argues that a substantial part of the failure comes from finding the right datasets in the first place rather than reasoning over them once found [2].

KramaBench points in a different direction. Its best system reaches about 55% end-to-end accuracy, and replacing its retrieval stage with perfect retrieval improves that only to 62% [3], which suggests that discovery is not the dominant bottleneck in that setting.

SANA later tested both benchmarks and found that neither result is universally true: search becomes a major limitation at LakeQA scale, while it matters much less on KramaBench, and data analysis remains difficult across both [4].

That left me with a practical problem. I was considering data discovery for agents as a possible research direction, but I did not want to spend months assuming the problem existed in my own setting simply because it appeared in one benchmark and not another, so I built a smaller experiment and tested the assumption directly.

The experiment

The experiment compares the same question under two conditions, using the same model and the same tools.

flowchart LR
    Q[Same question<br/>Same model<br/>Same tools]
    Q --> C
    Q --> D

    C["<b>Condition C</b><br/>7,985 files<br/>The model must find the right ones"]
    D["<b>Condition D</b><br/>Only the relevant files are available"]

    C --> R1[Answer]
    D --> R2[Answer]

I ran that comparison across three models of increasing capability: gpt-oss 20B, gpt-oss 120B, and GLM-5.3.

Both conditions receive the same toolbox, including grep, find, ls, file previews, and Python, so the only variable that changes is the search space. In Condition C, the model must discover the relevant files itself, while in Condition D those files have already been isolated for it.

The question is therefore very narrow:

How much performance do we lose when the model has to find the evidence before it can reason over it?

If Condition C performs about as well as Condition D, then discovery is probably no longer the main difficulty. If it performs substantially worse, then some of the loss is happening before the reasoning stage even begins.

What I measured

I kept two outcomes separate on purpose:

1. whether the model found the files it needed, and
2. whether it got the final answer right.

That distinction matters because “the model never found the evidence” and “the model found the evidence and still got the answer wrong” are different failures. The first points toward a search or retrieval problem, while the second points toward reasoning or analysis.

For evidence location, I did not count a file as found simply because the model mentioned its name. Its contents had to actually pass in front of the model through a tool call.

I also tracked how much of what the model opened was relevant, because an agent that reads all 7,985 files would technically have perfect recall while demonstrating nothing useful about data discovery.

The dataset

I used real football data from the StatsBomb/Hudl open release [5]. The corpus contains 1,627 matches from the complete 2015/16 Premier League, La Liga, Serie A, and Ligue 1 seasons, reshaped into 7,985 files across about 920 MB, with another 110 real matches from other seasons added as distractors.

I wanted the corpus to behave more like a real data collection than a benchmark that had been cleaned for the model, so I deliberately introduced several forms of messiness: stale exports such as events_v2_FINAL.csv, meaningless filenames such as Untitled spreadsheet.csv, a README describing the folder structure from two refactors ago, matches involving the correct teams but from the wrong season, empty and truncated files, near-duplicates, Latin-1 and UTF-16 encodings, two 130 MB dump files, stale-schema archives, and analyst notes quoting incorrect numbers.

Some of the hardest inconsistencies were already present in the source data.

Player names do not always match across files. A team sheet may say:

Coke
Tana

while the event data refers to the same players as:

Jorge Andújar Moreno
Pedro Taunausú Domínguez Placeres

A literal search cannot connect those names, so some questions require information from more than one file.

The source also contains a real escaping issue where:

N'Golo Kanté

appears as:

N''Golo Kanté

A literal search for the expected name therefore returns nothing. I did not add either of these issues; they were already waiting in the data.

The questions

There are 33 questions, and I did not write any of the answers by hand. Every answer is computed from the raw source data because a hand-written gold answer is another place where I could introduce an error without noticing.

The questions fall into five broad kinds.

flowchart TD
    subgraph RELEVANCE["Find the right information"]
      T1["<b>T1</b> Simple lexical lookup<br/>'Which teams did referee X officiate?'"]
      T2["<b>T2</b> Semantic lookup<br/>'Most passes into the attacking third'"]
      T4["<b>T4</b> Cross-file join<br/>'How many shots did Coke take?'"]
    end

    subgraph COMPLETENESS["Check enough evidence"]
      T3["<b>T3</b> Full-season aggregation<br/>requires many files"]
      T5["<b>T5</b> Prove a negative<br/>'Which team never scored from a corner?'"]
    end

The distinction that eventually mattered most was not the difference between T1, T2, and T4. It was the difference between finding something relevant and knowing that enough evidence has been checked.

I also ran a closed-book control with no tools. The model scored zero out of 33, which meant none of the questions could be answered from memory and all 33 remained valid for the experiment.

Then I discovered my experiment was broken

My first complete result looked dramatic: the model scored 42% when it had to search and 88% when I handed it the relevant files.

That was a large gap, and it was exactly the story I expected to find.

Instead of stopping there, I opened one execution trace and looked at what the agent had actually typed.

turn 1  rg -l "Las Palmas"     -> nothing
turn 2  rg -l "2015-09-23"     -> nothing
turn 3  rg -l "Sevilla"        -> nothing
turn 4  rg -l "Coke"           -> nothing

...

turn 12 rg "Las Palmas" dumps/laliga_all_events.csv
        -> 67507:267273,1,1,0,0,...

The same term failed near the start of the run and succeeded later. The difference was the path argument.

When ripgrep is called without a path and standard input is not a terminal, it searches standard input instead of the current directory [6]. My shell tool spawned every command with a pipe on standard input, and that pipe was always empty, so a command such as:

rg -l "Las Palmas"

was not searching the corpus at all. It was searching nothing.

The agent's main search tool had been silently returning empty results, which meant part of what looked like a discovery failure was actually a broken harness.

Once I started checking the traces more carefully, four more bugs surfaced.

Five bugs that could have changed the conclusion

I discarded 114 completed runs after fixing these issues.

The uncomfortable part was not simply that the harness had bugs. It was that the original result looked completely believable, and several of the bugs pushed the numbers in the same direction.

If I had looked only at the aggregate result, I could easily have concluded that I had found a large agent data-discovery problem when I was partly measuring my own infrastructure.

This kind of hazard is not unique to my setup. Xu et al. report that disabling exploration on Terminal-Bench dropped agent accuracy from 30.6% to 3.4%, which is a reminder that what the surrounding system permits can strongly affect what the model appears able to do [7].

The corrected results

After fixing the harness, I ran the same comparison on three models of increasing capability.

The two smaller models are OpenAI's open-weight gpt-oss models [8], while the strongest model in this experiment is GLM-5.3 through Ollama Cloud.

The gap is real: every model performs worse when it has to find its own evidence, and the paired results are strongly one-sided. When the two conditions disagreed, the version with the relevant files already isolated almost always won.

At the same time, the gap gets smaller as the models get stronger. It stays at 33.3 points for both gpt-oss models, then drops to 24.2 points for GLM-5.3, while the strongest model reaches 73% when searching on its own against 97% when given the files directly.

Searching also costs more. On GLM-5.3, the model consumed 42% more tool output while scoring 24 points lower, and only 26% of the files it opened were relevant, which means roughly three out of every four files it read were wasted effort.

That is enough to say that discovery still has a measurable cost in this setup, but it is not enough to say that generic data discovery is a durable research problem.

What this means for the original question

Before collecting the data, I wrote down the rule I would use to decide whether the direction was worth pursuing.

flowchart TD
    S{Does searching perform<br/>almost as well as being helped?}

    S -->|Yes| STOP1[Discovery is probably not the bottleneck.<br/>Drop it.]

    S -->|No| G{Does the gap shrink<br/>as models improve?}

    G -->|Yes| STOP2[Likely a capability problem<br/>that is already improving.]

    G -->|No| W{Why does it fail?}

    W -->|Ordinary search mistakes| STOP3[Probably drop it.]
    W -->|Fails mainly on exhaustive claims| GO[Investigate further.]

The result lands somewhere between the second and third branches.

There is still a discovery gap, but the strongest model closes part of it, which makes the broad claim that “agents cannot find the right data” a weak long-term research bet for me. If future models continue to reduce that gap without any special systems work, then building a research programme around generic discovery would mean chasing a problem that is already shrinking.

That is a useful negative result. The experiment cost me a night; committing to the wrong direction could have cost months.

The part I am not ready to dismiss

One pattern does not fit neatly into the generic discovery story.

Some questions do not simply ask the model to find useful evidence. They ask it to establish that it has checked enough evidence.

Consider:

Which team never scored from a corner?

Finding five relevant matches does not answer that question, and neither does finding 50. The model needs some reason to believe that it has covered the entire relevant set before it can make a universal claim.

That is different from ordinary retrieval.

A normal search system asks:

Can I find something relevant?

This kind of task asks:

Have I checked everything I need to check?

Those are not the same problem.

The distinction showed up most clearly in the aggregation and absence questions, where one particularly strange failure mode was the model giving the correct answer without reading the evidence required to justify it. From a benchmark perspective, that still counts as correct; from a systems perspective, it is indistinguishable from a lucky guess unless the execution is instrumented well enough to show what evidence was actually used.

I do not want to overstate this result. The sample is small, and on the strongest model this category contains only nine measurements, so one different outcome moves the result by more than 10 percentage points. The pattern across models is also noisy rather than monotonic, which means I do not yet have evidence for a stable trend.

For now, the remaining question is simply:

Do agents have a reliable way to know when they have examined enough data to support an exhaustive claim?

That is interesting enough to test further, but not yet strong enough to call a research problem.

What I learned from the experiment

The main lesson was not that agents are bad at finding files. It was that measuring agents is surprisingly easy to get wrong, especially when the agent sits inside a tool harness whose behaviour can quietly change the result.

My first headline number was wrong for reasons that had nothing to do with the model, yet it looked completely plausible.

A few things I would now do by default in any similar experiment:

• Read raw execution traces, not just aggregate metrics.
• Separate “found the evidence” from “reasoned correctly over the evidence.”
• Track whether correct answers were actually supported by evidence the agent saw.
• Treat tool failures and sandbox behaviour as part of the experiment, not as implementation details.
• Write down the decision rule before seeing the result.

Without those checks, it is very easy to measure the harness and blame the model.

Limitations

This is still a small experiment on one domain, one corpus, 33 questions, and three models, with fewer repetitions on the strongest model than I would want for a formal paper.

The question set is also underpowered for some of the tier-level comparisons I originally planned, so I trust the aggregate Condition C versus Condition D comparison much more than the fine-grained breakdown.

After finding five bugs in my own setup, I also do not want to pretend that one night's results deserve absolute confidence.

The conclusion I am comfortable with is narrower:

There is still a measurable cost when agents have to find their own data, but that cost gets smaller as model capability improves, which makes generic data discovery a weak long-term research bet.

The more interesting remaining question is whether exhaustive claims require stronger guarantees about what evidence has actually been checked.

That still needs more work.

Code

Everything is public on GitHub:

• corpus generation
• ground-truth pipeline
• agent loop
• tool sandbox
• scoring code
• 623 execution traces
• technical notes

The 920 MB corpus itself is not committed because it can be rebuilt from the original StatsBomb release. REGENERATE.md contains the rebuild instructions, while gold.jsonl contains the hash of the frozen question set so the regenerated corpus can be checked against the original experiment.

The project uses Python, Docker, and Ollama.

Models used:

• gpt-oss 20B locally on an M5 Pro
• gpt-oss 120B through Ollama Cloud
• GLM-5.3 through Ollama Cloud

References

[1] H. Wang et al., "LakeQA: An Exploratory QA Benchmark over a Million-Scale Data Lake," ICML 2026. arXiv:2606.10460.
https://arxiv.org/abs/2606.10460

[2] E. Ang et al., "Agentic Data Environments," IEEE Data Engineering Bulletin, vol. 50, no. 1, 2026. Invited paper, not peer reviewed. arXiv:2607.07397.
https://arxiv.org/abs/2607.07397

[3] E. Lai et al., "KramaBench: A Benchmark for AI Systems on Data-to-Insight Pipelines over Data Lakes," arXiv:2506.06541, 2025.
https://arxiv.org/abs/2506.06541

[4] A. S. Wijaya, J. Liu, H. Wang and E. Wu, "SANA: What Matters for QA Agents over Massive Data Lakes?," arXiv:2606.13904, 2026.
https://arxiv.org/abs/2606.13904

[5] Hudl StatsBomb, "Open Data."
https://github.com/hudl/open-data

[6] A. Gallant, "ripgrep."
https://github.com/BurntSushi/ripgrep

[7] J. Xu, T. Zhou, E. Wu and K. Kaffes, "Toward Systems Foundations for Agentic Exploration," Workshop on Systems for Agentic AI, SOSP 2025. arXiv:2510.05556.
https://arxiv.org/abs/2510.05556

[8] OpenAI, "gpt-oss-120b and gpt-oss-20b Model Card," arXiv:2508.10925, 2025.
https://arxiv.org/abs/2508.10925

[9] Q. McNemar, "Note on the sampling error of the difference between correlated proportions or percentages," Psychometrika, vol. 12, no. 2, pp. 153-157, 1947.