The Problem
In early 2026, Anthropic's Claude Code became progressively worse over six weeks after three unrelated product changes, and none of the three was caught before customers started reporting it [1]. A reduction in reasoning effort made the tool feel less capable; a caching bug that repeatedly cleared its working context meant it kept redoing reasoning it had already finished, appearing forgetful and repetitive; a system prompt change cut code-generation quality by 3% before being reverted four days later [1]. A September 2025 postmortem traced a separate six-week decline to three unrelated infrastructure bugs, again surfaced only after users noticed [2]. None of this came from the underlying model changing. It came from ordinary product decisions that nobody flagged as risky, going unnoticed for weeks because two structural gaps make this hard to catch: every run of an AI system varies even when nothing has changed, so one bad run looks the same as a real regression, and a vendor's announcement, or the absence of one, says nothing about whether a change was intended or harmful.
What Has Already Been Tried
AgentAssay already provides a calibrated statistical test for exactly this kind of regression: run a task several times, compare the spread against a prior release, decide whether a drop is real [4]. Its own experiments validate this against faults the author introduces deliberately as controlled mutations, not regressions that occurred naturally during real software evolution.
This research investigates that gap using Qwen Code CLI, Alibaba's open-source coding assistant, chosen for two practical reasons: its release history is fully public, so the method needs no access to a vendor's internal data, and an existing published study already evaluated it against the same benchmark used here, so the infrastructure is proven and results can be compared directly [5], [6]. That study analysed all 35 releases together to identify one overall trend, rather than testing whether each individual release was itself a statistically significant regression or improvement, and found real quality swings with no overall improvement despite constant development [5]. Its verdict on any one release also requires the full 35-release dataset already collected, and it identifies causes by manual pull-request review, not an automated test [5].
This research combines both: AgentAssay's calibrated statistical method, applied to Qwen Code CLI's real, naturally occurring release history, using SWE-bench Verified as the fixed benchmark [4]–[6].
Proposed Approach
In short: catch a regression the moment it ships by testing each release against its own recent history rather than one lucky or unlucky run, trace it to its exact cause by undoing changes one at a time, and never mistake silence for safety.
Each release is run several times on the same 50 tasks, since even one unchanged release's pass rate moves around from run to run; that spread sets the noise floor. Two releases are then compared as ranges rather than raw percentages, using a sequential test that keeps its false-alarm rate controlled no matter how many releases it is applied to in a row [7]. For any confirmed drop, the method finds its cause by reverting one changed part of the code at a time and rerunning until the result is statistically reliable [4], then checks that attribution against the release's own changelog and pull requests as an answer key, never the other way around. Separately, each confirmed drop is classified as intended, accidental, or unresolved using commit history, issue reports, and later reversals, since a vendor's own announcement, or its absence, is not reliable evidence of either.
One boundary is worth stating plainly: this measures whether a release still solves each task correctly, not how well. Two releases that both pass are treated as equivalent even if one is slower or less secure, a limitation inherited from the SWE-bench Verified pass/fail signal itself [6], not something this method currently addresses.
Figure 1: The four-part method. New releases are tested using only evidence available at release time. The dashed box is a separately built answer key used only to grade accuracy, never fed into the method.
Research Plan
Work through August 2026 applies the detection test to the twenty releases from v0.0.10 through v0.5.2, which already contain three documented swings to check the method against [5]. Confirmed drops proceed through cause-finding and intent classification in full. This targets a WSDM 2027 submission (abstract due August 17, full paper due August 24). Beyond that, the plan extends to live releases in real time, a second coding assistant, and a public dataset of disclosed and undisclosed AI system quality regressions, since none currently exists.
Expected Contribution and Who Benefits
This research contributes three things beyond existing work [4]–[6]: a detection test with a same-day, statistically bounded verdict rather than a retrospective one; automated, per-incident attribution by reverting and rerunning rather than manual review; and the first evidence-based separation of genuine regressions from disclosed or undisclosed deliberate changes in a real release history.
The beneficiaries are concrete. Developers who use these coding agents day to day gain a way to know whether a release they depend on helped, hurt, or made no difference. Engineering teams building AI coding agents gain a way to catch a genuine regression before customers do. Maintainers of Qwen Code CLI and similar tools gain a labelled record of exactly which historical releases improved, degraded, or held steady. Companies evaluating agent-testing platforms gain an independent test of a method so far demonstrated only by its own creator [4]. Researchers gain both a validated methodology and a set of real, labelled regression cases for future work.
Why a Coding Agent, Not RAG
RAG QA systems were the initial primary system for this research, then set aside in favour of a coding assistant, not because RAG lacks a clear correct answer: a factual question with one gold answer, as in FRAMES [3], is arguably as checkable as a coding task. The actual difficulty is that a RAG system's outcome depends on two separable steps, retrieval and generation, and a failure in either produces the same wrong final answer, an entanglement a coding task's single pass or fail signal never has. A smaller practical issue compounds this: matching generated text against a gold answer needs either an exact match, which can wrongly fail a correct paraphrase, or a judgement call on how close counts as close enough, neither of which a compiled test needs. RAG is retained as a second system, to test whether the method generalises once established where these two issues do not arise.
References
[1] Anthropic, "An update on recent Claude Code quality reports," Anthropic Engineering Blog, Apr. 23, 2026.
[2] Anthropic, "A postmortem of three recent issues," Anthropic Engineering Blog, Sept. 2025.
[3] S. Krishna, K. Krishna, A. Mohananey, S. Schwarcz, A. Stambler, S. Upadhyay, and M. Faruqui, "Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation," arXiv:2409.12941, 2024.
[4] V. P. Bhardwaj, "AgentAssay: Token-Efficient Regression Testing for Non-Deterministic AI Agent Workflows," arXiv:2603.02601, 2026.
[5] O. Ben Sghaier, H. Li, B. Adams, and A. E. Hassan, "Don't Blame the Large Language Model: How Scaffolding Evolution Shapes Coding Agent Quality," arXiv:2607.03691, 2026.
[6] OpenAI, "Introducing SWE-bench Verified," OpenAI Blog, 2024.
[7] A. Podkopaev and A. Ramdas, "Tracking the Risk of a Deployed Model and Detecting Harmful Distribution Shifts," in Proc. ICLR, 2022.