Research Proposal: Confidence-Bounded Behavioural Judgement for Composed, Evolving Production AI Systems

1. The Problem

A production AI system is a chain, not a single model answering a question. A user query passes through a filter, a retrieval system, one or more models, and a post-processor before it reaches the user:

flowchart LR
    A[User query] --> B[Filter]
    B --> C[Retrieval system]
    C --> D[Model]
    D --> E[Post-processor]
    E --> F[User]

Every link in that chain changes over time, often without anyone deciding it should: the provider updates the model without announcing it, someone edits a prompt, the retrieval corpus grows and gets reindexed, users shift what they ask for. Eventually someone on the team says the system has got worse, and someone else says the numbers look fine, and neither one is obviously wrong.

Chen, Zaharia, and Zou measured this happening directly: GPT-4's accuracy on a maths task dropped from 84% to 51% between March and June 2023, with no announced model change [1]. Ma, Yang, and Kästner found that 58.8% of prompt-and-model combinations dropped in accuracy across silent API updates, with 70.2% of those drops exceeding 5%, and concluded that regression testing large language models needs fundamentally different methods, because correctness itself is harder to pin down, prompts are brittle, and the APIs are nondeterministic [2]. Pan et al. found that 74% of production agent teams still rely mainly on human evaluation, and that the field has no agreed way to identify when errors or low-quality responses are actually occurring [3].

Even with logs, outputs, and scores from a running system, you can't tell whether a change is a genuine regression or ordinary day-to-day variation without a reference point for what acceptable behaviour looks like, and that reference point is almost never formally defined. When the system changes on purpose, the reference point has to change with it, or you end up comparing today's system to a standard that no longer applies.

2. The Research Question

Under what conditions can a reliable, confidence-bounded judgement be made about whether a composed, nondeterministic production AI system is behaving acceptably, using only the partial, sampled signals available during operation, as the system and its acceptable behaviour both evolve?

Every qualifier in that sentence is has a purpose:

• Composed: the system is built from interacting parts, the filter, retrieval, model, and post-processor chain from Section 1, rather than being one model you can point a benchmark at. A fault introduced in one component can surface several stages later, in output that looks fine at the point where the fault actually occurred [12].
• Nondeterministic: the same input can produce different outputs on repeat runs even when nothing is wrong. Ouyang, Zhang, Harman, and Wang found that 75.76% of code generation tasks produced zero identical outputs across repeated requests under default settings [16]. Treat any single observed difference as evidence of failure, and you'll spend most of your time chasing noise instead of faults.
• Confidence-bounded: the judgement comes with an explicit, quantified limit on how often it can be wrong, the kind of guarantee Podkopaev and Ramdas built for single-model risk monitoring [4], rather than a single score with no error rate attached to it.
• Partial, sampled signals: nobody observes every execution. Pan et al. found that 74% of production agent teams still rely mainly on human evaluation of a sample of interactions, not a full audit [3].
• Both evolve: the system changes (new prompts, models, retrieval indices) and so does the standard it's judged against. Shankar et al. showed that the act of grading outputs changes what evaluators believe their own criteria are, so a fixed rubric doesn't survive contact with a system that's still moving [8].

Put plainly: you can't see everything your system does, its outputs vary even when nothing's wrong, and both the system and your sense of what "good" looks like are moving. So when is it actually justified to conclude the system is fine, or that something has broken?

These are two questions that need to be answered in order, because the second one only makes sense once the first is settled:

RQ1: What must a description of acceptable behaviour look like for a composed, evolving production AI system, so that it stays valid and useful as the system changes?

RQ2: Given such a description, how can a genuine behavioural regression be told apart from expected variation, with statistical confidence, as the system's components, prompts, and environment change at the same time?

3. What Exists and Where It Stops

Three research communities have tackled parts of this problem, none cover the full combination.

Community 1: Statistical monitoring and risk detection

Podkopaev and Ramdas (ICLR 2022) introduced sequential risk monitoring (SRM), which tracks a model's running risk and flags when it crosses a predefined threshold, with anytime-valid false-alarm control [4]. Amoukou et al. extended this to unlabelled production data [5]. Prinster, Han, Liu, and Saria (ICML 2025) introduced WATCH, which uses weighted conformal martingales to adapt online as shifts occur [6].

These methods are sound, but they assume three things that don't hold for composed production AI systems: a single model rather than a pipeline, a scalar correctness signal per interaction (classification accuracy, error rate), and a reference distribution that drifts passively rather than one that engineers deliberately change. When a prompt edit, a retrieval reindex, and a silent model update all land in the same deployment window, the reference distribution is being changed on purpose, and neither SRM nor WATCH has any way to tell that apart from a harmful regression.

Community 2: LLM evaluation and judgement

Zheng et al. (NeurIPS 2023) established LLM-as-a-judge as the dominant automated evaluation approach, reporting over 80% agreement with human preferences in structured settings [7]. RAGAS decomposes RAG quality into faithfulness and relevance signals for reference-free evaluation. ARES trains lightweight judges for retrieval and generation scoring.

All of these assume the criteria for a good response are fixed in advance and stay stable. Shankar, Zamfirescu-Pereira, Hartmann, Parameswaran, and Arawjo showed that assumption fails: the act of grading outputs changes what evaluators believe their own criteria are, so no fixed rubric stays valid as the system and its users evolve [8]. LLM-as-a-judge also gives you a score per interaction with no confidence bound on whether that score reflects a real regression or ordinary noise; it grades individual outputs rather than reasoning about the system's statistical behaviour over time.

Community 3: Regression testing and governance for evolving LLM systems

Ma, Yang, and Kästner (CAIN 2024) documented the silent regression problem empirically and argued for fundamentally different regression testing in LLM contexts [2]. Chishti, Oyinloye, and Li (2026) built on this with a governance framework: production contracts (manually written behavioural rules with fixed thresholds), risk-category regression suites, and compatibility gates [9]. Their exploratory validation showed risk-category analysis can surface regressions that aggregate metrics miss.

The limitations are the ones they name themselves: thresholds are set without principled statistical calibration, coverage is limited to single-model API calls rather than composed pipelines, and there's no mechanism for updating contracts when the system legitimately changes [9]. Their contracts work for structured, checkable outputs, JSON formatting, code test pass rates, but they don't apply to open-ended language output, where there's no executable check for correctness.

The gap

No existing method covers a pipeline where several components change deliberately at once, outputs are open-ended language rather than scalar or structured, the reference for acceptable behaviour has to evolve, and the goal is to separate genuine regression from nondeterministic variation with formal statistical confidence. Every method reviewed above needs, as an input, exactly the thing none of them produces: a description of acceptable behaviour that stays valid while the system changes.

4. What This Research Will Do

The work breaks into two phases. Phase 1 works out what a valid behavioural description needs to look like. Phase 2 builds and tests a framework, not just an algorithm, that satisfies those conditions.

Phase 1: What a valid behavioural description requires

flowchart TD
    subgraph P1["Phase 1: what a valid description requires"]
        A1["Failure analysis: Chen et al, Ma et al, Chishti et al"] --> A4["Taxonomy of failure conditions"]
        A2["Gap analysis: SRM and WATCH assumptions"] --> A5["Statement of the theoretical gap"]
        A3["Criteria-drift finding: Shankar et al"] --> A6["Upper bound: no fixed artifact survives"]
        A4 --> B["Necessary conditions for a valid behavioural description"]
        A5 --> B
        A6 --> B
    end
    B --> P2["Phase 2: detection method built on those conditions"]

The goal is to establish, from evidence, what properties a description of acceptable behaviour must have to support reliable judgement as the system evolves: not the right wording, but the right properties.

Three steps:

First, a structured analysis of documented production failures, the Chen et al. GPT-4 accuracy drop [1], the Ma et al. silent API regression study [2], the Chishti et al. exploratory validation [9], and others. For each case: what caused the existing description of acceptable behaviour to fail? Was it fixed while the system moved? Did it assume scalar outputs? Did it break under non-determinism? This produces a taxonomy of failure conditions, the empirical grounding for what any description needs to handle.

Second, a formal analysis of SRM and WATCH [4], [6] to identify exactly which assumptions they take as external inputs but never produce themselves, which gives a precise statement of the theoretical gap: what a description must supply that these methods presuppose.

Third, the criteria-drift finding [8] sets an upper bound: a description can't be a fixed artefact, because fixed artefacts get invalidated as the system and its users co-evolve.

Phase 1's output: a formal account of the necessary conditions for a valid behavioural description, with documented counterexamples showing what happens when each condition is violated. This stands as a publishable empirical and analytical contribution on its own, the first systematic account of why existing descriptions fail and what a working one would require.

Phase 2: A framework built on those conditions

Phase 2 takes the necessary conditions from Phase 1 as constraints and builds a framework for making confidence-bounded judgements about whether a composed production AI system is behaving acceptably. The word "framework" is doing specific work here: an algorithm gives you an answer for one experimental setup, but the deliverable needs to be precise enough that another researcher could apply it to a different pipeline and get a comparable guarantee.

The framework extends the sequential testing foundation of Podkopaev and Ramdas [4] in three directions the existing work doesn't cover. It treats the pipeline, not a single model, as the unit of analysis, aggregating evidence across components with distributed tracing rather than treating each interaction as an independent scalar observation. It handles semantic rather than scalar outputs by keeping a behavioural reference as an embedding-based representation of the system's expected output distribution, one that updates when the system changes deliberately rather than staying fixed. And it uses anytime-valid confidence sequences to keep formal false-alarm control across those deliberate baseline resets, so an intentional change neither triggers a false alarm nor quietly resets the clock in a way that hides a genuine regression.

What the experiment looks like: a controlled production AI pipeline, a RAG-based question-answering system built on an open-source model with a retrieval index. Three conditions get introduced by design, with ground truth known in advance: a silent model swap (replacing the model with a degraded variant), a prompt edit that reduces output quality on a held-out set, and a retrieval index contamination that increases irrelevant retrieval. Each of these is a genuine regression, tested against a no-fault baseline of normal nondeterministic variation:

flowchart LR
    P[Controlled RAG pipeline] --> F1[Silent model swap]
    P --> F2[Prompt edit]
    P --> F3[Retrieval contamination]
    P --> F4[No fault: normal variation]
    F1 --> D[Detection method]
    F2 --> D
    F3 --> D
    F4 --> D
    D --> M1[Detection delay at 5% false-alarm rate]
    D --> M2[Fault attribution accuracy]

A fourth condition, not shown above, tests something existing methods can't handle at all: a legitimate improvement, a deliberate model upgrade or prompt rewrite that genuinely raises quality, which should update the behavioural reference rather than raise an alarm. Getting the first three right while misfiring on this one would mean the framework flags every genuine improvement as a regression, which defeats the point of letting acceptable behaviour evolve in the first place.

What it's compared against: two baselines, the closest existing methods. SRM applied to an LLM-judge quality score per interaction [4], [7], and the production contract framework with manually calibrated thresholds [9]. The point isn't to beat these on a shared benchmark. It's to show the framework handles cases neither baseline can, the semantic-output case and the deliberately-changing-reference case, while matching or beating them on the cases they do cover.

Expected outputs: the framework is evaluated on three things, not one. Detection delay, measured in interactions before it correctly alarms at a fixed false-alarm rate of 5%. Fault attribution accuracy, whether it correctly identifies which pipeline component caused the regression. And confidence calibration, whether a stated confidence level actually matches the observed error rate across repeated trials, so that when the framework reports 90% confidence a regression has occurred, it's right about 90% of the time. Calibration is the harder bar of the three: a method can post a good detection delay by getting lucky on thresholds, but it only earns the "confidence-bounded" part of the research question if the stated confidence and the observed error rate actually line up. Ground truth is known by construction, so the main evaluation needs no human labelling. Thirty trials per condition, 95% confidence intervals, paired statistical tests.

Mitigation: scoped as future work. This research establishes when and why a system has regressed and which component is responsible. What to do about it (rolling back a prompt, switching a model, reindexing retrieval) is a downstream engineering decision this research informs but doesn't prescribe.

The open question this doesn't resolve yet: how to formally express a confidence bound over semantic, open-ended output. For structured outputs, JSON compliance, code test pass rates, the statistical machinery already exists. For a RAG system's answer quality, the signal is softer, and there are two ways to go: define a measurable, defensible proxy signal, or state the framework in terms of any signal meeting certain properties and leave the choice of signal to the application. Which is right depends on what the evaluation literature says about semantic quality under uncertainty, and that reading needs to happen before the Phase 2 design is locked in.

5. Why This Produces Knowledge, Not Just a Tool

Three active research communities have left this question open: under what conditions is a confidence-bounded judgement about production AI system behaviour justified, given partial, sampled, semantically complex evidence from a system whose behaviour and acceptable standards are both moving? The taxonomy of failure conditions and the formal account of necessary properties are the knowledge contribution here. The framework demonstrates that this knowledge can actually be operationalised, and the tool built to run the experiments demonstrates the framework, nothing more.

The research also makes a transferability argument. The proposed conditions and framework come from properties of composed, nondeterministic, deliberately-evolving systems in general, not from any one domain. The validation uses one agent type to demonstrate the approach, and working out which conditions are domain-specific and which are general is part of the contribution: it's what connects this research to other agent types beyond the experimental setting.

6. Roadmap

Step 1: Read into the sequential testing and conformal prediction literature to pin down exactly what formal tools SRM, WATCH, and their extensions provide, and what each assumes. This directly shapes what Phase 1 needs to establish.

Step 2: Run the structured failure analysis across documented production cases and produce the taxonomy. First written output, goes to the supervisor early.

Step 3: Develop the formal account of necessary conditions, grounded in the taxonomy and the gap analysis of existing methods. This is the Phase 1 contribution.

Step 4: Design and build the controlled experimental pipeline, and run the Phase 2 validation experiments for the framework.

Step 5: Write up the fault attribution analysis and the transferability argument.

Step 6: Write the full paper. The taxonomy and failure analysis form the motivation. The formal conditions form the theoretical contribution. The framework and its validation experiment form the empirical contribution. The transferability argument forms the discussion.

References

[1] L. Chen, M. Zaharia, and J. Zou, "How Is ChatGPT's Behavior Changing Over Time?" Harvard Data Science Review, vol. 6, no. 2, Mar. 2024. DOI: 10.1162/99608f92.5317da47.

[2] W. Ma, C. Yang, and C. Kästner, "(Why) Is My Prompt Getting Worse? Rethinking Regression Testing for Evolving LLM APIs," in Proc. IEEE/ACM 3rd Int. Conf. AI Engineering: Software Engineering for AI (CAIN), Lisbon, Portugal, 2024, pp. 166-171.

[3] M. Z. Pan et al., "Measuring Agents in Production," arXiv:2512.04123, 2025.

[4] A. Podkopaev and A. Ramdas, "Tracking the Risk of a Deployed Model and Detecting Harmful Distribution Shifts," in Proc. 10th Int. Conf. Learning Representations (ICLR), 2022.

[5] S. Amoukou, M. Vaysse, T. Kolster, R. Samworth, and S. Saria, "Prediction-Powered Risk Monitoring of Deployed Models for Detecting Harmful Distribution Shifts," arXiv:2602.02229, 2026.

[6] D. Prinster, X. Han, A. Liu, and S. Saria, "WATCH: Adaptive Monitoring for AI Deployments via Weighted-Conformal Martingales," in Proc. Int. Conf. Machine Learning (ICML), 2025.

[7] L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica, "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena," in Advances in Neural Information Processing Systems (NeurIPS), vol. 36, Datasets and Benchmarks Track, 2023.

[8] S. Shankar, J. D. Zamfirescu-Pereira, B. Hartmann, A. G. Parameswaran, and I. Arawjo, "Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences," in Proc. 37th Annual ACM Symposium on User Interface Software and Technology (UIST), 2024, art. 131.

[9] M. S. Chishti, D. P. Oyinloye, and J. Li, "Test Before You Deploy: Governing Updates in the LLM Supply Chain," arXiv:2604.27789, 2026.

[10] E. T. Barr, M. Harman, P. McMinn, M. Shahbaz, and S. Yoo, "The Oracle Problem in Software Testing: A Survey," IEEE Transactions on Software Engineering, vol. 41, no. 5, pp. 507-525, 2015.

[11] V. Riccio, G. Jahangirova, A. Stocco, N. Humbatova, M. Weiss, and P. Tonella, "Testing Machine Learning Based Systems: A Systematic Mapping," Empirical Software Engineering, vol. 25, no. 6, pp. 5193-5254, 2020.

[12] D. Sculley, G. Holt, D. Golovin, E. Davydov, T. Phillips, D. Ebner, V. Chaudhary, M. Young, J.-F. Crespo, and D. Dennison, "Hidden Technical Debt in Machine Learning Systems," in Advances in Neural Information Processing Systems (NeurIPS), vol. 28, 2015, pp. 2503-2511.

[13] S. Shankar, R. Garcia, J. M. Hellerstein, and A. G. Parameswaran, "Operationalizing Machine Learning: An Interview Study," arXiv:2209.09125, 2022.

[14] D. Kreuzberger, N. Kühl, and S. Hirschl, "Machine Learning Operations (MLOps): Overview, Definition, and Architecture," IEEE Access, vol. 11, pp. 31866-31879, 2023.

[15] S. Rabanser, S. Günnemann, and Z. C. Lipton, "Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift," in Advances in Neural Information Processing Systems (NeurIPS), vol. 32, 2019, pp. 1394-1406.

[16] S. Ouyang, J. M. Zhang, M. Harman, and M. Wang, "An Empirical Study of the Non-Determinism of ChatGPT in Code Generation," ACM Transactions on Software Engineering and Methodology, vol. 34, no. 2, art. 42, 2025. DOI: 10.1145/3697010.

[17] A. Paleyes, R.-G. Urma, and N. D. Lawrence, "Challenges in Deploying Machine Learning: A Survey of Case Studies," ACM Computing Surveys, vol. 55, no. 6, art. 114, 2022.

[18] E. Breck, S. Cai, E. Nielsen, M. Salib, and D. Sculley, "The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction," in Proc. IEEE Int. Conf. Big Data, 2017, pp. 1123-1132.

[19] J. M. Zhang, M. Harman, L. Ma, and Y. Liu, "Machine Learning Testing: Survey, Landscapes and Horizons," IEEE Transactions on Software Engineering, vol. 48, no. 1, pp. 1-36, 2022.

[20] J. G. Moreno-Torres, T. Raeder, R. Alaiz-Rodríguez, N. V. Chawla, and F. Herrera, "A Unifying View on Dataset Shift in Classification," Pattern Recognition, vol. 45, no. 1, pp. 521-530, 2012.

[21] J. Gama, I. Žliobaitė, A. Bifet, M. Pechenizkiy, and A. Bouchachia, "A Survey on Concept Drift Adaptation," ACM Computing Surveys, vol. 46, no. 4, art. 44, pp. 1-37, 2014.

[22] J. Lu, A. Liu, F. Dong, F. Gu, J. Gama, and G. Zhang, "Learning under Concept Drift: A Review," IEEE Transactions on Knowledge and Data Engineering, vol. 31, no. 12, pp. 2346-2363, 2019.

[23] A. Yehudai, L. Eden, A. Li, G. Uziel, Y. Zhao, R. Bar-Haim, A. Cohan, and M. Shmueli-Scheuer, "Survey on Evaluation of LLM-based Agents," arXiv:2503.16416, 2025.

[24] C. Qi and X. Qiao, "Building Reliable AI Systems: A Framework That Bridges Architecture and Operations," SSRN working paper, 2026.

[25] K. Anitha Raja, "Continuous Evaluation and Observability for Enterprise AI Agents: A Unified Framework for LLM and ML Systems," International Journal of Computational and Experimental Science and Engineering, vol. 12, no. 1, 2026.

[26] P. Liang, R. Bommasani, T. Lee, D. Tsipras, D. Soylu, M. Yasunaga, Y. Zhang, D. Narayanan, Y. Wu, A. Kumar, et al., "Holistic Evaluation of Language Models," arXiv:2211.09110, 2022. (Also Transactions on Machine Learning Research, 2023.)

[27] M. T. Ribeiro, T. Wu, C. Guestrin, and S. Singh, "Beyond Accuracy: Behavioral Testing of NLP Models with CheckList," in Proc. 58th Annual Meeting of the Association for Computational Linguistics (ACL), 2020, pp. 4902-4912.

[28] S. Amershi, A. Begel, C. Bird, R. DeLine, H. Gall, E. Kamar, N. Nagappan, B. Nushi, and T. Zimmermann, "Software Engineering for Machine Learning: A Case Study," in Proc. IEEE/ACM 41st Int. Conf. Software Engineering: Software Engineering in Practice (ICSE-SEIP), 2019, pp. 291-300.