LAB 01 / MODEL REGRESSION DETECTOR / ACTIVE
A prompt change.
A measurable regression.
Same eight cases. Same pinned models. Same T4 runtime. The baseline classifies seven correctly; the intentionally weakened candidate classifies two.
Inspect recorded results ↓Real inference · synthetic fixture · captured September 28, 2026
PROBE V2 / OBSERVED RESULTS
Critical failure starts above an eight-percentage-point accuracy drop. This run exceeds that threshold. Invalid responses count as quality failures; missing execution evidence stays visible.
| Measurement | Baseline | Candidate |
|---|---|---|
| Correct classifications | 7 / 8 | 2 / 8 |
| Accuracy | 87.5% | 25% |
| Invalid responses | 0 | 0 |
| Execution errors / missing | 0 / 0 | 0 / 0 |
| Valid summary judgments | 8 / 8 | 8 / 8 |
| Advisory summary score | 4.75 / 5 | 4.875 / 5 |
Regressions: fixture-01, fixture-02, fixture-03, fixture-05, fixture-06, fixture-07.
RUN RECORD / KEEP THE FIRST ATTEMPT
The original baseline returned prose or fenced JSON for all eight cases. Strict validation rejected every response. Accuracy was 0%; the candidate scored 25%, so the relative gate returned PASS.
A passing comparison does not establish a useful baseline. Those original outputs remain preserved.
A new versioned baseline prompt specified an exact JSON shape. Both arms were captured in a fresh session with unchanged model revisions. All responses were valid; accuracy dropped from 87.5% to 25%.
The evidence was finalized before exporting the report. Saved Kaggle version 2 retains both sessions.
REPRODUCIBILITY / SCOPE
This is real model inference on a small synthetic fixture, not a human-reviewed gold evaluation or a general model-quality benchmark. Summary judgments have not been calibrated against human ratings.
989aa7980e4cf806f80c7fef2b1adb7bc71aa306aa8e72537993ba99e69dfaafa59ed015b17504d1support-json-v2 → support-demo-weakenedThis page records observed metrics. Raw bundles are saved in the private Kaggle notebook; a local download has not been verified. Human gold review, full-dataset capture, Databricks acceptance, and Space publication remain pending.