FIELD REPORT 01 / TRAINING SYSTEMS
Reliable AI training starts with reproducible runs.
Recovery and evaluation are part of the platform.
Hugging Face data → reproducible CPU/GPU training → calibrated intent classification. Checkpoint recovery, Ray/PyTorch execution and Databricks reporting support the model lifecycle.
Watch the narrated demo ↓Static evidence site · no account required
NEW / WORKING INTENT CLASSIFIER
A 77-intent classifier trained on Hugging Face BANKING77. Compare a strong linear baseline with a pretrained encoder, calibrate confidence, and route uncertain questions to human review.
Measured on 3,076 held-out questions after model and routing-policy selection. Training, model selection, calibration, and policy tuning use separate partitions.
The local prediction API returns a native intent, three calibrated alternatives, routing reasons, model version, and inference latency. Every output maps to the 77-label vocabulary.
Questions below the confidence or margin threshold go to human review. CPU warm p95: 11.7 ms, measured over 200 single-query model calls after 20 warmups.
One hundred frozen authored challenge queries test unrelated, ambiguous, and multi-intent requests; 6.0% were accepted. The downloadable report includes confidence intervals and per-intent results.
Jamin follows a customer’s missing-card question from intent recognition to a routing decision, then explains human review, measured results, Kaggle training, recovery and traceability. Narrated walkthrough · 5 min 19 sec · English captions. The original engineering walkthrough follows below.
Download new demo · MP4 ↗Download classifier measurements ↗Inspect recorded predictions ↗Classifier flowchart ↗
THE ENGINEERING WALKTHROUGH
Jamin walks through dataset lineage, checkpoint recovery, distributed training, GPU LoRA and Databricks reporting. Follow the architecture and measured results, then inspect the execution below.
01 / ACTUAL EXECUTION
What happens when a training job crashes halfway through? This recording follows a small model learning to sort banking questions into 77 categories. First it trains without interruption. Then a second run is deliberately stopped at update 7 and restarted from the saved checkpoint at update 5.
The test: can the restarted run finish all 20 updates and reach the same training state as the uninterrupted run? The final comparison passed.
How to explore: The terminal starts with the full transcript. Choose a numbered chapter to inspect one stage, or select Play recording to watch the captured output appear in order. Pause to read, change the playback speed, or choose Show all to restore the complete transcript. This replays saved output; it does not launch a new training job.
$ are commands; the lines below are their recorded results.state: failed, exit 1; expected 1 and Driver exited -9 show the planned worker shutdown. committed_step: 5 identifies the last complete saved checkpoint.final_step: 20 and the final “Full-state recovery comparison passed” message show that recovery completed and matched the reference within the stated tolerance.Recorded 2026-09-17 on macOS / arm64, Python 3.12.14. This replay uses a local CPU; the separate T4 LoRA experiment appears in GPU LoRA below.
[0.477s · lineage]
{
"dataset": "mteb/banking77",
"dataset_hash": "375427283fcef7a1e557ab1f9626fb731f7543419b2571e53d055017669dcb07",
"split_counts": {
"train": 2464,
"validation": 154,
"test": 3076
},
"model": "tiny-hash-mlp-banking77-v1",
"max_steps": 20
}
[0.478s · validate]
$ train-demo validate configs/cpu_smoke.yaml
[0.781s · validate]
{
[0.782s · validate]
"valid": true,
[0.782s · validate]
"global_batch_size": 8
[0.782s · validate]
}
[0.827s · validate]
[exit 0; expected 0]
[0.828s · baseline]
$ train-demo run configs/cpu_smoke.yaml
[3.706s · baseline]
{"run_id": "run-090fda28ddf7", "directory": "<repo>/runs/run-090fda28ddf7", "state": "success"}
[3.862s · baseline]
[exit 0; expected 0]
[3.863s · fault]
$ train-demo run configs/cpu_fault.yaml
[5.613s · fault]
{"run_id": "run-d9166c292dde", "directory": "<repo>/runs/run-d9166c292dde", "state": "failed"}
[5.665s · fault]
[exit 1; expected 1]
[5.735s · fault]
{"run_id": "run-d9166c292dde", "state": "failed", "error": "Driver exited -9; see driver-0.log", "committed_step": 5}
[5.736s · resume]
$ train-demo resume --run-id run-d9166c292dde
[8.043s · resume]
{"run_id": "run-d9166c292dde", "directory": "<repo>/runs/run-d9166c292dde", "state": "success"}
[8.242s · resume]
[exit 0; expected 0]
[8.314s · verify]
{
"baseline_run_id": "run-090fda28ddf7",
"recovered_run_id": "run-d9166c292dde",
"restored_steps": [
0,
5
],
"final_step": 20,
"raw_rows": 22,
"canonical_rows": 20,
"verified_state_fields": [
"model",
"optimizer",
"scheduler",
"rng_by_rank",
"global_step",
"epoch",
"sampler_position"
],
"rtol": 0,
"atol": 1e-07
}
[8.536s · evaluate]
{
"run_id": "run-d9166c292dde",
"base_macro_f1": 0.002908097932580854,
"trained_macro_f1": 0.012578938460299316
}
[8.536s · complete]
Full-state recovery comparison passed. CPU-only execution; no cloud training.Baseline run-090fda28ddf7 · recovered run-d9166c292dde. Raw metrics: 22; canonical metrics: 20. Repeated updates remain visible in the raw record. Local absolute paths are replaced with <repo> in this public copy.
02 / MEASURED, NOT PROJECTED
Base macro-F1: 0.0029. Trained: 0.0126. All 77 intents; 20 updates; batch size 8. This lightweight baseline exercises the complete training, evaluation and recovery contract.
Seven state fields match the uninterrupted baseline within the declared tolerance.
Three matched-batch repetitions per backend. Ray / Gloo recovery restored step 5 after rank-1 failure at step 7; final model difference: 0.0.
The matched-batch benchmark identifies where distributed coordination outweighs computation, informing workload sizing before adding workers.
| Backend | Run | Wall time | Macro-F1 |
|---|---|---|---|
| CPU / 1 worker | run-047d1aac07c4 | 3.60 s | 0.0202 |
| CPU / 1 worker | run-d6d704212b1d | 2.68 s | 0.0202 |
| CPU / 1 worker | run-152fb00bdb76 | 2.58 s | 0.0202 |
| Ray / 2 CPU workers | run-19bef4f74d0d | 11.02 s | 0.0201 |
| Ray / 2 CPU workers | run-b089bc348547 | 9.62 s | 0.0201 |
| Ray / 2 CPU workers | run-cdd4739d5375 | 9.96 s | 0.0201 |
Databricks preparation, Delta reads/writes, volume export/readback, repeated imports and comparison queries passed on 2026-09-17.
8 run summaries · 246 raw metrics · 240 canonical metrics · 24,608 paired predictions · zero duplicate-key groups.
Executed notebook results verify idempotent imports and canonical metrics. Imports use table-level keyed MERGE; cross-table writes have separate commit boundaries.
03 / REAL GPU EXECUTION
SmolLM2-135M-Instruct with PEFT LoRA on Kaggle's Tesla T4 allocation. Training and recovery were validated as independent single-GPU jobs.
Same pinned BANKING77 data. Batch 2 × accumulation 4. Peak allocated memory during the successful evaluation resume: 588.0 MiB. Median update: 0.502 s.
15.75 examples/s across updates 3–100. Setup and evaluation are excluded from this training throughput.
A separate short GPU run died at step 3 and resumed at step 2. Full checkpoint state matched the uninterrupted GPU reference; maximum tensor difference: 0.
Recovery restores adapter, optimizer, scheduler, RNG and sampler state, preserving the training trajectory.
| Model | Macro-F1 | Invalid labels |
|---|---|---|
| Pinned base model | 0.0000 | 100.00% |
| LoRA adapter | 0.0490 | 95.64% |
96 predictions improved; 0 regressed. Evaluation exposed a label-generation bottleneck: 95.64% of adapter outputs fall outside the accepted labels. The prompt omitted the 77-label vocabulary, and the base returned no exact valid labels. The classifier extension above addresses this interface directly with a native 77-class head and separate model-selection, calibration, and routing partitions. Combined runtime across the failed first attempt and successful evaluation resume: 56.9 minutes. Sequential evaluation hit the first attempt’s 45-minute deadline; batched evaluation resumed from checkpoint 100. The resume reused all 100 completed training updates.
Databricks imported all four LoRA runs twice: 111 raw metrics, 110 canonical metrics and 3,076 paired predictions. Counts stayed unchanged, with zero duplicate keys. Databricks used CPU serverless for reporting; GPU training and recovery ran on Kaggle.
Download GPU measurements and verification ↗Download all paired predictions ↗
04 / FOLLOW THE ARTIFACTS
FigJam access depends on board permissions. The architecture above and downloadable evidence remain available without sign-in.
mteb/banking77, revision 18072d2685ea682290f7b8924d94c62acc19c0b2. Data hash 375427283fcef7a1e557ab1f9626fb731f7543419b2571e53d055017669dcb07. Source holdout preserved; exact normalized duplicate groups audited. The next data-review stage covers semantic paraphrase overlap and independent label review.
The verified deployment combines single-host CPU/Ray workers, single-GPU LoRA jobs and Databricks reporting. The architecture separates compute, checkpoint state and reporting through versioned artifacts. Production expansion would add durable shared checkpoint storage, managed job orchestration, access controls, monitoring and workload-specific release gates, then validate multi-node throughput and host-loss recovery under load. Verified configuration, recovery and reporting contracts provide the foundation for that expansion.
Download measured evidence JSON ↗ Download presenter steps ↗