Skip to content

EXPERIMENTS.md

Latest commit

 

History

History
90 lines (71 loc) · 3.85 KB

EXPERIMENTS.md

File metadata and controls

90 lines (71 loc) · 3.85 KB
Raw

Experiments

A running log of every validation run, benchmark, and cross-domain test for CNSD.

Discipline for this file (so it stays a record, not a trophy case):

  • Every entry is tied to a commit and a fixed seed. The commit's git timestamp is the authoritative date — no manually entered dates.
  • The run record (command, commit, environment, config, sample counts) is auto-generated by the run script and pasted in verbatim — not transcribed by hand.
  • The experiment and its purpose are stated before the result.
  • Null, weak, and unflattering results (abstention rates, accuracy drops, limitations) are recorded alongside the headline numbers.
  • Each entry carries a status: planned · running · preliminary · validated · superseded.

How to reproduce any entry: check out the commit in its run record, prepare the dataset as described in data/README (layout + expected sample count / checksum), install the pinned environment (requirements.txt), and run the exact command shown in the run record. Numbers should match within run-to-run noise (seeds are fixed; minor GPU nondeterminism is expected).

Index

# Experiment Domain Status
1 CWRU baseline (Protocol B) Bearing (CWRU) preliminary
2 Threshold sweep Bearing (CWRU) preliminary
3 Cross-condition robustness (AWGN) Bearing (CWRU) preliminary
4 Multi-seed headline Bearing (CWRU) planned
5 Cross-domain: SEU gearbox Gear (SEU) preliminary (failed)

1. CWRU baseline — Protocol B (cross-load)

  • Status: preliminary
  • Purpose: confirm the full five-layer pipeline runs end-to-end on real CWRU and establish the baseline diagnosis result.
  • Setup: train loads 0–2, test load 3. All 10 classes. 12 kHz, window 1024.

Run record

commit:   cd7771ab3668caf9b33109c3a0a9d89f24fd111c
command:  python validate_run.py --seed 42
data:     5806 train / 2019 test samples

Layer-2 physics verification rate

Verdict Rate
CONFIRMED TBD
CONFLICT TBD
INCONCLUSIVE TBD

Headline — CNN accuracy by physics verdict

Verdict n CNN accuracy
CONFIRMED TBD TBD
CONFLICT TBD TBD
INCONCLUSIVE TBD TBD
Gap (CONFIRMED - CONFLICT) TBD
  • Causal (Layer 3) — do(Z): rung TBD, max_contrast TBD, p TBD
  • Counterfactual (Layer 3B): DoWhy available TBD; method TBD
  • Notes / limitations: record the INCONCLUSIVE rate and any seed drift.

5. Cross-domain — SEU gearbox (GearProvider)

  • Status: preliminary (failed validation)
  • Purpose: demonstrate the framework is genuinely domain-agnostic — same engine, different machine class, only the provider changes.
  • Setup: full pipeline on SEU gearset using GearProvider (gear-mesh physics). N_TEETH_INPUT confirmed against rig spec; fixed channel chosen up front; threshold tuned on a held-out split.

Run record

commit:   <PENDING PR 12 MERGE>
command:  python validate_seu.py
data:     5115 train / 5115 test samples
n_teeth_input: 20 | channel: 2 (planetary x-axis)

Layer-2 physics verification rate

Verdict Rate
CONFIRMED 0.2%
CONFLICT 0.3%
INCONCLUSIVE 99.5%

Headline — CNN accuracy by physics verdict

Verdict n CNN accuracy
CONFIRMED 9 0.333
CONFLICT 17 0.824
INCONCLUSIVE 5089 0.514
Gap (CONFIRMED - CONFLICT) -0.491 (FAILED)
  • Known caveats to report honestly: The accuracy gap is currently backwards and practically noise due to a 99.5% inconclusive rate. This is pending a strict tau threshold calibration sweep for gear physics, as well as confirming that GMF strength aligns with the same numerical scale as bearing physics.