|
| 1 | +# Experiments |
| 2 | +A running log of every validation run, benchmark, and cross-domain test for CNSD. |
| 3 | + |
| 4 | +**Discipline for this file** (so it stays a record, not a trophy case): |
| 5 | +- Every entry is tied to a **commit** and a **fixed seed**. The commit's git timestamp is the authoritative date — no manually entered dates. |
| 6 | +- The **run record** (command, commit, environment, config, sample counts) is **auto-generated by the run script** and pasted in verbatim — not transcribed by hand. |
| 7 | +- The experiment and its purpose are stated *before* the result. |
| 8 | +- Null, weak, and unflattering results (abstention rates, accuracy drops, limitations) are recorded alongside the headline numbers. |
| 9 | +- Each entry carries a status: `planned` · `running` · `preliminary` · `validated` · `superseded`. |
| 10 | + |
| 11 | +**How to reproduce any entry**: check out the commit in its run record, prepare the dataset as described in `data/README` (layout + expected sample count / checksum), install the pinned environment (`requirements.txt`), and run the exact command shown in the run record. Numbers should match within run-to-run noise (seeds are fixed; minor GPU nondeterminism is expected). |
| 12 | + |
| 13 | +--- |
| 14 | + |
| 15 | +## Index |
| 16 | + |
| 17 | +| # | Experiment | Domain | Status | |
| 18 | +|---|------------|--------|--------| |
| 19 | +| 1 | CWRU baseline (Protocol B) | Bearing (CWRU) | preliminary | |
| 20 | +| 2 | Threshold sweep | Bearing (CWRU) | preliminary | |
| 21 | +| 3 | Cross-condition robustness (AWGN) | Bearing (CWRU) | preliminary | |
| 22 | +| 4 | Multi-seed headline | Bearing (CWRU) | planned | |
| 23 | +| 5 | Cross-domain: SEU gearbox | Gear (SEU) | preliminary (failed) | |
| 24 | + |
| 25 | +--- |
| 26 | + |
| 27 | +## 1. CWRU baseline — Protocol B (cross-load) |
| 28 | + |
| 29 | +* **Status:** preliminary |
| 30 | +* **Purpose:** confirm the full five-layer pipeline runs end-to-end on real CWRU and establish the baseline diagnosis result. |
| 31 | +* **Setup:** train loads 0–2, test load 3. All 10 classes. 12 kHz, window 1024. |
| 32 | + |
| 33 | +**Run record** |
| 34 | +```text |
| 35 | +commit: cd7771ab3668caf9b33109c3a0a9d89f24fd111c |
| 36 | +command: python validate_run.py --seed 42 |
| 37 | +data: 5806 train / 2019 test samples |
| 38 | +``` |
| 39 | + |
| 40 | +**Layer-2 physics verification rate** |
| 41 | +| Verdict | Rate | |
| 42 | +|---------|------| |
| 43 | +| CONFIRMED | TBD | |
| 44 | +| CONFLICT | TBD | |
| 45 | +| INCONCLUSIVE | TBD | |
| 46 | + |
| 47 | +**Headline — CNN accuracy by physics verdict** |
| 48 | +| Verdict | n | CNN accuracy | |
| 49 | +|---------|---|--------------| |
| 50 | +| CONFIRMED | TBD | TBD | |
| 51 | +| CONFLICT | TBD | TBD | |
| 52 | +| INCONCLUSIVE | TBD | TBD | |
| 53 | +| **Gap (CONFIRMED - CONFLICT)** | | **TBD** | |
| 54 | + |
| 55 | +* **Causal (Layer 3)** — `do(Z)`: rung *TBD*, max_contrast *TBD*, p *TBD* |
| 56 | +* **Counterfactual (Layer 3B)**: DoWhy available *TBD*; method *TBD* |
| 57 | +* **Notes / limitations**: record the INCONCLUSIVE rate and any seed drift. |
| 58 | + |
| 59 | +--- |
| 60 | + |
| 61 | +## 5. Cross-domain — SEU gearbox (GearProvider) |
| 62 | + |
| 63 | +* **Status:** preliminary (failed validation) |
| 64 | +* **Purpose:** demonstrate the framework is genuinely domain-agnostic — same engine, different machine class, only the provider changes. |
| 65 | +* **Setup:** full pipeline on SEU gearset using `GearProvider` (gear-mesh physics). `N_TEETH_INPUT` confirmed against rig spec; fixed channel chosen up front; threshold tuned on a held-out split. |
| 66 | + |
| 67 | +**Run record** |
| 68 | +```text |
| 69 | +commit: <PENDING PR 12 MERGE> |
| 70 | +command: python validate_seu.py |
| 71 | +data: 5115 train / 5115 test samples |
| 72 | +n_teeth_input: 20 | channel: 2 (planetary x-axis) |
| 73 | +``` |
| 74 | + |
| 75 | +**Layer-2 physics verification rate** |
| 76 | +| Verdict | Rate | |
| 77 | +|---------|------| |
| 78 | +| CONFIRMED | 0.2% | |
| 79 | +| CONFLICT | 0.3% | |
| 80 | +| INCONCLUSIVE | 99.5% | |
| 81 | + |
| 82 | +**Headline — CNN accuracy by physics verdict** |
| 83 | +| Verdict | n | CNN accuracy | |
| 84 | +|---------|---|--------------| |
| 85 | +| CONFIRMED | 9 | 0.333 | |
| 86 | +| CONFLICT | 17 | 0.824 | |
| 87 | +| INCONCLUSIVE | 5089 | 0.514 | |
| 88 | +| **Gap (CONFIRMED - CONFLICT)** | | **-0.491 (FAILED)** | |
| 89 | + |
| 90 | +* **Known caveats to report honestly:** The accuracy gap is currently backwards and practically noise due to a 99.5% inconclusive rate. This is pending a strict `tau` threshold calibration sweep for gear physics, as well as confirming that GMF strength aligns with the same numerical scale as bearing physics. |
0 commit comments