Update threshold sweep logic and add formal validation report

Author: TasfinMahmud

threshold_sweep.py

@@ -6,7 +6,7 @@
 from cnsd.diagnosis.system import CNSD
 
 # Reuse CWRU loader and configs from validate_run
-from validate_run import load_cwru, CWRU, TAXONOMY
+from validate_run import load_cwru, CWRU, TAXONOMY, headline_accuracy_by_verdict
 
 def main():
     print('=' * 50)
@@ -26,18 +26,25 @@ def main():
     model = CNSD()
     model.fit(train_data, epochs=30)
 
-    print('\nStarting Threshold Sweep on Test Data:')
+    print('\nStarting Threshold Sweep on Calibration/Train Data:')
     thresholds = [3.0, 2.5, 2.0, 1.5]
 
     for t in thresholds:
-        print('-' * 40)
+        print('-' * 60)
         print(f'Testing Threshold: {t}')
         model.symbolic.tau = t  # Hot-swap the threshold!
-        report = model.diagnose(test_data)
+        report = model.diagnose(train_data)
         vr = report.verification_rate()
         print(f"  CONFIRMED   : {vr.get('CONFIRMED', 0.0):.1%}")
         print(f"  CONFLICT    : {vr.get('CONFLICT', 0.0):.1%}")
         print(f"  INCONCLUSIVE: {vr.get('INCONCLUSIVE', 0.0):.1%}")
+        
+        hb = headline_accuracy_by_verdict(report, train_data.y)
+        if 'CONFIRMED' in hb and 'CONFLICT' in hb:
+            gap = hb['CONFIRMED']['cnn_accuracy'] - hb['CONFLICT']['cnn_accuracy']
+            print(f"  ACCURACY GAP: +{gap:.3f} (CONFIRMED vs CONFLICT)")
+        else:
+            print("  ACCURACY GAP: N/A (Missing CONFIRMED or CONFLICT samples)")
 
 if __name__ == '__main__':
     main()

validation_report.md

@@ -0,0 +1,41 @@
+# CNSD Pipeline Validation Report (Issue #8)
+
+This document serves as a professional record of the experiments, fixes, and validation runs performed to harden the CWRU baseline for the CNSD causal-neurosymbolic system.
+
+## 1. Loader Bug Fix (Nested Subdirectories)
+The `load_cwru()` function was missing the `@3`, `@6`, and `@12` outer-race classes due to a rigid directory parsing logic. This was fixed by utilizing `os.walk` to comprehensively scan all nested subdirectories for `.mat` files. 
+
+**Result**: All 10 CWRU taxonomy classes are now correctly ingested.
+```text
+[train_data] Dataset 'CWRU_Train': 5806 samples, window=1024, fs=12000Hz, classes=[0, 1, 2, 3, 4, 5, 6, 7, 8, 9], conditions=[0, 1, 2], physics=yes
+[test_data] Dataset 'CWRU_Test': 2019 samples, window=1024, fs=12000Hz, classes=[0, 1, 2, 3, 4, 5, 6, 7, 8, 9], conditions=[3], physics=yes
+```
+
+## 2. Root-Cause Printing Fix
+The root-cause print loop was previously dumping raw physics vectors and showing "No defect detected". The logic was updated to specifically filter for `CONFIRMED` faults and uniquely print the human-readable physical descriptions of each fault class exactly once.
+
+**Result**: Clean, auditable, and human-readable root-cause statements.
+```text
+[examples] auditable root-cause diagnoses (CONFIRMED faults only):
+    [Class 2] [HIGH_CONFIDENCE] Defect on the rolling element (Ball), evidenced by a peak at the BSF (135.9 Hz, strength 3.5). a defect on a rolling element striking both races as it spins.
+    [Class 4] [HIGH_CONFIDENCE] Defect on the inner race (Inner Race), evidenced by a peak at the BPFI (156.1 Hz, strength 4.4). a defect on the rotating inner race, modulated by shaft rotation.
+    [Class 5] [HIGH_CONFIDENCE] Defect on the inner race (Inner Race), evidenced by a peak at the BPFI (156.1 Hz, strength 3.0). a defect on the rotating inner race, modulated by shaft rotation.
+    [Class 6] [HIGH_CONFIDENCE] Defect on the inner race (Inner Race), evidenced by a peak at the BPFI (156.1 Hz, strength 3.3). a defect on the rotating inner race, modulated by shaft rotation.
+    [Class 7] [HIGH_CONFIDENCE] Defect on the outer race (Outer Race), evidenced by a peak at the BPFO (103.4 Hz, strength 4.6). a defect on the stationary outer race struck by each rolling element.
+```
+
+## 3. Threshold Sweep (Calibration Data)
+To reduce the high "INCONCLUSIVE" abstention rate, the symbolic envelope-prominence threshold (`tau`) was swept across the **calibration (training) split** to prevent test-set tuning (data leakage). At each threshold, the accuracy gap between `CONFIRMED` and `CONFLICT` samples was measured. 
+
+**Sweep Results (Calibration Data)**:
+*   **Threshold 3.0**: 49.8% Inconclusive | 2.3% Conflict | Gap: +0.000
+*   **Threshold 2.5**: 37.4% Inconclusive | 4.9% Conflict | Gap: +0.000
+*   **Threshold 2.0**: 23.9% Inconclusive | 12.5% Conflict | Gap: +0.000
+*   **Threshold 1.5**: 8.7% Inconclusive | 27.5% Conflict | Gap: +0.000
+
+*Note: The CNN gap is +0.000 on the calibration set because the CNN perfectly memorizes the training data (100% accuracy).*
+
+**Conclusion**: We officially lock the threshold at **`2.5`**. Lowering the threshold to 2.5 successfully cuts the inconclusive abstention rate by >10% while keeping the false alarm conflicts under a strict 5.0% tolerance. 
+
+## 4. Terminology Clarification
+The script `cross_domain_validation.py` was renamed to `cross_condition_robustness.py` to correctly reflect the protocol (AWGN noise injection). "Cross-domain" is reserved for multi-rig experiments (e.g., SEU/Paderborn).