fix: evaluate tau robustness on PU test set and flag limitations

Author: TasfinMahmud

EXPERIMENTS.md

@@ -240,19 +240,25 @@ n_teeth_input: 20 | channel: 2 (planetary x-axis)
 
 * **Known caveats to report honestly:** The accuracy gap is currently backwards and practically noise due to a 99.5% inconclusive rate. This is pending a strict `tau` threshold calibration sweep for gear physics, as well as confirming that GMF strength aligns with the same numerical scale as bearing physics.
 
-## 4. Paderborn University (PU) Dataset - Cross-Domain Domain Shift (Speed)
+## 6. Paderborn University (PU) Dataset - Cross-Domain Domain Shift (Speed)
 
 **Dataset**: Authentic bearing fatigue damages (FAG 6203 deep groove ball bearings).
 **Objective**: Eliminate data leakage by explicitly testing the model's ability to generalize across changing physical operating conditions (Domain Shift). The CNN is trained exclusively on 900 RPM data and tested exclusively on 1500 RPM data.
-**Physics Config**: D=28.5mm, d=6.75mm, N=8, f_s=64kHz. `tau` calibrated to 2.5 on the 1500 RPM calibration split.
+**Physics Config**: D=28.5mm, d=6.75mm, N=8, f_s=64kHz. Baseline CNN Accuracy: 0.704.
 
-### Headline — CNN accuracy by physics verdict (Domain Shift: 900 RPM -> 1500 RPM)
-| Verdict | n | CNN accuracy |
-|---------|---|--------------|
-| CONFIRMED | 2340 | 0.933 |
-| CONFLICT | 1724 | 0.544 |
-| INCONCLUSIVE | 3393 | 0.504 |
-| **Gap (CONFIRMED - CONFLICT)** | | **+0.389** |
+### Test-Set Robustness Check (1500 RPM Test Split)
+To ensure the gap is robust and not just overfitted to a specific threshold, the test set was evaluated across multiple `tau` values:
+
+| Threshold (`tau`) | CONFIRMED Acc | CONFLICT Acc | **Gap** | Inconclusive Rate |
+|-------------------|---------------|--------------|---------|-------------------|
+| 1.0 | 0.918 | 0.553 | **+0.365** | 0.1% |
+| 2.0 | 0.953 | 0.562 | **+0.390** | 25.9% |
+| 2.5 | 0.977 | 0.560 | **+0.417** | 46.3% |
+| 3.0 | 0.987 | 0.566 | **+0.421** | 58.5% |
 
 **Notes**:
-This experiment perfectly proves the necessity of the causal physics engine. Because the baseline CNN was trained only on 900 RPM data, its pattern matching failed catastrophically when tested on 1500 RPM data (Baseline Accuracy crashed to 64.8%). However, the Physics Engine mathematically adjusts for RPM dynamically. It successfully caught the CNN's failures, flagging over 1,700 predictions as CONFLICTS, while isolating 2,340 verified predictions that maintained a 93.3% accuracy. The massive Accuracy Gap of **+38.9%** undeniably proves that the physics engine acts as a robust, domain-aware verification layer for unreliable neural networks.
+Because the baseline CNN was trained only on 900 RPM data, its pattern matching degraded when tested on 1500 RPM data (Baseline Accuracy crashed to 70.4%). The Physics Engine dynamically adjusts for RPM and isolates reliable predictions. As shown above, the gap remains strongly positive across all thresholds, peaking at +0.421.
+
+**Known Limitations**: 
+- **High Inconclusive Rate**: At the optimally calibrated threshold (`tau=2.5`), the engine flags ~46% of predictions as INCONCLUSIVE. This is a known trade-off of the strict verification process.
+- **Scope**: Demonstrated strong robustness on the PU speed-shift task (single dataset, single seed).

validate_pu.py

@@ -188,33 +188,41 @@ def headline_accuracy_by_verdict(report, y_true):
             best_tau = float(tau)
 
     print(f'\n=> Selected optimal tau: {best_tau}')
-    model.symbolic.tau = best_tau
 
-    print('\n[3] Evaluating on Test Set (1500 RPM)...')
+    print('\n[3] Evaluating on Test Set (1500 RPM) Across Multiple Taus...')
+    
+    # Calculate baseline CNN accuracy first (independent of tau)
+    model.symbolic.tau = best_tau
     report = model.diagnose(test_data)
-
-    hb = headline_accuracy_by_verdict(report, y_test)
-
-    # Calculate baseline CNN accuracy
     pred = np.array([r['predicted_class'] for r in report.records])
     baseline_acc = float((pred == np.asarray(y_test)).mean())
-
     print('\n--- FINAL TEST RESULTS (CROSS-DOMAIN PU) ---')
     print(f'Baseline CNN Acc:       {baseline_acc:.3f}')
-    if 'CONFIRMED' in hb:
-        print(
-            f'Physics-Confirmed Acc:  {hb["CONFIRMED"]["cnn_accuracy"]:.3f} (n={hb["CONFIRMED"]["n"]})'
-        )
-    if 'CONFLICT' in hb:
-        print(
-            f'Physics-Conflict Acc:   {hb["CONFLICT"]["cnn_accuracy"]:.3f} (n={hb["CONFLICT"]["n"]})'
-        )
-    if 'INCONCLUSIVE' in hb:
-        print(
-            f'Physics-Inconclusive Acc:{hb["INCONCLUSIVE"]["cnn_accuracy"]:.3f} (n={hb["INCONCLUSIVE"]["n"]})'
-        )
-
-    if 'CONFIRMED' in hb and 'CONFLICT' in hb:
-        gap = hb['CONFIRMED']['cnn_accuracy'] - hb['CONFLICT']['cnn_accuracy']
-        print(f'GAP (CONF - CNFL):      {gap:+.3f}')
+    print('--------------------------------------------')
+
+    taus_to_test = [1.0, 2.0, 2.5, 3.0]
+    if best_tau not in taus_to_test:
+        taus_to_test.append(best_tau)
+        taus_to_test = sorted(list(set(taus_to_test)))
+
+    for test_tau in taus_to_test:
+        model.symbolic.tau = test_tau
+        report = model.diagnose(test_data)
+        hb = headline_accuracy_by_verdict(report, y_test)
+        
+        print(f'\n[Tau = {test_tau:.1f}]')
+        
+        if 'CONFIRMED' in hb:
+            print(f'  Physics-Confirmed Acc:   {hb["CONFIRMED"]["cnn_accuracy"]:.3f} (n={hb["CONFIRMED"]["n"]})')
+        if 'CONFLICT' in hb:
+            print(f'  Physics-Conflict Acc:    {hb["CONFLICT"]["cnn_accuracy"]:.3f} (n={hb["CONFLICT"]["n"]})')
+        
+        if 'INCONCLUSIVE' in hb:
+            inc_n = hb["INCONCLUSIVE"]["n"]
+            inc_pct = (inc_n / len(y_test)) * 100
+            print(f'  Physics-Inconclusive Acc:{hb["INCONCLUSIVE"]["cnn_accuracy"]:.3f} (n={inc_n}, {inc_pct:.1f}%)')
+        
+        if 'CONFIRMED' in hb and 'CONFLICT' in hb:
+            gap = hb['CONFIRMED']['cnn_accuracy'] - hb['CONFLICT']['cnn_accuracy']
+            print(f'  GAP (CONF - CNFL):       {gap:+.3f}')
     print('--------------------------------------------')