Skip to content

fix(resolution): validate rule priority + harden parent-deny immutability against unsat bias#3169

Open
Haruka-Kaya wants to merge 1 commit into
microsoft:mainfrom
Haruka-Kaya:fix/3137-merge-deny-immutability
Open

fix(resolution): validate rule priority + harden parent-deny immutability against unsat bias#3169
Haruka-Kaya wants to merge 1 commit into
microsoft:mainfrom
Haruka-Kaya:fix/3137-merge-deny-immutability

Conversation

@Haruka-Kaya

Copy link
Copy Markdown

Addresses the manifest_resolution/merge.py findings from the robustness review #3137 (per @imran-siddique's request to track fixes as separate per-file PRs).

Unhandled TypeError on non-numeric priority

merge_documents sorts with key=lambda r: r.get("priority", 0). .get(..., 0) only substitutes 0 when the key is absent — an explicit priority: null (→ None) or a typo (priority: highstr) reaches the sort and
raises a bare TypeError comparing against ints. That is not a ResolutionError,
so it escapes the documented contract and may surface as a crash instead of a
fail-closed deny (ADR-0013).

Fix: validate priority as numeric (non-bool int/float) in the upfront
rule-validation loop and raise ResolutionError.invalid_governance otherwise.

Deny-immutability hardening (ADR-0014)

_conditions_disjoint short-circuits to "disjoint" when either side is
classified unsatisfiable. Because the parent deny condition is the left
operand on the deny-immutability path, a false positive in the unsatisfiability
heuristic would declare the parent deny disjoint from a child allow, so
_conditions_overlap returns False and the child allow is kept — silently
neutralising a parent deny.

Fix: _conditions_overlap (used only by the deny-immutability check) now
passes ignore_left_unsatisfiable=True, so the parent (left) side's
unsatisfiability classification can never establish "no overlap". The child
(right) side's unsatisfiability is still honored (an unsatisfiable child allow
never fires, so it overrides nothing).

Honest scope note

_condition_unsatisfiable looks sound today (its and/or/in [] cases all
classify only genuinely-unsatisfiable conditions, and a genuinely-unsatisfiable
deny is inert), so I could not exhibit a concrete exploitable fail-open — the
empty-or repro from the review is an inert deny, not a true neutralisation.
This change is therefore defensive hardening: it removes the structural
dependence of a security-critical check on the heuristic being false-positive
free. For sound classifications it is behavior-preserving and only ever more
conservative (stronger deny immutability), as the regression tests confirm.
Happy to drop it if maintainers prefer to keep merge semantics untouched.

Tests

Adds tests/test_merge_deny.py: non-numeric priorityResolutionError,
priority sort ordering, and deny-immutability behavior (overlapping child allow
dropped, disjoint child allow survives, satisfiable compound parent deny still
protects).

Verified locally against real OPA 0.70.0 (WSL):

platform linux -- Python 3.13.14, pytest-9.0.3
tests/test_merge_deny.py + tests/test_manifest_resolution.py + tests/scenarios/
======================== 195 passed in 3.53s ========================

Refs #3137

🤖 Generated with Claude Code

@github-actions github-actions Bot added tests size/M Medium PR (< 200 lines) labels Jun 24, 2026
@github-actions

github-actions Bot commented Jun 24, 2026

Copy link
Copy Markdown
🤖 AI Agent: docs-sync-checker — Docs Sync

AI-generated review output. Treat it as untrusted analysis and verify before acting.

Docs Sync

  • merge_documents() in merge.py -- missing docstring
  • README.md -- no updates found for the changes in merge_documents() behavior
  • CHANGELOG -- missing entry for changes to merge_documents() behavior regarding priority validation and deny-immutability hardening

@github-actions

github-actions Bot commented Jun 24, 2026

Copy link
Copy Markdown
🤖 AI Agent: breaking-change-detector — API Compatibility

AI-generated review output. Treat it as untrusted analysis and verify before acting.

API Compatibility

Severity Change Impact
High Validation added for priority in merge_documents to ensure it is numeric (int or float). Non-numeric values (e.g., None, str, bool) now raise a ResolutionError.invalid_governance. Existing code passing non-numeric priority values to merge_documents will now fail with an exception.
Medium _conditions_disjoint now includes an ignore_left_unsatisfiable parameter, which changes behavior when used in _conditions_overlap. The parent condition's unsatisfiability is ignored in certain cases to strengthen deny-immutability. This change may alter the behavior of _conditions_overlap in cases where the parent condition is classified as unsatisfiable. This could lead to stricter deny-immutability enforcement, potentially breaking workflows relying on the previous behavior.

@github-actions

github-actions Bot commented Jun 24, 2026

Copy link
Copy Markdown
🤖 AI Agent: code-reviewer — View details

AI-generated review output. Treat it as untrusted analysis and verify before acting.

TL;DR: 0 blockers, 1 warning. The PR improves robustness and security but introduces a minor backward compatibility concern.

# Sev Issue Where
1 Warn Potential backward compatibility issue due to stricter priority validation merge_documents in merge.py

Action items:

  1. None.

Warnings:

# Issue Where Follow-up Action
1 Stricter priority validation may reject previously accepted inputs merge_documents in merge.py Fine as follow-up PRs

No issues found. Clean change.

@github-actions

Copy link
Copy Markdown
🤖 AI Agent: contributor-guide — View details

AI-generated review output. Treat it as untrusted analysis and verify before acting.

Welcome, and thank you for your detailed and well-documented contribution!

What you did well:

  • Excellent problem description and clear explanation of the fixes implemented.
  • Comprehensive test coverage for the changes, including edge cases.

Actionable items before merge:

  1. Ensure that the new test file test_merge_deny.py is included in the appropriate test suite and runs as part of the CI pipeline.
  2. Verify that the changes align with the project's coding style and guidelines. Refer to CONTRIBUTING.md for details.

Let us know if you need any assistance!

@github-actions

github-actions Bot commented Jun 24, 2026

Copy link
Copy Markdown
🤖 AI Agent: test-generator — View details

AI-generated review output. Treat it as untrusted analysis and verify before acting.

Test coverage looks good. No gaps identified.

@github-actions

github-actions Bot commented Jun 24, 2026

Copy link
Copy Markdown
🤖 AI Agent: security-scanner — View details

AI-generated review output. Treat it as untrusted analysis and verify before acting.

No security issues found.

@github-actions

Copy link
Copy Markdown

PR Review Summary

Check Status Details
🔍 Code Review ⚠️ Missing No current-run comment
🛡️ Security Scan ⚠️ Missing No current-run comment
🔄 Breaking Changes ⚠️ Missing No current-run comment
📝 Docs Sync ⚠️ Missing No current-run comment
🧪 Test Coverage ⚠️ Missing No current-run comment

Verdict: ⚠️ AI review incomplete; ready for human review

AI review comments are untrusted advisory output. The summary reports workflow-generated completion status only, not model-authored pass/fail claims.

@MohammadHaroonAbuomar MohammadHaroonAbuomar left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Priority validation is correct (int/float, bool rejected, placed before the single-doc early return). The ignore_left_unsatisfiable flag is threaded through all four recursive arms with a safe default, so the second caller at merge.py:231 is unaffected.

One required rewrite: tests/test_merge_deny.py:101-107 (test_conditions_overlap_protects_parent_against_left_unsat_bias) uses two identical condition dicts. _conditions_overlap returns True at merge.py:294 via _condition_key(parent) == _condition_key(child) before the new code path is reached. The test passes unchanged on main and exercises none of the new logic. Please rewrite with a parent condition that _condition_unsatisfiable actually classifies as unsatisfiable (for example {"and": [{"field":"x","operator":"in","value":[]}, {"field":"tool","operator":"eq","value":"shell"}]}) versus a non-identical child, so the test fails on main and passes here.

None of the other three tests are RED on main either (they pin existing behaviour). Fine to keep as regression guards, but the PR needs at least one RED-to-GREEN test to substantiate the unsat-bias claim, or the description should frame the change as defense-in-depth.

Minor: merge.py:260 _flag = ignore_left_unsatisfiable alias slightly hurts grep-ability; non-blocking.

Haruka-Kaya added a commit to Haruka-Kaya/agent-governance-toolkit that referenced this pull request Jun 25, 2026
…soft#3169@MohammadHaroonAbuomar のレビュー対応:

- 【必須】test_conditions_overlap_..._unsat_bias は同一 condition 2つで
  _condition_key 一致の早期 return を踏むだけで、main でも pass=新ロジックを
  検証できていなかった。レビュアー提案の具体例で書き直し:
  親 deny = {and:[{x in []}, {tool eq shell}]}(_condition_unsatisfiable が
  unsatisfiable と分類するが tool==shell で子と重なる)vs 非同一の子 allow。
  → main では子 allow が生き残る(fail-open)= RED、本ブランチでは drop = GREEN
  を実機で確認済み。これにより本指摘が defense-in-depth ではなく実在の
  fail-open であることも実証。
- _condition_key 早期 return の sanity guard を別テストに分離。
- minor: _flag エイリアスを廃し ignore_left_unsatisfiable を直接渡す(grep性)。

検証: test_merge_deny.py + test_manifest_resolution.py 58 passed。
新テストは main で RED / 本ブランチで GREEN を確認。

Refs microsoft#3137

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VT2CfGX3tmPsQaofuuWwED
@Haruka-Kaya

Copy link
Copy Markdown
Author

@MohammadHaroonAbuomar good catch — you're right that the original test only exercised the _condition_key fast-path. Rewritten in 8238dd8 using your suggested shape:

  • Parent deny {and: [{x in []}, {tool eq shell}]} (which _condition_unsatisfiable classifies unsatisfiable via the empty-in conjunct) vs a non-identical child allow {tool eq shell}.
  • I verified it is RED on main and GREEN here: on main the merge returns ['allow-shell', 'deny-shell'] (the child allow survives — the parent deny is neutralised), and with the fix the child allow is dropped.

So this also settles the framing: it is a genuine fail-open, not just defense-in-depth — I'd been overly cautious in the description. Kept a separate _condition_key fast-path sanity test, and inlined the _flag alias for grep-ability. test_merge_deny.py + test_manifest_resolution.py (58) green.

@imran-siddique imran-siddique left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The two fixes are correct and the test suite is solid after the 8238dd8 rewrite.\n\nPriority validation\n\nisinstance(priority, bool) or not isinstance(priority, (int, float)) is the right guard -- bool subclasses int so the explicit bool check must come first. The validation runs inside the pre-sort loop, before both sort paths (single-doc early return and the multi-doc merge path), so neither sort can receive a non-numeric key.\n\nDeny-immutability hardening\n\nThe ignore_left_unsatisfiable flag is threaded through all four recursive arms (left_or, right_or, left_and, right_and) and defaults to False, so the second call site inside _condition_unsatisfiable (which checks intra-conjunct disjointness, not deny-path logic) is unaffected. The _conditions_overlap caller passes ignore_left_unsatisfiable=True, which is the correct scope.\n\nTests\n\ntest_unsat_classified_parent_deny_still_protects_overlapping_child uses the shape MohammadHaroonAbuomar requested ({and: [{x in []}, {tool eq shell}]} vs {tool eq shell}), which is genuinely RED on main and GREEN here -- confirming this is a real fail-open fix, not just defense-in-depth. The _condition_key fast-path sanity test is correctly kept separate.\n\nAll required CI gates pass. No issues blocking merge.

@Haruka-Kaya Haruka-Kaya force-pushed the fix/3137-merge-deny-immutability branch from 8238dd8 to 901028a Compare June 25, 2026 19:34
…ity against unsat bias

Addresses the manifest_resolution/merge.py findings from review microsoft#3137.

- merge_documents sorted on r.get("priority", 0); an explicit priority: null or
  a non-numeric value raised a bare TypeError. Validate priority as a non-bool
  int/float up front and raise ResolutionError otherwise.
- Deny-immutability (ADR-0014): _conditions_disjoint short-circuited to
  "disjoint" when either side was classified unsatisfiable. On the deny path the
  parent deny is the left operand, so this could neutralise a parent deny.
  _conditions_overlap now passes ignore_left_unsatisfiable so the parent's
  unsatisfiability cannot declare "no overlap"; the child side is still honored.

Adds tests/test_merge_deny.py, including a test that is RED on main (the child
allow survives, neutralising the deny) and GREEN here, demonstrating the fix
closes a real fail-open. Verified with the resolution suite.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VT2CfGX3tmPsQaofuuWwED

Signed-off-by: Himanaraba <kayaharuka@hotmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/M Medium PR (< 200 lines) tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants