Skip to content

Commit 0e6c30c

Browse files
Cuneyt Ozturkclaude
andcommitted
examples: lm-eval bridge as flagship reference integration (+ in-toto attest beat)
Make the lm-evaluation-harness <-> PRML bridge the host-facing reference integration: - add an [ATTEST] beat emitting the locked claim as an in-toto (ITE-6) Statement (falsify >=0.3.8) — the embed hook a host that ingests SLSA/in-toto can drop in. - README: frame it as the flagship reference integration, note it uses the REAL falsify_prml reference (no parallel schema), link docs/EMBED.md + lm-eval PR #3752. This is the faithful, runnable proof for the embed bet — already on real PRML v0.1, unlike the inspect/giskard adapters (which are on the obsolete schema; migration queued). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 48bcede commit 0e6c30c

2 files changed

Lines changed: 22 additions & 2 deletions

File tree

patterns/examples/README.md

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -25,5 +25,13 @@ python3 lm_eval_to_prml.py --mode lock --results results.json \
2525
--task hellaswag --metric acc_norm --threshold 0.75 # real run
2626
```
2727

28-
Real: PRML canonicalisation/hashing/verdicts (`falsify_prml`). Sample-modelled: a faithful
29-
in-file lm-eval results dict so it runs with no lm-eval install; `--results` accepts a real one.
28+
Real: PRML canonicalisation/hashing/verdicts (`falsify_prml` — the same reference the spec and the
29+
4 byte-equivalent impls use, no parallel schema). Sample-modelled: a faithful in-file lm-eval results
30+
dict so it runs with no lm-eval install; `--results` accepts a real one.
31+
32+
**This is the flagship reference integration** — what it looks like to drop a pre-committed eval claim
33+
into a tool a platform already uses (lm-evaluation-harness, the most-used LLM harness). The demo's
34+
`[ATTEST]` step emits the locked claim as an **in-toto / ITE-6 Statement** (`falsify >= 0.3.8`), so a
35+
host that already ingests SLSA/in-toto can treat a pre-registered eval as one more predicate type — the
36+
3-function embed path is in [`docs/EMBED.md`](https://github.com/studio-11-co/falsify/blob/main/docs/EMBED.md).
37+
Related upstream thread: [lm-evaluation-harness PR #3752](https://github.com/EleutherAI/lm-evaluation-harness/pull/3752).

patterns/examples/lm_eval_to_prml.py

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -181,6 +181,18 @@ def main():
181181
print("\n[VERIFY] after the run — check the result against the sealed bar")
182182
cmd_verify(lock, args)
183183

184+
# The embed hook: hand a host an in-toto (ITE-6) attestation of the locked claim,
185+
# so a platform that already ingests SLSA/in-toto can treat the pre-registered eval
186+
# as one more predicate type. (falsify >= 0.3.8)
187+
if hasattr(prml, "to_intoto_statement"):
188+
print("\n[ATTEST] emit the locked claim as an in-toto (ITE-6) Statement")
189+
stmt = prml.to_intoto_statement(lock["manifest"])
190+
print(f" _type : {stmt['_type']}")
191+
print(f" predicateType : {stmt['predicateType']}")
192+
print(f" subject[0] : {stmt['subject'][0]['name']} "
193+
f"sha256={stmt['subject'][0]['digest']['sha256'][:16]}… (== the PRML lock)")
194+
print(" -> drop this into your evidence bundle / transparency log. See docs/EMBED.md.")
195+
184196
print("\n[ADVERSARIAL] someone lowers the threshold 0.75 -> 0.78 post-hoc")
185197
moved = json.loads(json.dumps(lock))
186198
moved["manifest"]["threshold"] = 0.78 # edit the manifest, keep the old locked hash

0 commit comments

Comments
 (0)