Skip to content

[STG-2253] ci: validate skills against agent-skills standards#134

Merged
shrey150 merged 3 commits into
mainfrom
add-skills-standards-ci
Jun 16, 2026
Merged

[STG-2253] ci: validate skills against agent-skills standards#134
shrey150 merged 3 commits into
mainfrom
add-skills-standards-ci

Conversation

@shrey150

@shrey150 shrey150 commented Jun 11, 2026

Copy link
Copy Markdown
Contributor

Adds CI enforcement of the agent-skills standards plus a contributor guide.

Linear: STG-2253

What's in this PR

  • scripts/validate-skills.mjs — zero-dependency Node 20+ validator (own minimal strict frontmatter parser, node: builtins only). Supports --skill <name> for single-skill runs.
  • .github/workflows/validate-skills.yml — runs the validator on every PR and on pushes to main.
  • CONTRIBUTING.md — skill anatomy, required frontmatter, license rule, README-table rule, scripts convention, evals, and what reviewers check that CI can't.

What the validator checks

Check Severity
SKILL.md missing, or frontmatter block missing/unparseable error
frontmatter name missing or not equal to the directory name error
frontmatter description missing/empty or >1024 chars error
frontmatter license missing or not MIT error
LICENSE.txt missing, not MIT, or copyright line not containing Browserbase, Inc. error
README table missing a row linking skills/<name>/SKILL.md error
README rows linking to nonexistent skill dirs error
Banned files in a skill dir: setup.json, node_modules/, .env* error
frontmatter compatibility missing warning
frontmatter allowed-tools missing warning
SKILL.md >500 lines with no REFERENCE.md/references/ split warning

Errors exit 1 (fail CI); warnings are printed and exit 0.

Note: the validate-skills check on THIS PR will be red until the standards-update PR merges — that red is the validator correctly flagging main's pre-existing violations (missing LICENSE.txt files, stale/missing README rows, non-MIT license fields). Rebase/rerun after it merges.

E2E Test Matrix

Command / flow Observed output Confidence / sufficiency
node scripts/validate-skills.mjs (against current main tree) Exit 1. Flagged all known violations: browser-to-api/company-research/event-prospecting missing LICENSE.txt; README stale rows site-debugger + bb-usage; README missing rows for agent-experience, autobrowse, browser-to-api, company-research, event-prospecting; autobrowse license See LICENSE.txt; cookie-sync missing license; plus tracked skills/autobrowse/.env.example (matches the .env* ban). Summary: 14 skill(s) checked: 8 passed, 6 failed, 13 error(s), 8 warning(s). Full excerpt below. Proves every error and warning class fires on real repo content, including parsing of all 14 real frontmatter variants (quoted scalars, | block scalars, nested metadata maps and lists).
node scripts/validate-skills.mjs --skill browser [OK ] browser / 1 skill(s) checked: 1 passed... 0 error(s) / exit 0 Proves the pass path and single-skill mode exit 0 on a compliant skill.
node scripts/validate-skills.mjs --skill cookie-sync Exit 1 with license error + compatibility/allowed-tools warnings Proves single-skill mode scopes checks and still fails on errors.
node scripts/validate-skills.mjs --skill nope / --bogus error: skill "nope" not found in skills/ (exit 1); usage message (exit 1) Proves arg-handling error paths.
Live GitHub Actions run of validate-skills.yml on this PR (run 27369208287) Workflow triggered on pull_request, ran node scripts/validate-skills.mjs on ubuntu-latest, produced output identical to the local run, exited 1 (red as predicted). Proves the workflow wiring (checkout + setup-node + script) works end-to-end on GitHub Actions, not just locally.
Validator (this branch) run against origin/update-skills-to-agent-standards (#135) via git archive export All 14 skills [OK]14 skill(s) checked: 14 passed, 0 failed, 0 error(s), 0 warning(s), exit 0 Proves the validator goes green once #135's fixes land — merge #135 first, then rerun this PR's CI.
Full validator output against main (proof of detection)
[FAIL] agent-experience
  error: README.md table has no row linking skills/agent-experience/SKILL.md
  warning: frontmatter `compatibility` is missing
[FAIL] autobrowse
  error: frontmatter `license` must be "MIT" (found "See LICENSE.txt")
  error: banned file: .env.example
  error: README.md table has no row linking skills/autobrowse/SKILL.md
[OK  ] browser
[FAIL] browser-to-api
  error: LICENSE.txt is missing
  error: README.md table has no row linking skills/browser-to-api/SKILL.md
[OK  ] browser-trace
[OK  ] browserbase-cli
[FAIL] company-research
  error: LICENSE.txt is missing
  error: README.md table has no row linking skills/company-research/SKILL.md
[FAIL] cookie-sync
  error: frontmatter `license` is missing (must be `MIT`)
  warning: frontmatter `compatibility` is missing
  warning: frontmatter `allowed-tools` is missing
[FAIL] event-prospecting
  error: LICENSE.txt is missing
  error: README.md table has no row linking skills/event-prospecting/SKILL.md
[WARN] fetch
  warning: frontmatter `compatibility` is missing
[WARN] functions
  warning: frontmatter `compatibility` is missing
  warning: frontmatter `allowed-tools` is missing
[WARN] safe-browser
  warning: frontmatter `compatibility` is missing
[WARN] search
  warning: frontmatter `compatibility` is missing
[OK  ] ui-test
[FAIL] README.md
  error: README.md links to nonexistent skill dir skills/bb-usage/
  error: README.md links to nonexistent skill dir skills/site-debugger/

14 skill(s) checked: 8 passed, 6 failed, 13 error(s), 8 warning(s)

No changeset — CI/docs/tooling only, no published package behavior changes.

🤖 Generated with Claude Code


Note

Low Risk
Docs and CI-only tooling with no runtime or published-package behavior changes; CI may stay red until a separate standards-fix PR lands on main.

Overview
Adds CI enforcement and a contributor guide for agent-skills layout and metadata.

A new zero-dependency scripts/validate-skills.mjs (Node 20+, built-in node: modules only) walks every skills/<name>/ directory and enforces machine-checkable rules: required SKILL.md with a parseable frontmatter block (name matches the folder, non-empty description ≤1024 chars, license: MIT), LICENSE.txt with MIT + Browserbase copyright, no banned paths (setup.json, node_modules, .env*), and README table links that match real skill dirs (including stale-link detection). Missing compatibility / allowed-tools, or an oversized SKILL.md without REFERENCE.md/references/, are warnings only; everything else fails with exit 1. Supports --skill <name> for local single-skill runs.

.github/workflows/validate-skills.yml runs that script on every PR and on pushes to main (Node 20, package-manager-cache: false so setup-node does not require pnpm). CONTRIBUTING.md documents anatomy, frontmatter, license/README/script/eval expectations, reviewer-only checks, and the same local validate commands CI uses.

Reviewed by Cursor Bugbot for commit fb4df7d. Bugbot is set up for automated code reviews on this repo. Configure here.

shrey150 and others added 2 commits June 11, 2026 11:36
Adds scripts/validate-skills.mjs (zero-dependency Node 20+ validator),
a GitHub Actions workflow that runs it on PRs and pushes to main, and
CONTRIBUTING.md documenting the skill standards it enforces.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
setup-node@v5 defaults package-manager-cache to true, reads the
packageManager (pnpm) field from package.json, and tries to prime a
pnpm cache. pnpm isn't installed in the job, so it fails with
"Unable to locate executable file: pnpm" before the validator runs.

validate-skills.mjs uses only Node built-ins (no deps to install),
so disable the auto-cache rather than adding a pnpm setup step.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@shrey150 shrey150 merged commit 86f5d51 into main Jun 16, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants