Skip to content

feat: add contract test suite for sdk-to-spec conformance#18

Open
omermorad wants to merge 17 commits into
mainfrom
feat/docker-contract-tests
Open

feat: add contract test suite for sdk-to-spec conformance#18
omermorad wants to merge 17 commits into
mainfrom
feat/docker-contract-tests

Conversation

@omermorad

@omermorad omermorad commented May 6, 2026 •
edited
Loading

Copy link
Copy Markdown

Summary

  • Adds contract-tests/ workspace with vitest-based contract tests validating every public SDK method against the OpenAPI spec
  • CI fetches the bundled OpenAPI spec and starts Prism locally (no Docker dependency)
  • Each test asserts three things: SDK call resolves (Prism accepted the request and response), response is defined, and correct HTTP method + path via a request-capture harness
  • Prism sentinel test verifies that --errors flag is actively enforcing the spec
  • Coverage guardrail introspects the SDK client at runtime and fails if any public method lacks a test
  • 13 tests with known spec bugs are marked test.fails so CI stays green; they auto-flip when spec fixes land

Architecture

Gateway (openapi.public.yaml)        This PR (contract-tests/):
        |                              tests/ (10 test files, 52 tests)
   bundled as                          src/setup.ts (starts Prism locally)
   spec/openapi.yaml                   src/client.ts (request-capture harness)
        |                              scripts/ (coverage + spec fetch)
        v                                 |
   Prism mock server (local)              v
        +--------- tests run against ----+

Test plan

  • Tests run locally
  • Coverage guardrail verifies all public SDK methods have tests
  • Prism sentinel verifies --errors enforcement is active

@github-actions

github-actions Bot commented May 6, 2026 •
edited
Loading

Copy link
Copy Markdown

SDK Contract Tests Coverage

39/0 tests passed across 12 test files

Code Coverage (v8, source-mapped)

${COVERAGE_TABLE}

Coverage is measured on the built SDK. Resource files at 100% means every public method is exercised by contract tests.

OpenAPI Endpoint Coverage

${OPENAPI_TABLE}

@omermorad omermorad changed the title ci: add contract tests via Dockerized test runner ci: add contract tests via dockerized test runner May 6, 2026
@omermorad omermorad self-assigned this May 10, 2026
omermorad added 3 commits May 11, 2026 19:02
@omermorad
feat: add contract test suite for SDK-to-spec conformance
03663ca 
Run SDK methods against a Prism mock server (with --errors) to validate
that each public method maps to the correct route, sends spec-valid
requests, and deserializes responses without throwing.

Tests use a request-capture harness (custom fetch) to assert HTTP method
and path per call. A prism-sentinel test verifies that Prism's --errors
flag is actively enforcing the spec. A coverage guardrail introspects the
SDK client and fails if any public method lacks a test.

Spec is fetched from Gateway at test time via gh CLI, removing the need
for a Docker image or cross-repo artifact.

13 tests currently fail due to spec issues (missing required fields,
enum mismatches, undocumented routes), not SDK bugs. These are real
conformance findings that need spec fixes.
@omermorad
fix: resolve prism binary directly, add workspace dep, fix CI runner
28b9459 
- Use absolute path to prism binary instead of npx (ensures it resolves
  from contract-tests/node_modules regardless of cwd)
- Add @nimble-way/nimble-js as workspace:* dependency so pnpm links the
  built SDK into the contract-tests workspace
- Use pnpm --filter to run vitest in the correct workspace context
- Apply SDK prettier formatting (printWidth 110, 2-space indent)
@omermorad
refactor: use Docker mock server from GHCR instead of fetching spec
a2bcbcf 
CI starts ghcr.io/nimbleway/sdk-mock-server (Prism + spec) as a
container, then runs tests against it. setup.ts detects if Prism is
already running (Docker case) or starts it locally (dev case with
fetched spec).
@omermorad omermorad changed the title ci: add contract tests via dockerized test runner feat: add contract test suite for SDK-to-spec conformance May 11, 2026
@omermorad omermorad force-pushed the feat/docker-contract-tests branch from 24a716c to a2bcbcf Compare May 11, 2026 16:22
@omermorad omermorad requested a review from davidr-nimble May 11, 2026 16:24
omermorad added 8 commits May 11, 2026 19:24
@omermorad
style: format pnpm-lock.yaml
9e34f57
@omermorad
ci: temporarily use old GHCR image until mock server image is built
dea9fdb
@omermorad
fix(contract-tests): allow package imports in ESLint and drop Docker …
e81c33d 
…from CI

ESLint: add contract-tests/ to the override that disables no-restricted-imports,
since it legitimately imports @nimble-way/nimble-js as a workspace dependency.

CI: replace Docker-based Prism with fetch-spec + local Prism startup via setup.ts.
The sdk-mock-server image does not exist yet (Gateway PR #122 not merged), and the
old sdk-contract-tests image does not work with the entrypoint override.
@omermorad
fix(contract-tests): bundle OpenAPI spec instead of fetching cross-re…
11da94a 
…po in CI

The GITHUB_TOKEN cannot access the internal Gateway repo (404). Bundle
the spec directly so CI does not need cross-repo access. Developers can
update the spec by running: pnpm --filter sdk-contract-tests run fetch-spec
@omermorad
fix(contract-tests): mark 13 tests as expected failures for known spe…
250a5cd 
…c bugs

All failures are OpenAPI spec conformance issues, not test bugs:
- shared batch schema leaks search_engine as required for non-SERP endpoints
- response schemas missing required fields (status_url, _query, url, parsing)
- type/enum mismatches (next_cursor, crawl status, api_type)
- media endpoint only declares binary content type
- agent.publish and domain-knowledge.getDriver routes not in public spec

Using test.fails so CI passes green. When a spec fix lands and the
bundled spec is updated, any fixed test will flip to unexpected-pass,
signaling it should be converted back to a normal test.

Also excludes contract-tests/spec/ from prettier (bundled YAML).
@omermorad
fix(contract-tests): remove agent.publish test (private endpoint, not…
58dfd8f 
… in public spec)
@omermorad
fix(contract-tests): exclude agent.publish from coverage guardrail (p…
9159891 
…rivate method)
@omermorad
fix(contract-tests): fix trailing blank line in agent.test.ts
ccfe305
@omermorad omermorad changed the title feat: add contract test suite for SDK-to-spec conformance feat: add contract test suite for sdk-to-spec conformance May 12, 2026
davidr-nimble
davidr-nimble reviewed May 12, 2026
View reviewed changes
Comment thread contract-tests/src/setup.ts Outdated
omermorad added 6 commits May 12, 2026 18:58
@omermorad
feat(contract-tests): switch CI to Docker mock server from GHCR
c836100 
Gateway PR #122 merged, ghcr.io/nimbleway/sdk-mock-server:latest is
now available. CI pulls the image and runs Prism in Docker instead of
starting it locally.
@omermorad
fix(contract-tests): fix Docker health check (Prism returns 404 on /)
342c4c1
@omermorad
refactor(contract-tests): use testcontainers for Prism mock server
f603d20 
Replace manual process spawn and Docker CLI steps with testcontainers.
It handles image pull, container lifecycle, port mapping, and cleanup.

CI no longer needs manual docker run/stop steps; testcontainers in
setup.ts manages everything. GHCR login is kept for pulling the
private image.
@omermorad
fix: format pnpm-lock.yaml
41c97cd
@omermorad
fix(contract-tests): remove bundled OpenAPI spec (Docker image has it)
ed2d462
@omermorad
fix(contract-tests): remove fetch-spec script (Docker image bundles t…
0c2bb63 
…he spec)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@davidr-nimble davidr-nimble davidr-nimble approved these changes

Assignees

@omermorad omermorad

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@omermorad @davidr-nimble