Xamt++: Adapter-Aware Cross-Framework API Differential Testing

Xamt++ is the release artifact for the current cross-framework API matching and differential testing pipeline. It extends the original XAMT prototype from a small set of framework shims to an adapter-aware matcher and runner covering 10 numerical and deep-learning libraries:

chainer, jax, keras, mindspore, mxnet, numpy, paddle, scipy, tensorflow, and torch.

The artifact contains the code used to mine APIs, form cross-library equivalence groups, validate those groups through adapters, fuzz the executable groups, and replay the current differential bug reproductions.

Artifact Contents

Xamt/
|- tools/
|  |- api_match_common.py              # namespaces, aliases, categories, role rules
|  |- compare_api_matchers.py          # API mining, grouping, confidence scoring
|  |- diff_static_candidate_groups.py  # adapters, execution, output normalization
|  |- timed_group_fuzz.py              # timed fuzzing over matched groups
|  |- artifact_check.py                # metadata and packaging sanity checks
|  `- build_artifact.py                # reproducible source-archive builder
|- bug_repros/
|  |- metadata.json                    # 41 filtered reproduction records (>1e-3 numeric)
|  |- bug_*.py                         # one-command live replayers (original IDs preserved)
|  `- README.md                        # filter criteria and retained-bug audit
|- ARTIFACT.md                         # step-by-step reproduction guide
`- README.md

The older functions/, inputs/, run_tasks/, tests/, and utilities/ directories are retained for compatibility with the original XAMT artifact. The current Xamt++ pipeline is driven by tools/ and bug_repros/.

Current Coverage and Result Snapshot

Current artifact snapshot: 2026-05-25.

Metric	Count
Libraries	10
Pairwise adapter-aware groups	650
Unique APIs in executable groups	4372
Raw mined APIs	8782
Static pairwise DIFF groups	46
Current unique DIFF candidates	194 keys / 137 base groups
Raw live reproducible DIFF scripts	177
Filtered scripts (finite numeric diff > 1e-3)	41

The headline reproduction set is now the 41 scripts that survive a strict numeric filter: a live DIFF whose minority/majority disagreement contains at least one element where both sides are finite and differ by more than 1e-3. Disagreements caused only by NaN/Inf propagation, signed zero, sub-ULP precision, or index/ordering tie-breaks are excluded. Passing the filter means the divergence is a genuine finite-value difference, not that every script is a confirmed upstream defect; subnormal flush-to-zero, branch-cut/tie-method conventions, and ill-conditioned inputs may still explain some entries.

The filter criteria and the per-script retained-bug audit are recorded in bug_repros/README.md; bug_repros/metadata.json holds the machine-readable records for the 41 retained scripts.

Environment Model

The full 10-library run uses one main Python process plus optional external Python workers for libraries that conflict with the main environment.

Main process:

• numpy
• scipy
• torch
• tensorflow
• keras
• jax

External workers:

• XAMT_PADDLE_PY for Paddle
• XAMT_MINDSPORE_PY for MindSpore
• XAMT_CHAINER_PY for Chainer
• XAMT_MXNET_PY for MXNet

If an external worker is not configured, the runner skips that library during API collection or reports it as unavailable during execution. This lets partial runs work in a single environment while keeping the full artifact reproducible on a machine with all worker environments installed.

Observed local main environment:

Python 3.13.5
numpy 2.2.6
scipy 1.16.3
torch 2.11.0+cu128
tensorflow 2.21.0
keras 3.14.0
jax 0.9.2

For a new machine, start with requirements-main.txt, then create external worker environments for Paddle, MindSpore, Chainer, and MXNet as described in ARTIFACT.md.

Quick Start

Run the artifact sanity check. This does not import heavyweight DL libraries.

cd Xamt
python -B tools/artifact_check.py

Inspect matched API coverage from the snapshot table in this README.

sed -n '/## Current Coverage/,/## Environment Model/p' README.md

Replay one curated bug. The exact result depends on which libraries are installed and which external worker variables are set.

python -B bug_repros/bug_001_clip_generic_3.py

Run the deterministic adapter-aware validation over mined APIs.

python -B -m tools.diff_static_candidate_groups \
  --strategy pairwise-adapter-aware \
  --details 20

Run a timed fuzzing pass over already matched groups.

python -B -m tools.timed_group_fuzz \
  --strategy pairwise-adapter-aware \
  --seconds-per-group 60 \
  --include-edge-values \
  --include-nonfinite \
  --stop-on-diff \
  --jsonl results/pairwise_edge_nonfinite.jsonl

Build a clean source artifact archive.

python -B tools/build_artifact.py --out dist/xamtplusplus-artifact.tar.gz

How the Pipeline Works

1. api_match_common.py defines the target namespaces, alias rules, category terms, and parameter role normalization.
2. compare_api_matchers.py mines callables, records signatures/docs, groups APIs by canonical name/category/arity/roles, and assigns confidence scores.
3. diff_static_candidate_groups.py maps a group to executable inputs through per-library adapters, normalizes outputs, and labels the group as PASS, DIFF, ERROR, or SKIP.
4. The pairwise-adapter-aware strategy validates executable pair matches and unions passing pairs into connected API components.
5. timed_group_fuzz.py refreshes canonical inputs with random, edge-value, and nonfinite states, then repeatedly executes each group within a time budget.
6. bug_repros/ replays curated candidates against the current runner and counts only live status: DIFF outputs as reproducible bugs.

Reporting Counts

Use the following default language when reporting this artifact:

Xamt++ covers 10 libraries, 650 adapter-aware API groups, and 4372 APIs in executable matched groups. The current candidate table contains 194 unique DIFF keys across 137 base API groups, of which 177 reproduce live. Applying a strict numeric filter — a live DIFF with at least one finite element-wise difference greater than 1e-3, excluding NaN/Inf, signed-zero, sub-ULP, and index/ordering tie-break disagreements — yields the headline set of 41 filtered reproduction scripts.

Citation

@INPROCEEDINGS{Xamt,
  author={Duan, Bin and Dong, Ruican and Dong, Naipeng and Kim, Dan Dongseong and Yang, Guowei},
  booktitle={2025 IEEE 36th International Symposium on Software Reliability Engineering (ISSRE)},
  title={XAMT: Cross-Framework API Matching for Testing Deep Learning Libraries},
  year={2025},
  pages={191-202},
  keywords={Fuzzing;Software reliability;Testing;Deep Learning Libraries},
  doi={10.1109/ISSRE66568.2025.00030}}