Merge branch 'develop' into lindbladblock-simplification

Author: Corey Ostrove

.github/workflows/autodeploy.yml

@@ -30,7 +30,7 @@ jobs:
       - name: Build wheels
         uses: pypa/cibuildwheel@v3.0.0
         env:
-          CIBW_BUILD: cp39-* cp310-* cp311-* cp312-*
+          CIBW_BUILD: cp310-* cp311-* cp312-* cp313-*
           CIBW_BUILD_VERBOSITY: 1
           CIBW_BEFORE_ALL_LINUX: ./.github/ci-scripts/before_install.sh
           CIBW_BEFORE_ALL_MACOS: ./.github/ci-scripts/before_install_macos.sh

.github/workflows/beta-master.yml

@@ -14,18 +14,21 @@ jobs:
       fail-fast: false
       matrix:
         os: [macos-latest, ubuntu-latest, windows-latest]
-        python-version: [3.9, '3.10', '3.11', '3.12']
+        python-version: ['3.10', '3.11', '3.12', '3.13', '3.14']
         use-cython: ['true', 'false']
     uses: ./.github/workflows/reuseable-main.yml
     name: Run pyGSTi tests
     with:
       os: ${{ matrix.os }}
       python-version: ${{ matrix.python-version }}
+      run-coverage: ${{ matrix.os == 'ubuntu-latest' && matrix.python-version == '3.13' && matrix.use-cython == 'true' }} # cover exactly one job
       use-cython: ${{ matrix.use-cython }}
       run-unit-tests: 'true'
       run-integration-tests: 'true'
       run-extra-tests: 'true'
-      run-notebook-tests: 'false' # TODO: Turn off failing notebook tests for runner failures starting end of May 2024
+      # Run the docs notebook regression on exactly one job (Linux + Python 3.13 +
+      # Cython). The suite is slow, so we don't fan it out across the full OS/Python matrix.
+      run-notebook-tests: ${{ matrix.os == 'ubuntu-latest' && matrix.python-version == '3.13' && matrix.use-cython == 'true' }}
 
 
 

.github/workflows/bugfix.yml

@@ -15,13 +15,14 @@ jobs:
       fail-fast: false
       matrix:
         os: [ubuntu-latest, windows-latest] # No Mac
-        python-version: [3.9, '3.10', '3.11', '3.12']
+        python-version: ['3.10', '3.11', '3.12', '3.13']
         use-cython: ['true', 'false']
     uses: ./.github/workflows/reuseable-main.yml
     name: Run pyGSTi tests
     with:
       os: ${{ matrix.os }}
       python-version: ${{ matrix.python-version }}
+      run-coverage: ${{ matrix.os == 'ubuntu-latest' && matrix.python-version == '3.13' && matrix.use-cython == 'true' }} # cover exactly one job
       use-cython: 'true' # Cython only
       run-unit-tests: 'true'
       run-integration-tests: 'true'

.github/workflows/develop.yml

@@ -15,18 +15,22 @@ jobs:
       fail-fast: false
       matrix:
         os: [ubuntu-latest, windows-latest] # No Mac
-        python-version: [3.9, '3.10', '3.11', '3.12']
+        python-version: ['3.10', '3.11', '3.12', '3.13']
         use-cython: ['true', 'false']
     uses: ./.github/workflows/reuseable-main.yml
     name: Run pyGSTi tests
     with:
       os: ${{ matrix.os }}
       python-version: ${{ matrix.python-version }}
+      run-coverage: ${{ matrix.os == 'ubuntu-latest' && matrix.python-version == '3.13' && matrix.use-cython == 'true' }} # cover exactly one job
       use-cython: 'true' # Cython only
       run-unit-tests: 'true'
       run-integration-tests: 'true'
       run-extra-tests: 'false' # No integration tests
-      run-notebook-tests: 'false' # No notebook tests 
+      # Run the docs notebook regression on exactly one job (Linux + newest Python +
+      # Cython), the same single-job pattern used for run-coverage above. The suite is
+      # slow, so we don't fan it out across the whole matrix.
+      run-notebook-tests: ${{ matrix.os == 'ubuntu-latest' && matrix.python-version == '3.13' && matrix.use-cython == 'true' }}
 
   push: # Push to stable "beta" branch on successful build
     runs-on: ubuntu-latest
@@ -47,6 +51,52 @@ jobs:
         git checkout beta
         git merge --ff-only ${GITHUB_SHA} && git push origin beta
 
+  docs-notebooks: # Rebuild the ReadTheDocs "develop-with-notebooks" branch
+    # ReadTheDocs builds the hosted docs from `develop-with-notebooks`, NOT from
+    # `develop`. That branch is `develop` plus jupytext-generated *.ipynb
+    # companions committed next to each notebook *.md, so the Colab/Binder launch
+    # buttons (which fetch from GitHub) resolve to real notebooks. We regenerate
+    # the whole branch as a single atomic, force-pushed commit so RTD never sees
+    # a half-built state.
+    runs-on: ubuntu-latest
+
+    # Only after tests pass on a push to develop.
+    needs: build
+    if: github.ref == 'refs/heads/develop' && github.event_name == 'push'
+
+    steps:
+    - uses: actions/checkout@v4
+      with:
+        fetch-depth: 0
+        token: ${{ secrets.PYGSTI_TOKEN }}
+    - uses: actions/setup-python@v5
+      with:
+        python-version: '3.11'
+    - name: Install jupytext
+      run: pip install 'jupytext>=1.16'
+    - name: Build develop-with-notebooks branch
+      run: |
+        git config --global user.name 'PyGSTi'
+        git config --global user.email 'pygsti@noreply.github.com'
+        # Start the branch from the exact develop commit that just passed tests.
+        git checkout -B develop-with-notebooks ${GITHUB_SHA}
+        # Generate an .ipynb next to each notebook .md. --sync honors
+        # docs/jupytext.toml (formats = "ipynb,md:myst") and skips plain-markdown
+        # pages that lack a jupytext header (index/landing pages).
+        shopt -s globstar
+        jupytext --sync docs/markdown/**/*.md
+        # Point the hosted build's launch links at this branch (the
+        # edit_button_branch extension keeps Show source / Suggest edit on develop).
+        # Only the repository.branch key (2-space indent) is rewritten; the
+        # pygsti_edit_branch value stays on develop. The trailing comment is kept.
+        sed -i -E 's|^(  branch: )develop\b(.*)$|\1develop-with-notebooks\2|' docs/_config.yml
+        grep -q '^  branch: develop-with-notebooks' docs/_config.yml  # fail loudly if the rewrite missed
+        # Force-add the generated notebooks (gitignored on develop) + config change.
+        git add -f docs/markdown
+        git add docs/_config.yml
+        git commit -m "Auto-build notebooks for ReadTheDocs [skip ci]"
+        git push --force origin develop-with-notebooks
+
 
 
 

.github/workflows/feature-branches.yml

@@ -19,12 +19,13 @@ jobs:
       fail-fast: false
       matrix:
         os: [ubuntu-latest, windows-latest]
-        python-version: [3.9, '3.12'] # Only extremal Python versions
+        python-version: ['3.10', '3.13'] # Only extremal Python versions
     uses: ./.github/workflows/reuseable-main.yml
     name: Run pyGSTi tests
     with:
       os: ${{ matrix.os }}
       python-version: ${{ matrix.python-version }}
+      run-coverage: ${{ matrix.os == 'ubuntu-latest' && matrix.python-version == '3.13' }} # cover exactly one job
       use-cython: 'true' # Only test environment with Cython
       run-unit-tests: 'true'
       run-integration-tests: 'true'

.github/workflows/reuseable-main.yml

@@ -28,10 +28,29 @@ on:
       run-notebook-tests:
         required: true
         type: string
+      # Whether THIS matrix entry measures coverage. Callers set it true for exactly
+      # one job (ubuntu + highest Python, where coverage.py's sysmon core keeps the
+      # tracer cheap on 3.12+). CI never uploads the report, so measuring it on every
+      # job is pure overhead; one job preserves a human-readable number in the logs.
+      run-coverage:
+        required: false
+        type: string
+        default: 'false'
 
 env:
   SKIP_DEAP: 1
   PYGSTI_NO_CUSTOMLM_SIGINT: 1
+  # Resolve PyTorch (an explicit CI-build test dependency) from the CPU-only wheel index
+  # instead of pip's default CUDA build. The latter drags in ~2.5 GB of nvidia-* wheels
+  # that CI never uses. uv honors this for every resolution in the job.
+  UV_TORCH_BACKEND: cpu
+  # Use coverage.py's sys.monitoring core (PEP 669) for --cov runs. Python 3.12's
+  # specializing adaptive interpreter de-optimizes the legacy C trace function,
+  # which made coverage of call-heavy tests (e.g. the errgenproptools error-generator
+  # composition tests) ~40x slower on 3.12 than on 3.9 -- a Python-version effect
+  # previously misread as a numpy-2 regression. sysmon restores near-native speed;
+  # it falls back gracefully (with a CoverageWarning) on Python < 3.12.
+  COVERAGE_CORE: sysmon
 
 jobs:
   build-and-test:
@@ -56,25 +75,39 @@ jobs:
       uses: actions/setup-python@v5
       with:
         python-version: ${{ inputs.python-version }}
-    - name: Cache pip packages
+    # uv gives fast, parallel installs and CPU-only torch resolution (see env above).
+    # The astral-sh/setup-uv action is blocked by the org Actions allowlist, so we
+    # bootstrap uv from PyPI with the stock pip (a `run:` step is not an action) and
+    # cache its downloads with the GitHub-owned actions/cache.
+    - name: Set up uv
+      run: python -m pip install --upgrade pip uv
+    - name: Cache uv downloads
       uses: actions/cache@v4
       with:
-        path: ~/.cache/pip
-        key: ${{ runner.os }}-pip-${{ hashFiles('setup.py') }}-${{ hashFiles('**/*requirements.txt') }}
-    - name: Install pip packages
-      run: |
-        python -m pip install --upgrade pip
-        python -m pip install wheel
-        python -m pip install flake8
+        # Default uv cache dir, listed for each OS; non-matching paths are skipped.
+        path: |
+          ~/.cache/uv
+          ~/Library/Caches/uv
+          ~\AppData\Local\uv\cache
+        # Key on the real dependency manifests (deps live in pyproject.toml).
+        key: ${{ runner.os }}-uv-${{ hashFiles('pyproject.toml', 'setup.py') }}
+        restore-keys: |
+          ${{ runner.os }}-uv-
+    - name: Install build and lint tools
+      # flake8 for the lint step; setuptools/wheel kept on hand for any dependency that
+      # still imports pkg_resources at runtime.
+      run: uv pip install --system setuptools wheel flake8
     - name: Install package (Cython)
       if: ${{ inputs.use-cython == 'true' }}
+      # The PEP 660 editable install compiles the Cython extensions in-place via build
+      # isolation, so a separate `setup.py build_ext --inplace` is redundant (verified:
+      # the editable install alone yields importable .so for all 19 extensions).
       run: |
-        python -m pip install -e .[testing]
-        python setup.py build_ext --inplace
+        uv pip install --system -e .[testing,pytorch]
     - name: Install package (No Cython, Linux only)
       if: ${{ inputs.use-cython != 'true' && inputs.os == 'ubuntu-latest' }}
       run: |
-        PYGSTI_CYTHON_SKIP=1 python -m pip install -e .[testing_no_cython]
+        PYGSTI_CYTHON_SKIP=1 uv pip install --system -e .[testing_no_cython,pytorch]
     - name: Lint with flake8 (Linux only)
       if: ${{ inputs.os == 'ubuntu-latest'}}
       run: |
@@ -84,26 +117,36 @@ jobs:
         flake8 . --exit-zero --statistics
     - name: Run unit tests
       if: ${{ inputs.run-unit-tests == 'true' }}
+      # bash on every runner (Git Bash on Windows) so $COV_FLAG expands portably.
+      shell: bash
+      env:
+        COV_FLAG: ${{ inputs.run-coverage == 'true' && '--cov=pygsti' || '' }}
       run: |
         python -Ic "import pygsti; print(pygsti.__version__); print(pygsti.__path__)"
-        python -m pytest -n auto --dist loadscope --cov=pygsti --durations=25 test/unit
+        python -m pytest -n auto --dist loadscope $COV_FLAG --durations=25 test/unit
     - name: Run workflow tests
       if: ${{ inputs.run-integration-tests == 'true' }}
+      shell: bash
+      env:
+        COV_FLAG: ${{ inputs.run-coverage == 'true' && '--cov=pygsti' || '' }}
       run: |
         python -Ic "import pygsti; print(pygsti.__version__); print(pygsti.__path__)"
-        python -m pytest -n auto --dist loadscope --cov=pygsti test/integration
+        python -m pytest -n auto --dist loadscope $COV_FLAG test/integration
     - name: Run test_packages
       if: ${{ inputs.run-extra-tests == 'true' }}
       run: |
         python -Ic "import pygsti; print(pygsti.__version__); print(pygsti.__path__)"
         python -m pytest -v -n auto --dist loadscope --ignore=test/test_packages/notebooks --durations=25 test/test_packages
     - name: Run notebook regression
-      if: ${{ inputs.run-notebook-tests == 'true' }}
+      # Linux-only: this step compiles CHP with gcc and writes it to /usr/local/bin,
+      # so it cannot run on the Windows/macOS runners even if a caller requests it.
+      if: ${{ inputs.run-notebook-tests == 'true' && inputs.os == 'ubuntu-latest' }}
       run: |
-        # Tutorials live as MyST Markdown under docs/markdown/; jupytext
-        # materializes the paired .ipynb mirror under docs/notebooks/ for
-        # nbval to consume.
-        pip install -e .[docs]
+        # Tutorials live as MyST Markdown under docs/markdown/; jupytext pairs each
+        # .md with an .ipynb sitting NEXT TO IT in the same directory (see
+        # docs/jupytext.toml), so the materialized notebooks land back under
+        # docs/markdown/ for nbval to consume.
+        uv pip install --system -e .[docs]
         (cd docs && jupytext --sync 'markdown/**/*.md')
 
         # Download and compile CHP for the Clifford-simulation notebook;
@@ -112,4 +155,4 @@ jobs:
         gcc -o /usr/local/bin/chp /tmp/chp.c
 
         python -Ic "import pygsti; print(pygsti.__version__); print(pygsti.__path__)"
-        python -m pytest -n auto --nbval-lax --dist loadscope --nbval-current-env --durations=25 docs/notebooks
+        python -m pytest -n auto --nbval-lax --dist loadscope --nbval-current-env --durations=25 docs/markdown

.gitignore

@@ -46,10 +46,21 @@ test/test_packages/cmp_chk_files/Fake_Dataset_none.txt.cache
 # Documentation autogenerated #
 ###############################
 docs/_autosummary
+docs/_autosummary_manual
 docs/_build
-docs/notebooks/**/*.ipynb
-docs/tutorial_files
-docs/example_files
+# jupytext-generated notebook companions (see docs/jupytext.toml). These are
+# build artifacts on develop; CI commits them on the develop-with-notebooks branch.
+docs/markdown/**/*.ipynb
+# Scratch dirs the tutorials read/write at run time (notebooks reference them as
+# ../../tutorial_files and ../../example_files). Track an empty .keep so the dirs
+# exist on a fresh checkout, but ignore everything the notebooks generate inside.
+docs/tutorial_files/*
+!docs/tutorial_files/.keep
+docs/example_files/*
+!docs/example_files/.keep
+# Offline JS/CSS asset bundle that pyGSTi report writing drops next to the running
+# notebook (CWD) when generating reports; a build artifact of running the tutorials.
+docs/markdown/**/offline/
 
 # Compiled source #
 ###################
@@ -89,6 +100,12 @@ parsetab_string.py
 *.tar
 *.zip
 
+# Uncompressed differential circuit corpus (test/performance/circuit_corpus.py).
+# A full corpus is ~120 MB; only the compressed baseline is tracked (see the
+# negation in test/performance/.gitignore). Guarded here too in case it is
+# generated from the repo root, not just under test/performance/.
+*.jsonl
+
 # Logs and databases #
 ######################
 *.log
@@ -110,3 +127,7 @@ HighImpactPaper1/texFiles/.DS_Store
 local
 pygsti/_version.py
 1_1.profile/0.prof
+
+# committed golden fixtures for the Circuit characterization tests
+!test/unit/circuits/golden/*.pkl
+!test/unit/circuits/golden/*.pkl.gz

.readthedocs.yaml

@@ -10,7 +10,11 @@ build:
   commands:
     # rtd-requirements.txt installs `.[docs]` which pulls in pyGSTi itself
     # plus jupyter-book, jupytext, matplotlib, numpy.
-    - pip install -r rtd-requirements.txt
+    # PYGSTI_SKIP_CYTHON: the API docs only need pyGSTi importable; the
+    # pure-Python fallback reps suffice, so don't spend build time compiling.
+    - PYGSTI_SKIP_CYTHON=1 pip install -r rtd-requirements.txt
     - jb build docs/
+    # .doctrees is Sphinx's intermediate cache (~130 MB); don't publish it.
+    - rm -rf docs/_build/html/.doctrees
     - mkdir -p $READTHEDOCS_OUTPUT/html
     - cp -r docs/_build/html/. $READTHEDOCS_OUTPUT/html/

README.md

@@ -36,7 +36,7 @@ In particular, there are a number of characterization protocols currently implem
 
 PyGSTi is designed with a modular structure so as to be highly customizable
 and easily integrated to new or existing python software.  It runs using
-python 3.9 or higher.  To facilitate integration with software for running
+python 3.10 or higher.  To facilitate integration with software for running
 cloud-QIP experiments, pyGSTi `Circuit` objects can be converted to IBM's
 **OpenQASM** and Rigetti Quantum Computing's **Quil** circuit description languages.
 
@@ -188,23 +188,21 @@ Assuming you've followed the *local installation* directions above:
 * Change to the docs directory, by running:
     ``cd docs``
 
-* Convert the Markdown files to their corresponding notebooks:
-    ``jupytext --sync markdown/**/*.md``
-
-* Change to the newly populated `notebooks` directory:
-    ``cd notebooks``
+* Build the notebooks from their Markdown sources, by running:
+    ``jupytext --to notebook markdown/**/*.md`` 
+  This writes an `.ipynb` next to each `.md` under `markdown/`.
 
 * Start up the Jupyter notebook server by running ``jupyter notebook`` or a JupyterLab server by running ``jupyter lab``.
 
 The Jupyter server should open up your web browser to the server root, from
-where you can start the first `intro.ipynb` notebook.  Note that the key
+where you can start the first `markdown/intro.ipynb` notebook.  Note that the key
 command to execute a cell within the Jupyter notebook is ``Shift+Enter``, not
 just ``Enter``.
 
 #### Contributing notebook changes
-**Only the `docs/markdown/*.md` files are version-controlled.** The `docs/notebooks/` directory is gitignored —
-it's a build artifact materialized by `jupytext --sync`. The canonical source is the `.md` file, and edits there
-"win" on the next sync. So:
+**Only the `docs/markdown/*.md` files are version-controlled.** The paired `.ipynb` files — generated next to
+each `.md` under `docs/markdown/` — are gitignored build artifacts. The canonical source is the `.md` file, and
+edits there "win" on the next sync. So:
 
 - If you find it more convenient to edit the `.ipynb` (e.g. for interactive iteration), **sync your edits back to
   Markdown before committing**: from the `docs/` folder run `jupytext --sync markdown/**/*.md`, then commit the updated `.md`.

agent-docs/01-representation.md

@@ -150,6 +150,8 @@ A few notes on the structure:
 
 The third layer has a non-obvious consequence: **representation degeneracies in state prep** (and to a lesser extent measurement). Multiple distinct parameter-vector values can map to the same physical noisy SPAM, because what you're modeling is the *combined* prep-plus-noisy-op, and the split between "ideal prep" and "noisy op" is non-unique. This is a sort of "gauge freedom" (although not the kind relevant to gauge optimization). If you're debugging a fit and SPAM's contributions to the parameter vector look weird, this might be why.
 
+**Instruments** extend the same pattern to mid-circuit measurements. Each member factors as a measurement effect followed by a post-measurement CPTP gate, `I_k(ρ) = G_k(E_k^½ ρ E_k^½)` with `E_k = I_k†(I)`, and is represented as a single `ComposedOp([RootConjOperator(E_k), G_k])`. The effects `{E_k}` are gathered into one shared [`ComposedPOVM`](../pygsti/modelmembers/povms/composedpovm.py) — its completeness `Σ_k E_k = I` is exactly the instrument's joint trace preservation — and each `G_k` is parameterized independently (a CP-constrained `G_k` makes *that member* CP). Build these with the classmethods `Instrument.from_effects` / `Instrument.from_cptr_superops` (logic in [`modelmembers/instruments/_construction.py`](../pygsti/modelmembers/instruments/_construction.py)), or reach them via `convert(instrument, 'CPTPLND', basis)` / `set_all_parameterizations('CPTPLND')`. An n-outcome instrument needs only **n effects and n gates**. Note the shared-POVM layout makes the *later* members' parameter indices **non-contiguous** (they share the early POVM indices but own late gate indices); this is handled, but it's why `_submember_rpindices` can be an array rather than a slice.
+
 ## Pitfalls and gotchas
 
 - **`gpindices` is the central glue.** Every `ModelMember` has a `gpindices` slice or array that says "I own indices `[a, b)` of the parent Model's `_paramvec`." A `gpindices` of `None` means "unallocated — will be assigned on add-to-model." Copying a member from one Model to another typically requires rebinding indices; `Model._rebuild_paramvec()` is the canonical way to force a fresh allocation.