ci: add publish-from-tag workflow for PyPI#2758
Merged
Merged
Conversation
Adds .github/workflows/publish.yaml: builds an already-tagged release and uploads to Soda PyPI + public PyPI. Unlike release.yaml it never tags, bumps, or creates releases. Closes the gap where releases driven by release-buddy (which tags + creates the GitHub Release itself) never reached public PyPI: release.yaml refuses to run against an existing tag, so nothing published publicly between v4.7.0 (2026-04-17) and v4.14.0 — only dev PyPI kept flowing. - release: published -> auto-publishes new releases (the permanent fix) - workflow_dispatch -> backfill/recovery for a specific tag - twine --skip-existing -> idempotent; safe re-runs, tolerates a registry already having the version - verifies built artifacts are the clean tag version (not .devN) before upload Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
m1n0
reviewed
Jun 16, 2026
| # twine runs with --skip-existing, so re-runs are safe and the automatic path | ||
| # does not fail when a registry already has the version (e.g. Soda PyPI). | ||
|
|
||
| on: |
Contributor
There was a problem hiding this comment.
shouldn't this run by default on tags?
Contributor
Author
There was a problem hiding this comment.
Yes good catch, fixed
A GitHub Release is best-effort (release-buddy tolerates `gh release create` failing), so `release: published` could silently skip a publish — the very gap this workflow closes. The release tag is always pushed, so key off `push: tags: [v*]` instead. workflow_dispatch is retained for backfill. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
m1n0
approved these changes
Jun 16, 2026
m1n0
left a comment
m1n0
left a comment
Contributor
There was a problem hiding this comment.
thanks for addressing this!
paulteehan
added a commit
that referenced
this pull request
Jun 16, 2026
Reverses the "drop Soda PyPI" half of this branch — soda-core clean releases are still needed on pypi.cloud.soda.io. The original failure was not that the registry is wrong, but that #2758 passed --skip-existing to every leg, and pypi.cloud.soda.io (devpi) rejects that flag: UnsupportedConfiguration: 'https://pypi.cloud.soda.io' does not have support for the following features: --skip-existing Fix per-registry instead of removing the leg: - public PyPI -> keeps `twine upload --skip-existing` (supported) - pypi.cloud.soda.io -> upload plain; capture output and treat an "already exists" / 409 rejection as success, so re-runs and backfills stay idempotent Keeps this branch's single-approval gate (one approval email per run, not one per matrix leg). Restores the AWS secrets fetch the Soda leg needs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
paulteehan
added a commit
that referenced
this pull request
Jun 16, 2026
* ci(publish): public PyPI only + single approval gate Two fixes to publish.yaml from the first 4.14.0 run: 1. Drop the Soda PyPI (pypi.cloud.soda.io) upload. It errored on `--skip-existing` (unsupported by that index), and it shouldn't be a target: that index serves the soda-library 1.x line that scan-launcher builds from, and soda-library's connectors share package names with soda-core's (soda-postgres, soda-snowflake, ...). Publishing soda-core's 4.x there collides with soda-library's identically-named 1.x packages. Nothing consumes soda-core clean releases from cloud (launchers pull soda-core dev builds from the dev platform index). soda-core clean releases belong on public PyPI only. Removes the now-unused AWS secrets-manager fetch too. 2. Collapse the 14 environment approvals into one. The matrix publish job referenced `production-release` directly, so each of the 14 legs raised its own approval (14 emails). A tiny `approve` gate job now carries the environment; the matrix depends on it and drops the environment. Safe because PYPI_API_TOKEN is a repo-level secret. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci(publish): restore Soda PyPI upload, fix --skip-existing on devpi Reverses the "drop Soda PyPI" half of this branch — soda-core clean releases are still needed on pypi.cloud.soda.io. The original failure was not that the registry is wrong, but that #2758 passed --skip-existing to every leg, and pypi.cloud.soda.io (devpi) rejects that flag: UnsupportedConfiguration: 'https://pypi.cloud.soda.io' does not have support for the following features: --skip-existing Fix per-registry instead of removing the leg: - public PyPI -> keeps `twine upload --skip-existing` (supported) - pypi.cloud.soda.io -> upload plain; capture output and treat an "already exists" / 409 rejection as success, so re-runs and backfills stay idempotent Keeps this branch's single-approval gate (one approval email per run, not one per matrix leg). Restores the AWS secrets fetch the Soda leg needs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Releases are now driven by release-buddy (bump → PR → merge → tag → GitHub Release). But the only workflow that publishes to public PyPI is
release.yaml("one-button release"), which isworkflow_dispatch-only and refuses to run against an already-existing tag (itsvalidatejob hard-fails onCheck tag does not exist). Since the tag already exists by the time we'd publish,release.yamlcan't be used.Result: nothing reached public pypi.org between v4.7.0 (2026-04-17) and v4.14.0 — only dev PyPI kept flowing. Internal consumers pull core from private/enterprise PyPI so it went unnoticed; only OSS users were affected (stuck on 4.7.0). Affects all 13 release-matrix packages (soda-core + soda-postgres, soda-bigquery, …).
What
New
.github/workflows/publish.yamlthat only builds a given ref and uploads — it never tags, bumps, or creates releases.release: published→ auto-publishes new releases. This is the permanent fix: release-buddy creates the Release, this publishes it.workflow_dispatch(input:tag,dry_run) → backfill/recovery for a specific existing tag (e.g.v4.14.0, and the 4.8.0–4.13.0 backlog).release.yaml(PYPI_API_TOKEN, AWS-secrets Soda creds,production-releaseenvironment).twine upload --skip-existing→ idempotent; safe re-runs and tolerates a registry already having the version..devN) before any upload — PyPI uploads are irreversible.Immediate use
After merge: Actions → Publish to PyPI → Run workflow → tag
v4.14.0(run once withdry_run: truefirst to confirm builds). This backfills the current release for OSS users.Follow-ups (not in this PR)
release.yaml's inline build-and-publish becomes redundant oncerelease: publishedis live — consider trimming it to avoid double work (--skip-existingalready makes the overlap harmless).checkReleaseArtifact()already hits pypi.org) so a stalled publish can never go unnoticed again.🤖 Generated with Claude Code