Skip to content

Commit 1cd3557

Browse files
paulteehanclaude
paulteehan
and
claude
authored
chore: add publish-from-tag workflow for PyPI (#2758)
* ci: add publish-from-tag workflow for PyPI Adds .github/workflows/publish.yaml: builds an already-tagged release and uploads to Soda PyPI + public PyPI. Unlike release.yaml it never tags, bumps, or creates releases. Closes the gap where releases driven by release-buddy (which tags + creates the GitHub Release itself) never reached public PyPI: release.yaml refuses to run against an existing tag, so nothing published publicly between v4.7.0 (2026-04-17) and v4.14.0 — only dev PyPI kept flowing. - release: published -> auto-publishes new releases (the permanent fix) - workflow_dispatch -> backfill/recovery for a specific tag - twine --skip-existing -> idempotent; safe re-runs, tolerates a registry already having the version - verifies built artifacts are the clean tag version (not .devN) before upload Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci: trigger publish on tag push instead of release published A GitHub Release is best-effort (release-buddy tolerates `gh release create` failing), so `release: published` could silently skip a publish — the very gap this workflow closes. The release tag is always pushed, so key off `push: tags: [v*]` instead. workflow_dispatch is retained for backfill. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent a999906 commit 1cd3557

1 file changed

Lines changed: 207 additions & 0 deletions

File tree

‎.github/workflows/publish.yaml‎

Lines changed: 207 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,207 @@
1+
---
2+
name: Publish to PyPI
3+
4+
# Publishes an ALREADY-TAGGED release to Soda PyPI and public PyPI.
5+
#
6+
# Why this exists, separate from release.yaml:
7+
# release.yaml ("one-button release") resolves a version, tags it, builds,
8+
# publishes, and bumps the next prerelease — all in one shot, and it refuses
9+
# to run against a tag that already exists. Releases are now driven by
10+
# release-buddy (bump -> PR -> merge -> tag -> GitHub Release), so the tag
11+
# already exists by the time we want to publish, and release.yaml can't be
12+
# used. As a result nothing published to PUBLIC PyPI between v4.7.0
13+
# (2026-04-17) and v4.14.0 — only dev PyPI kept flowing. See incident notes.
14+
#
15+
# This workflow only BUILDS A GIVEN REF AND UPLOADS. It never tags, bumps, or
16+
# creates releases.
17+
#
18+
# Triggers:
19+
# push: tags (v*) -> standard path. The release tag is the canonical
20+
# "this version exists" signal and is always pushed
21+
# by release-buddy, so we key off the tag directly.
22+
# (A GitHub Release is best-effort — release-buddy
23+
# tolerates `gh release create` failing — so keying
24+
# off `release: published` could silently skip a
25+
# publish, the very gap this closes. A tag can't be
26+
# missed.)
27+
# workflow_dispatch -> backfill / recovery for an already-existing tag
28+
# (e.g. v4.14.0, or the 4.8.0-4.13.0 backlog); a tag
29+
# pushed before this workflow existed won't re-fire.
30+
#
31+
# twine runs with --skip-existing, so re-runs are safe and the automatic path
32+
# does not fail when a registry already has the version (e.g. Soda PyPI).
33+
34+
on:
35+
push:
36+
tags:
37+
- "v*"
38+
workflow_dispatch:
39+
inputs:
40+
tag:
41+
description: "Existing tag to publish (e.g. v4.14.0)."
42+
required: true
43+
type: string
44+
dry_run:
45+
description: "Build and verify only — do not upload."
46+
required: false
47+
type: boolean
48+
default: false
49+
50+
concurrency:
51+
group: publish-${{ inputs.tag || github.ref_name }}
52+
cancel-in-progress: false
53+
54+
jobs:
55+
resolve:
56+
name: Resolve tag & modules
57+
runs-on: ubuntu-24.04
58+
outputs:
59+
tag: ${{ steps.ref.outputs.tag }}
60+
version: ${{ steps.ref.outputs.version }}
61+
modules: ${{ steps.modules.outputs.modules }}
62+
steps:
63+
- name: Resolve tag
64+
id: ref
65+
run: |
66+
TAG="${{ inputs.tag || github.ref_name }}"
67+
if [ -z "$TAG" ]; then
68+
echo "::error::No tag provided."; exit 1
69+
fi
70+
echo "tag=$TAG" >> "$GITHUB_OUTPUT"
71+
echo "version=${TAG#v}" >> "$GITHUB_OUTPUT"
72+
echo "Resolved tag '$TAG' -> version '${TAG#v}'"
73+
74+
- uses: actions/checkout@v4
75+
with:
76+
ref: ${{ steps.ref.outputs.tag }}
77+
fetch-depth: 0
78+
79+
- name: Verify tag exists
80+
run: |
81+
git rev-parse "refs/tags/${{ steps.ref.outputs.tag }}" >/dev/null 2>&1 \
82+
|| { echo "::error::Tag ${{ steps.ref.outputs.tag }} not found."; exit 1; }
83+
84+
- name: Define release matrix
85+
id: modules
86+
run: echo "modules=$(bash scripts/release_matrix.sh)" >> "$GITHUB_OUTPUT"
87+
88+
publish:
89+
name: Publish ${{ matrix.module }}
90+
needs: [resolve]
91+
runs-on: ubuntu-24.04
92+
environment: production-release
93+
strategy:
94+
fail-fast: false
95+
matrix:
96+
module: ${{ fromJSON(needs.resolve.outputs.modules) }}
97+
steps:
98+
- uses: actions/checkout@v4
99+
with:
100+
ref: ${{ needs.resolve.outputs.tag }}
101+
102+
- name: Set up Python 3.10
103+
uses: actions/setup-python@v5
104+
with:
105+
python-version: "3.10"
106+
107+
- name: Set up UV
108+
uses: astral-sh/setup-uv@e58605a9b6da7c637471fab8847a5e5a6b8df081 # v5
109+
110+
- name: Get external secrets
111+
uses: aws-actions/aws-secretsmanager-get-secrets@a9a7eb4e2f2871d30dc5b892576fde60a2ecc802 # v2.0.10
112+
env:
113+
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_BUILD_ACCESS_KEY_ID }}
114+
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_BUILD_SECRET_ACCESS_KEY }}
115+
AWS_REGION: ${{ secrets.AWS_BUILD_DEFAULT_REGION }}
116+
with:
117+
secret-ids: |
118+
PYPI_USERNAME,/soda/github/common/SODA_PYPI_CLOUD_WRITE_USERNAME
119+
PYPI_PASSWORD,/soda/github/common/SODA_PYPI_CLOUD_WRITE_PASSWORD
120+
121+
- name: Build ${{ matrix.module }}
122+
run: |
123+
uv venv .venv
124+
source .venv/bin/activate
125+
uv pip install build twine
126+
cd ${{ matrix.module }}
127+
python3 -m build
128+
129+
- name: Verify built artifacts match the tag version
130+
env:
131+
VERSION: ${{ needs.resolve.outputs.version }}
132+
run: |
133+
# Built artifacts MUST be the clean release version (not a .devN).
134+
# PyPI uploads are irreversible, so fail loudly on a mismatch.
135+
shopt -s nullglob
136+
MATCHED=("${{ matrix.module }}"/dist/*-"$VERSION"-*.whl "${{ matrix.module }}"/dist/*-"$VERSION".tar.gz)
137+
if [ ${#MATCHED[@]} -eq 0 ]; then
138+
echo "::error::No artifacts matching version $VERSION in ${{ matrix.module }}/dist:"
139+
ls -1 "${{ matrix.module }}"/dist || true
140+
exit 1
141+
fi
142+
echo "Artifacts for $VERSION:"; printf ' %s\n' "${MATCHED[@]}"
143+
144+
- name: Stagger uploads
145+
run: |
146+
# Spread concurrent matrix uploads to reduce PyPI 503s.
147+
DELAY=$(echo -n "${{ matrix.module }}" | cksum | awk '{print $1 % 91}')
148+
echo "Staggering upload by ${DELAY}s..."
149+
sleep "$DELAY"
150+
151+
- name: Publish to Soda PyPI
152+
id: soda_pypi
153+
if: ${{ !inputs.dry_run }}
154+
env:
155+
TWINE_REPOSITORY_URL: ${{ vars.CLOUD_PYPI_REPOSITORY }}
156+
TWINE_USERNAME: ${{ env.PYPI_USERNAME }}
157+
TWINE_PASSWORD: ${{ env.PYPI_PASSWORD }}
158+
run: |
159+
source .venv/bin/activate
160+
RESULT="failure"
161+
for attempt in 1 2 3; do
162+
if twine upload --skip-existing "${{ matrix.module }}"/dist/*; then
163+
RESULT="success"; echo "Soda PyPI upload succeeded on attempt $attempt"; break
164+
fi
165+
if [ "$attempt" -lt 3 ]; then
166+
DELAY=$((attempt * 15))
167+
echo "::warning::Soda PyPI upload failed (attempt $attempt/3), retrying in ${DELAY}s..."
168+
sleep "$DELAY"
169+
else
170+
echo "::error::Soda PyPI upload failed after 3 attempts"
171+
fi
172+
done
173+
echo "result=$RESULT" >> "$GITHUB_OUTPUT"
174+
175+
- name: Publish to public PyPI
176+
id: public_pypi
177+
if: ${{ !inputs.dry_run }}
178+
env:
179+
TWINE_USERNAME: __token__
180+
TWINE_PASSWORD: ${{ secrets.PYPI_API_TOKEN }}
181+
run: |
182+
source .venv/bin/activate
183+
RESULT="failure"
184+
for attempt in 1 2 3; do
185+
if twine upload --skip-existing "${{ matrix.module }}"/dist/*; then
186+
RESULT="success"; echo "Public PyPI upload succeeded on attempt $attempt"; break
187+
fi
188+
if [ "$attempt" -lt 3 ]; then
189+
DELAY=$((attempt * 15))
190+
echo "::warning::Public PyPI upload failed (attempt $attempt/3), retrying in ${DELAY}s..."
191+
sleep "$DELAY"
192+
else
193+
echo "::error::Public PyPI upload failed after 3 attempts"
194+
fi
195+
done
196+
echo "result=$RESULT" >> "$GITHUB_OUTPUT"
197+
198+
- name: Report status
199+
if: ${{ always() && !inputs.dry_run }}
200+
run: |
201+
echo "### ${{ matrix.module }} @ ${{ needs.resolve.outputs.tag }}" >> "$GITHUB_STEP_SUMMARY"
202+
echo "- Soda PyPI: ${{ steps.soda_pypi.outputs.result }}" >> "$GITHUB_STEP_SUMMARY"
203+
echo "- Public PyPI: ${{ steps.public_pypi.outputs.result }}" >> "$GITHUB_STEP_SUMMARY"
204+
if [ "${{ steps.soda_pypi.outputs.result }}" != "success" ] || [ "${{ steps.public_pypi.outputs.result }}" != "success" ]; then
205+
echo "::error::${{ matrix.module }} did not publish cleanly to both registries — see logs."
206+
exit 1
207+
fi

0 commit comments

Comments
 (0)