andygrove opened a new pull request, #2512:
URL: https://github.com/apache/datafusion-ballista/pull/2512
# Which issue does this PR close?
Closes #2511.
# Rationale for this change
Our release includes the Python client, and the Python client can't be built
until datafusion-python publishes the matching DataFusion version. That has
held up Ballista 55.0.0 since we moved to DataFusion 55 five weeks ago (#2375).
#2511 proposes releasing the Python client separately, with its own vote, and
this PR implements the tooling and docs for that. It's a draft while the
discussion in #2511 is open.
A few design points:
- **No separate release branches.** Python release candidates are tagged on
the same `branch-NN` as the Rust release, with a `python-` prefix
(`python-55.0.0-rc1`, then `python-55.0.0`), after the Python update has been
cherry-picked there.
- **The Python release builds against the published crates.**
`python/Cargo.toml` pins the `ballista*` crates from crates.io, so the wheels
contain exactly the Rust code that was voted on in the Rust release, and later
Rust commits on the branch can't leak into them. This is already how
`python/Cargo.toml` looks on `main` today.
- **Same version number.** The Python client takes the version of the crates
it pins, so the Ballista 55.0.0 crates are followed by `ballista` 55.0.0 on
PyPI.
- **The source release is just `python/`.** The tarball is built from the
`python/` subtree, so it is self-contained and much smaller than the Rust one.
# What changes are included in this PR?
New `python/dev/release/`, with the Python release tooling:
- `create-tarball.sh` builds a source tarball of `python/` at the
`python-X.Y.Z-rcN` tag. It first checks that `python/Cargo.toml` has the
expected version and no `path` dependencies, then runs RAT, signs the tarball,
uploads it to dist/dev and prints the vote email. The email includes the
TestPyPI link, the Ballista crate version the client is built against, and a
link to the history of `python/` as the list of changes.
- `release-tarball.sh` copies an approved release candidate to
`dist/release/datafusion/datafusion-ballista-python-X.Y.Z`.
- `verify-release-candidate.sh` downloads the release candidate, checks the
signature and checksums, then builds the client and runs the Python tests with
uv, the same way CI does.
- `download-python-wheels.py` moved here from `dev/release/`. Only the help
text changed.
- `README.md` describes the Python release process. The TestPyPI and PyPI
sections moved here from `dev/release/README.md`, reordered so the wheels are
downloaded and uploaded to TestPyPI before the vote and the same files go to
PyPI after it. The expected wheel list no longer includes a Windows wheel,
since `build.yml` stopped building one.
Also:
- `python/NOTICE.txt`, so the Python source release has a NOTICE file.
`python/LICENSE.txt` already existed.
- `dev/release/create-tarball.sh` leaves `python/` out of the Rust tarball.
Its vote email drops the TestPyPI link and says the Python client is released
separately.
- `dev/update_ballista_versions.py` no longer bumps `python/Cargo.toml`. The
Python client pins published crates, so bumping it with the Rust crates would
point it at versions that aren't on crates.io yet.
- `dev/release/README.md` points to the Python process instead of describing
the PyPI steps.
- `.github/workflows/build.yml` builds wheels on `python-*-rc*` tags instead
of every `*-rc*` tag.
- `dev/release/rat_exclude_files.txt` excludes `uv.lock`, which sits at the
root of the Python tarball.
- A short paragraph in the DataFusion dependency section of the contributor
guide.
How I checked it:
- Ran both `create-tarball.sh` scripts and the Python `release-tarball.sh`
end to end against local test tags, with `gpg` and `svn` replaced by stubs. RAT
passes on both tarballs, the checksums verify, the Rust tarball has no
`python/` entries, and the vote emails render as expected.
- Checked that the Python `create-tarball.sh` refuses an unknown tag, a
version mismatch, and path dependencies (by tagging `54.1.0`, whose
`python/Cargo.toml` used path dependencies).
- Ran `update_ballista_versions.py` in a scratch worktree and confirmed it
leaves `python/` alone.
- RAT over the repo, prettier and ruff (the CI commands) all pass.
- `verify-release-candidate.sh` hasn't been run end to end, because that
needs a real release candidate on dist/dev. Its build and test step runs the
same commands as the `Test Python Release` CI job.
Two things this doesn't settle:
- **Python-only fixes.** A fix to the Python client with no Rust change has
no version number to use, because `python/Cargo.toml` can't express a PEP 440
post-release like `55.0.0.post1`. One option is a Rust patch release, another
is setting a static version in `pyproject.toml` for that release.
- **TestPyPI and a second RC.** TestPyPI only accepts each filename once, so
`rc2` for the same version can't be uploaded there. The README says to skip
that step and drop the link from the vote email in that case.
# Are there any user-facing changes?
No changes to the crates or the Python package. Release managers follow the
new process, and the Python client gets its own vote.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
