martin-g commented on code in PR #1610: URL: https://github.com/apache/datafusion-ballista/pull/1610#discussion_r3154428467
########## dev/release/README.md: ########## @@ -281,6 +282,158 @@ dot -Tsvg dev/release/crate-deps.dot > dev/release/crate-deps.svg (cd ballista-cli && cargo publish) ``` +### Publish Python Wheels to PyPI + +Only approved releases of the tarball should be published to PyPI, in order to +conform to Apache Software Foundation governance standards. The Python wheels +that get uploaded must be the same artifacts that the community voted on — they +are downloaded from the release candidate's CI run, not rebuilt. + +#### Prerequisites + +A DataFusion committer can publish the [`ballista` package on Review Comment: ```suggestion A DataFusion PMC member can publish the [`ballista` package on ``` ########## dev/release/README.md: ########## @@ -281,6 +282,158 @@ dot -Tsvg dev/release/crate-deps.dot > dev/release/crate-deps.svg (cd ballista-cli && cargo publish) ``` +### Publish Python Wheels to PyPI + +Only approved releases of the tarball should be published to PyPI, in order to +conform to Apache Software Foundation governance standards. The Python wheels +that get uploaded must be the same artifacts that the community voted on — they +are downloaded from the release candidate's CI run, not rebuilt. + +#### Prerequisites + +A DataFusion committer can publish the [`ballista` package on +PyPI](https://pypi.org/project/ballista/) after an official project release has +been made. One-time setup: + +- Create accounts on [pypi.org](https://pypi.org) and + [test.pypi.org](https://test.pypi.org) (separate accounts). +- Ask an existing maintainer of the `ballista` PyPI project — listed on the + project page — to add you as a maintainer. The request should be made on the + dev mailing list so it is publicly tracked. +- Generate project-scoped API tokens for both PyPI and TestPyPI. +- Configure `~/.pypirc`: + + ```ini + [distutils] + index-servers = + pypi + testpypi + + [pypi] + username = __token__ + password = pypi-... + + [testpypi] + repository = https://test.pypi.org/legacy/ + username = __token__ + password = pypi-... + ``` + +- Install `twine`: + + ```bash + pip install twine Review Comment: https://github.com/apache/datafusion-ballista/blob/68dfeddeeedec3382cfcae0a4139af7005dc3eac/dev/release/download-python-wheels.py#L23 also says that `requests` is needed ########## dev/release/README.md: ########## @@ -281,6 +282,158 @@ dot -Tsvg dev/release/crate-deps.dot > dev/release/crate-deps.svg (cd ballista-cli && cargo publish) ``` +### Publish Python Wheels to PyPI + +Only approved releases of the tarball should be published to PyPI, in order to +conform to Apache Software Foundation governance standards. The Python wheels +that get uploaded must be the same artifacts that the community voted on — they +are downloaded from the release candidate's CI run, not rebuilt. + +#### Prerequisites + +A DataFusion committer can publish the [`ballista` package on +PyPI](https://pypi.org/project/ballista/) after an official project release has +been made. One-time setup: + +- Create accounts on [pypi.org](https://pypi.org) and + [test.pypi.org](https://test.pypi.org) (separate accounts). +- Ask an existing maintainer of the `ballista` PyPI project — listed on the + project page — to add you as a maintainer. The request should be made on the + dev mailing list so it is publicly tracked. +- Generate project-scoped API tokens for both PyPI and TestPyPI. +- Configure `~/.pypirc`: + + ```ini + [distutils] + index-servers = + pypi + testpypi + + [pypi] + username = __token__ + password = pypi-... + + [testpypi] + repository = https://test.pypi.org/legacy/ + username = __token__ + password = pypi-... + ``` + +- Install `twine`: + + ```bash + pip install twine + ``` + +#### Download the Voted-On Wheels + +Once the vote passes and the final tag has been created from the RC commit, +download the same wheels that were voted on from the RC's CI run. Retagging the +RC commit does not trigger a fresh build, so the RC artifacts remain the +canonical source. + +```bash +export GH_TOKEN=... # GitHub PAT with read access to actions +mkdir ballista-pypi-<version> && cd ballista-pypi-<version> +python ../dev/release/download-python-wheels.py <version>-rc<N> +ls *.whl *.tar.gz # confirm filenames carry the right version +``` + +The merged artifact should contain one of each of the following platform wheels +(file naming uses [PEP 425](https://peps.python.org/pep-0425/) tags): + +- `ballista-<version>-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl` +- `ballista-<version>-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl` +- `ballista-<version>-cp310-abi3-macosx_*_arm64.whl` +- `ballista-<version>-cp310-abi3-win_amd64.whl` +- `ballista-<version>.tar.gz` (sdist) + +> **Known CI caveat:** the merged artifact currently contains the macOS arm64 +> wheel twice (jobs `build-python-mac-win`'s macOS leg and `build-macos-x86_64` +> both run on `macos-latest`, which is now arm64) and **no** macOS x86_64 wheel. +> Keep one copy of the arm64 wheel and delete any duplicate before upload. +> Tracked in [#1608](https://github.com/apache/datafusion-ballista/issues/1608). + +#### Validate the Artifacts + +```bash +twine check *.whl *.tar.gz +``` + +The `download-python-wheels.py` script also writes `.asc` GPG signatures and +`.sha256` / `.sha512` checksum files alongside each artifact. Those are for ASF +SVN — PyPI rejects them. Pass explicit globs to `twine` so only the wheels and +sdist are considered. + +#### TestPyPI Dry-Run + +PyPI uploads are immutable: once a version is published it cannot be replaced +or re-uploaded, only yanked. A TestPyPI dry-run takes a few minutes and catches +the common ways a release goes wrong. + +```bash +twine upload --repository testpypi *.whl *.tar.gz + +python -m venv /tmp/ballista-pypi-smoke +source /tmp/ballista-pypi-smoke/bin/activate +pip install -i https://test.pypi.org/simple/ \ + --extra-index-url https://pypi.org/simple/ \ + ballista==<version> +python -c "from ballista import BallistaSessionContext; print('ok')" +deactivate +``` + +`--extra-index-url` is required because TestPyPI does not mirror dependencies +like `pyarrow` and `datafusion`. + +#### Upload to PyPI + +```bash +twine upload *.whl *.tar.gz +``` + +If the upload fails partway through, re-run with `--skip-existing` to retry only +the files that did not get through. + +#### Verify + +Confirm the new version appears at +`https://pypi.org/project/ballista/<version>/`. Then in another fresh +virtual environment: + +```bash +python -m venv /tmp/ballista-pypi-verify +source /tmp/ballista-pypi-verify/bin/activate +pip install ballista==<version> Review Comment: Idea: Instead of using `<version>` I'd suggest to use `${BALLISTA_VERSION}` that is defined earlier. This way the release manager could just copy/paste/run the code without any manual edits. ########## dev/release/README.md: ########## @@ -281,6 +282,158 @@ dot -Tsvg dev/release/crate-deps.dot > dev/release/crate-deps.svg (cd ballista-cli && cargo publish) ``` +### Publish Python Wheels to PyPI + +Only approved releases of the tarball should be published to PyPI, in order to +conform to Apache Software Foundation governance standards. The Python wheels +that get uploaded must be the same artifacts that the community voted on — they +are downloaded from the release candidate's CI run, not rebuilt. + +#### Prerequisites + +A DataFusion committer can publish the [`ballista` package on +PyPI](https://pypi.org/project/ballista/) after an official project release has +been made. One-time setup: + +- Create accounts on [pypi.org](https://pypi.org) and + [test.pypi.org](https://test.pypi.org) (separate accounts). +- Ask an existing maintainer of the `ballista` PyPI project — listed on the + project page — to add you as a maintainer. The request should be made on the + dev mailing list so it is publicly tracked. +- Generate project-scoped API tokens for both PyPI and TestPyPI. +- Configure `~/.pypirc`: + + ```ini + [distutils] + index-servers = + pypi + testpypi + + [pypi] + username = __token__ + password = pypi-... + + [testpypi] + repository = https://test.pypi.org/legacy/ + username = __token__ + password = pypi-... + ``` + +- Install `twine`: + + ```bash + pip install twine + ``` + +#### Download the Voted-On Wheels + +Once the vote passes and the final tag has been created from the RC commit, +download the same wheels that were voted on from the RC's CI run. Retagging the +RC commit does not trigger a fresh build, so the RC artifacts remain the +canonical source. + +```bash +export GH_TOKEN=... # GitHub PAT with read access to actions +mkdir ballista-pypi-<version> && cd ballista-pypi-<version> +python ../dev/release/download-python-wheels.py <version>-rc<N> +ls *.whl *.tar.gz # confirm filenames carry the right version +``` + +The merged artifact should contain one of each of the following platform wheels +(file naming uses [PEP 425](https://peps.python.org/pep-0425/) tags): + +- `ballista-<version>-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl` +- `ballista-<version>-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl` +- `ballista-<version>-cp310-abi3-macosx_*_arm64.whl` +- `ballista-<version>-cp310-abi3-win_amd64.whl` +- `ballista-<version>.tar.gz` (sdist) + +> **Known CI caveat:** the merged artifact currently contains the macOS arm64 +> wheel twice (jobs `build-python-mac-win`'s macOS leg and `build-macos-x86_64` +> both run on `macos-latest`, which is now arm64) and **no** macOS x86_64 wheel. +> Keep one copy of the arm64 wheel and delete any duplicate before upload. +> Tracked in [#1608](https://github.com/apache/datafusion-ballista/issues/1608). Review Comment: Not sure this note is needed here. The issue will be resolved soon (before next release). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
