andygrove opened a new pull request, #2576:
URL: https://github.com/apache/datafusion-ballista/pull/2576
# Which issue does this PR close?
N/A. Supersedes #2385, which built against a fork branch of
datafusion-python before there was a release candidate.
# Rationale for this change
The Python client is still built against the Ballista 54.0.0 crates and
DataFusion 54. Ballista 55.0.0 is on crates.io, and the vote for
datafusion-python 55.0.0 has started, with `55.0.0-rc1` tagged in
apache/datafusion-python and its wheels on TestPyPI.
Moving `python/` to 55 now means CI exercises the Python client against the
release candidate during the vote, and preparing the Python release afterwards
is just swapping the release candidate for the published release.
It also matters for compatibility. Since #2514, a 55 scheduler rejects
clients older than 55, so the Python client on `main`, built against the 54.0.0
crates, cannot run queries on a 55 cluster.
# What changes are included in this PR?
- Bump `pyballista` to 55.0.0, and pin `ballista`, `ballista-core`,
`ballista-executor` and `ballista-scheduler` to the published `=55.0.0` crates.
- Point `datafusion-python` at the `55.0.0-rc1` tag of
apache/datafusion-python, and pin `datafusion` and `datafusion-proto` to
`=55.2.0`, which is what datafusion-python's `Cargo.lock` uses at that tag.
- Bump `pyo3` from 0.28 to 0.29 to match datafusion-python. Both link the
native Python library, so there can only be one version.
- Require `datafusion==55` in `pyproject.toml`. Until it is on PyPI, uv
takes it from TestPyPI through an `explicit` index, so no other package can
resolve from there.
- Bump `python/requirements.txt` from `datafusion==52.0.0` to
`datafusion==55.0.0`. The pip setup in the README uses it, and it was missed in
the last few bumps.
- `ExecutorProcessConfig::concurrent_tasks` is now `vcores`, so `cluster.rs`
uses the new name and drops the two TODO workarounds.
- Call `LogicalPlan.to_bytes()` instead of `to_proto()`. datafusion-python
55 deprecates `to_proto()` as an alias of `to_bytes()`, and it emitted a
`DeprecationWarning` on every query.
- Refresh `python/Cargo.lock` and `python/uv.lock`. The Cargo lockfile was
re-resolved rather than fully updated, so only crates the bump requires have
moved. In `uv.lock`, only `datafusion` changes, apart from the lockfile
`revision` that current uv writes.
In `python/`, `cargo check --locked` and `cargo clippy --locked
--all-targets -- -D warnings` pass. `uv sync` followed by `uv run pytest`, as
in the `Test Python Release` job, passes against the TestPyPI `datafusion`
55.0.0 wheel (73 passed, 1 skipped because IPython is not installed).
Once the vote passes, a follow-up needs to replace the git dependency with
`datafusion-python = { version = "=55.0.0" }` and remove the TestPyPI index.
`check-cargo-lock.py` only accepts crates.io dependencies, so
`python-55.0.0-rc1` cannot be tagged before then.
# Are there any user-facing changes?
The Python client requires `datafusion==55` instead of `datafusion==54`, and
needs a Ballista 55 scheduler because of the version check from #2514. It also
picks up the Ballista 55 defaults, so the adaptive planner is now on by default
for Python users, as it already is for the Rust client (#2315).
There are no changes to the Python API.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
