Hyukjin Kwon created SPARK-62039:
------------------------------------
Summary: Split the Rust migration notes into the 4.3 line and the
unreleased 4.4 line, and qualify the claim that the Rust API mirrors PySpark's
surface
Key: SPARK-62039
URL: https://issues.apache.org/jira/browse/SPARK-62039
Project: Spark
Issue Type: Improvement
Components: Connect
Affects Versions: 5.0.0
Reporter: Hyukjin Kwon
docs/compatibility.md's "Rust migration for the 4.3 line" mixes what v4.3.0
shipped with the Rust-API changes made after v4.3.0 (tagged 2026-11-16), which
ship in 4.4.0, and some of its 4.3-line passages state what is no longer true
of v4.3.0's API (the interval map keys, the new Value variants, the Pipeline
stages). The docs also say that the native Rust API mirrors the same surface as
PySpark's, which overstates it.
Split the section into the 4.3 line, at v4.3.0's meaning with its corrections,
and a "Rust migration for the 4.4 line (unreleased)" of every Rust-API change
since v4.3.0, and qualify the surface claim with the APIs that need Python
itself (of Python functions, pandas or Python's implicit session).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]