Hyukjin Kwon created SPARK-59109:
------------------------------------
Summary: Fix PySpark API-surface parity in the Rust drop-in:
properties, show() default, camelCase kwargs, __version__
Key: SPARK-59109
URL: https://issues.apache.org/jira/browse/SPARK-59109
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.2.0
Reporter: Hyukjin Kwon
An audit of the Rust-backed drop-in against real PySpark 4.2.0 found basic
API-surface parity gaps that the transport-level official-suite gate never
exercises (it runs upstream PySpark over the Rust transport, not over this
Python API surface).
- pyspark.__version__ was "0.1.0"; now sourced from pyspark.version ("4.2.0").
- SparkSession.version and DataFrame.dtypes / executionInfo were methods;
exposed as properties to match PySpark attribute access.
- DataFrame.show() had a required n; now defaults to (n=20, truncate=True,
vertical=False).
- ~30 PyO3 method parameters were snake_case; renamed to PySpark camelCase so
keyword calls match (e.g. sql(sqlQuery), cast(dataType), limit(num),
withColumn(colName), between(lowerBound/upperBound)).
Adds offline introspection guards; verified to zero diffs against a real
PySpark 4.2.0 signature/kind audit and against a live Spark Connect server.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
