[
https://issues.apache.org/jira/browse/SPARK-60902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60902:
-----------------------------------
Labels: pull-request-available (was: )
> Apply column-name lists and pandas columns in Spark Connect Rust
> createDataFrame as PySpark does
> ------------------------------------------------------------------------------------------------
>
> Key: SPARK-60902
> URL: https://issues.apache.org/jira/browse/SPARK-60902
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> The Rust-backed Python client's {{createDataFrame}} applies column-name lists
> and the columns of a pandas DataFrame differently from PySpark Connect 4.3.0:
> * PySpark applies {{toDF}} when it has column names to apply: for a list or
> tuple of names, and for every pandas DataFrame (its column names, or the
> field names of a given schema). The server resolves {{toDF}} into a
> {{Project}}, so {{isLocal()}} is False. The Rust client puts the names in the
> local relation's schema instead, so {{isLocal()}} is True, for example for
> {{createDataFrame([(1,)], ["a"])}} and {{createDataFrame(pandas_df)}}.
> * PySpark extends a list of names that is shorter than plain rows or a pandas
> DataFrame with {{_2}}, {{_3}}, ... (an empty list gives {{_1}}, {{_2}}, ...);
> the Rust client raises an error instead. A longer list raises
> {{AXIS_LENGTH_MISMATCH}} in PySpark, and an atomic schema for a pandas
> DataFrame {{UNSUPPORTED_DATA_TYPE_FOR_ARROW}}; the Rust client raises an
> unclassified error for both.
> Rows with no schema or with a StructType, DDL, or atomic schema, and PyArrow
> tables, already match. NumPy ndarray input, which the client does not accept
> yet, is tracked by SPARK-60772.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
