Hyukjin Kwon created SPARK-60902:
------------------------------------
Summary: Apply toDF in Spark Connect Rust createDataFrame where
PySpark does, so isLocal() matches
Key: SPARK-60902
URL: https://issues.apache.org/jira/browse/SPARK-60902
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
PySpark Connect's {{createDataFrame}} applies {{toDF}} to the local relation
when it has column names to apply: for a list or tuple of column names, and for
every pandas DataFrame (its column names, or the field names of a given
schema). The server resolves {{toDF}} into a {{Project}}, so {{isLocal()}} is
False for those DataFrames.
The Rust-backed Python client puts the names in the local relation's schema
instead, so {{isLocal()}} is True for them:
* {{createDataFrame([(1,)], ["a"]).isLocal()}} is True; PySpark returns False.
* {{createDataFrame(pandas_df).isLocal()}}, with no schema, a list of names, a
StructType or a DDL string, is True; PySpark returns False.
Rows with no schema or with a StructType, DDL, or atomic schema, and PyArrow
tables, already match. Apply {{toDF}} in the same cases as PySpark. NumPy
ndarray input, which the client does not accept yet, is tracked by SPARK-60772.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
