[
https://issues.apache.org/jira/browse/SPARK-60974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-60974:
---------------------------------
Fix Version/s: connect-rust-4.3.0
> Accept a DDL string schema in mapInPandas, applyInPandas and the other map
> functions in Spark Connect Rust as PySpark does
> --------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-60974
> URL: https://issues.apache.org/jira/browse/SPARK-60974
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> In PySpark, the schema of {{DataFrame.mapInPandas}}, of
> {{GroupedData.applyInPandas}} and {{applyInArrow}}, and of the cogrouped
> {{applyInPandas}} and {{applyInArrow}} can be a DDL string, as can the output
> type of {{applyInPandasWithState}} and {{transformWithStateInPandas}}. The
> client parses the string into a {{StructType}} and pickles the function with
> that type. The Rust-backed Python client pickles the string itself, so the
> Python worker fails. For example:
> {code:python}
> df.groupBy("k").applyInPandas(lambda pdf: pdf, "k long, v double").collect()
> {code}
> This works in PySpark 4.3.0. The client raises {{PythonException}} with
> {{AssertionError: dataType k long, v double should be an instance of <class
> 'pyspark.sql.types.DataType'>}}. With a DDL string, five of the six batch
> functions fail, and {{applyInArrow}} raises
> {{UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION}}. Pickle the parsed type as
> PySpark does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
