[
https://issues.apache.org/jira/browse/SPARK-61018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-61018:
---------------------------------
Fix Version/s: connect-rust-4.3.0
> Name the UDFs of mapInPandas, mapInArrow, applyInPandas, applyInArrow and
> applyInPandasWithState after their functions in the Python client as PySpark
> does
> -----------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61018
> URL: https://issues.apache.org/jira/browse/SPARK-61018
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> In PySpark, the UDF of {{df.mapInPandas(func, schema)}}, {{df.mapInArrow}},
> {{df.groupBy(...).applyInPandas}}, {{applyInArrow}} and
> {{applyInPandasWithState}}, and of the {{applyInPandas}} and {{applyInArrow}}
> of {{cogroup}}, has the name of the function, {{func.__name__}} (or the name
> of its class for a callable object), as {{UserDefinedFunction}} names it. The
> server shows it in the plan: {{df.mapInArrow(my_func, "id long").explain()}}
> shows {{MapInArrow my_func(id#0L)#3, [id#4L], false}}.
> The Rust-backed Python client names the UDF after the method, so the plan
> shows {{MapInArrow mapInArrow(id#0L)#3, [id#4L], false}}, and
> {{FlatMapGroupsInPandas [id#0L], applyInPandas(id#0L)#3, ...}} for
> {{applyInPandas}}. Name them as PySpark does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
