Hyukjin Kwon created SPARK-61018:
------------------------------------
Summary: Name the UDFs of mapInPandas, mapInArrow, applyInPandas,
applyInArrow and applyInPandasWithState after their functions in the Python
client as PySpark does
Key: SPARK-61018
URL: https://issues.apache.org/jira/browse/SPARK-61018
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
In PySpark, the UDF of {{df.mapInPandas(func, schema)}}, {{df.mapInArrow}},
{{df.groupBy(...).applyInPandas}}, {{applyInArrow}} and
{{applyInPandasWithState}}, and of the {{applyInPandas}} and {{applyInArrow}}
of {{cogroup}}, has the name of the function, {{func.__name__}} (or the name of
its class for a callable object), as {{UserDefinedFunction}} names it. The
server shows it in the plan: {{df.mapInArrow(my_func, "id long").explain()}}
shows {{MapInArrow my_func(id#0L)#3, [id#4L], false}}.
The Rust-backed Python client names the UDF after the method, so the plan shows
{{MapInArrow mapInArrow(id#0L)#3, [id#4L], false}}, and {{FlatMapGroupsInPandas
[id#0L], applyInPandas(id#0L)#3, ...}} for {{applyInPandas}}. Name them as
PySpark does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
