[
https://issues.apache.org/jira/browse/SPARK-42893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xinrong Meng updated SPARK-42893:
---------------------------------
Description:
Considering the upcoming improvements on the result inconsistencies between
traditional Pickled Python UDFs and Arrow-optimized Python UDFs, we'd better
block the feature, otherwise, users who try out the feature will expect
behavior changes in the next release.
In addition, since Spark Connect Python Client(SCPC) has been introduced in
Spark 3.4, we'd better ensure the feature is ready in both vanilla PySpark and
SCPC at the same time for compatibility.
was:Considering the upcoming improvements on the result inconsistencies
between traditional Pickled Python UDFs and Arrow-optimized Python UDFs, we'd
better block the feature, otherwise, users who try out the feature will expect
behavior changes in the next release.
> Block the usage of Arrow-optimized Python UDFs
> ----------------------------------------------
>
> Key: SPARK-42893
> URL: https://issues.apache.org/jira/browse/SPARK-42893
> Project: Spark
> Issue Type: Sub-task
> Components: PySpark
> Affects Versions: 3.4.0
> Reporter: Xinrong Meng
> Priority: Major
>
> Considering the upcoming improvements on the result inconsistencies between
> traditional Pickled Python UDFs and Arrow-optimized Python UDFs, we'd better
> block the feature, otherwise, users who try out the feature will expect
> behavior changes in the next release.
> In addition, since Spark Connect Python Client(SCPC) has been introduced in
> Spark 3.4, we'd better ensure the feature is ready in both vanilla PySpark
> and SCPC at the same time for compatibility.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
