[
https://issues.apache.org/jira/browse/SPARK-53029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takuya Ueshin reassigned SPARK-53029:
-------------------------------------
Assignee: Shujing Yang
> Support return type coercion for Arrow Python UDTFs
> ---------------------------------------------------
>
> Key: SPARK-53029
> URL: https://issues.apache.org/jira/browse/SPARK-53029
> Project: Spark
> Issue Type: Sub-task
> Components: PySpark
> Affects Versions: 4.1.0
> Reporter: Allison Wang
> Assignee: Shujing Yang
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.1.0
>
>
> Address this follow up discussion
> [https://github.com/apache/spark/pull/51692#discussion_r2241413025]
> Currently
> @arrow_udtf(returnType="x int")
> ...: class MyArrowUDTF:
> ...: def eval(self, batch: pa.RecordBatch):
> ...: yield pa.table(\{"x": batch.column(0)})
> Will throw this error
> In [12]: MyArrowUDTF(df.asTable()).show()
> 25/08/22 11:55:23 ERROR Executor: Exception in task 0.0 in stage 4.0 (TID 22)
> org.apache.spark.api.python.PythonException: Traceback (most recent call
> last):
> File "pyarrow/table.pxi", line 4865, in pyarrow.lib.Table.from_batches
> File "pyarrow/error.pxi", line 155, in
> pyarrow.lib.pyarrow_internal_check_status
> File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
> pyarrow.lib.ArrowInvalid: Schema at index 0 was different:
> x: int32
> vs
> x: int64
> at
> org.apache.spark.api.python.BasePythonRunner$ReaderIterator.handlePythonException(PythonRunner.scala:622)
> at
> org.apache.spark.sql.execution.python.PythonArrowOutput$$anon$1.read(PythonArrowOutput.scala:130)
> at
> org.apache.spark.api.python.BasePythonRunner$ReaderIterator.hasNext(PythonRunner.scala:573)
> at
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
> at scala.collection.Iterator$$anon$9.hasNext(Iterator.scala:583)
> at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:601)
>
> We should look into arrow_cast and add arrow casting for UDTF return types.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
