[ 
https://issues.apache.org/jira/browse/SPARK-61445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

L. C. Hsieh resolved SPARK-61445.
---------------------------------
    Fix Version/s: 4.4.0
       Resolution: Fixed

Issue resolved by pull request 60327
[https://github.com/apache/spark/pull/60327]

> Pass through only the required child columns in Python UDF evaluation and 
> return Python results directly in columnar mode
> -------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61445
>                 URL: https://issues.apache.org/jira/browse/SPARK-61445
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 4.4.0
>            Reporter: L. C. Hsieh
>            Assignee: L. C. Hsieh
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 4.4.0
>
>
> EvalPythonExec (BatchEvalPython / ArrowEvalPython) always outputs 
> child.output ++ results, so every column of its child, including the UDF 
> arguments, is copied into a HybridRowQueue and joined with the results, even 
> when the parent drops it, e.g. df.select(udf(a)). With columnar input 
> (spark.sql.execution.arrow.pythonUDF.columnarInput.enabled), the joined rows 
> are also converted back to columnar batches.
> Like the Python UDTF nodes, the evaluation should pass through only the child 
> columns that its parent uses (requiredChildOutput, narrowed by 
> ColumnPruning). When no column is passed through, the row path needs no 
> queue, and the columnar path can return the result batches of Python directly.
> This is a follow-up of SPARK-60325 
> (https://github.com/apache/spark/pull/59582).



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to