L. C. Hsieh created SPARK-61516:
-----------------------------------

             Summary: Return rows directly from ArrowEvalPythonExec with 
columnar input to a row-based parent
                 Key: SPARK-61516
                 URL: https://issues.apache.org/jira/browse/SPARK-61516
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 5.0.0
            Reporter: L. C. Hsieh


With spark.sql.execution.arrow.pythonUDF.columnarInput.enabled, 
ArrowEvalPythonExec only outputs columnar batches, so the planner inserts a 
ColumnarToRowExec above it for a row-based parent. With pass-through columns, 
the evaluator joins the pass-through rows with the Python results as rows, 
converts them back to columnar batches with RowToColumnConverter, and 
ColumnarToRowExec turns them into rows again. This makes ArrowEvalPythonExec 
with columnar input also support row-based output, so that a row-based parent 
gets the joined rows (or the rows of the result batches) directly. This is a 
follow-up promised in SPARK-60325.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to