L. C. Hsieh created SPARK-61516:
-----------------------------------
Summary: Return rows directly from ArrowEvalPythonExec with
columnar input to a row-based parent
Key: SPARK-61516
URL: https://issues.apache.org/jira/browse/SPARK-61516
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: L. C. Hsieh
With spark.sql.execution.arrow.pythonUDF.columnarInput.enabled,
ArrowEvalPythonExec only outputs columnar batches, so the planner inserts a
ColumnarToRowExec above it for a row-based parent. With pass-through columns,
the evaluator joins the pass-through rows with the Python results as rows,
converts them back to columnar batches with RowToColumnConverter, and
ColumnarToRowExec turns them into rows again. This makes ArrowEvalPythonExec
with columnar input also support row-based output, so that a row-based parent
gets the joined rows (or the rows of the result batches) directly. This is a
follow-up promised in SPARK-60325.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]