[
https://issues.apache.org/jira/browse/SPARK-60325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
L. C. Hsieh reassigned SPARK-60325:
-----------------------------------
Assignee: L. C. Hsieh
> Remove the holding of input batches from columnar Python UDFs
> -------------------------------------------------------------
>
> Key: SPARK-60325
> URL: https://issues.apache.org/jira/browse/SPARK-60325
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.4.0
> Reporter: L. C. Hsieh
> Assignee: L. C. Hsieh
> Priority: Major
> Labels: pull-request-available
>
> SPARK-56350 added an Arrow columnar path to Arrow Python UDFs, which keeps
> the pass-through columns of each input batch until its result comes back from
> Python, many batches later. SPARK-60294 disabled
> spark.sql.execution.arrow.pythonUDF.columnarInput.enabled by default, because
> a producer may release or overwrite the vectors it returned before then, e.g.
> a reader that refills the same vectors for every batch, the Arrow cache, or
> the Python output of a stacked ArrowEvalPythonExec.
> ColumnVector does not promise that a vector stays valid after the next batch
> is requested, and only the holding of the pass-through columns needs that:
> the UDF inputs of a batch are written to Python before the next batch is
> requested. This ticket removes the Arrow columnar path, so that columnar
> input always copies the pass-through columns into the row queue and joins the
> results by row. The results of Python can then be sliced with the output
> batch limits, and the conversion of the joined rows back to batches supports
> NullType and UDT columns, a non-positive maxRecordsPerBatch, and
> maxBytesPerBatch. The configuration stays disabled by default.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]