[
https://issues.apache.org/jira/browse/SPARK-60325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
L. C. Hsieh updated SPARK-60325:
--------------------------------
Description:
SPARK-56350 added an Arrow columnar path to Arrow Python UDFs, which keeps the
pass-through columns of each input batch until its result comes back from
Python, many batches later. SPARK-60294 disabled
spark.sql.execution.arrow.pythonUDF.columnarInput.enabled by default, because a
producer may release or overwrite the vectors it returned before then, e.g. a
reader that refills the same vectors for every batch, the Arrow cache, or the
Python output of a stacked ArrowEvalPythonExec.
ColumnVector does not promise that a vector stays valid after the next batch is
requested, and only the holding of the pass-through columns needs that: the UDF
inputs of a batch are written to Python before the next batch is requested.
This ticket removes the Arrow columnar path, so that columnar input always
copies the pass-through columns into the row queue and joins the results by
row. The results of Python can then be sliced with the output batch limits, and
the conversion of the joined rows back to batches supports NullType and UDT
columns, a non-positive maxRecordsPerBatch, and maxBytesPerBatch. The
configuration stays disabled by default.
was:
SPARK-56350 added an Arrow columnar path to Arrow Python UDFs, which keeps the
pass-through columns of each input batch until its result comes back from
Python, many batches later. SPARK-60294 disabled
spark.sql.execution.arrow.pythonUDF.columnarInput.enabled by default, because a
producer may release or overwrite the vectors it returned before then: e.g. a
reader that refills the same vectors for every batch, the Arrow cache, which
closed the previous root once the next batch was read, or the Python output of
a stacked ArrowEvalPythonExec.
SPARK-60306 added an internal marker, ReusedColumnVector, for vectors that
their producer may release or overwrite after returning them. This ticket makes
the Arrow columnar path hold only vectors without the marker, and take the row
queue path for the others; makes the Arrow cache leave the release of its
batches to their consumer, so it stays on the columnar path; and enables the
configuration by default again.
> Remove the holding of input batches from columnar Python UDFs
> -------------------------------------------------------------
>
> Key: SPARK-60325
> URL: https://issues.apache.org/jira/browse/SPARK-60325
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.4.0
> Reporter: L. C. Hsieh
> Priority: Major
> Labels: pull-request-available
>
> SPARK-56350 added an Arrow columnar path to Arrow Python UDFs, which keeps
> the pass-through columns of each input batch until its result comes back from
> Python, many batches later. SPARK-60294 disabled
> spark.sql.execution.arrow.pythonUDF.columnarInput.enabled by default, because
> a producer may release or overwrite the vectors it returned before then, e.g.
> a reader that refills the same vectors for every batch, the Arrow cache, or
> the Python output of a stacked ArrowEvalPythonExec.
> ColumnVector does not promise that a vector stays valid after the next batch
> is requested, and only the holding of the pass-through columns needs that:
> the UDF inputs of a batch are written to Python before the next batch is
> requested. This ticket removes the Arrow columnar path, so that columnar
> input always copies the pass-through columns into the row queue and joins the
> results by row. The results of Python can then be sliced with the output
> batch limits, and the conversion of the joined rows back to batches supports
> NullType and UDT columns, a non-positive maxRecordsPerBatch, and
> maxBytesPerBatch. The configuration stays disabled by default.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]