Yicong Huang created SPARK-60901:
------------------------------------
Summary: Simplify the pipelined UDF input path in the Python worker
Key: SPARK-60901
URL: https://issues.apache.org/jira/browse/SPARK-60901
Project: Spark
Issue Type: Improvement
Components: PySpark
Affects Versions: 5.0.0
Reporter: Yicong Huang
The pipelined mode added in SPARK-56642 duplicates the worker's processing path
in invoke_udf. Keep a single path and move the background input reader into a
context manager in worker_util, so pipelined mode only decides whether the
input is read ahead in a thread. Also expose
ArrowStreamSerializer.flush_per_batch instead of setting a private attribute
from worker.py. No behavior change.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
