L. C. Hsieh created SPARK-61767:
-----------------------------------

             Summary: Extract the batch export of the in-process Python UDF 
evaluator into InProcessArrowBatchExporter
                 Key: SPARK-61767
                 URL: https://issues.apache.org/jira/browse/SPARK-61767
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 5.0.0
            Reporter: L. C. Hsieh


InProcessArrowEvalPythonEvaluatorFactory.evaluateBatches (SPARK-59718) 
interleaves three things in one method of about 300 lines: pulling input rows 
under the iterator's lock, building and exporting Arrow batches to Python, and 
releasing both at task completion.

This moves the second into a class of its own, InProcessArrowBatchExporter, 
which owns the Arrow writer of the current batch, the imported result vectors, 
and the registrations of the UDFs. evaluateBatches keeps the orchestration: it 
pulls rows into the queue and the exporter until the batch is full, evaluates 
it, and joins the results.

This is a refactoring without behavior change. The exporter can be tested 
without a Python interpreter.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to