GitHub user sshiv012 added a comment to the discussion: Support Arrow batches as an explicit Python UDF input and output mode
+1 on the direction. The conversion cost is real: DataProcessor._set_output_tuple finalizes every output row individually, OutputManager.tuple_to_batch feeds partitioners one tuple at a time, and tuple_to_frame rebuilds the Arrow table from tuples. For a batch or table UDF that already produced a DataFrame, that round trip is pure overhead. Please check #8914 (per-port output, I implemented a prototype on a branch). You might want to take into account 1. Output port. A UDF can emit on more than one port, and ports may have different schemas (table on port 0, html-content on port 1). An Arrow output needs a port, e.g. yield PortOutput(batch, port=1), and schema validation has to happen per port, at the batch level (cast the Arrow schema to the port schema once, instead of row by row). 2. Partitioning and completion flush. Hash partitioning hashes field values per row, and emit_ecm flushes each link's buffered tail with that link's port schema. The Arrow path has to keep both, otherwise rows can land on a different worker than today or a tail batch can go out with the wrong schema. Arrow compute can hash the key columns, but the hash must match the current per-row result. Happy to coordinate so that the two branches don't have conflicts in design, merge conflicts can be resolved :) GitHub link: https://github.com/apache/texera/discussions/8477#discussioncomment-18807058 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
