Rich-T-kid opened a new issue, #25515: URL: https://github.com/apache/datafusion/issues/25515
Several DataFusion operators share the same two-copy pattern when producing output: 1. Copy 1 : interleave_record_batch (or build_batch_from_indices) gathers selected rows from multiple source batches into a new fully-materialized RecordBatch. 2. Copy 2 : BatchCoalescer::push_batch copies those rows into the coalescer's internal column builders. Prerequisite: https://github.com/apache/arrow-rs/issues/11141 Affected call sites ``` ┌──────────────────┬────────────────────┬───────────────────────────────────────┬────────┐ │ Operator │ File │ Current pattern │ Copies │ ├──────────────────┼────────────────────┼───────────────────────────────────────┼────────┤ │ Top-K │ topk/mod.rs:1016 │ interleave_record_batch → push_batch │ 2 │ ├──────────────────┼────────────────────┼───────────────────────────────────────┼────────┤ │ Top-K rank │ topk/mod.rs:1875 │ same │ 2 │ ├──────────────────┼────────────────────┼───────────────────────────────────────┼────────┤ │ Hash join │ hash_join/… │ build_batch_from_indices → push_batch │ 2 │ ├──────────────────┼────────────────────┼───────────────────────────────────────┼────────┤ │ Nested loop join │ nested_loop_join/… │ interleave_record_batch → push_batch │ 2 │ ├──────────────────┼────────────────────┼───────────────────────────────────────┼────────┤ │ Sort merge join │ sort_merge_join/… │ materialization step → push_batch │ 2 │ └──────────────────┴────────────────────┴───────────────────────────────────────┴────────┘ ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
