dwsmith1983 opened a new pull request, #25658: URL: https://github.com/apache/datafusion/pull/25658
## Which issue does this PR close? - Related to #12131. ## Rationale for this change Three pieces of per-chunk work on the hash join probe path repeat work whose inputs do not change within a probe batch. The multi-column key comparator was rebuilt for every output chunk, and building it scans the entire build side for negative zero when a key is a float. The integer-map path cloned both index buffers per chunk. Every output batch deep-cloned the output schema, metadata included, to wrap it in a new `Arc`. ## What changes are included in this PR? - The key comparator is built once per probe batch, cached on the stream, and cleared when the next batch arrives or the stream stops. - The integer-map path takes and returns its index buffers instead of cloning them. - `build_batch_from_indices` and `build_batch_empty_build_side` accept a `SchemaRef` and clone the `Arc`. Two-column join on `(l_orderkey, CAST(l_quantity AS DOUBLE))` over TPC-H SF1 with a 60K row build side and 17M output rows, median of five: 150 ms before, 108 ms after; `HashJoinExec` `elapsed_compute` 147 ms to 104 ms. Low-fanout float joins, the `hj` benchmark queries 23 to 25, and TPC-H q7 and q9 stay within run-to-run noise. ## What is the testing strategy for this PR? New tests in `datafusion/physical-plan/src/joins/hash_join`: `join_inner_multi_key_float_zero_and_nulls_across_chunks`, `join_inner_multi_key_across_probe_batches` (fails if the per-batch comparator reset is removed), and `join_inner_array_map_across_chunks`. Existing join tests and the `join` sqllogictest files pass unchanged. ## Are there any user-facing changes? No. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
