sagiweiss commented on issue #24819: URL: https://github.com/apache/datafusion/issues/24819#issuecomment-6786835265
The same pattern shows up in two more operators in 55.2.0, in case it helps scope the fix here (paths are relative to `datafusion/physical-plan/src/`): - **`CrossJoinExec`**, `joins/cross_join.rs:229` (`load_left_input`). Each left batch is reserved as it arrives (`reservation.try_grow(batch_size)`, line 217), and then `merged_batch` is built with `concat_batches(&left_schema, &batches)` without reserving the copy. As with `NestedLoopJoinExec`, the original batches and the concatenated copy coexist, so the transient peak is about twice what the pool accounts. - **`ExternalSorter::in_mem_sort_stream`**, `sorts/sort.rs:624`. The in-place branch concatenates `in_mem_batches` with `concat_batches` before `try_resize` sets the reservation to the merged batch. The comment at `sorts/sort.rs:608-612` says this function performs no reservation by design, and the branch only runs while the reservation is under `sort_in_place_threshold_bytes` (default 1 MiB, `datafusion/common/src/config.rs:916`), so the unaccounted copy is small. Mentioning it only for completeness. For the cross join, reserving the buffered size before `concat_batches`, then dropping the originals and shrinking the reservation by their share, closed the gap in a local patch. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
