lakshaydevoo-arch opened a new pull request, #27334: URL: https://github.com/apache/datafusion/pull/27334
## Which issue does this PR close? - Closes #26805. ## Rationale for this change When a hash join uses computed build-side keys (for example, `ON a + 1 = b`), evaluating those expressions allocates new arrays. Previously, their memory was not reserved until after the hash map or ArrayMap had been allocated. This could temporarily leave computed key memory unaccounted for, allowing actual memory usage to exceed the configured pool limit before the key reservation was attempted. ## What changes are included in this PR? - Reserve the evaluated `left_values` immediately after evaluation, before either hash map allocation path. - Reserve the additional `normalized_values` memory separately afterward. - Reuse the same `RecordBatchMemoryCounter` across both reservations to avoid double-counting buffers shared with the input batch or evaluated keys. - Add a regression test covering both hash-map and ArrayMap paths. The total memory accounting remains unchanged; only the reservation order changes. ## What is the testing strategy for this PR? Added `join_build_key_arrays_are_reserved_before_map`, which uses a computed join key and a constrained memory pool to verify that key memory is reserved before the map allocation. The regression test fails on the original implementation and passes with the fix for both hash-map and ArrayMap paths. **Validation:** - Extended workspace tests: 13,655 passed, 0 failed. - Hash join tests: 692 passed, 0 failed. - `cargo fmt --all -- --check`: passed. - `cargo clippy -p datafusion-physical-plan --all-targets --all-features -- -D warnings`: passed. - `git diff --check`: passed. `./dev/rust_lint.sh` passed its formatting, Clippy, and TOML formatting stages. The license-header check failed on an unrelated untracked local file (`.commandcode/taste/taste.md`), which is not included in this PR. The extended workspace tests ran before a one-line regression-test limit adjustment. The affected hash join tests, formatting, and Clippy passed afterward. ## Are there any user-facing changes? No changes to query results, public APIs, or configuration. The change improves memory-limit enforcement for hash joins with computed build-side keys by ensuring their memory is accounted for before allocating the hash map. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
