Kristofferjoha opened a new pull request, #27346: URL: https://github.com/apache/datafusion/pull/27346
TITLE: fix: don't double count column join keys in SortMergeJoinExec buffered batch reservation BODY: ## Which issue does this PR close? - Closes #27322. ## Rationale for this change `BufferedBatch::try_new` reserves `batch.get_array_memory_size()` plus the size of each evaluated join key array. For a column key, the key array is the batch's own column, so that column is counted twice. Joins on wide string keys reserve about twice what they hold, and spill or fail under a memory limit earlier than they should. ## What changes are included in this PR? - `BufferedBatch::try_new` only charges key buffers the batch doesn't already hold, using `RecordBatchMemoryCounter`. - When a spilled batch is read back, the join keys are re-evaluated against it so the pre-spill key buffers are released. With the repro from the issue both queries now report 19.5 MB (was 38.8 MB and 19.7 MB). ## What is the testing strategy for this PR? Added `column_join_key_not_double_counted`, which checks that `peak_mem_used` is the same for an int key and a string key over the same data. Updated the expected `peak_mem_used` in `explain_analyze.slt` (432 B -> 288 B) and the size formula in three existing spill tests. The re-evaluation on restore is not covered by a test. ## Are there any user-facing changes? No. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
