adriangb commented on PR #23390: URL: https://github.com/apache/datafusion/pull/23390#issuecomment-4916454801
## `large_values` benchmark results (new suite in this PR) Since existing suites decode to ~1MiB batches (the normalizer is metrics-provably pure pass-through on TPC-H/ClickBench scans), commit 075ac3220f adds a `large_values` dfbench suite: incompressible multi-KiB string values, so each parquet file decodes to a single 100MB+ batch within the 8192-row limit. Queries stress scan+filter, sort (small/large key), aggregation, and join. A/B = `DATAFUSION_EXECUTION_TARGET_BATCH_SIZE_BYTES=16MiB` vs unset, same binary. **16KiB values (134MB decoded batches), laptop:** | scenario | off | on | |---|---|---| | unlimited memory, Q1-Q5 | baseline | +5-10% (worst case: every batch is split, ~1GiB copied per query) | | `--memory-limit 4G` Q2 sort, n=4 | 819ms | **630ms (-23%)** | | `--memory-limit 4G` Q2 sort, n=12 | 427ms | **360ms (-16%)** | | `--memory-limit 2G` Q2/Q3 sorts | FAIL in `ExternalSorter` insert (cannot reserve 2x134MB against fair share) | FAIL, but much later: merge phase | **64KiB values (536MB decoded batches), `--memory-limit 2G`:** `off` fails instantly in `ExternalSorter`; `on` sorts + spills fine and then fails in the merge with a telling profile: ``` ExternalSorterMerge[0]#2(can spill: false) consumed 1106.2 MB ExternalSorterMerge[1]#4(can spill: false) consumed 878.9 MB Failed to allocate additional 639.1 MB for SortPreservingMergeExec[0] ``` A 639MB single allocation despite 16MiB input chunks: the sorter re-chunks its own output at `batch_size` **rows** (8192 x 64KiB = 536MB), reconstructing oversized batches downstream of the normalizer. Input-side normalization visibly helps (spill-heavy sorts get 16-23% faster, and the failure point moves from "cannot buffer one batch" to the merge), but completing these queries under tight limits needs byte-aware output chunking in sort/merge — the operator-emit-path follow-up called out in the PR description. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01UcPTREZVLXsSZDRCae33gm -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
