adriangb commented on PR #23390: URL: https://github.com/apache/datafusion/pull/23390#issuecomment-4917014089
## Update: byte-aware sort/merge output chunking (commit 88bdade5af) Following up on the finding above (the sorter re-chunks its own output at `batch_size` **rows**, rebuilding oversized batches downstream of a normalized scan): when `target_batch_size_bytes` is set, sorted-output chunking (`IncrementalSortIterator`, also used by the aggregation spill path) and the streaming merge (`BatchBuilder` emit, plumbed through `StreamingMergeBuilder`/`MultiLevelMergeBuilder` into `SortExec`, `SortPreservingMergeExec`, and `GroupedHashAggregateStream`) now emit when the row count **or** the estimated byte target is reached. Unset keeps row-only chunking byte-for-byte. Result on the `large_values` benchmark, `--memory-limit 2G`: | dataset | off | on (before this commit) | on (now) | |---|---|---|---| | 64KiB values (536MB decoded batches), Q2 sort, n=4 | FAIL (`ExternalSorter` cannot reserve 2x536MB) | FAIL in merge (`ExternalSorterMerge` ~1.1GB reservations, 639MB single SPM grow) | **completes, 2.6s** | | 16KiB values (134MB batches), all 5 queries, n=4 | Q2/Q3 FAIL | Q2/Q3 FAIL | **all 5 complete** | The mechanism is visible in the memory profiles: spill-file batches now respect the byte target, so per-partition merge reservations (sized from the largest spilled batch) drop from ~1.1GB to ~128MB. Remaining boundary: at 12 concurrent sort partitions in a pool ~2x the data size, the aggregate of per-partition merge reservations plus the top-level SPM buffering can still exhaust the fair pool — that is a genuine concurrent-capacity limit (and FairSpillPool interplay) rather than byte-blindness, left for follow-up. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01UcPTREZVLXsSZDRCae33gm -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
