Weijun-H opened a new pull request, #25565: URL: https://github.com/apache/datafusion/pull/25565
## What Adds a sorter-local byte target for spill merge output batches. Before `BatchBuilder` materializes a merge output batch, it estimates the largest supported output prefix that fits the target, reserves that output-construction budget, and interleaves only that prefix. The target is currently derived from the selected spill runs' recorded maximum batch memory. This keeps a merge pass from rebuilding a multi-row output batch that is substantially wider than the spilled input batches it is merging. ## Policy / Compatibility This does not try to make sort memory a hard resident-memory cap. It is a pre-construction accounting and output-shaping guard for avoidably large multi-row sort output batches. A multi-row reservation failure retries with a smaller prefix. A single row may exceed the byte target, but it must still obtain its construction reservation; if that final reservation fails, the query returns the existing memory error. That compatibility point is related to the policy discussion in #25183. Unsupported schemas retain the current path. In this first cut, the byte target is applied only when every output column is a supported flat type; any unsupported column makes the whole batch fall back to current behavior. View arrays and containers such as Struct/List/Map are intentionally not estimated here. #23390 is relevant background for retained-buffer concerns around view arrays. ## Tests The focused tests cover: - selecting the largest supported prefix under the byte target; - output-construction reservation against the real pool for spill merge; - bitmap-capacity accounting for nullable and Boolean output; - unsupported mixed schemas falling back as a whole batch; - a wide Binary spill merge that must split output batches by byte target while preserving complete sorted output. No performance improvement is claimed here. Related to #25183. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
