sunchao opened a new pull request, #25383: URL: https://github.com/apache/datafusion/pull/25383
## Which issue does this PR close? Prerequisite for #25172, which will enforce one memory allowance across sibling reservations. ## Rationale for this change Aggregate spill replay merges sorted files while rebuilding aggregate state. Both components share a memory consumer. If the merge reserves the whole allowance, the aggregate cannot process the rows it receives. ## What changes are included in this PR? A private merge option asks the actual memory pool to admit merge buffers plus an equal amount of replay headroom, then releases the spare capacity before starting the merge. All supported aggregate replay paths enable it. Ordinary sort callers retain their existing behavior. The merge can split oversized batches. If a row cannot shrink, only the minimum merge may retry without spare headroom, and the real pool must still admit it. Decoded-size inspection and actual batch-row limits prevent unnecessary spill rewrites and under-reserved output buffers. Legacy replay also releases unused capacity in its newly created grouping state. This uses one admission policy for all pools and adds no public memory-pool API. ## What is the testing strategy for this PR? Regression coverage checks all aggregate replay implementations, competing consumers, growing ARRAY_AGG state, indivisible rows, short and odd-sized batches, decoded string views, full disk quotas, exact output, and memory/disk cleanup. Validated replay alone, with the existing FairSpillPool implementation: - 2,263 physical-plan tests and 2,281 core/CLI tests passed. - The existing permanent-pressure regression passed with its original success expectation. - All 520 SQL logic test files passed. - `cargo fmt --all` and strict all-target/all-feature Clippy passed. - Full `./dev/rust_lint.sh` passed, including private Rust documentation and local Markdown links. Local validation uses Rust 1.98.1 and upstream revision `22651d24` with its unchanged dependency lockfile; newer main's dependency versions are unavailable in the local registry. Merge compatibility with current main is checked separately, and GitHub CI validates its merged revision. ## Are there any user-facing changes? Aggregate replay leaves memory available for processing merged rows. Temporary headroom reservations can increase recorded reservation peaks without allocating additional data buffers. The headroom is released before replay, so later concurrent allocations can still cause resource exhaustion. The dependent accounting PR #25172 must merge after this prerequisite. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
