JanKaul opened a new pull request, #22043: URL: https://github.com/apache/datafusion/pull/22043
Adds an async hook that lets a `MemoryPool` ask other consumers to free memory before failing an allocation. This PR is **complementary to [#21425](https://github.com/apache/datafusion/pull/21425)** — it is not a replacement. It exists to broaden the design discussion there with a concrete alternative, not to supersede that work. ## Design - **`trait MemoryReclaimer`** (async) attached to a `MemoryConsumer` via `with_reclaimer`. Implements: `reclaim(target)`, optional `reclaimable_bytes`, optional `priority`. - **`MemoryPool::try_grow_async`** — default delegates to sync `try_grow`. `TrackConsumersPool` overrides it to walk registered reclaimers (priority desc, size desc) on OOM, retry the grow after each, then fall through to `inner.try_grow_async` so a wrapped reclaim-aware pool isn't shadowed. - **Operator-side state machine** (`SortExec`): a channel-based `ExternalSorterReclaimer` hands a oneshot to the partition's stream loop; `tokio::select! biased { reclaim_rx.recv() … ; input.next() … }` spills end-to-end before replying with the freed-byte count. The stream loop is the sole owner of the sorter's batches, so the spill is ordered before the report — the bytes the pool sees are bytes already on disk. ## How this differs from #21425 - Async trait + `try_grow_async` instead of sync `pool.reclaim(...)` — matches the channel hand-off pattern needed for cooperative spill inside DataFusion's async execution. - Auto-triggered on OOM rather than caller-driven. - Includes operator wiring for `SortExec` to demonstrate the full flow. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
