Kontinuation opened a new pull request, #599: URL: https://github.com/apache/sedona-db/pull/599
## Summary - Add `SedonaFairSpillPool`, a new memory pool that reserves a configurable fraction of total memory for unspillable consumers, preventing spillable operators from exhausting all available memory (addresses [datafusion#17334](https://github.com/apache/datafusion/issues/17334) in the Sedona context) - Add `--memory-limit`, `--mem-pool-type`, and `--unspillable-reserve-ratio` CLI arguments to `sedona-cli` for configuring memory pool behavior - Refactor `SedonaContext::new_local_interactive()` to expose `new_local_interactive_with_runtime_env()`, allowing callers to inject a custom `RuntimeEnv` with pre-configured memory pools ## Motivation When running out-of-core spatial joins, spillable operators (e.g., `SpatialJoinExec`) could consume all available memory, causing unspillable operators (e.g., `RepartitionExec`'s merge consumer) to fail with OOM errors. The `SedonaFairSpillPool` mitigates this by reserving a configurable portion (default 20%) of the memory pool exclusively for unspillable allocations. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
