Phoenix500526 commented on issue #25183: URL: https://github.com/apache/datafusion/issues/25183#issuecomment-5659011108
@alamb @jayzhan211 Following the discussion in #24923, one possible approach is to reserve memory for spill output buffers before materializing the batch. If the reservation cannot be satisfied, even with available workspace, we could reduce the output row count and try again, then hold that budget until the async write finishes. I tried a small local prototype limited to non-nullable Int32 batches in multi-run spill merging. It avoided memory-pool accounting overshoot in two small cases, but increased writer calls; the performance impact has not been measured. The trade-off is less avoidable overshoot versus smaller batches, potentially higher encoding/write overhead, and added complexity. The prototype preserves the existing best-effort spill fallback when even one row cannot be reserved, rather than introducing a hard failure for that shortfall. It does not enforce a strict memory limit. Before expanding this, do you think this trade-off is worth pursuing, or should we keep and document the current behavior? Are there workloads or benchmarks that should guide that decision? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
