pepijnve commented on PR #19287: URL: https://github.com/apache/datafusion/pull/19287#issuecomment-3669553386
@alamb https://github.com/apache/datafusion/pull/19287/commits/9f2f22ef0c7ecfd2d8a504308079c1831be79949 makes the `external_aggr` benchmark succeed. Because I was setting `can spill: false` for `OutOfMemoryMode::EmitEarly`, there was in effect still greedy memory reservation despite the use of `FairSpillPool`. Additionally the 'emit early' code would only kick in if there was at least one full batch of group values to emit. I've modified the code to still try to do that if it can, but fall back to emitting a smaller batch if memory pressure requires that. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
