alamb commented on issue #25183: URL: https://github.com/apache/datafusion/issues/25183#issuecomment-5766446739
> One iteration of https://github.com/apache/datafusion/pull/24923 reused available spill workspace but returned ResourcesExhausted if the workspace and a new reservation could not cover the output batch. This introduced a new failure condition for tight-budget queries that could previously continue spilling. The reviewer also pointed out that rejecting an already-built batch does not prevent its allocation. While this is true, it seems like the fix should be properly accounting for the already-built batch at construction time? Alternately, if the challenge is that the spill code starts writing the batch in some other task via async then I would think we need to account for the batch that would be held. But Perhaps now I see that this is not so simple 🤔 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
