adriangb commented on issue #20715: URL: https://github.com/apache/datafusion/issues/20715#issuecomment-4004381733
> There is the total amount within a pool, but there is also memory reservation bytes which essentially acts as two watermarks, albeit in a roundabout way. I do kind of agree with this. It feels like there are two watermarks, but it's not fully explicit in the system. This is just a gut feeling but my guess is that making the low/high watermark system explicit and configurable will allow more operators to participate in this and make it easier for users to reason about the tradeoff of making them ~ the same (query might error) vs. making them ~ wide (we might spill when we didn't really need to). Another way to think about it is that we have two different memory limits: one for spillable operators and one for non-spillable operators. The spill reservation is the space between those two. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
