LiaCastaneda commented on issue #22758: URL: https://github.com/apache/datafusion/issues/22758#issuecomment-4916406038
Thanks for sharing, about what you said here > Stop any active queries > Record a heap dump so we can see what's going on > Cancel any active queries If a single query is causing the memory pressure, you'd end up cancelling all the other queries running on that node too. Have you found this problematic? In a scenario where all the queries (including the heavy one) retry on a different node, wouldn't you just see the same issue again, cancelling more unrelated queries each time? > I feel that, given our production experiences, the only way to really knock this problem on its head is that we need to account for all allocations within datafusion and arrow. It is a bit of a painful, but probably needed, refactor. If we don't do this, we are going to end up in our current whack-a-mole state. I agree, ideally this would live in Arrow, although the community has expressed concern over how breaking it would be to integrate a `claim()` API into each Arrow buffer creation call and/or to adopt the still-unstable Allocator API. It looks like the main concern is that we shouldn't leave this in a mid-state, and that we should get it right the first time, given how breaking a change like this would be to public APIs like kernels, constructors, etc. I wonder if in the medium term, it would make sense to come up with a solution in DataFusion that doesn't require changes in arrow-rs, at least until we're able to use the rust Allocator (such as `claim`, even if it's not 100% accurate since it reports after allocation) -- which would be the permanent approach. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
