Dandandan commented on PR #6929: URL: https://github.com/apache/arrow-datafusion/pull/6929#issuecomment-1634099130
> If it turns out that bounding memory usage inevitably reduces performance in a non-negligible way, I propose we introduce a configuration flag to control this. We can use the high-performance/unbounded behavior the default one, but one should still be able to choose the lower performance/bounded version for memory conscious use cases. Sounds reasonable to me like a simple solution. Slightly more complex: track the memory usage and maybe reserve up to 10-100MB for buffering (configurable). I guess at some point much more buffering won't really help anymore for performance. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
