16pierre commented on issue #17334: URL: https://github.com/apache/datafusion/issues/17334#issuecomment-3436178925
Thanks for https://github.com/apache/datafusion/pull/18014 @adriangb, this could definitely fix the issue I previously hit here. Back then I ended up working around without `RepartitionExec` nodes, and trashed my original drafts. My work-around was to manually control the parallelism on the datasources and coalesce some `UnionExec` nodes to control parallelism etc. We could potentially close this issue as no longer relevant on our end. There's some interesting wider design discussions regarding how to handle the spillable vs non-spillable pools here, I quite like @milenkovicm approach here, next time we hit spilling problems we'll give it a shot, in previous iterations we definitely spent quite a few dev cycles fine-tuning the memory settings (especially before multi-level merge was released). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
