Dandandan commented on pull request #1732: URL: https://github.com/apache/arrow-datafusion/pull/1732#issuecomment-1028782119
> > For context one of the reasons I'm interested in reducing the concurrency is somewhat stupid, the plans and the resulting metrics and tracing become HUGE, but I also can't help but feel that always pursuing the maximum concurrency is liable to hit a wall somewhere - probably memory grin > > I think pursuing the maximum concurrency specified in `target_partitions` is a good strategy, but it needs to be complimented by a more sophisticated scheduler, that breaks the plans down into smaller pieces, perhaps as described / explained in #64 Agreed with this approach :100: -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
