jorgecarleitao commented on pull request #8283: URL: https://github.com/apache/arrow/pull/8283#issuecomment-699658056
It makes sense, @andygrove. Note that I do not disagree with us having a custom scheduler. I was noting that we could separate the two problems: one problem is creation of tasks, and the other is the scheduling of said tasks. When we spawn tasks in `executionPlan::execute`, we may be mixing these two. The same way that we expose execution details of a node via (`output_partitioning`), we can expose other execution details (e.g. `is_scan()`), that our custom scheduler can use to e.g. limit concurrency. Regardless, this is in no way a blocker or anything, and we can leave this discussion for a future iteration - I agree that what you are doing here is really great stuff and we should go for it 🚀 ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
