Dandandan commented on PR #21138: URL: https://github.com/apache/datafusion/pull/21138#issuecomment-4187523225
> no separate IO threads My current view: I think we currently are using `spawn_blocking` fine to keep it of the main / query threads, which is not optimal (because we create too much/unbounded threads), but OK from separating IO calls / CPU. I think though that we: * Don't perfectly balance work - that's where morselization will help to balance work evenly across threads (i.e. all threads are 90% busy most of the time * Don't interleave IO and CPU - that's where prefetching might help to get the last ~10-15% or so extra utilization * There is also some IO-level efficiency improvements, like file open / close on each read, which might help a bit (although it will matter even less when it is prefetched). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
