tschwarzinger commented on PR #24921: URL: https://github.com/apache/datafusion/pull/24921#issuecomment-5537802340
After a good night's sleep, I am leaning towards an implementation that delegates the eager fetch decision to the `DataSource` might be a better approach. The primary reason is that the current approach can trigger increased CPU usage. For example, if the buffering requires decompressing a data page, the decompression algorithm could block an additional CPU core which might not be expected by users (assuming number of target partitions ~ maximum number of used CPU cores) per query. Moving the eager fetch decision to the `DataSource` would enable an implementation where only the requests are triggered and the decoding is postponed until the actual stream is polled. What are your opinions of that? It also trades more fine-grained control of the `DataSource` (with likely more necessary code) against an approach that is automatically applied for all sources. One way could be to make `DataSource::open` async or even just create the futures in the open and await them when the stream is being polled. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
