xudong963 commented on issue #26682: URL: https://github.com/apache/datafusion/issues/26682#issuecomment-6416794894
> 3\. [@alamb](https://github.com/alamb) / [@adriangb](https://github.com/adriangb) / [@Dandandan](https://github.com/Dandandan): make the I/O policy in DataFusion extensible, so the scan can choose between one request per row group, prefetch with a byte budget, or batch-granular fetch depending on the storage system > > * in progress: [WIP: Add parquet `progressive_io` option using `ParquetPushDecoder::scan_plan` (one-shot row group I/O with pushdown filters) #26387](https://github.com/apache/datafusion/pull/26387) > * in progress: [Prefetch Parquet row groups with a bounded I/O budget #23492](https://github.com/apache/datafusion/pull/23492) > * proof of concept: [POC: batch-granular Parquet scans with the push decoder and scan_plan read-ahead #24086](https://github.com/apache/datafusion/pull/24086) These three ongoing PRs each added a configuration option: #26387 Added `progressive_io` (bool), which ensures that only one request is made per row group when pushdown is enabled. #23492 Added `prefetch_size` (bytes), which enables prefetching at the row group granularity. #24086 adds `read_ahead_bytes` (bytes), which enables read-ahead at the batch level. The semantics of the latter two are nearly identical—both involve “reading ahead by a specified number of bytes”—but they differ in granularity. If merged in the order they were submitted, users would be faced with four separate toggles: `pushdown`, `progressive`, `prefetch`, and `read_ahead`. It would be impossible to test all possible combinations. My recommendation: #26387 Merge first. This is essentially a bug fix: `pushdown` should not alter the I/O mode. Only after merging will `pushdown` have the opportunity to be enabled by default. Merge #23492 and #24086 into a single read-ahead budget parameter. The granularity is determined by the decoder’s capabilities: before `arrow-rs#11234` was merged, it was based on row groups; after the merge, it is based on batches. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
