alamb commented on issue #26682: URL: https://github.com/apache/datafusion/issues/26682#issuecomment-6426292491
> It might be reasonable to say that we only read ahead within a row group and thus setting read_ahead_bytes = 32Gi or something large (could be 0 or -1) is equivalent to progressive_io = false I am thinking progressive_io is sort of a policy choice rather than a resource cap (and the ideal setting is based on how rows that match the predicate are distributed over the input pages). In general I think the parquet reader should be reading ahead to the next row group if there is memory budget for it, rather than being restricted to just a single row group -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
