alamb commented on issue #26682: URL: https://github.com/apache/datafusion/issues/26682#issuecomment-6425681510
> https://github.com/apache/datafusion/pull/26387 Merge first. This is essentially a bug fix: pushdown should not alter the I/O mode. Only after merging will pushdown have the opportunity to be enabled by default. Sounds good -- I agree. I think once arrow 60.1 is out (eta in a few days) we'll have that one ready to merge > Merge https://github.com/apache/datafusion/pull/23492 and https://github.com/apache/datafusion/pull/24086 into a single read-ahead budget parameter. The granularity is determined by the decoder’s capabilities: before arrow-rs#11234 was merged, it was based on row groups; after the merge, it is based on batches. I think the ideal budget is not based on rows / row groups, but instead on "target_buffer_memory" as that is the limiting resource that prevents infinite buffering and what will run out earlier. Things like "num_row_groups" is just a proxy for buffer / memory usage. It may be that "read_ahead_bytes" in https://github.com/apache/datafusion/pull/24086 is the right measure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
