mohitgurav20 commented on PR #25811: URL: https://github.com/apache/datafusion/pull/25811#issuecomment-6599946893
Makes total sense @alamb, this is definitely the same trade-off as #24393. A unified prefetch policy for the scanner would be awesome to have long-term. Since current `main` already awaits Bloom filters one-by-one in a loop anyway, checking them incrementally doesn't change today's I/O timing, but saves whole round-trips whenever an early column prunes. If you think having control right now is useful, I can easily add a config knob (like `parquet.bloom_filter_incremental_pruning`) so users can toggle it if needed. Otherwise happy to keep it as-is until the broader #24393 work lands. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
