alamb commented on PR #23492: URL: https://github.com/apache/datafusion/pull/23492#issuecomment-5197454577
> * Should we just configure a fixed budget or should we do something with bandwidth? A classic way to model this would be with another "number of outstanding requests" limit -- which would limit how many outstanding requests could be going (which is a reasonable proxy for bandwidth). For users with large bandwidth / trying to maxxx out the execution, they could crank it up to 10 or 20 or something. > * The interaction with filter pushdown is interesting - what is the best way to avoid serialization behind a large number of row filters while also minimizing bandwidth / decoding usage? The current implementation just plans the filters serially (`f1 -> f2 -> f3 -> f4 -> ... -> full projection`) without considering IO latency. If we really wanted to get fancy, what I think we would do is 1. start fetching all the filter columns in parallel (subject to the outstanding request limit) 2. evaluate the filters as today serially 3. if we evaluates an earlier filter (say `f1` in your diagram) and pruned pages for a subsequent filter (say `f2`) we could start *another* racing read for the pruned pages in `f2` and if that finished before the original outstanding request we could cancel Such a strategy seems somewhat complicated and on one end of the performance spectrum (maximum performance, at the cost of more requests and memory..) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
