Dandandan commented on PR #20665: URL: https://github.com/apache/datafusion/pull/20665#issuecomment-4011280821
> > Last tiem @adriangb looked into that I think he concluded that a bunch of the variability is related to if the TopK finds the top values quickly enough to skip some of the files entirely but that depends on a race to read the "right" rows. > > My hope is that after #20362 we can get #20304 across the line which should nuke the issue of a race with randomly ordered files. In theory the filter pushdown should only be essential when there is no stats sorting between files (i.e. the values being ordered by are randomly distributed). I agree - I saw also some huge variability in speedups for https://github.com/apache/datafusion/pull/20481 - I think with some great (row group or otherwise) ordering, some queries could have 3x+ speedups -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
