tschwarzinger commented on issue #25610: URL: https://github.com/apache/datafusion/issues/25610#issuecomment-5886087060
Thank you for opening this! I was wondering whether better statistics on the table scans could help you with this task. I've been looking into doing some of the pruning work during planning such that the optimizer has better statistics available. See https://github.com/apache/datafusion/issues/24944 . The issue contains an example from the imdb benchmark but maybe TPC-H or TPC-DS have similar potential benefits. Getting good results might require sorting the parquet files though. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
