Dandandan commented on issue #20324: URL: https://github.com/apache/datafusion/issues/20324#issuecomment-3929241338
> I think one step to get this would be to add an disabled() -> bool API Yes, that seems a good idea to me. I think arrow-rs would need to check this while filtering so it can also stop doing it _while_ processing a row group (to avoid having the filter anyway > We should implement this by starting all filters as row filters (similar to current main) but "demoting" ineffective ones to be run as post-scan filters. Hm yes. For dynamic filters like topk ones, I think it would also make sense to support the other way around as they start ineffective (i.e. they start out filtering out nothing) and then progressively become more effective as data is consumed at the upstream `SortExec/TopK` (at that point already lot of rowgroups have been read). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
