alamb commented on issue #8846: URL: https://github.com/apache/arrow-rs/issues/8846#issuecomment-4769809793
Context from @adriangb in https://github.com/apache/arrow-rs/pull/9956#issuecomment-4769463376: > I think we had been planning to put more of the adaptivity at a higher level (DataFusion specifically) as it has more information about things like statistics, and cross file predicate selectivity. The pattern that seems to work well is: - Seed with cheap heuristics - Adapt once runtime information is available - Have multiple levels of adaptivity That last case specifically applies here: - Datafusion can decide what even is a row filter or not - arrow-rs can optimize how it applies those row filters (e.g. when it pays the price of flattening / rolling over a mask, etc.) at a more fine-grained level -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
