Dandandan commented on a change in pull request #9704:
URL: https://github.com/apache/arrow/pull/9704#discussion_r593962933
##########
File path: rust/datafusion/src/physical_plan/parquet.rs
##########
@@ -645,8 +654,29 @@ fn build_predicate_expression(
use crate::logical_plan;
// predicate expression can only be a binary expression
let (left, op, right) = match expr {
- Expr::BinaryExpr { left, op, right } => (left, *op, right),
+ Expr::BinaryExpr { left, op, right } => (left.clone(), *op,
right.clone()),
+ Expr::Between {
+ expr,
+ negated,
+ low,
+ high,
+ } if !negated // NOT BETWEEN not supported
+ => {
+ let left = Box::new(Expr::BinaryExpr {
+ left: expr.clone(),
+ op: Operator::GtEq,
+ right: low.clone(),
+ });
+ let right = Box::new(Expr::BinaryExpr {
+ left: expr.clone(),
+ op: Operator::LtEq,
+ right: high.clone(),
+ });
+
+ (left, Operator::And, right)
+ }
_ => {
+ debug!("Filter expression not supported in ParquetExec {:?}",
expr);
Review comment:
Yeah this is very useful! I found that the filter pushdown doesn't
filter out *anything* on the TPC-H benchmark (even when adding some extra
support), as the data is distributed so evenly, so every row group contains
almost the complete range of values.
I think @andygrove was thinking about some more general support for keeping
statistics like this, but some debug logging is something easy to add.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
