Gengliang Wang created SPARK-27698: -------------------------------------- Summary: Add new method for getting pushed down filters in Parquet file reader Key: SPARK-27698 URL: https://issues.apache.org/jira/browse/SPARK-27698 Project: Spark Issue Type: Task Components: SQL Affects Versions: 3.0.0 Reporter: Gengliang Wang
To return accurate pushed filters in Parquet file scan(https://github.com/apache/spark/pull/24327#pullrequestreview-234775673), we can process the original data source filters in the following way: 1. For "And" operators, split the conjunctive predicates and try converting each of them. After that : 1.1 if partially predicate pushed down is allowed, return convertible results; 1.2 otherwise, return the whole predicate if convertible, or empty result if not convertible. 2. For other operators, it is either entirely pushed down, or not pushed down. In the current push down strategy, the "Non-And" operators are not able to be partially pushed down. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org