[ https://issues.apache.org/jira/browse/SPARK-28983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
huajian lan updated SPARK-28983: -------------------------------- Description: Assume the hive partition table has p1/p2 partition column and c1 non-partition column, the below where clause can not prunning partitions in sparksql, but we can prunning by "p1 = 'a' or p2 = 'b'" in principal. {code:java} where (p1 = 'a' and c1 = 1) or (p2 = 'b' and c1 = 2){code} The problem is in HiveTableScans#apply(plan: LogicalPlan), when the predicate has any child that contains non-partition column, it will make a outer FilterExec and empty predicate in HiveTableScanExec. was: Assume the hive partition table has p1/p2 partition column and c1 non-partition column, the below where clause can not prunning partitions in sparksql, but we can prunning by "p1 = 'a' or p2 = 'b'" in principal. {code:java} where (p1 = 'a' and c1 = 1) or (p2 = 'b' and c1 = 2){code} The problem is in the object HiveTableScans#apply(plan: LogicalPlan), when the predicate has any child that contains non-partition column, it will make a outer FilterExec and empty predicate in HiveTableScanExec. > Can not pruning hive partitions by Or predicate > ----------------------------------------------- > > Key: SPARK-28983 > URL: https://issues.apache.org/jira/browse/SPARK-28983 > Project: Spark > Issue Type: Improvement > Components: SQL > Affects Versions: 2.4.4 > Reporter: huajian lan > Priority: Minor > > Assume the hive partition table has p1/p2 partition column and c1 > non-partition column, > the below where clause can not prunning partitions in sparksql, but we can > prunning by "p1 = 'a' or p2 = 'b'" in principal. > {code:java} > where > (p1 = 'a' and c1 = 1) > or > (p2 = 'b' and c1 = 2){code} > > The problem is in HiveTableScans#apply(plan: LogicalPlan), when the > predicate has any child that contains non-partition column, it will make a > outer FilterExec and empty predicate in HiveTableScanExec. > -- This message was sent by Atlassian Jira (v8.3.2#803003) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org