Hu Fuwang created SPARK-30525: --------------------------------- Summary: HiveTableScanExec do not need to prune partitions again after pushing down to hive metastore Key: SPARK-30525 URL: https://issues.apache.org/jira/browse/SPARK-30525 Project: Spark Issue Type: Improvement Components: SQL Affects Versions: 3.1.0 Reporter: Hu Fuwang
In HiveTableScanExec, it will push down to hive metastore for partition pruning if spark.sql.hive.metastorePartitionPruning is true, and then it will prune the returned partitions again using partition filters, because some predicates, eg. "b like 'xyz'", are not supported in hive metastore. But now this problem is already fixed in HiveExternalCatalog.listPartitionsByFilter, the HiveExternalCatalog.listPartitionsByFilter can return exactly what we want now. So it is not necessary any more to double prune in HiveTableScanExec. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org