[ https://issues.apache.org/jira/browse/SPARK-30525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hu Fuwang updated SPARK-30525: ------------------------------ Description: In HiveTableScanExec, it will push down to hive metastore for partition pruning if _spark.sql.hive.metastorePartitionPruning_ is true, and then it will prune the returned partitions again using partition filters, because some predicates, eg. "b like 'xyz'", are not supported in hive metastore. But now this problem is already fixed in HiveExternalCatalog.listPartitionsByFilter, the HiveExternalCatalog.listPartitionsByFilter can return exactly what we want now. So it is not necessary any more to double prune in HiveTableScanExec. (was: In HiveTableScanExec, it will push down to hive metastore for partition pruning if spark.sql.hive.metastorePartitionPruning is true, and then it will prune the returned partitions again using partition filters, because some predicates, eg. "b like 'xyz'", are not supported in hive metastore. But now this problem is already fixed in HiveExternalCatalog.listPartitionsByFilter, the HiveExternalCatalog.listPartitionsByFilter can return exactly what we want now. So it is not necessary any more to double prune in HiveTableScanExec.) > HiveTableScanExec do not need to prune partitions again after pushing down to > hive metastore > -------------------------------------------------------------------------------------------- > > Key: SPARK-30525 > URL: https://issues.apache.org/jira/browse/SPARK-30525 > Project: Spark > Issue Type: Improvement > Components: SQL > Affects Versions: 3.1.0 > Reporter: Hu Fuwang > Priority: Major > > In HiveTableScanExec, it will push down to hive metastore for partition > pruning if _spark.sql.hive.metastorePartitionPruning_ is true, and then it > will prune the returned partitions again using partition filters, because > some predicates, eg. "b like 'xyz'", are not supported in hive metastore. But > now this problem is already fixed in > HiveExternalCatalog.listPartitionsByFilter, the > HiveExternalCatalog.listPartitionsByFilter can return exactly what we want > now. So it is not necessary any more to double prune in HiveTableScanExec. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org