[
https://issues.apache.org/jira/browse/SPARK-30525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hu Fuwang updated SPARK-30525:
------------------------------
Description: In HiveTableScanExec, it will push down to hive metastore for
partition pruning if _spark.sql.hive.metastorePartitionPruning_ is true, and
then it will prune the returned partitions again using partition filters,
because some predicates, eg. "b like 'xyz'", are not supported in hive
metastore. But now this problem is already fixed in
HiveExternalCatalog.listPartitionsByFilter, the
HiveExternalCatalog.listPartitionsByFilter can return exactly what we want now.
So it is not necessary any more to double prune in HiveTableScanExec. (was: In
HiveTableScanExec, it will push down to hive metastore for partition pruning if
spark.sql.hive.metastorePartitionPruning is true, and then it will prune the
returned partitions again using partition filters, because some predicates, eg.
"b like 'xyz'", are not supported in hive metastore. But now this problem is
already fixed in HiveExternalCatalog.listPartitionsByFilter, the
HiveExternalCatalog.listPartitionsByFilter can return exactly what we want now.
So it is not necessary any more to double prune in HiveTableScanExec.)
> HiveTableScanExec do not need to prune partitions again after pushing down to
> hive metastore
> --------------------------------------------------------------------------------------------
>
> Key: SPARK-30525
> URL: https://issues.apache.org/jira/browse/SPARK-30525
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.1.0
> Reporter: Hu Fuwang
> Priority: Major
>
> In HiveTableScanExec, it will push down to hive metastore for partition
> pruning if _spark.sql.hive.metastorePartitionPruning_ is true, and then it
> will prune the returned partitions again using partition filters, because
> some predicates, eg. "b like 'xyz'", are not supported in hive metastore. But
> now this problem is already fixed in
> HiveExternalCatalog.listPartitionsByFilter, the
> HiveExternalCatalog.listPartitionsByFilter can return exactly what we want
> now. So it is not necessary any more to double prune in HiveTableScanExec.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
