srielau commented on code in PR #58255:
URL: https://github.com/apache/spark/pull/58255#discussion_r3863258983
##########
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala:
##########
@@ -408,9 +408,30 @@ private[client] class Shim_v2_0 extends Shim with Logging {
}
}
- if (!SQLConf.get.metastorePartitionPruningFastFallback ||
- predicates.isEmpty ||
- predicates.exists(hasTimeZoneAwareExpression)) {
+ // CHAR/VARCHAR partition keys are excluded from the metastore filter (see
+ // SupportedAttribute), because Hive compares them with its own
trailing-blank rules. When
+ // a predicate actually mentions such a key, prune on the client instead.
Other empty-filter
+ // or MetaException fallbacks still honor
metastorePartitionPruningFastFallback.
+ def referencesCharVarcharPartitionKey: Boolean = {
+ SQLConf.get.charVarcharStandardSemantics && {
+ val charVarcharPartNames = catalogTable.partitionSchema.fields.collect
{
+ case f if CharVarcharUtils.hasCharVarchar(f.dataType) => f.name
+ }
+ charVarcharPartNames.nonEmpty && {
+ val resolver = SQLConf.get.resolver
+ predicates.exists(_.exists {
+ case a: Attribute => charVarcharPartNames.exists(n => resolver(n,
a.name))
+ case _ => false
+ })
+ }
+ }
+ }
+ val useClientSidePrune = SQLConf.get.metastorePartitionPruningFastFallback
||
Review Comment:
Addressed in 82c6ea109c2. The CHAR/VARCHAR-reference check now runs before
the metastore-filter branch and forces client-side pruning for the full
predicate. Added a mixed `(ds INT, p CHAR(5))` partition test for `ds = 1 AND p
= 'a '`; it asserts exactly one partition is fetched. The targeted
SPARK-59001 Hive test passes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
