cloud-fan commented on code in PR #58255:
URL: https://github.com/apache/spark/pull/58255#discussion_r3859459744
##########
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala:
##########
@@ -408,9 +408,30 @@ private[client] class Shim_v2_0 extends Shim with Logging {
}
}
- if (!SQLConf.get.metastorePartitionPruningFastFallback ||
- predicates.isEmpty ||
- predicates.exists(hasTimeZoneAwareExpression)) {
+ // CHAR/VARCHAR partition keys are excluded from the metastore filter (see
+ // SupportedAttribute), because Hive compares them with its own
trailing-blank rules. When
+ // a predicate actually mentions such a key, prune on the client instead.
Other empty-filter
+ // or MetaException fallbacks still honor
metastorePartitionPruningFastFallback.
+ def referencesCharVarcharPartitionKey: Boolean = {
+ SQLConf.get.charVarcharStandardSemantics && {
+ val charVarcharPartNames = catalogTable.partitionSchema.fields.collect
{
+ case f if CharVarcharUtils.hasCharVarchar(f.dataType) => f.name
+ }
+ charVarcharPartNames.nonEmpty && {
+ val resolver = SQLConf.get.resolver
+ predicates.exists(_.exists {
+ case a: Attribute => charVarcharPartNames.exists(n => resolver(n,
a.name))
+ case _ => false
+ })
+ }
+ }
+ }
+ val useClientSidePrune = SQLConf.get.metastorePartitionPruningFastFallback
||
Review Comment:
**Non-blocking:**
Move the CHAR/VARCHAR-reference check before the `filter.isEmpty` branch.
`convertFilters` keeps supported conjuncts, so `ds = 1 AND p = 'a'` yields a
non-empty metastore filter and never calls this helper. Spark then fetches
every partition matching `ds` before residual pruning. Please route any
predicate that references a CHAR/VARCHAR key through client-side pruning and
cover the mixed-key case with the fetched-partition metric.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
