hemanthboyina commented on code in PR #58666:
URL: https://github.com/apache/spark/pull/58666#discussion_r4048931529
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala:
##########
@@ -602,11 +604,23 @@ object DataSourceV2Relation {
val catalystColStat = ColumnStat(distinct, min, max, nullCount,
avgLen, maxLen, histogram)
- output.foreach(attribute => {
- if (attribute.name.equals(key.describe())) {
- colStats = colStats :+ (attribute -> catalystColStat)
+ // Catalyst statistics are keyed by top-level Attribute, so only
single-part references
+ // can be matched to an output column.
+ val fieldNames = key.fieldNames
+ if (fieldNames.length == 1) {
+ val fieldName = fieldNames.head
+ val matches = output.filter(attribute => resolver(attribute.name,
fieldName))
+ // On an ambiguous case-insensitive match (e.g. outputs "id" and
"ID"), require a unique
+ // exact-name match, otherwise skip so CBO is not fed the wrong
column.
+ val matched = matches match {
Review Comment:
Right, the retained stat was traversal-order dependent. Now grouping matches
by output attribute (exprId) and preferring the exact-name key, skipping when
it's still ambiguous, so the result is deterministic. Added a regression test
for the many-keys-to-one-output case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
