yyanyy commented on code in PR #58298:
URL: https://github.com/apache/spark/pull/58298#discussion_r4021211282
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/util/SchemaUtils.scala:
##########
@@ -529,12 +529,23 @@ private[spark] object SchemaUtils {
if (field.nullable) s"$name $dataType" else s"$name $dataType NOT NULL"
}
+ /**
+ * Folds a name to the key used to decide whether two names refer to the
same column or field.
+ *
+ * This is the identity rule name resolution is built on: `AttributeSeq`
looks attributes up by
+ * this key and only then filters the candidates with the resolver, and the
duplicate-name checks
+ * above reject a schema that holds two names folding to one key. Matching
by folded name
+ * therefore finds exactly the field resolution would, while comparing with
the resolver alone
+ * can match several fields a schema is allowed to keep apart
(`equalsIgnoreCase` equates U+017F
+ * LONG S with `s`, which this fold does not).
+ */
+ def foldName(name: String, caseSensitiveAnalysis: Boolean): String = {
+ if (caseSensitiveAnalysis) name else name.toLowerCase(Locale.ROOT)
Review Comment:
Good catch. I fixed this by removing the invalid assumption that a ROOT-fold
bucket is unique rather than changing the existing duplicate-name check.
Compatibility now keeps every folded candidate and filters them with the
resolver, while data-column rebinding uses `AttributeSeq.resolve`, so both make
the same choice as a fresh query instead of taking the first folded match.
I also added the requested Turkish-locale end-to-end test with the new
U+0130 column first; it verifies that both fresh and stale queries read the
retained i+U+0307 column.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
