ksbeyer commented on code in PR #51466:
URL: https://github.com/apache/spark/pull/51466#discussion_r2237974818
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TableOutputResolver.scala:
##########
@@ -180,12 +180,38 @@ object TableOutputResolver extends SQLConfHelper with
Logging {
} else {
CharVarcharUtils.stringLengthCheck(casted, attr.dataType)
}
- Alias(exprWithStrLenCheck, attr.name)(explicitMetadata =
Some(attr.metadata))
+ makeNamed(exprWithStrLenCheck, attr)
} else {
value
}
}
+
+ /**
+ * Convert the expression into a NamedExpression that gets the name and
metadata from
+ * the given attribute. The metadata may be used by writers to get certain
table
+ * properties. For example
[[org.apache.spark.sql.catalyst.json.JacksonGenerator]]
+ * looks for default value metadata to control some behavior. We also
remove any
+ * remaining char/varchar metadata because it should be handled by this time.
+ * If we don't do this,
+ * DeltaBasedMergeIntoTableUpdateAsDeleteAndInsertSuite.
+ * test("SPARK-51513: Fix RewriteMergeIntoTable rule produces
unresolved plan")
+ * fails because the query attribute has varchar metadata and gets false
from:
+ *
org.apache.spark.sql.catalyst.plans.logical.V2WriteCommand.areCompatible
+ * which prevents the write operation from resolving.
+ */
+ private def makeNamed(expr: Expression, attr: Attribute): NamedExpression = {
+ val requiredMetadata = CharVarcharUtils.cleanMetadata(attr.metadata)
+ expr match {
+ case expr: NamedExpression if expr.name == attr.name && expr.metadata ==
requiredMetadata =>
+ expr
+ // Save an Alias if we can change the name directly.
Review Comment:
This code has been changed; I got rid of the direct Attribute name/metadata
changing and always use a new Alias when I want to change the name or metadata.
At least one reason mucking with attributes wasn't a good idea is the
RemoveNoopOperators rule was discarding my changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
