dongjoon-hyun commented on code in PR #59919:
URL: https://github.com/apache/spark/pull/59919#discussion_r4416495397
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveSchemaEvolution.scala:
##########
@@ -111,6 +113,27 @@ object ResolveSchemaEvolution extends Rule[LogicalPlan] {
plan.rewriteAttrs(attrMap).asInstanceOf[T]
}
+ /**
+ * Rewrites references to the metadata columns of the old relation (e.g. in
MERGE conditions)
+ * to the metadata columns of the new relation, which exposes them with new
expression IDs.
+ * Metadata columns are matched by their logical name, which stays the same
even if a data column
+ * of the new table renames a conflicting metadata column. References to
metadata columns that
+ * the new relation does not expose are left unchanged.
+ */
+ private def rewriteMetadataAttrs[T <: LogicalPlan](
+ plan: T,
+ oldRelation: DataSourceV2Relation,
+ newRelation: DataSourceV2Relation): T = {
+ val newMetadataAttrs: Map[String, Attribute] =
newRelation.metadataOutput.collect {
+ case MetadataAttributeWithLogicalName(attr, name) => name -> attr
+ }.toMap
+ val attrMap = AttributeMap(oldRelation.metadataOutput.collect {
Review Comment:
Thank you for the review. I added `SPARK-60709: type widening top-level
field and target metadata column in update condition` to
`MergeIntoSchemaEvolutionTypeWideningTests`, so it runs in all five suites. It
widens `salary` from INT to BIGINT with `UPDATE SET *` and a `t._partition =
'hr'` condition.
I also checked that it reaches the state where the target metadata columns
are already part of the old relation's output. With the remap limited to the
metadata attributes that are not in that output, only its tests with the
evolution clause fail with `MISSING_ATTRIBUTES`, while the three existing tests
still pass. Without the remap, all four tests fail with the evolution clause.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
