shangxinli commented on code in PR #20492:
URL: https://github.com/apache/hudi/pull/20492#discussion_r4444187186
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/catalog/HoodieCatalog.scala:
##########
@@ -229,7 +230,36 @@ class HoodieCatalog extends DelegatingCatalogExtension
case _ => false
}
val catalogPropertyChanges = if (propertyChanges.isEmpty) Seq.empty
else catalogPropertyChangesToApply(table, propertyChanges)
- otherChanges.groupBy(c => c.getClass).foreach {
+ // SDP reconciles its business schema against the catalog's schema on
each refresh.
+ // Hudi owns its metadata columns; deleting them must not change the
stored schema.
+ val schemaChanges = otherChanges.filterNot {
+ case delete: DeleteColumn =>
+ delete.fieldNames().length == 1 &&
HoodieRecord.HOODIE_META_COLUMNS_WITH_OPERATION.contains(delete.fieldNames()(0))
+ // Hudi exposes nullable fields and does not enforce NOT NULL
constraints. SDP
+ // re-sends nullability on refresh; retain the stored schema in
either direction.
+ case change: UpdateColumnNullability =>
Review Comment:
This now silently drops any UpdateColumnNullability change for an existing
column on every alter table call, not just SDP's reconciliation pass, since
schemaChanges filters it out before the apply loop ever sees it. Worth
confirming that's intended for a user's explicit ALTER COLUMN NULL or NOT NULL
too, since it'll now no-op with no error or log.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
