shangxinli commented on code in PR #20492:
URL: https://github.com/apache/hudi/pull/20492#discussion_r4444137853
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/catalog/HoodieCatalog.scala:
##########
@@ -229,7 +230,36 @@ class HoodieCatalog extends DelegatingCatalogExtension
case _ => false
}
val catalogPropertyChanges = if (propertyChanges.isEmpty) Seq.empty
else catalogPropertyChangesToApply(table, propertyChanges)
- otherChanges.groupBy(c => c.getClass).foreach {
+ // SDP reconciles its business schema against the catalog's schema on
each refresh.
+ // Hudi owns its metadata columns; deleting them must not change the
stored schema.
+ val schemaChanges = otherChanges.filterNot {
+ case delete: DeleteColumn =>
+ delete.fieldNames().length == 1 &&
HoodieRecord.HOODIE_META_COLUMNS_WITH_OPERATION.contains(delete.fieldNames()(0))
+ // Hudi exposes nullable fields and does not enforce NOT NULL
constraints. SDP
+ // re-sends nullability on refresh; retain the stored schema in
either direction.
Review Comment:
This now silently drops any UpdateColumnNullability change for an existing
column on every alter table call, not just SDP's reconciliation pass, since
it'll now no-op with no error or log. Worth confirming that's intended for a
user's explicit ALTER COLUMN NULL/NOT NULL too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
