nsivabalan commented on code in PR #20492:
URL: https://github.com/apache/hudi/pull/20492#discussion_r4452612086
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/catalog/HoodieCatalog.scala:
##########
@@ -229,7 +230,36 @@ class HoodieCatalog extends DelegatingCatalogExtension
case _ => false
}
val catalogPropertyChanges = if (propertyChanges.isEmpty) Seq.empty
else catalogPropertyChangesToApply(table, propertyChanges)
- otherChanges.groupBy(c => c.getClass).foreach {
+ // SDP reconciles its business schema against the catalog's schema on
each refresh.
+ // Hudi owns its metadata columns; deleting them must not change the
stored schema.
+ val schemaChanges = otherChanges.filterNot {
+ case delete: DeleteColumn =>
+ delete.fieldNames().length == 1 &&
HoodieRecord.HOODIE_META_COLUMNS_WITH_OPERATION.contains(delete.fieldNames()(0))
+ // Hudi exposes nullable fields and does not enforce NOT NULL
constraints. SDP
+ // re-sends nullability on refresh; retain the stored schema in
either direction.
+ case change: UpdateColumnNullability =>
Review Comment:
+1 to this. SDP reconciliation needs the nullability no-op, but an explicit
user `ALTER COLUMN ... SET/DROP NOT NULL` should not silently do nothing.
Either reject it, or at least log a warning when the change is dropped.
Non-blocking for this PR; a follow-up is fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
