nsivabalan commented on code in PR #20492:
URL: https://github.com/apache/hudi/pull/20492#discussion_r4439072004
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/catalog/HoodieCatalog.scala:
##########
@@ -229,7 +230,20 @@ class HoodieCatalog extends DelegatingCatalogExtension
case _ => false
}
val catalogPropertyChanges = if (propertyChanges.isEmpty) Seq.empty
else catalogPropertyChangesToApply(table, propertyChanges)
- otherChanges.groupBy(c => c.getClass).foreach {
+ // SDP reconciles its business schema against the catalog's schema on
each refresh.
+ // Hudi owns its metadata columns; deleting them must not change the
stored schema.
+ val schemaChanges = otherChanges.filterNot {
+ case delete: DeleteColumn =>
+ delete.fieldNames().length == 1 &&
HoodieRecord.HOODIE_META_COLUMNS.contains(delete.fieldNames()(0))
Review Comment:
**`_hoodie_operation` is not covered.** `HOODIE_META_COLUMNS` holds only the
five core metadata columns. On a table with
`hoodie.allow.operation.metadata.field=true`, `_hoodie_operation` is part of
the catalog schema, so SDP will send a `DeleteColumn` for it on every refresh,
and this request will fail with `UnsupportedOperationException`. Could this use
`HoodieRecord.HOODIE_META_COLUMNS_WITH_OPERATION`, which is already used
elsewhere in the code? Adding a table with the operation field enabled to the
test would cover it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
