rahulsmahadev opened a new pull request, #17957: URL: https://github.com/apache/iceberg/pull/17957
Implements Spark 4.2's `SupportsSchemaEvolution` mix-in on Iceberg's `SparkTable`, so Iceberg participates in DSv2 automatic schema evolution during writes. During automatic schema evolution Spark passes each candidate column change to `supportsColumnChange(TableChange.ColumnChange)` and skips the ones the source reports as unsupported. `SparkTable` now reports support consistent with what Iceberg's schema update actually applies: - add column: only when nullable, without a default, and with a Spark type Iceberg can convert; - update type: only when the new type is a primitive and an allowed Iceberg type promotion; - update nullability: only when relaxing to nullable; - delete column: rejected for identifier fields and for any field whose subtree contains an identifier field (matching `SchemaUpdate`), and for unknown fields; - rename, comment, and position changes are supported. `SparkTable` already advertises `TableCapability.AUTOMATIC_SCHEMA_EVOLUTION` (gated by the `write.spark.auto-schema-evolution.enabled` table property), which is what makes Spark consult this hook. A unit test is added in `TestSparkTable`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
