rahulsmahadev opened a new pull request, #17958: URL: https://github.com/apache/iceberg/pull/17958
Exposes stable Iceberg field IDs to Spark as DSv2 column IDs, so Spark 4.2's column-ID validation for nested fields (SPARK-57544) works with Iceberg tables. Spark 4.2 populates `Column.id()` from `StructField.id()`, and `SparkTable.columns()` goes through Spark's default `CatalogV2Util.structTypeToV2Columns(schema(), true)`, which reads those IDs recursively for top-level and nested struct fields. Iceberg's `TypeToSparkType` did not set them, so `Column.id()` was always null. This sets `StructField.withId(Integer.toString(fieldId))` during `Schema -> StructType` conversion, using each field's Iceberg field ID. `withId` writes Spark's official field-ID metadata key and preserves existing `StructField` metadata; it does not conflict with Iceberg's separate metadata-column marker. Named struct fields, including those nested inside arrays and maps, receive IDs through the existing recursive visitor; collection element/key/value pseudo-fields are not named Spark fields and are left unchanged. A test is added in `TestSparkSchemaUtil`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
