xr-chen opened a new issue, #765: URL: https://github.com/apache/incubator-xtable/issues/765
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/incubator-xtable/issues?q=is%3Aissue) and found no similar issues. ### Please describe the bug 🐞 Duplicated field IDs will be generated in the converted schema when converting a columnMapping-enabled Delta table that also has columns of array type into Iceberg format. For example ``` table_name = "people" local_base_path = "file:/tmp/delta-dataset" data = [ ("Alice", [1, 2, 3]), ("Bob", [4, 5]), ("Charlie", [6, 7, 8, 9]) ] df = spark.createDataFrame(data, ["name", "scores"]) ( df.write .format("delta") .save(f"{local_base_path}/{table_name}", mode="overwrite") ) df.printSchema() root |-- name: string (nullable = true) |-- scores: array (nullable = true) | |-- element: long (containsNull = true) ``` xTable will work for the above Delta table with one array column, but after enabling the columnMapping on the source Delta table ``` spark.sql(f"""ALTER TABLE delta.`{local_base_path}/{table_name}` SET TBLPROPERTIES ( 'delta.minReaderVersion' = '2', 'delta.minWriterVersion' = '5', 'delta.columnMapping.mode' = 'name' )""") ``` Converting it into an Iceberg table will cause the following error ``` 2025-12-09 05:23:25 ERROR org.apache.xtable.spi.sync.TableFormatSync:83 - Failed to sync snapshot java.lang.IllegalArgumentException: Multiple entries with same key: 1=name and 1=scores.element at org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap.conflictException(ImmutableMap.java:378) ~[xtable-utilities_2.12-0.2.0-SNAPSHOT-bundled.jar:0.2.0-SNAPSHOT] at org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap.checkNoConflict(ImmutableMap.java:372) ~[xtable-utilities_2.12-0.2.0-SNAPSHOT-bundled.jar:0.2.0-SNAPSHOT] at org.apache.iceberg.relocated.com.google.common.collect.RegularImmutableMap.checkNoConflictInKeyBucket(RegularImmutableMap.java:246) ~[xtable-utilities_2.12-0.2.0-SNAPSHOT-bundled.jar:0.2.0-SNAPSHOT] at org.apache.iceberg.relocated.com.google.common.collect.RegularImmutableMap.fromEntryArrayCheckingBucketOverflow(RegularImmutableMap.java:133) ~[xtable-utilities_2.12-0.2.0-SNAPSHOT-bundled.jar:0.2.0-SNAPSHOT] at org.apache.iceberg.relocated.com.google.common.collect.RegularImmutableMap.fromEntryArray(RegularImmutableMap.java:95) ~[xtable-utilities_2.12-0.2.0-SNAPSHOT-bundled.jar:0.2.0-SNAPSHOT] at org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap$Builder.build(ImmutableMap.java:572) ~[xtable-utilities_2.12-0.2.0-SNAPSHOT-bundled.jar:0.2.0-SNAPSHOT] at org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap$Builder.buildOrThrow(ImmutableMap.java:600) ~[xtable-utilities_2.12-0.2.0-SNAPSHOT-bundled.jar:0.2.0-SNAPSHOT] at org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap$Builder.build(ImmutableMap.java:587) ~[xtable-utilities_2.12-0.2.0-SNAPSHOT-bundled.jar:0.2.0-SNAPSHOT] at org.apache.iceberg.types.IndexByName.byId(IndexByName.java:81) ~[xtable-utilities_2.12-0.2.0-SNAPSHOT-bundled.jar:0.2.0-SNAPSHOT] at org.apache.iceberg.types.TypeUtil.indexNameById(TypeUtil.java:172) ~[xtable-utilities_2.12-0.2.0-SNAPSHOT-bundled.jar:0.2.0-SNAPSHOT] at org.apache.iceberg.Schema.lazyIdToName(Schema.java:183) ~[xtable-utilities_2.12-0.2.0-SNAPSHOT-bundled.jar:0.2.0-SNAPSHOT] at org.apache.iceberg.Schema.<init>(Schema.java:112) ~[xtable-utilities_2.12-0.2.0-SNAPSHOT-bundled.jar:0.2.0-SNAPSHOT] at org.apache.iceberg.Schema.<init>(Schema.java:91) ~[xtable-utilities_2.12-0.2.0-SNAPSHOT-bundled.jar:0.2.0-SNAPSHOT] at org.apache.iceberg.Schema.<init>(Schema.java:83) ~[xtable-utilities_2.12-0.2.0-SNAPSHOT-bundled.jar:0.2.0-SNAPSHOT] at org.apache.iceberg.Schema.<init>(Schema.java:79) ~[xtable-utilities_2.12-0.2.0-SNAPSHOT-bundled.jar:0.2.0-SNAPSHOT] at org.apache.xtable.iceberg.IcebergSchemaExtractor.toIceberg(IcebergSchemaExtractor.java:79) ~[xtable-utilities_2.12-0.2.0-SNAPSHOT-bundled.jar:0.2.0-SNAPSHOT] at org.apache.xtable.iceberg.IcebergConversionTarget.initializeTableIfRequired(IcebergConversionTarget.java:157) ~[xtable-utilities_2.12-0.2.0-SNAPSHOT-bundled.jar:0.2.0-SNAPSHOT] at org.apache.xtable.iceberg.IcebergConversionTarget.beginSync(IcebergConversionTarget.java:145) ~[xtable-utilities_2.12-0.2.0-SNAPSHOT-bundled.jar:0.2.0-SNAPSHOT] at org.apache.xtable.spi.sync.TableFormatSync.getSyncResult(TableFormatSync.java:162) ~[xtable-utilities_2.12-0.2.0-SNAPSHOT-bundled.jar:0.2.0-SNAPSHOT] at org.apache.xtable.spi.sync.TableFormatSync.syncSnapshot(TableFormatSync.java:74) [xtable-utilities_2.12-0.2.0-SNAPSHOT-bundled.jar:0.2.0-SNAPSHOT] at org.apache.xtable.conversion.ConversionController.syncSnapshot(ConversionController.java:283) [xtable-utilities_2.12-0.2.0-SNAPSHOT-bundled.jar:0.2.0-SNAPSHOT] at org.apache.xtable.conversion.ConversionController.syncTableFormats(ConversionController.java:203) [xtable-utilities_2.12-0.2.0-SNAPSHOT-bundled.jar:0.2.0-SNAPSHOT] at org.apache.xtable.conversion.ConversionController.sync(ConversionController.java:106) [xtable-utilities_2.12-0.2.0-SNAPSHOT-bundled.jar:0.2.0-SNAPSHOT] at org.apache.xtable.utilities.RunSync.syncTableMetdata(RunSync.java:186) [xtable-utilities_2.12-0.2.0-SNAPSHOT-bundled.jar:0.2.0-SNAPSHOT] at org.apache.xtable.utilities.RunSync.runSync(RunSync.java:297) [xtable-utilities_2.12-0.2.0-SNAPSHOT-bundled.jar:0.2.0-SNAPSHOT] at org.apache.xtable.utilities.RunSync.main(RunSync.java:282) [xtable-utilities_2.12-0.2.0-SNAPSHOT-bundled.jar:0.2.0-SNAPSHOT] 2025-12-09 05:23:25 ERROR org.apache.xtable.conversion.ConversionController:217 - Sync failed for the following formats ICEBERG ``` ### Are you willing to submit PR? - [x] I am willing to submit a PR! - [ ] I am willing to submit a PR but need help getting started! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
