xr-chen commented on code in PR #766:
URL: https://github.com/apache/incubator-xtable/pull/766#discussion_r2611473746
##########
xtable-core/src/test/java/org/apache/xtable/ITConversionController.java:
##########
@@ -746,6 +746,29 @@ public void testIcebergCorruptedSnapshotRecovery() throws
Exception {
}
}
+ @Test
+ public void testColumnMappingEnabledDeltaToIceberg() {
Review Comment:
@the-other-tim-brown Yes, I think so, but the code actually can't pass this
test case now, so it probably won't work on any rename column type of schema
change. It seems to me that only populating fieldId doesn't work, and the
converted Iceberg doesn't know which 'physical' column in the data file to read
data from for a 'logic' column name in the table schema, and it returns null
values for all columns if we read from the generated Iceberg table. This issue
is probably due to:
1. We don't extract the `delta.columnMapping.physicalName` from the Delta
table's schema in
https://github.com/apache/incubator-xtable/blob/64f38b813291747a5aecff14664a19354ee02d60/xtable-core/src/main/java/org/apache/xtable/delta/DeltaSchemaExtractor.java#L56,
so we don't know where the column is actually stored
2. In the converted Iceberg, it doesn't have a [name
mapping](https://iceberg.apache.org/spec/?h=name+mapping#column-projection) to
recognize which Parquet column corresponds to a given Iceberg field ID
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
