JingsongLi commented on code in PR #9497:
URL: https://github.com/apache/paimon/pull/9497#discussion_r4284080184
##########
paimon-core/src/main/java/org/apache/paimon/iceberg/metadata/IcebergSchema.java:
##########
@@ -52,14 +58,63 @@ public class IcebergSchema {
@JsonProperty(FIELD_FIELDS)
private final List<IcebergDataField> fields;
+ /** Paimon column IDs are 0-based; Iceberg IDs must be >= 1, so every ID
is shifted by one. */
+ private static final int ID_OFFSET = 1;
+
+ /**
+ * Builds the Iceberg schema for a Paimon table schema.
+ *
+ * <p>Iceberg field IDs must be positive, but Paimon assigns a single
0-based counter to both
+ * top-level and nested fields (see {@code Schema.Builder#column}). Every
Paimon ID - top-level
+ * and nested ROW/ARRAY/MAP/MULTISET alike - is shifted by {@link
#ID_OFFSET} so the whole
+ * emitted Iceberg schema uses one consistent, positive ID space. The same
mapping therefore
+ * applies to the manifest header, the table metadata schemas, the
partition source IDs and the
+ * manifest metrics maps, which are all derived from this schema.
+ */
public static IcebergSchema create(TableSchema tableSchema) {
return new IcebergSchema(
(int) tableSchema.id(),
tableSchema.fields().stream()
- .map(IcebergDataField::new)
+ .map(IcebergSchema::positiveField)
.collect(Collectors.toList()));
}
+ private static IcebergDataField positiveField(DataField field) {
+ return new IcebergDataField(shiftIds(field));
+ }
+
+ private static DataField shiftIds(DataField field) {
+ return new DataField(
+ field.id() + ID_OFFSET, field.name(), shiftIds(field.type()),
field.description());
Review Comment:
[P1] Keep Iceberg field IDs aligned with the physical data files. This
shifts the table/header schema IDs, but ParquetSchemaConverter still writes the
original Paimon DataField IDs into Parquet. In a real two-INT table, physical
a=0/b=1 is now described as a=1/b=2: committing (11,101),(22,202) and reading
with Iceberg returns (101,null),(202,null), and filters a=11 or b=101 return no
rows. A table partitioned by a also loses b. Keeping all other HEAD code
(including these new headers) and substituting only the exact-base
IcebergSchema restores all full/filtered reads. This also affects existing
files after metadata regeneration, so a metadata-only +1 mapping cannot be
deployed safely; the mapping needs to remain consistent with physical IDs and
include an explicit compatibility strategy for existing data.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
