Hi Sylwester, Thanks for raising this issue. I am also working to resolve the inconsistency in iceberg-cpp [1][2].
Your proposal looks reasonable, and it aligns with what the Avro reader does in the Java implementation (whereas ORC follows the Parquet rule). Whichever rule we decide to use, we should make sure it aligns with the specification. Could you please open a spec PR so we can have a concrete discussion there? [1] https://github.com/apache/iceberg-cpp/pull/1108 [2] https://github.com/apache/iceberg-cpp/pull/1109 Best regards, Manu On Tue, Nov 3, 2026 at 8:20 AM Sylwester Lachiewicz via dev < [email protected]> wrote: > Hi all, > > The spec is ambiguous about Parquet data files that carry field IDs for > some fields but not all, and no two of the four implementations I checked > resolve such a file the same way. I'd like to agree on one reading and > write it into the spec. > > Column Projection [1] says a reader should "use > schema.name-mapping.default metadata to map field id to columns without > field id", which reads as per column. The paragraph that follows says the > mappings "provide fallback field ids to be used when a data file does not > contain field id information", which reads as per file. For a file with no > IDs at all, both readings agree. They diverge when only some fields carry > an ID. > > What the readers do today, on their main branches: > > - Java (ReadConf, ParquetSchemaUtil.HasIds): if any field in the file has > an ID, the file is resolved by ID only and the name mapping is not > consulted. > - PyIceberg (pyarrow_to_schema, _HasIds): IDs are used only if every > field, including list elements and map keys and values, has one; otherwise > the whole file is resolved through the name mapping. > - iceberg-rust (arrow/reader/pipeline.rs): the first top-level column > decides; if it has an ID, the file is resolved by ID only. > - iceberg-go (table/internal/parquet_files.go, > pruneParquetSchema.fieldID): Parquet column pruning resolves per field; an > embedded ID wins, and the mapping fills in the fields without one. > > This happens in practice. Delta tables with column mapping, written > without IcebergCompatV2, store IDs for top-level columns but none for map > keys and values or list elements. When such a table gets Iceberg metadata > [2], PyIceberg reads the map columns through the name mapping, while the > Java reader resolves by ID only and does not find the map's key and value. > The same table gives different results depending on the reader. > > I propose the per-field reading, which matches the Column Projection rule: > a field with an embedded ID is resolved by that ID, and a field without one > falls back to the name mapping. It never overrides an ID a writer recorded, > it changes nothing for files with full IDs or with none, and it makes the > partial case readable. > > If there is agreement, I can open a spec PR that clarifies the paragraph, > and contribute a fixture to iceberg-verification [3]: a file with IDs on > its top-level fields only, as a companion to the base name-mapping table > that issue covers. > > Is there a reason for the per-file behaviour that I'm missing, for example > a correctness concern with mixing ID and name resolution within one file? > > [1] > https://www.google.com/url?q=https://iceberg.apache.org/spec/%23column-projection&source=gmail&ust=1793746430284000&sa=E > [2] > https://www.google.com/url?q=https://github.com/apache/incubator-xtable/issues/911&source=gmail&ust=1793746430284000&sa=E > [3] > https://www.google.com/url?q=https://github.com/apache/iceberg-verification/issues/14&source=gmail&ust=1793746430284000&sa=E > > Thanks, > Sylwester >
