thswlsqls opened a new pull request, #18037: URL: https://github.com/apache/iceberg/pull/18037
Closes #18033 ## Summary - `IcebergRecordReader.openTask()` builds the reader without `idToConstant(...)`, so identity partition values that live only in the manifest never reach the reader and come back as `null`. - This is a regression from a2802c44c (#15333), which merged the Avro, ORC and Parquet branches into one `ReadBuilder`; all three used to pass `constantsMap(task, IdentityPartitionConverters::convertConstant)`. `data/GenericReader.openFile()`, touched by the same PR, kept that call, and so do Flink's `RowDataFileScanTaskReader` and Spark's `BaseRowReader`. - Only data files that do not physically store the partition columns are affected. `TableMigrationUtil.listPartition(...)` takes partition values from Hive metadata rather than from the file, so tables imported with `add_files`/`migrate` hit this. ## Testing done - Added `TestIcebergInputFormats#identityPartitionValuesMissingFromDataFile`, which writes a data file whose schema omits the identity partition columns and asserts the read-back values come from the manifest. It fails before the fix on all 6 parameterized variants (`IcebergInputFormat`/`MapredIcebergInputFormat` x Avro/ORC/Parquet) and passes after. - `./gradlew :iceberg-mr:check` — 164 tests, 2 skipped, 0 failures. - `:iceberg-mr` is not a REVAPI module, so no API compatibility check applies. --- **AI Disclosure** - Model: Claude Opus 5 - Platform/Tool: Claude Code - Human Oversight: [unknown - human to fill in] - Prompt Summary: Restore the identity partition constants that `IcebergInputFormat.IcebergRecordReader.openTask()` stopped passing to the format model `ReadBuilder` in a2802c44c, mirroring the sibling `data/GenericReader.openFile()`, and add a regression test in `TestIcebergInputFormats`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
