kevinjqliu opened a new issue, #3957: URL: https://github.com/apache/iceberg-python/issues/3957
### Apache Iceberg version main (development) ### Please describe the bug 🐞 Follow-up to [the discussion on referenced_data_file](https://redirect.github.com/apache/iceberg-python/pull/3953#discussion_r3997076081). `Record` doesn't retain its schema, and `DataFile` getters assume v3 field positions. This means callers need to know the in-memory layout separately from the file version. Reproduced locally on `308768d99`: - Writing a default `DataFile.from_args()` record through `AvroOutputFile` with a v2 file schema silently writes a non-null `referenced_data_file` as null unless we also pass the v3 `record_schema`. The writer reads index 16 (`first_row_id`) instead of 17. - A `DataFile.from_args(_table_format_version=2, ...)` puts the reference at index 16, but the getters still assume v3: `first_row_id` returns the path and `referenced_data_file` raises `IndexError`. Reading a v2 file into `DataFile` without projecting to v3 has the same problem. The normal manifest read/write helpers already supply the v3 projection, but field access still depends on everything using the expected positions. Records should retain their schema and use field IDs instead. Callers should only need to specify the output version and let manifest IO handle the conversion. ### Willingness to contribute N/A -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
