rdblue commented on code in PR #6997:
URL: https://github.com/apache/iceberg/pull/6997#discussion_r1125740765
##########
python/pyiceberg/io/pyarrow.py:
##########
@@ -492,11 +695,8 @@ def _file_to_table(
schema_raw = None
if metadata := parquet_schema.metadata:
schema_raw = metadata.get(ICEBERG_SCHEMA)
- if schema_raw is None:
- raise ValueError(
- "Iceberg schema is not embedded into the Parquet file, see
https://github.com/apache/iceberg/issues/6505";
- )
- file_schema = Schema.parse_raw(schema_raw)
+ # TODO: if field_ids are not present, another fallback level should be
implemented to look them up in the table schema
Review Comment:
This is Iceberg's "name mapping" feature.
The basics:
1. Name mapping is applied ONLY if there are not field IDs in the Parquet
file
2. If a field is present in the name mapping, the ID from the mapping is
used. Multiple names can be mapped to the same ID
3. If a field is not present in the name mapping, it is omitted.
For PyIceberg, the name mapping would probably be implemented by
pre-processing the PyArrow schema to add the `PYTHON:field_id` properties. That
way this code doesn't need to change.
As a result, if there is no field ID for any field, it should be omitted
from the Iceberg schema.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
