p1c2u commented on issue #1798:
URL:
https://github.com/apache/iceberg-python/issues/1798#issuecomment-2737554034
@kevinjqliu
after investigation I found out it happens after I append pyarrow table
without list field specified in schema as optional.
Example table with schema
```python
from pyiceberg.catalog import load_catalog
catalog = load_catalog(**dict(type="in-memory"))
from pyiceberg.schema import Schema
from pyiceberg.types import NestedField, StringType, DoubleType, LongType,
ListType
schema = Schema(
NestedField(field_id=1, name="name", field_type=StringType(),
required=False),
NestedField(
field_id=3,
name="my_list",
field_type=ListType(
element_id=45, element=StringType(), element_required=False
),
required=False,
),
)
catalog.create_namespace_if_not_exists("test")
catalog.create_table_if_not_exists("test.table", schema)
```
I append dataset with my_list:
```python
import pyarrow as pa
df_1 = pa.Table.from_pylist([
{"name": "one", "my_list": ["test"]},
{"name": "another", "my_list": ["test"]},
])
catalog.load_table("test.table").append(df_1)
```
Read works
```python
catalog.load_table("test.table").scan().to_arrow()
```
I append dataset without my_list:
```python
import pyarrow as pa
df_2 = pa.Table.from_pylist([
{"name": "one"},
{"name": "another"},
])
catalog.load_table("test.table").append(df_2)
```
This time it won't work
```python
catalog.load_table("test.table").scan().to_arrow()
```
it will throw
> ValueError: Parquet file does not have field-ids and the Iceberg table
does not have 'schema.name-mapping.default' defined
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
