b-phi opened a new issue, #1619:
URL: https://github.com/apache/iceberg-python/issues/1619
### Apache Iceberg version
0.8.1 (latest release)
### Please describe the bug 🐞
Given a simple schema with a DayTransform over a Timestamp column, an error
is thrown during write related to mixing downcasted and non-downcasted
timestamps.
This would also be resolved with support for nanosecond timestamp types.
```python
schema = Schema(
NestedField(field_id=1, name='timestamp', field_type=TimestampType(),
required=False),
)
partition_spec = PartitionSpec(
PartitionField(source_id=1, field_id=1001, transform=DayTransform(),
name='date'),
)
schema = pa.schema([pa.field('timestamp', pa.timestamp('ns'))])
data = pa.Table.from_pydict({'timestamp': [1]}, schema=schema)
table.append(data)
```
>.local/lib/python3.10/site-packages/pyiceberg/table/__init__.py:984: in
append
tx.append(df=df, snapshot_properties=snapshot_properties)
.local/lib/python3.10/site-packages/pyiceberg/table/__init__.py:417: in
append
for data_file in data_files:
.local/lib/python3.10/site-packages/pyiceberg/io/pyarrow.py:2636: in
_dataframe_to_data_files
partitions = _determine_partitions(spec=table_metadata.spec(),
schema=table_metadata.schema(), arrow_table=df)
.local/lib/python3.10/site-packages/pyiceberg/io/pyarrow.py:2715: in
_determine_partitions
partition_values_table = pa.table({
.local/lib/python3.10/site-packages/pyiceberg/io/pyarrow.py:2716: in
<dictcomp>
str(partition.field_id):
partition.transform.pyarrow_transform(field.field_type)(arrow_table[field.name])
.local/lib/python3.10/site-packages/pyiceberg/transforms.py:560: in <lambda>
return lambda v: pc.days_between(pa.scalar(epoch), v) if v is not None
else None
.local/lib/python3.10/site-packages/pyarrow/compute.py:247: in wrapper
return func.call(args, None, memory_pool)
pyarrow/_compute.pyx:393: in pyarrow._compute.Function.call
???
pyarrow/error.pxi:155: in pyarrow.lib.pyarrow_internal_check_status
???
E pyarrow.lib.ArrowNotImplementedError: Function 'days_between' has no
kernel matching input types (timestamp[us], timestamp[ns])
pyarrow/error.pxi:92: ArrowNotImplementedError
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from
the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
