aoelvp94 opened a new issue, #76338:
URL: https://github.com/apache/airflow/issues/76338
### Description
`IcebergHook` (`apache-iceberg` provider) can list namespaces/tables, load a
table, read its schema, partition spec, properties and snapshots, and return an
OAuth2 token. It has no method that writes data.
Proposal: add a small `append()` method that uses the catalog the hook
already builds from the connection:
```python
IcebergHook().append("demo.events", [{"id": 1, "name": "alpha"}])
IcebergHook().append("demo.events", pa_table) # pyarrow.Table also accepted
```
- Accepts a `pyarrow.Table` or a `list[dict]`, aligned to the Iceberg table
schema (keys not in the schema are dropped instead of failing the write).
- On `CommitFailedException`, reloads the table from the catalog and retries
a bounded number of times.
- Does not accept a `RecordBatchReader`: it is consumed on the first
attempt, so a retry would append nothing.
Out of scope: overwrite, upsert, delete, merge-on-read, and new operators.
### Use case/motivation
I want a DAG task to append a small batch of rows to an Iceberg table using
the existing Iceberg connection, without each DAG building its own pyiceberg
catalog client.
Today the task has to reach into `hook.catalog` /
`hook.load_table(...).append(...)` and handle `CommitFailedException` itself.
Recent pyiceberg releases retry the commit inside a transaction when
`commit.retry.num-retries` > 0, but that does not cover a table that sets
retries to `0`, and the provider still allows `pyiceberg>=0.8.0`.
A local reproduction (SQLite catalog, no cloud) covering append, unknown
key, stale handle and reload-after-conflict:
https://github.com/aoelvp94/airflow-iceberg-append-demo (`just check`).
### Related issues
None found for writing through `IcebergHook`. Related provider work: #62634
(catalog introspection in `IcebergHook`), #71387
(`IcebergTableSnapshotTrigger`).
### Are you willing to submit a PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
