eugenegujing opened a new pull request, #7579:
URL: https://github.com/apache/texera/pull/7579
### What changes were proposed in this PR?
This PR adds
`amber/src/test/python/core/storage/iceberg/test_iceberg_table_writer.py` with
14 pure unit tests using mocked catalog/table objects (no real Iceberg catalog,
no Postgres, no network). Covered behaviors:
- Constructor: loads the table via
`catalog.load_table(f"{namespace}.{name}")` and takes `buffer_size` from
`StorageConfig.ICEBERG_TABLE_COMMIT_BATCH_SIZE`.
- Buffer-threshold flush: `put_one` below the threshold does not flush;
reaching `buffer_size` triggers a flush (serde called with schema + buffered
items, `table.append` called with serde's result, buffer cleared); items added
after a flush start a fresh buffer.
- `close()` flushes the remaining items when the buffer is non-empty, and
performs no append when the buffer is empty.
- `open()` clears a previously dirty buffer.
- `remove_one()` removes a buffered item; removing an item already flushed
out of the buffer raises `ValueError` (pinning the current `list.remove`
behavior).
- `_flush_buffer()` returns early on an empty buffer (no
serde/append/refresh calls).
- Retry path: `table.append` raising pyiceberg's `CommitFailedException`
twice then succeeding completes the flush, with `table.refresh()` called once
per attempt; a permanent failure is reraised after 10 attempts and the buffer
is NOT cleared.
- A serde failure propagates without touching the table: serde runs outside
the retry loop, so it is called exactly once, no refresh/append happens, and
the buffer is kept.
- The retry decorator sets no exception filter, so a non-conflict error
(e.g. `ValueError`) is also retried for all 10 attempts before being reraised
(pinning the current behavior).
The retry tests patch `tenacity.nap.time.sleep`, so the exponential-backoff
waits (`wait_random_exponential(0.001, 10)`) never sleep for real; the whole
file runs in about a second.
No production code is changed.
### Any related issues, documentation, discussions?
Closes #7575
### How was this PR tested?
This PR is itself test-only. The new spec was run with:
```
cd amber && pytest
src/test/python/core/storage/iceberg/test_iceberg_table_writer.py -v
```
Result: 14 passed in about a second. The file is formatted with `black`
(unchanged by `--check`). The suite was mutation-checked: targeted mutations of
the writer (flipping the `>=` threshold to `>`, dropping the `buffer.clear()`
after append, removing `table.refresh()` in the retry body, making `close()`
skip the flush, removing the empty-buffer early return, and lowering
`stop_after_attempt`) each caused at least one test to fail, and the source was
restored afterwards.
### Was this PR authored or co-authored using generative AI tooling?
Co-authored by: Claude Code (Claude Fable 5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
