1fanwang opened a new pull request, #3769: URL: https://github.com/apache/iceberg-python/pull/3769
<!-- Relates to #737 --> # Rationale for this change pyiceberg can stage a write on a branch and validate it there, but cannot publish it. Write and Audit work; Publish does not, so the last step happens outside Python — typically a Spark session calling `system.cherrypick_snapshot`. This adds `ManageSnapshots.cherry_pick_snapshot()`, mirroring Java's `CherryPickOperation`: - An **append** is replayed as a new snapshot on top of current, carrying the data files it added and recording `source-snapshot-id`, plus `published-wap-id` when staged with a `wap.id`. - Any **other operation** is fast-forwarded to when its parent is already current, and rejected otherwise with Java's message. - Picking an existing ancestor is a no-op, and a `wap.id` may only be published once. **Not covered:** dynamic overwrite (`OVERWRITE` with `replace-partitions`), which Java also replays. It raises here rather than silently doing nothing. Worth a follow-up. # Prior art [#750](https://github.com/apache/iceberg-python/pull/750) by @chinmay-bhat implemented this in 2024 and was closed by the stale bot with no review on merit; credit for getting there first is theirs. I rewrote rather than rebased, since `ManageSnapshots` has moved to `pyiceberg/table/update/snapshot.py` and the producer API changed underneath it. This one is narrower: a single method, no `publish_changes(wap_id)` and no dedicated producer class, since `fast_append()` already does the work. @chinmay-bhat — glad to hand this back if you would rather carry it. Touches the same class as [#3649](https://github.com/apache/iceberg-python/pull/3649). Happy to rebase onto whichever lands first and reuse its helper and exception types. # Are these changes tested? **Integration**, against the REST catalog and Hive metastore from `dev/docker-compose-integration.yml`. It stages a branch write with a `wap.id`, advances `main` so the pick is a replay not a fast-forward, publishes, then checks the summary properties and that a second publish is refused. <details><summary>Red, with <code>pyiceberg/table/update/snapshot.py</code> at upstream/main</summary> ``` $ git checkout upstream/main -- pyiceberg/table/update/snapshot.py $ pytest tests/integration/test_snapshot_operations.py -m integration -k cherry_pick -q E AttributeError: 'ManageSnapshots' object has no attribute 'cherry_pick_snapshot' tests/integration/test_snapshot_operations.py:366: AttributeError FAILED ...::test_cherry_pick_snapshot_publishes_staged_write[session_catalog_hive] FAILED ...::test_cherry_pick_snapshot_publishes_staged_write[session_catalog] 2 failed, 23 deselected ``` </details> <details><summary>Green, with the change restored</summary> ``` $ pytest tests/integration/test_snapshot_operations.py -m integration -k cherry_pick -v test_cherry_pick_snapshot_publishes_staged_write[session_catalog_hive] PASSED test_cherry_pick_snapshot_publishes_staged_write[session_catalog] PASSED ``` The table goes from `[1, 9]` to `[1, 2, 3, 9]`, the published snapshot carries `source-snapshot-id` and `published-wap-id=etl-001`, and re-publishing raises `Duplicate request to cherry pick wap id that was published already: etl-001`. </details> Ran the docs example verbatim too: 1 row before publish, 3 after, wap id recorded. **Unit**, across the `memory`, `sql`, and `sql_without_rowcount` catalogs: replay onto a moved-on `main`, wap id recorded, duplicate publish rejected, append replayed when its parent is current, non-append fast-forwarded, ancestor no-op, and both rejection paths. Same red on unpatched source. Whole unit and integration suites pass; `prek run -a` clean. # Are there any user-facing changes? `ManageSnapshots.cherry_pick_snapshot(snapshot_id)`, plus a Write-Audit-Publish section in `mkdocs/docs/api.md`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
