lawofcycles opened a new pull request, #3320: URL: https://github.com/apache/iceberg-python/pull/3320
Closes #3319 Closes #819 Closes #269 # Rationale for this change PyIceberg currently fails immediately with `CommitFailedException` when a concurrent transaction commits first, regardless of whether the writes actually conflict. Java Iceberg handles this transparently through its retry loop in `SnapshotProducer.commit()`. This PR adds automatic commit retry with exponential backoff and data conflict validation to PyIceberg, matching Java Iceberg's behavior. On `CommitFailedException`, the retry loop refreshes table metadata, re-runs validation, and regenerates manifests. If validation detects a real data conflict, the operation aborts with `ValidationException` instead of retrying. The retry loop is placed in `Transaction.commit_transaction()` rather than in individual snapshot producers. This is necessary because `Transaction.delete()` uses two producers (`_DeleteFiles` + `_OverwriteFiles`) that must be committed atomically. Retrying at the producer level would break this atomicity. Validation behavior follows Java's `BaseOverwriteFiles.validate()`, using the existing validation functions from `validate.py` that were contributed through #1935, #1938, #2050, and #3049. ## Are these changes tested? Yes. 54 unit tests and 8 integration tests covering retry success, `ValidationException` abort, retry exhaustion, isolation levels, partition-level conflict detection (both user-specified and auto-computed filters), manifest cleanup, and producer state reset. ## Are there any user-facing changes? Yes. Previously, all concurrent write conflicts resulted in `CommitFailedException`. Now: - Compatible concurrent writes (e.g. concurrent appends) are retried automatically and succeed transparently - Incompatible concurrent writes (e.g. concurrent deletes on the same data) raise `ValidationException` instead of `CommitFailedException` The following new table properties are supported. - `commit.retry.num-retries` (default: 4) - `commit.retry.min-wait-ms` (default: 100) - `commit.retry.max-wait-ms` (default: 60000) - `write.delete.isolation-level` (default: serializable) - `write.update.isolation-level` (default: serializable) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
