jonathanc-n commented on issue #964:
URL: https://github.com/apache/iceberg-rust/issues/964#issuecomment-2764259328
> 2. create UpdateAction and apply them to the metadata
>
> * When applying, there is the conflict detection process based on the
current local metadata load at step 1
> * The conflict detection process is specific for the Update Action
type. e.g. FastAppend just appends data files so it doesn't have conflict
detection.
> * If conflict detection in the apply function fails, it means that the
table has some conflict and we can't commit. This process abort
I'm a bit confused about this part. First of all should we do a operation
match and not have to append requirements if there are only append operations.
What do you think about a metadata check (a check if new metadata was
already committed, if table already exists, etc.) before we append updates or
requirements.
The process would be to:
1. Call refresh on the table metadata in `SnapshotProduceAction`
- This would mean that we would need to pass in a reference of `Catalog`
to `TableBuilder` to be able to perform the refresh.
- `SnapshotProduceAction` can use its transaction instance to use its
referenced `Table` to call the refresh.
2. Perform the metadata check before appending the updates and requirements
to the transaction. This will check the metadata that was used during the
snapshot produce process against the current refreshed metadata.
3. If it fails we can call a `CommitExceptionError` and have it be retried.
Update and requirements never get appended to the Transaction so we do not need
to connect to catalog.
Any alternate suggestions or fixes? cc @liurenjie1024 @Fokko @Xuanwo @sdd
@ZENOTME @c-thiel
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
