[GitHub] [hudi] prashantwason commented on a diff in pull request #8604: [HUDI-6151] Rollback previously applied commits to MDT when operations are retried.
prashantwason commented on code in PR #8604: URL: https://github.com/apache/hudi/pull/8604#discussion_r1213467835 ## hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/metadata/SparkHoodieBackedTableMetadataWriter.java: ## @@ -159,6 +162,13 @@ protected void commit(String instantTime, Maphttps://github.com/apache/hudi/pull/8684 where the new partition enabling has been changed to: 1. Use bulkInsert for initial commit 2. Always use a unique timestamp on MDT -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] prashantwason commented on a diff in pull request #8604: [HUDI-6151] Rollback previously applied commits to MDT when operations are retried.
prashantwason commented on code in PR #8604:
URL: https://github.com/apache/hudi/pull/8604#discussion_r1203427378
##
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/metadata/SparkHoodieBackedTableMetadataWriter.java:
##
@@ -161,27 +161,28 @@ protected void commit(String instantTime,
Map alreadyCompletedInstant =
metadataMetaClient.getActiveTimeline().filterCompletedInstants().filter(entry
-> entry.getTimestamp().equals(instantTime)).lastInstant();
-if (alreadyCompletedInstant.isPresent()) {
- // this code path refers to a re-attempted commit that got committed
to metadata table, but failed in datatable.
- // for eg, lets say compaction c1 on 1st attempt succeeded in
metadata table and failed before committing to datatable.
- // when retried again, data table will first rollback pending
compaction. these will be applied to metadata table, but all changes
- // are upserts to metadata table and so only a new delta commit will
be created.
- // once rollback is complete, compaction will be retried again,
which will eventually hit this code block where the respective commit is
- // already part of completed commit. So, we have to manually remove
the completed instant and proceed.
- // and it is for the same reason we enabled
withAllowMultiWriteOnSameInstant for metadata table.
- HoodieActiveTimeline.deleteInstantFile(metadataMetaClient.getFs(),
metadataMetaClient.getMetaPath(), alreadyCompletedInstant.get());
- metadataMetaClient.reloadActiveTimeline();
+LOG.info(String.format("%s completed commit at %s being applied to
metadata table",
+alreadyCompletedInstant.isPresent() ? "Already" : "Partially",
instantTime));
Review Comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] prashantwason commented on a diff in pull request #8604: [HUDI-6151] Rollback previously applied commits to MDT when operations are retried.
prashantwason commented on code in PR #8604: URL: https://github.com/apache/hudi/pull/8604#discussion_r1203427058 ## hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/metadata/SparkHoodieBackedTableMetadataWriter.java: ## @@ -161,27 +161,28 @@ protected void commit(String instantTime, Map
[GitHub] [hudi] prashantwason commented on a diff in pull request #8604: [HUDI-6151] Rollback previously applied commits to MDT when operations are retried.
prashantwason commented on code in PR #8604: URL: https://github.com/apache/hudi/pull/8604#discussion_r1203423423 ## hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/metadata/SparkHoodieBackedTableMetadataWriter.java: ## @@ -161,27 +161,28 @@ protected void commit(String instantTime, Map
