danny0405 commented on code in PR #13269:
URL: https://github.com/apache/hudi/pull/13269#discussion_r2094669943
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java:
##########
@@ -940,46 +954,54 @@ public String startCommit(String actionType,
HoodieTableMetaClient metaClient) {
*/
public void startCommitWithTime(String instantTime) {
HoodieTableMetaClient metaClient = createMetaClient(true);
- startCommitWithTime(instantTime, metaClient.getCommitActionType(),
metaClient);
+ startCommitWithTime(Option.of(instantTime),
metaClient.getCommitActionType(), metaClient);
}
/**
* Completes a new commit time for a write operation
(insert/update/delete/insert_overwrite/insert_overwrite_table) with specified
action.
*/
public void startCommitWithTime(String instantTime, String actionType) {
HoodieTableMetaClient metaClient = createMetaClient(true);
- startCommitWithTime(instantTime, actionType, metaClient);
+ startCommitWithTime(Option.of(instantTime), actionType, metaClient);
}
/**
* Starts a new commit time for a write operation (insert/update/delete)
with specified action.
*/
- private void startCommitWithTime(String instantTime, String actionType,
HoodieTableMetaClient metaClient) {
+ private String startCommitWithTime(Option<String> providedInstantTime,
String actionType, HoodieTableMetaClient metaClient) {
if (needsUpgrade(metaClient)) {
// unclear what instant to use, since upgrade does have a given instant.
executeUsingTxnManager(Option.empty(), () -> tryUpgrade(metaClient,
Option.empty()));
}
CleanerUtils.rollbackFailedWrites(config.getFailedWritesCleanPolicy(),
HoodieTimeline.COMMIT_ACTION, () ->
tableServiceClient.rollbackFailedWrites(metaClient));
- LOG.info("Generate a new instant time: {} action: {}", instantTime,
actionType);
- // check there are no inflight restore before starting a new commit.
- HoodieTimeline inflightRestoreTimeline =
metaClient.getActiveTimeline().getRestoreTimeline().filterInflightsAndRequested();
- ValidationUtils.checkArgument(inflightRestoreTimeline.countInstants() == 0,
- () -> "Found pending restore in active timeline. Please complete the
restore fully before proceeding. As of now, "
- + "table could be in an inconsistent state. Pending restores: "
- +
Arrays.toString(inflightRestoreTimeline.getInstantsAsStream().map(HoodieInstant::requestedTime).toArray()));
-
- if (config.getFailedWritesCleanPolicy().isLazy()) {
- this.heartbeatClient.start(instantTime);
- }
+ txnManager.beginTransaction(Option.empty(),
lastCompletedTxnAndMetadata.map(Pair::getLeft));
+ String instantTime;
+ HoodieInstant instant = null;
+ try {
+ instantTime = providedInstantTime.orElseGet(() ->
createNewInstantTime(false));
+ // check there are no inflight restore before starting a new commit.
+ HoodieTimeline inflightRestoreTimeline =
metaClient.getActiveTimeline().getRestoreTimeline().filterInflightsAndRequested();
Review Comment:
Can we fire a JIRA to refactor it out, it's just too heavy to add a
protection check with metadata file listing for creating requested file every
time, I mean, restore is a very infrequent operation compared to regular
commits.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
