danny0405 commented on code in PR #12250:
URL: https://github.com/apache/hudi/pull/12250#discussion_r1841744869
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/upgrade/UpgradeDowngradeUtils.java:
##########
@@ -107,4 +132,95 @@ static void updateMetadataTableVersion(HoodieEngineContext
context, HoodieTableV
throw new HoodieIOException("Error while updating metadata table
version", e);
}
}
+
+ static void upgradeToLSMTimeline(HoodieTable table, HoodieEngineContext
engineContext, HoodieWriteConfig config) {
+ // migrate archived timeline
+ LSMTimelineWriter lsmTimelineWriter =
LSMTimelineWriter.getInstance(config, table);
+ HoodieArchivedTimeline archivedTimeline =
table.getMetaClient().getArchivedTimeline();
+ Consumer<Exception> exceptionHandler = e -> {
+ if (config.isFailOnTimelineArchivingEnabled()) {
+ throw new HoodieException(e);
+ }
+ };
+
lsmTimelineWriter.write(Collections.singletonList(ActiveAction.fromInstants(archivedTimeline.getInstants())),
Review Comment:
The list for archived timeline is huge, we need to split the list into small
batchs and write into the LSM timeline per-batch, by default, just use 10
instants as a batch which is in line with the current behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
