danny0405 commented on code in PR #12250:
URL: https://github.com/apache/hudi/pull/12250#discussion_r1842322272
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/upgrade/UpgradeDowngradeUtils.java:
##########
@@ -107,4 +132,95 @@ static void updateMetadataTableVersion(HoodieEngineContext
context, HoodieTableV
throw new HoodieIOException("Error while updating metadata table
version", e);
}
}
+
+ static void upgradeToLSMTimeline(HoodieTable table, HoodieEngineContext
engineContext, HoodieWriteConfig config) {
+ // migrate archived timeline
+ LSMTimelineWriter lsmTimelineWriter =
LSMTimelineWriter.getInstance(config, table);
+ HoodieArchivedTimeline archivedTimeline =
table.getMetaClient().getArchivedTimeline();
+ Consumer<Exception> exceptionHandler = e -> {
+ if (config.isFailOnTimelineArchivingEnabled()) {
+ throw new HoodieException(e);
+ }
+ };
+
lsmTimelineWriter.write(Collections.singletonList(ActiveAction.fromInstants(archivedTimeline.getInstants())),
Review Comment:
Another concern is that the legacy archived timeline may contain enormous
instants there (like several GBs of avro logs), it would be very time-consuming
to load the whole legacy archived timeline, maybe we just load the latest avro
log(which should be enough for file slicing).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
