snleee commented on code in PR #9825:
URL: https://github.com/apache/pinot/pull/9825#discussion_r1026045441
##########
pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/BaseMultipleSegmentsConversionExecutor.java:
##########
@@ -242,6 +256,10 @@ public List<SegmentConversionResult>
executeTask(PinotTaskConfig pinotTaskConfig
new
BasicHeader(FileUploadDownloadClient.CustomHeaders.SEGMENT_ZK_METADATA_CUSTOM_MAP_MODIFIER,
segmentZKMetadataCustomMapModifier.toJsonString());
+ URI outputSegmentTarURI =
moveSegmentToOutputPinotFS(pinotTaskConfig.getConfigs(),
convertedTarredSegmentFile);
+ LOGGER.info("Moved generated segment from [{}] to location: [{}]",
Review Comment:
What happens if the Minion somehow got killed (e.g. restart) after finish
the segment move to outputPinotFS but before finishing the metadata push to
controller?
I think that this can potentially leave some files in the deep storage
without any segment metadata pointing those files. This will potentially become
the problem unless we have the cleanup mechanism. Do we have any
design/solution to address this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
