9aman commented on code in PR #15142:
URL: https://github.com/apache/pinot/pull/15142#discussion_r1991325038
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/retention/RetentionManager.java:
##########
@@ -189,6 +219,110 @@ private boolean shouldDeleteInProgressLLCSegment(String
segmentName, IdealState
}
}
+ private List<String> getSegmentsToDeleteFromDeepstore(String
tableNameWithType, RetentionStrategy retentionStrategy,
+ List<SegmentZKMetadata> segmentZKMetadataList, int
untrackedSegmentsDeletionBatchSize) {
+ List<String> segmentsToDelete = new ArrayList<>();
+
+ if (untrackedSegmentsDeletionBatchSize <= 0) {
+ // return an empty list in case untracked segment deletion is not
configured
+ LOGGER.info(
+ "Not scanning deep store for untracked segments for table: {} as
untrackedSegmentsDeletionBatchSize is set "
+ + "to: {}",
+ tableNameWithType, untrackedSegmentsDeletionBatchSize);
+ return segmentsToDelete;
+ }
+
+ List<String> segmentsPresentInZK =
+
segmentZKMetadataList.stream().map(SegmentZKMetadata::getSegmentName).collect(Collectors.toList());
+ try {
+ LOGGER.info("Fetch segments present in deep store that are beyond
retention period for table: {}",
+ tableNameWithType);
+ segmentsToDelete =
+ findUntrackedSegmentsToDeleteFromDeepstore(tableNameWithType,
retentionStrategy, segmentsPresentInZK);
+ _controllerMetrics.setValueOfTableGauge(tableNameWithType,
ControllerGauge.UNTRACKED_SEGMENTS_COUNT,
+ segmentsToDelete.size());
+
+ if (segmentsToDelete.size() > untrackedSegmentsDeletionBatchSize) {
Review Comment:
The ideal way would be to do pagination on the FS and bring only
untrackedSegmentsDeletionBatchSize number of files/ segments from deepstore.
We can add this improvement later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
