tibrewalpratik17 commented on code in PR #15863:
URL: https://github.com/apache/pinot/pull/15863#discussion_r2112106937
##########
pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/upsertcompactmerge/UpsertCompactMergeTaskGenerator.java:
##########
@@ -462,4 +472,54 @@ protected String
getSegmentCrcList(List<SegmentMergerMetadata> segmentMergerMeta
segmentMergerMetadataList.stream().map(x ->
String.valueOf(x.getSegmentZKMetadata().getCrc()))
.collect(Collectors.toList()), ",");
}
+
+ /**
+ * Retrieves the maximum creation time (in milliseconds) among the specified
segments across all servers.
+ *
+ * <p>This method filters the provided server-to-segments map to only
include the segments specified in
+ * {@code segmentNames}, then queries the servers for the creation time
metadata of those segments using
+ * {@link
ServerSegmentMetadataReader#getSegmentCreationMetadataFromServers}. It returns
the maximum creation
+ * time found. If no creation time is found for the given segments, a {@link
RuntimeException} is thrown.
+ *
+ * @param tableNameWithType The name of the table with type (e.g.,
myTable_OFFLINE)
+ * @param segmentNames The list of segment names to query
+ * @param serverToSegments A map from server instance to the list of
segments it hosts
+ * @param serverToEndpoints A BiMap from server instance to its admin
endpoint
+ * @param serverSegmentMetadataReader The reader to fetch segment metadata
from servers
+ * @return The maximum creation time in milliseconds among the specified
segments
+ * @throws RuntimeException If no creation time is found for the given
segments or if an I/O error occurs
+ */
+ @VisibleForTesting
+ protected Long getMaxCreationTimeMillis(String tableNameWithType,
List<String> segmentNames,
+ Map<String, List<String>> serverToSegments, BiMap<String, String>
serverToEndpoints,
+ ServerSegmentMetadataReader serverSegmentMetadataReader) {
+
+ // Filter serverToSegments to only include segments present in segmentNames
+ Set<String> segmentNameSet = new HashSet<>(segmentNames);
+ Map<String, List<String>> filteredServerToSegments = new HashMap<>();
+ for (Map.Entry<String, List<String>> entry : serverToSegments.entrySet()) {
+ List<String> filteredSegments =
+
entry.getValue().stream().filter(segmentNameSet::contains).collect(Collectors.toList());
+ if (!filteredSegments.isEmpty()) {
+ filteredServerToSegments.put(entry.getKey(), filteredSegments);
+ }
+ }
+ Map<String, List<Long>> creationTimeMap;
+ try {
+ creationTimeMap =
+
serverSegmentMetadataReader.getSegmentCreationMetadataFromServers(tableNameWithType,
filteredServerToSegments,
Review Comment:
Can we instead take this as a response in the
`getValidDocIdsMetadataFromServer` call itself else we end up making 2 calls to
each server. We can update `ValidDocIdsMetadataInfo` POJO to return
creationTime as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
