tarun11Mavani commented on code in PR #15863:
URL: https://github.com/apache/pinot/pull/15863#discussion_r2113291757
##########
pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/upsertcompactmerge/UpsertCompactMergeTaskGenerator.java:
##########
@@ -462,4 +472,54 @@ protected String
getSegmentCrcList(List<SegmentMergerMetadata> segmentMergerMeta
segmentMergerMetadataList.stream().map(x ->
String.valueOf(x.getSegmentZKMetadata().getCrc()))
.collect(Collectors.toList()), ",");
}
+
+ /**
+ * Retrieves the maximum creation time (in milliseconds) among the specified
segments across all servers.
+ *
+ * <p>This method filters the provided server-to-segments map to only
include the segments specified in
+ * {@code segmentNames}, then queries the servers for the creation time
metadata of those segments using
+ * {@link
ServerSegmentMetadataReader#getSegmentCreationMetadataFromServers}. It returns
the maximum creation
+ * time found. If no creation time is found for the given segments, a {@link
RuntimeException} is thrown.
+ *
+ * @param tableNameWithType The name of the table with type (e.g.,
myTable_OFFLINE)
+ * @param segmentNames The list of segment names to query
+ * @param serverToSegments A map from server instance to the list of
segments it hosts
+ * @param serverToEndpoints A BiMap from server instance to its admin
endpoint
+ * @param serverSegmentMetadataReader The reader to fetch segment metadata
from servers
+ * @return The maximum creation time in milliseconds among the specified
segments
+ * @throws RuntimeException If no creation time is found for the given
segments or if an I/O error occurs
+ */
+ @VisibleForTesting
+ protected Long getMaxCreationTimeMillis(String tableNameWithType,
List<String> segmentNames,
+ Map<String, List<String>> serverToSegments, BiMap<String, String>
serverToEndpoints,
+ ServerSegmentMetadataReader serverSegmentMetadataReader) {
+
+ // Filter serverToSegments to only include segments present in segmentNames
+ Set<String> segmentNameSet = new HashSet<>(segmentNames);
+ Map<String, List<String>> filteredServerToSegments = new HashMap<>();
+ for (Map.Entry<String, List<String>> entry : serverToSegments.entrySet()) {
+ List<String> filteredSegments =
+
entry.getValue().stream().filter(segmentNameSet::contains).collect(Collectors.toList());
+ if (!filteredSegments.isEmpty()) {
+ filteredServerToSegments.put(entry.getKey(), filteredSegments);
+ }
+ }
+ Map<String, List<Long>> creationTimeMap;
+ try {
+ creationTimeMap =
+
serverSegmentMetadataReader.getSegmentCreationMetadataFromServers(tableNameWithType,
filteredServerToSegments,
Review Comment:
The `/tables/{tableName}/validDocIdsMetadata` call returns the metadata from
a single server and returns the result.
In this case, I want to fetch creation time from each server that hosts this
server hence I am using `getSegmentMetadataFromServer` which calls each server
to get the creation time.
Also, we are only making this calls once the selected group for merge
compact is finalized. Which means we will make this call for very few segments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
