jtuglu1 commented on code in PR #19532:
URL: https://github.com/apache/druid/pull/19532#discussion_r3597452016
##########
server/src/main/java/org/apache/druid/server/coordinator/loading/SegmentReplicationStatus.java:
##########
@@ -81,4 +96,84 @@ public Map<String, Object2LongMap<String>>
getTierToDatasourceToUnderReplicated(
return tierToUnderReplicated;
}
+
+ /**
+ * Computes unavailable, under-replicated and deep-storage-only segment
counts in a single
+ * pass over {@code usedSegments}, instead of three independent full
iterations. Produces
+ * results identical to calling {@link #getReplicaCountsInCluster} and
+ * {@link #getTierToDatasourceToUnderReplicated} independently for each
segment.
+ *
+ * @param ignoreMissingServers same semantics as in {@link
#getTierToDatasourceToUnderReplicated}.
+ */
+ public SegmentStatsSnapshot computeSegmentStats(Iterable<DataSegment>
usedSegments, boolean ignoreMissingServers)
+ {
+ final Object2IntOpenHashMap<String> datasourceToUnavailable = new
Object2IntOpenHashMap<>();
+ final Object2IntOpenHashMap<String> datasourceToDeepStorageOnly = new
Object2IntOpenHashMap<>();
+ final Map<String, Object2LongMap<String>> tierToUnderReplicated = new
HashMap<>();
+
+ for (DataSegment segment : usedSegments) {
+ final SegmentId segmentId = segment.getId();
+ final String datasource = segment.getDataSource();
+
+ final SegmentReplicaCount totalCount = totalReplicaCounts.get(segmentId);
+ if (totalCount != null && (totalCount.totalLoaded() > 0 ||
totalCount.required() == 0)) {
+ datasourceToUnavailable.addTo(datasource, 0);
+ } else {
+ datasourceToUnavailable.addTo(datasource, 1);
+ }
+ if (totalCount != null && totalCount.totalLoaded() == 0 &&
totalCount.required() == 0) {
+ datasourceToDeepStorageOnly.addTo(datasource, 1);
+ }
+
+ final Map<String, SegmentReplicaCount> tierToReplicaCount =
replicaCountsInTier.get(segmentId);
+ if (tierToReplicaCount != null) {
+ tierToReplicaCount.forEach((tier, counts) -> {
+ final int underReplicated = ignoreMissingServers ? counts.missing()
: counts.missingAndLoadable();
+ if (underReplicated >= 0) {
+ Object2LongOpenHashMap<String> datasourceToUnderReplicated =
(Object2LongOpenHashMap<String>)
+ tierToUnderReplicated.computeIfAbsent(tier, ds -> new
Object2LongOpenHashMap<>());
+ datasourceToUnderReplicated.addTo(datasource, underReplicated);
+ }
+ });
Review Comment:
The goal for this change was to avoid another full pass over usedSegments. I
suppose we can do something like:
```java
final Map<...> tierToUnderReplicated =
getTierToDatasourceToUnderReplicated(usedSegments, ignoreMissingServers);
```
as well to avoid any n^2 behavior during computeSegmentStats().
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
