somandal commented on code in PR #15368:
URL: https://github.com/apache/pinot/pull/15368#discussion_r2013091776
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/RebalanceSummaryResult.java:
##########
@@ -240,6 +240,45 @@ public Map<String, ServerSegmentChangeInfo>
getServerSegmentChangeInfo() {
}
}
+ public static class ConsumingSegmentSummary {
+ private final int _numConsumingSegmentsToBeMoved;
+ @JsonInclude(JsonInclude.Include.NON_NULL)
+ private final Integer _maxBytesConsumingSegmentsToCatchUp;
Review Comment:
we can't actually calculate bytes AFAIK. Thus just looking at number of
offsets to catch up or age alone doesn't tell that much about how much
resources it will take to catch up when the consuming segment moves.
I would think someone looking at this output knows characteristics of their
Kafka topic (e.g. maybe avg message size) and can better decide whether a
forceCommit is needed or not.
When I think about it, I think of two aspects I might care about:
Age - helps with identifying how stale the data might be if the consuming
segment moves and starts re-consuming on a new server
Number offsets lag - helps with estimating the potential resource
utilization on the servers when the segment is moved (provided you can
translate that to resource utilization in some way)
Let me know if this doesn't make sense, and we can discuss more on this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
