alanlau28 commented on code in PR #23285:
URL: https://github.com/apache/kafka/pull/23285#discussion_r3881657067
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/StoreChangelogReader.java:
##########
@@ -228,6 +228,11 @@ long calculateRestoreTime(final long restoreEndTimeNs) {
// probe that is working is never suppressed.
private static final Duration PROBE_RETRY_BACKOFF = Duration.ofSeconds(60);
+ // Smallest checkpoint gap worth a probe: below it, replaying the gap
costs about what the probe
+ // spends, and the probe pauses every restoring partition while it runs.
Tied to the widest probe
+ // window so the relationship survives retuning.
+ private static final long PROBE_MIN_CHECKPOINT_GAP = 8 *
PROBE_WINDOWS[PROBE_WINDOWS.length - 1];
Review Comment:
Load testing gave the number 8 * 2048, so a probe only fires when there is a
big gap in offsets from the stored offset X committed durably in the store + 1.
A higher gap would mean fewer probes/less overhead, but we possibly go back to
replaying expired data we could've skipped. Lower catches more possible skips,
but probes more eagerly causing more overhead and may not be cheaper than just
replaying from X+1.
Overall, an arbitrary number chosen.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
