sepuri sai krishna created KAFKA-20968:
------------------------------------------
Summary: Connect retry backoff overflows with infinite retries,
dropping the delay to zero and restarting the exponential ramp
Key: KAFKA-20968
URL: https://issues.apache.org/jira/browse/KAFKA-20968
Project: Kafka
Issue Type: Bug
Components: connect
Reporter: sepuri sai krishna
Assignee: sepuri sai krishna
RetryWithToleranceOperator.backoff()
(connect/runtime/src/main/java/org/apache/kafka/connect/runtime/errors/RetryWithToleranceOperator.java,
lines 299-315) computes the exponential backoff with an unchecked long shift:
void backoff(int attempt, long deadline) {
int numRetry = attempt - 1;
long delay = RETRIES_DELAY_MIN_MS << numRetry; // 300L <<
numRetry
if (delay > errorMaxDelayInMillis) {
delay = ThreadLocalRandom.current().nextLong(errorMaxDelayInMillis);
}
long currentTime = time.milliseconds();
if (delay + currentTime > deadline) {
delay = Math.max(0, deadline - currentTime);
}
log.debug("Sleeping for up to {} millis", delay);
try {
stopRequestedLatch.await(delay, TimeUnit.MILLISECONDS);
} catch (InterruptedException e) {
// ignore
}
}
"errors.retry.timeout = -1" is a documented configuration value --
ConnectorConfig.java:180 states "Use -1 for infinite retries." With it,
execAndRetry() sets deadline = Long.MAX_VALUE (line 203), so "attempt" grows
without bound for as long as the operation keeps throwing RetriableException.
That is precisely the scenario infinite retries exists for: a sink connector
whose downstream system is unavailable for an extended outage.
Because "numRetry" is unbounded, "300L << numRetry" eventually overflows. Two
distinct failures follow.
1. The delay becomes non-positive, and the jitter clamp is bypassed.
At numRetry = 55 (attempt 56), 300L << 55 overflows to -7638104968020361216. A
negative delay is not greater than errorMaxDelayInMillis, so the "delay >
errorMaxDelayInMillis" branch that applies jitter is skipped. With an infinite
deadline the second clamp is skipped as well, so the negative value reaches
CountDownLatch.await(), which returns immediately for a non-positive timeout.
The backoff disappears entirely and the connector retries in a tight loop
against the already-failing resource.
The same happens at numRetry = 58, 60, 61 (negative) and numRetry = 62, 63
(exactly 0).
2. Beyond numRetry = 63 the ramp silently restarts.
Java masks the shift distance for a long to its six lowest-order bits, so at
numRetry = 64 the shift is 0 and the delay returns to 300 ms, then 600 ms, 1200
ms, and so on. The backoff therefore cycles with period 64 instead of remaining
capped at errors.retry.delay.max.ms, repeatedly dropping back to sub-second
retries no matter how long the outage has lasted.
Both outcomes defeat the protection that errors.retry.delay.max.ms is
documented to provide -- ConnectorConfig.java:185-186 states "Jitter will be
added to the delay once this limit is reached to prevent thundering herd
issues."
The overflow is reached in ordinary operating time, not at some unreachable
extreme. With the default errors.retry.delay.max.ms of 60000, attempts 1
through 8 use the real exponential values (300 ms through 38400 ms) and
attempts 9 through 55 each wait a jittered interval uniformly distributed in
[0, 60000) ms, averaging 30 seconds. A connector retrying a downstream system
that is down therefore reaches the first zero-backoff attempt after roughly 25
minutes, and then re-enters the degraded region every 64 attempts thereafter.
RetryWithToleranceOperatorTest already contains a case asserting that the
deadline clamp does not produce a negative wait ("We may try to begin backing
off after the deadline has already passed; make sure that we don't wait with a
negative timeout"), but no case covers a negative wait arising from the
overflow itself.
Proposed fix: treat a shift that would overflow as an effectively infinite
delay so it falls into the existing jitter branch, keeping the delay within [0,
errors.retry.delay.max.ms] for every attempt number.
I'm happy to submit a PR for this fix.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
