Artur Dudko created KAFKA-20735:
-----------------------------------
Summary: Streams logs ERROR "Failed to acquire lock while closing
the state store" for lock contention during back-to-back rebalances
Key: KAFKA-20735
URL: https://issues.apache.org/jira/browse/KAFKA-20735
Project: Kafka
Issue Type: Improvement
Components: streams
Affects Versions: 3.9.1
Reporter: Artur Dudko
_Summary_
With the state updater enabled, Kafka Streams logs an ERROR when it cannot
acquire a task's state-directory lock while closing the state manager:
{code:java}
stream-thread [-StreamThread-N] standby-task [1_42] Failed to acquire lock
while closing the state store for STANDBY task 1_42 {code}
This occurs during normal operation - typically when two rebalances happen in
quick succession and the lock is still held elsewhere as the task is moved off
the thread. It is harmless (no data loss, no stuck lag), but being at ERROR it
pollutes error-based alerting and cannot be filtered without disabling the
whole logger.
_Background / prior discussion_
This was discussed on PR [https://github.com/apache/kafka/pull/12875] (which
introduced both the INFO "reattempt" log on the init path and this ERROR log on
the close path, under KAFKA-12679). In that thread [~cadonna] confirmed both
messages are harmless and explained the timing (lock still held by another
stream thread on the same client / two fast consecutive rebalances). The init
path already has a backoff/retry mechanism (PRs #17116, #17209, present in
3.9.1), but the close-path ERROR remains. [~lucasbru] requested this JIRA be
filed.
_Proposed change_
Preferred (per the PR discussion): before attempting to acquire the lock to
close the state store, check whether the store / state manager is still open,
and skip the lock-and-close attempt (and the ERROR log) when there is nothing
to close. This avoids the misleading error without losing signal for genuinely
unexpected failures.
_How to reproduce_
Run a multi-instance Streams app with num.standby.replicas >= 1 and
num.stream.threads > 1, state updater enabled, and trigger frequent/successive
rebalances (rolling restarts, scaling, pod eviction). Observe the ERROR above
with no accompanying task failure or lag.
References
PR & discussion: [https://github.com/apache/kafka/pull/12875]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
