[
https://issues.apache.org/jira/browse/KAFKA-20735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nilesh Kumar reassigned KAFKA-20735:
------------------------------------
Assignee: Nilesh Kumar
> Streams logs ERROR "Failed to acquire lock while closing the state store" for
> lock contention during back-to-back rebalances
> ----------------------------------------------------------------------------------------------------------------------------
>
> Key: KAFKA-20735
> URL: https://issues.apache.org/jira/browse/KAFKA-20735
> Project: Kafka
> Issue Type: Improvement
> Components: streams
> Affects Versions: 3.9.1
> Reporter: Artur Dudko
> Assignee: Nilesh Kumar
> Priority: Minor
> Labels: logging, states, streams
>
> _Summary_
> With the state updater enabled, Kafka Streams logs an ERROR when it cannot
> acquire a task's state-directory lock while closing the state manager:
> {code:java}
> stream-thread [-StreamThread-N] standby-task [1_42] Failed to acquire lock
> while closing the state store for STANDBY task 1_42 {code}
> This occurs during normal operation - typically when two rebalances happen in
> quick succession and the lock is still held elsewhere as the task is moved
> off the thread. It is harmless (no data loss, no stuck lag), but being at
> ERROR it pollutes error-based alerting and cannot be filtered without
> disabling the whole logger.
> _Background / prior discussion_
> This was discussed on PR [https://github.com/apache/kafka/pull/12875] (which
> introduced both the INFO "reattempt" log on the init path and this ERROR log
> on the close path, under KAFKA-12679). In that thread [~cadonna] confirmed
> both messages are harmless and explained the timing (lock still held by
> another stream thread on the same client / two fast consecutive rebalances).
> The init path already has a backoff/retry mechanism (PRs #17116, #17209,
> present in 3.9.1), but the close-path ERROR remains. [~lucasbru] requested
> this JIRA be filed.
> _Proposed change_
> Preferred (per the PR discussion): before attempting to acquire the lock to
> close the state store, check whether the store / state manager is still open,
> and skip the lock-and-close attempt (and the ERROR log) when there is nothing
> to close. This avoids the misleading error without losing signal for
> genuinely unexpected failures.
> _How to reproduce_
> Run a multi-instance Streams app with num.standby.replicas >= 1 and
> num.stream.threads > 1, state updater enabled, and trigger
> frequent/successive rebalances (rolling restarts, scaling, pod eviction).
> Observe the ERROR above with no accompanying task failure or lag.
> References
> PR & discussion: [https://github.com/apache/kafka/pull/12875]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
