[
https://issues.apache.org/jira/browse/KAFKA-19853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18093073#comment-18093073
]
Colt McNealy commented on KAFKA-19853:
--------------------------------------
[~lucasbru] yes, we can continue to reproduce this occasionally with the
following sequence of events.
# Build a cluster with 200GB state per instance, 3 instances (600GB total), 1
standby replica, 4 CPU's, 3 stream threads
# Kill one instance completely.
# The cluster has no offline tasks since standby's recover, but both remaining
instances are now full-speed-ahead restoring the missing standby's.
# Trigger a rebalance among the last standing servers, normally by restarting
one of them.
We can reduce the likelihood of the stall issue observed above happening by:
# Increasing the available CPU's
# Increasing disk bandwidth
# Reducing the number of stream threads (so there are fewer state updaters
thrashing the rocksdb compaction threadpool and disk bandwidth)
I believe this bug is still valid; also, our RocksDB configurations aren't that
crazy from normal. We mostly follow the Confluent guide for limiting memory,
and set it to use about 60% of container memory. 25% is used by the JVM heap,
and the remaining 15% is buffer overhead.
> Transaction Failure when StreamThread blocks on StateUpdater during
> onAssignment()
> ----------------------------------------------------------------------------------
>
> Key: KAFKA-19853
> URL: https://issues.apache.org/jira/browse/KAFKA-19853
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Affects Versions: 3.9.0
> Reporter: Colt McNealy
> Assignee: Colt McNealy
> Priority: Major
> Attachments: image (3).png, image (4).png, image (5).png
>
>
> We've observed that the `StreamThread` blocks waiting for a `Future` from the
> `StateUpdater` in the `StreamsPartitionAssigner#onAssignment()` method when
> we are moving a task out of the `StateUpdater` and onto the `StreamThread`.
>
> This can cause problems because, during restoration or with warmup replicas,
> the `StateUpdater#runOnce()` method can take a long time (upwards of 20
> seconds) when RocksDB stalls writes to allow compaction to keep up. In EOS
> this blockage may cause the transaction to time out, which is a big mess.
> This is because the `StreamThread` may have an open transaction before the
> `StreamsPartitionAssignor#onAssignment()` method is called.
>
> Some screenshots from the JFR below (credit to [~eduwerc]).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
