[
https://issues.apache.org/jira/browse/KAFKA-19853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18035625#comment-18035625
]
Colt McNealy commented on KAFKA-19853:
--------------------------------------
KIP-1035 would _help_ the write stalls but not remove them. Note that the write
stalls are not "global" but rather per-rocksdb instance, because (in the case I
observed) they were specifically caused by the number of L0 files in the
RocksDB for the state store under restoration.
As an experiment, I disabled manual flushing (changing
`StateManagerUtil.OFFSET_DELTA_THRESHOLD_FOR_CHECKPOINT` to 1M instead of 10k).
The results:
* I still got write stalls on the restoring tasks + warmup replicas, but they
were less bad.
* Restoration throughput improved 3x (from 4MB/s to 12MB/s) with identical
conditions as measured by restore consumer throughput.
> StreamThread blocks on StateUpdater during onAssignment()
> ---------------------------------------------------------
>
> Key: KAFKA-19853
> URL: https://issues.apache.org/jira/browse/KAFKA-19853
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Affects Versions: 3.9.0
> Reporter: Colt McNealy
> Priority: Major
> Attachments: image (3).png, image (4).png, image (5).png
>
>
> We've observed that the `StreamThread` blocks waiting for a `Future` from the
> `StateUpdater` in the `StreamsPartitionAssigner#onAssignment()` method when
> we are moving a task out of the `StateUpdater` and onto the `StreamThread`.
>
> This can cause problems because, during restoration or with warmup replicas,
> the `StateUpdater#runOnce()` method can take a long time (upwards of 20
> seconds) when RocksDB stalls writes to allow compaction to keep up. In EOS
> this blockage may cause the transaction to time out, which is a big mess.
> This is because the `StreamThread` may have an open transaction before the
> `StreamsPartitionAssignor#onAssignment()` method is called.
>
> Some screenshots from the JFR below (credit to [~eduwerc]).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
