Hi Murilo,
Have you checked out the following blog post on tuning performance of
RocksDB state stores [1] especially the section on high disk I/O and
write stalls [2]?
Do you manage the off-heap memory used by RocksDB as described in the
Streams docs [3]?
I do not know what may have caused the increased I/O. We upgraded
RocksDB from 5.18.3 to 5.18.4. And we disabled bulk loading mode for
RocksDB. But I cannot say if one of these might be the reason.
Have you looked at the RocksDB metrics [4] to get better insight into
the situation?
Best,
Bruno
[1]
https://www.confluent.io/blog/how-to-tune-rocksdb-kafka-streams-state-stores-performance
[2]
https://www.confluent.io/blog/how-to-tune-rocksdb-kafka-streams-state-stores-performance/#write-stalls
[3]
https://kafka.apache.org/30/documentation/streams/developer-guide/memory-mgmt.html
[4] https://kafka.apache.org/documentation/#kafka_streams_rocksdb_monitoring
On 20.12.21 18:23, Murilo Tavares wrote:
Hi
I have a KafkaStreams application with a reasonably complex, stateful
topology.
By monitoring it, we can say for sure that it is bounded by writing I/O.
This has become way worse after we upgraded KafkaStreams from 2.4 to 2.8.
(even though we disabled warm-up replicas by setting
"acceptable.recovery.lag" to Long.MAX_VALUE)
So my question is: how can we decrease writing to disk frequency in a
KafkaStreams application? Maybe tweaking memtable in the underlying
Rocksdb? Any other suggestions?
Also, any ideas why upgrading KafkaStreams would affect its i/o
performance? Any changes that might have caused an instance to do more
writes?
Thanks
Murilo
