Perf tuning is always tricky... 350 rec/sec sounds pretty low though.
You would first need to figure out where the bottleneck is. Kafka Streams exposes all kind of metrics: https://kafka.apache.org/documentation/#kafka_streams_monitoring
Might be good to inspect them as a first step -- maybe something is off and gives a first direction.
In general, it would be good to limit it to Kafka network I/O, local RocksDB disk I/O, or CPU utilization -- each one could be the bottleneck and we would need to first know which one before you can take any action to change configurations.
HTH. -Matthias On 4/4/24 7:21 PM, Nirmal Das wrote:
Hi All, My streams application is not processing more than 350 records/sec on a high load of 3milliom records produced every 2-3 minutes. My scenarios are as below - I am on Kafka and streams version of 3.5.1 . My key-value pair is in protobuf format . I do a groupbykey followed by TimeWindow of 10 mins with grace period of 6 hours . It is then followed by a aggregate function which stores the first and last offset of the record along with partition for that message key. Am I doing something wrong? Am I doing something anti-pattern which is throttling the system ? How can I improve this? Regards, Dev Lover
