Re: Kafka 20k topics metadata update taking long time

This is interesting. But 20K topics is also not a realistic assumption to have. I don't see an alarming difference in the latency results from the two scenarios. Also even when cached, the metadata needs to be refreshed anyway. On Thu, Jul 4, 2024 at 4:37 AM Amit Chopra wrote: > Hey, > > I

Re: Kafka Broker - High CPU

You can do a top -H to see which thread pid(s) is/are contributing to this. And then map to hex of that in thread dump to identify the culprits. Might lead to some additional clues. Regards Sab On Sat, 28 Sep 2019, 1:38 am Antony A, wrote: > Partition Leaders are pretty evenly matched between

Re: Kafka Consumer Hung on Certain Partitions - Single Kafka Consumer

Can you try executing kafka-consumer-groups to see what it says? Are your log offsets increasing and the lag increasing when processing for some of your partitions are stuck? May be the problem is in the producer side. Regards Sab On Mon, 9 Jul 2018, 11:43 am Steve Tian, wrote: > Are you

Re: Kafka Streams application Unable to Horizontally scale and the application on other instances refusing to start.

Is there anything abnormal in network performance for your Kafka brokers ec2 instances and Kafka stream processor ec2 instances? Is the network link getting saturated? If so you might want to upgrade Kafka brokers to an instance type with more network bandwidth. I would also go check the memory

Re: Improving Kafka State Store performance

T2 instances are credit based instances that do not provide constant throughout in cpu and io. You would want to reconsider using t2. That said we too face issues with scaling rocksdb. Although I don't remember the performance benchmark numbers but I do remember it was at least 3x slower when

Re: On rebalance consumer group start from old offset

This is similar to a problem I am also grappling with. We store the processed offset for each partition in state store. And after restarts we see that sometimes the start offset that Kafka Streams uses is a few thousands to a couple million behind per partition. To compound it, this is not

Re: Out of order message processing with Kafka Streams

Hans What you state would work for aggregations, but not for state machines and CEP. Regards Sab On 19 Mar 2017 12:01 a.m., "Hans Jespersen" wrote: > The only way to make sure A is consumed first would be to delay the > consumption of message B for at least 15 minutes which

Re: Potential memory leak in rocksdb

Pierre Do you see keys persisting in Rocksdb even after deleting them? Imagine k1 got deleted in first execution of punctuate and then you see it in the second execution of punctuate as well. Do you see such behaviour? That could explain why Rocksdb size keeps increasing. Regards Sab On 21 Feb

Re: RocksDB error when running Kafka with a larger data set

Sometimes this happens because the lock file is not cleared. Did you look for the .lock file? Regards Sab On 15 Feb 2017 8:59 p.m., "Frank Lyaruu" wrote: > No, it sure isn't... The log file doesn't seem to have been touched for > hours. The whole log file seems to be of the