AW: Configuration of log compaction

Hi Liam, My brokers are running in version 2.0.0 Restarting was not a problem, it just seemed "a lot" for just restarting the log cleaner. I will think about piping the logs of the log cleaner to elk. And I will definitely keep an eye on the logs to see if the error occurs again. Thanks again!

Monitoring Mirror Maker Lag & source

Hi Team, We have few kafka clusters replicating & we would like to monitor the lag per topic on source, in mirror maker we have configured CONSUMER_GROUP_ID & replication is working fine. When i run "kafka-consumer-groups.sh --list --bootstrap-server bootstrap.kafka:9092" , i dnt find the consum

Re: Kafka rebalancing message lost

For very large number of consumers, you can manually manage the offsets and/or assign partitions yourself per consumer to avoid rebalancing. On Dec 18, 2018 9:58 AM, "Ryanne Dolan" wrote: > Parth, I am skeptical that you actually need 500+ consumers. A well tuned > consumer can process hundreds

Re: Configuration of log compaction

Kia ora Claudia, What version Kafka are you running? I can't find the specified code in Kafka 1.0.0. With regards to your questions: 1) I don't think so, but I'm just guessing here. 2) The log cleaner logs, if piped into something like Elasticsearch could be used to monitor it? 3) No, it's a thr

Re: Cannot read messages using a specific group

Karim, can you inspect the offsets using kafka-consumer-groups.sh and let us know what you see? Also, is it possible that "group_name" is being used by a consumer group that is still running? For example, say you have a topic with one partition and a consumer with "group_name" already. In that cas

Re: Kafka rebalancing message lost

Parth, I am skeptical that you actually need 500+ consumers. A well tuned consumer can process hundreds of thousands of records per second. Some notes to consider: - You'll need at least 500 partitions if you have 500 consumers. - You almost never need more consumers than you have brokers in your

Cannot read messages using a specific group

Hi, One of our brokers went down and when it came back up the consumer of one topic couldn’t read using the original consumer group. This doesn’t output anything: bin/kafka-console-consumer --bootstrap-server server --topic ASSETS --group group_name However, if I change the name of the group it

Re: Kafka rebalancing message lost

Rebalancing of partitions consumers does not necessarily mean loss of message. But I understand it can be annoying. If Kafka is rebalancing between consumers frequently, It means your consumer code is not polling within the expected timeout, as a result of which Kafka thinks the consumer is gone.

Cannot read messages using a specific group.

Hi, One of our brokers went down and when it came back up the consumer of one topic couldn’t read using the original consumer group. This doesn’t output anything: bin/kafka-console-consumer --bootstrap-server server --topic ASSETS --group group_name However, if I change the name of the group it

Kafka rebalancing message lost

Team, We want to build a scalable kafka system for pub sub message and want to run consumers (500+) on docker. We want the system to scale up the consumer based on the message inflow. However in kafka this triggers a rebalancing and we fear loss of message. What is the best practices/way to achieve

AW: Configuration of log compaction

Hi Liam, thanks for the pointer. I found out, that the log cleaner on all kafka brokers died with the following error message: [2018-12-04 15:33:24,886] INFO Cleaner 0: Caught segment overflow error during cleaning: Detected offset overflow at offset -1 in segment LogSegment(baseOffset=3605669