Re: Question how to drop a unused group id?

Yes it is clearly there is no process committing offset for those group id, but because of monitor(kafka manager and self coding monitor) always try to detect the latest offset, does this effect the remove action? Regards, Peyton 湖南福米信息科技有限责任公司 Hunan Fumi Financial Technology Co., Ltd.

Kafka Distributed Connect - connect-offsets compacted topic grows without bound?

We have experienced a production issue on Kafka 0.11.0.1 where we believe the contents of the connect-offsets topic (and hence the in-memory map in KafkaOffsetBackingStore) grow without bound for Source Connectors that make use of source partitions with a non-bounded key space. (1) Compacted T

Kafka Distributed Connect - connect-offsets compacted topic grows without bound?

We have experienced a production issue on Kafka 0.11.0.1 where we believe the contents of the connect-offsets topic (and hence the in-memory map in KafkaOffsetBackingStore) grow without bound for Source Connectors that make use of source partitions with a non-bounded key space. (1) Compacted T

Re: Question how to drop a unused group id?

It should be automatically removed if there is no offset commit in the group for 24 hours. Are you sure there is no process committing offset for the group? If kafka-manager lists them they're probably still there. You can also use the consumer group command to check this: > bin/kafka-consumer-gr

Re: Question how to drop a unused group id?

Thanks! Hmmm.. the last question, then by default, the unused group id was already removed by this setting? We use kafka-manager to manage the kafka cluster, and we can still see the expired group ids, as we talked previously, it dos not exists actually, right? Best Regards, Peyton 湖南福米信息科技有限责

Re: Question how to drop a unused group id?

If you did not set the config, the default will be used. For your version of Kafka (0.10.2.1) the default is 1 day: https://github.com/apache/kafka/blob/0.10.2.1/core/src/main/scala/kafka/server/KafkaConfig.scala#L152 --Vahid From: "ÅíÅô(Peyton)" To: Vahid S Hashemian Cc: users@kaf

Re: Question how to drop a unused group id?

Ok, thanks again for the help, and another question, by default we did not set the config item, so does not it take effect with the default situation? I just searched the code with the configuration key “offset.retention.minutes", but I can get any reference, be changed? Best Regards, Peyton 湖南

Re: Question how to drop a unused group id?

Hi Peyton, Yes, if you'd like to change the value a broker restart is required for the change to take effect. --Vahid From: "ÅíÅô(Peyton)" To: Vahid S Hashemian Cc: users@kafka.apache.org, "彭鹏" Date: 08/13/2018 08:49 AM Subject:Re: Question how to drop a unused group id

Re: Question how to drop a unused group id?

Hi Vahid, Thank you very much for the reply, we use the kafka ver 2.11-0.10.2.1. So when I apply the "offset.retention.minutes" to the system that request a restart to be effected? Best Regards, Peyton 湖南福米信息科技有限责任公司 Hunan Fumi Financial Technology Co., Ltd. ===

Re: Java API to read metrics via JMX

Hi, you can try with: https://github.com/Segence/kamon-jmx-collector that uses Kamon underlying and allow report metrics to several backends like : Prometheus, InfluxDB, Datadog. Cheers, Diego. On Sun, Aug 12, 2018 at 10:16 AM Matt Farmer wrote: > I

Re: Question how to drop a unused group id?

Hi Peyton, What version of Kafka are you using? Starting from version 1.1.0 there is a DELETE_GROUP API: https://cwiki.apache.org/confluence/display/KAFKA/KIP-229%3A+DeleteGroups+API You can also use the consumer group command with `--delete` to delete a group. For prior versions, the safest op

Re: I have issue in Kafka 2.0

Have you checked the javadoc of KafkaConsumer? On Mon, Aug 13, 2018, 11:10 PM Kailas Biradar wrote: > I have issue more time this ConcurrentModificationException because > KafkaConsumer is not safe for multi-threaded access > > -- > kailas >

Question how to drop a unused group id?

Hi kafka team, I have an issue about drop the unused group id which is created with the new consumer api, but I found no way to process it, can you please give mi a hint how to do it? Thank you very much. Regards, Peyton 湖南福米信息科技有限责任公司 Hunan Fumi Financial Technology Co., Ltd. ===

I have issue in Kafka 2.0

I have issue more time this ConcurrentModificationException because KafkaConsumer is not safe for multi-threaded access -- kailas

Re: Cli (or other tool?) with structured output like json for administration?

Hi again, I am using confluent OSS, however there is nothing magical there that helps me with this problem. Let me explain the problem. Setting up the cluster + zookeeper is not a problem, making changes to existing configurations regarding security and other configs (with cli) is a problem. In

Re: Cli (or other tool?) with structured output like json for administration?

If you are working on automating your kafka setup, wouldn't it be better for you to use confluent OSS? Or, are you aiming for something very much home-grown? On 13 August 2018 at 13:25, Johan Rask wrote: > Hi, > we are working on automating our kafka setup (topics, users, acls etc) but > we are

Cli (or other tool?) with structured output like json for administration?

Hi, we are working on automating our kafka setup (topics, users, acls etc) but we are missing structured output (think json) from the cli tools (i.e kafka-acls.sh, kafka-topics.sh). It only gives us human readable output which is hard to parse. We would like to be able to use it with common tools

Re: Usage of cleanup.policy=compact,delete

Issue opened: https://issues.apache.org/jira/browse/KAFKA-7281 The config command accepts the new format as described, but the error message still references the old format/restrictions. br, Patrik On 12 August 2018 at 23:47, Matthias J. Sax wrote: > I think, at command line tool level, you need