Re: How do I create a consumer group
Yes - see the offsets.topic.num.partitions and offsets.topic.replication.factor broker configs. Joel, that exactly what I was looking for. I'll look into that and the source for OffsetsMessageFormatter later today! thanks svante
Re: How do I create a consumer group
What's your use case for knowing all consumer groups? For admin? Thanks, Jun On Fri, Dec 12, 2014 at 2:45 AM, svante karlsson s...@csi.se wrote: Disregard the creation question - we must have done something wrong because now our code is working without obvious changes (on another set of brokers). However it turns out to be difficult to know the existing consumer group strings. Is the message format in __consumer_offsets public/stable in any way or is there a better way of listing the existing group names? svante 2014-12-11 20:59 GMT+01:00 svante karlsson s...@csi.se: We're using 0.82 beta and a homegrown c++ async library based on boost asio that has support for the offset api. (apikeys OffsetCommitRequest = 8, OffsetFetchRequest = 9, ConsumerMetadataRequest = 10) If we use a java client and commit an offset then the consumer group shows up in the response from ConsumerMetadataRequest. However I cant figure out how to create a new one using the Kafka API. Also, my __consumer_offsets topic shows up with a replication factor of 1. Is that changeable? thanks, svante
Re: How do I create a consumer group
Also, my __consumer_offsets topic shows up with a replication factor of 1. Is that changeable? Yes - see the offsets.topic.num.partitions and offsets.topic.replication.factor broker configs. -- Joel
Re: How do I create a consumer group
However it turns out to be difficult to know the existing consumer group strings. Is the message format in __consumer_offsets public/stable in any way or is there a better way of listing the existing group names? Yes it is stable but not described in the user documentation since it is an internal implementation detail. You can take a look at the OffsetManager code and specifically the OffsetsMessageFormatter class. https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/OffsetManager.scala Joel 2014-12-11 20:59 GMT+01:00 svante karlsson s...@csi.se: We're using 0.82 beta and a homegrown c++ async library based on boost asio that has support for the offset api. (apikeys OffsetCommitRequest = 8, OffsetFetchRequest = 9, ConsumerMetadataRequest = 10) If we use a java client and commit an offset then the consumer group shows up in the response from ConsumerMetadataRequest. However I cant figure out how to create a new one using the Kafka API. Also, my __consumer_offsets topic shows up with a replication factor of 1. Is that changeable? thanks, svante
Re: How do I create a consumer group
What's your use case for knowing all consumer groups? For admin? Yes, I need to monitor producer and consumer lags from several interconnected kafka sites. /svante 2014-12-16 0:34 GMT+01:00 Jun Rao j...@confluent.io: What's your use case for knowing all consumer groups? For admin? Thanks, Jun On Fri, Dec 12, 2014 at 2:45 AM, svante karlsson s...@csi.se wrote: Disregard the creation question - we must have done something wrong because now our code is working without obvious changes (on another set of brokers). However it turns out to be difficult to know the existing consumer group strings. Is the message format in __consumer_offsets public/stable in any way or is there a better way of listing the existing group names? svante 2014-12-11 20:59 GMT+01:00 svante karlsson s...@csi.se: We're using 0.82 beta and a homegrown c++ async library based on boost asio that has support for the offset api. (apikeys OffsetCommitRequest = 8, OffsetFetchRequest = 9, ConsumerMetadataRequest = 10) If we use a java client and commit an offset then the consumer group shows up in the response from ConsumerMetadataRequest. However I cant figure out how to create a new one using the Kafka API. Also, my __consumer_offsets topic shows up with a replication factor of 1. Is that changeable? thanks, svante
Re: How do I create a consumer group
Disregard the creation question - we must have done something wrong because now our code is working without obvious changes (on another set of brokers). However it turns out to be difficult to know the existing consumer group strings. Is the message format in __consumer_offsets public/stable in any way or is there a better way of listing the existing group names? svante 2014-12-11 20:59 GMT+01:00 svante karlsson s...@csi.se: We're using 0.82 beta and a homegrown c++ async library based on boost asio that has support for the offset api. (apikeys OffsetCommitRequest = 8, OffsetFetchRequest = 9, ConsumerMetadataRequest = 10) If we use a java client and commit an offset then the consumer group shows up in the response from ConsumerMetadataRequest. However I cant figure out how to create a new one using the Kafka API. Also, my __consumer_offsets topic shows up with a replication factor of 1. Is that changeable? thanks, svante
Re: How do I create a consumer group
KAFKA-1476 is addressing this need, but it¹s not checked in yet. Currently maybe you can use zookeeper client to check the zookeeper path. ‹Jiangjie (Becket) Qin On 12/12/14, 2:45 AM, svante karlsson s...@csi.se wrote: Disregard the creation question - we must have done something wrong because now our code is working without obvious changes (on another set of brokers). However it turns out to be difficult to know the existing consumer group strings. Is the message format in __consumer_offsets public/stable in any way or is there a better way of listing the existing group names? svante 2014-12-11 20:59 GMT+01:00 svante karlsson s...@csi.se: We're using 0.82 beta and a homegrown c++ async library based on boost asio that has support for the offset api. (apikeys OffsetCommitRequest = 8, OffsetFetchRequest = 9, ConsumerMetadataRequest = 10) If we use a java client and commit an offset then the consumer group shows up in the response from ConsumerMetadataRequest. However I cant figure out how to create a new one using the Kafka API. Also, my __consumer_offsets topic shows up with a replication factor of 1. Is that changeable? thanks, svante
Re: How do I create a consumer group
If I understand KAFKA-1476 it is only a command line tool that gives access by using ZKUtils not an API to Kafka. We're looking for a Kafka API so I guess that this functionality is missing. thanks for the pointer Svante Karlsson 2014-12-12 19:03 GMT+01:00 Jiangjie Qin j...@linkedin.com.invalid: KAFKA-1476 is addressing this need, but it¹s not checked in yet. Currently maybe you can use zookeeper client to check the zookeeper path. ‹Jiangjie (Becket) Qin On 12/12/14, 2:45 AM, svante karlsson s...@csi.se wrote: Disregard the creation question - we must have done something wrong because now our code is working without obvious changes (on another set of brokers). However it turns out to be difficult to know the existing consumer group strings. Is the message format in __consumer_offsets public/stable in any way or is there a better way of listing the existing group names? svante 2014-12-11 20:59 GMT+01:00 svante karlsson s...@csi.se: We're using 0.82 beta and a homegrown c++ async library based on boost asio that has support for the offset api. (apikeys OffsetCommitRequest = 8, OffsetFetchRequest = 9, ConsumerMetadataRequest = 10) If we use a java client and commit an offset then the consumer group shows up in the response from ConsumerMetadataRequest. However I cant figure out how to create a new one using the Kafka API. Also, my __consumer_offsets topic shows up with a replication factor of 1. Is that changeable? thanks, svante
