Re: kafka-consumer-groups.sh doesn't work when consumers are off
Thanks for the added info. For the mean time we'll rely on the older ConsumerOffsetChecker for our monitoring tools. Thanks, Cliff On Fri, Jan 29, 2016 at 10:56 AM, Guozhang Wang wrote: > Tao, > > You are right, ConsumerOffsetChecker can still get offsets from the offset > manager in Kafka. > > Guozhang > > On Thu, Jan 28, 2016 at 9:36 PM, tao xiao wrote: > > > it first issues an offsetrequest to broker and check if offset is stored > in > > broker if not it will queries zk > > > > > > > https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/tools/ConsumerOffsetChecker.scala#L171 > > > > On Fri, 29 Jan 2016 at 13:11 Guozhang Wang wrote: > > > > > Tao, > > > > > > Hmm that is a bit wired since ConsumerOffsetChecker itself does not > talk > > to > > > brokers at all, but only through ZK. > > > > > > Guozhang > > > > > > On Thu, Jan 28, 2016 at 6:07 PM, tao xiao > wrote: > > > > > > > Guozhang, > > > > > > > > The old ConsumerOffsetChecker works for new consumer too with offset > > > stored > > > > in Kafka. I tested it with mirror maker with new consumer enabled. it > > is > > > > able to show offset during mirror maker running and after its > shutdown. > > > > > > > > On Fri, 29 Jan 2016 at 06:34 Guozhang Wang > wrote: > > > > > > > > > Once the offset is written to the log it is persistent and hence > > should > > > > > survive broker failures. And its retention policy is configurable. > > > > > > > > > > It may be a bit misleading in saying "in-memory cache" in my > previous > > > > > email: the brokers just keep the in-memory map of [group, > partition] > > -> > > > > > latest_offset, while the offset commit history is kept in the log. > > When > > > > we > > > > > delete the group, we remove the corresponding entry from memory map > > and > > > > put > > > > > a tombstone into log as well so that the old offsets will be > > compacted > > > > > eventually according to compaction policy. > > > > > > > > > > The old ConsumerOffsetChecker only works for old consumer that > stores > > > > > offset in ZK. > > > > > > > > > > Guozhang > > > > > > > > > > On Thu, Jan 28, 2016 at 1:43 PM, Cliff Rhyne > > wrote: > > > > > > > > > > > Hi Guozhang, > > > > > > > > > > > > That looks like it might help but feels like there might be some > > > gaps. > > > > > > Would it be able to survive restarts of the kafka broker? How > long > > > > would > > > > > > it stay in the cache (and is that configurable)? If it expires > > from > > > > the > > > > > > cache, what's the cache-miss operation look like? (yes, a lot of > > > this > > > > > > depends on the data still being in the logs to recover) > > > > > > > > > > > > In the mean time, can I rely on the deprecated > > ConsumerOffsetChecker > > > > > (which > > > > > > looks at zookeeper) even though I'm using the new KafkaConsumer? > > > > > > > > > > > > Thanks, > > > > > > Cliff > > > > > > > > > > > > On Thu, Jan 28, 2016 at 3:30 PM, Guozhang Wang < > wangg...@gmail.com > > > > > > > > wrote: > > > > > > > > > > > > > Hi Cliff, > > > > > > > > > > > > > > Short answer to your question is it is just the current > > > > implementation. > > > > > > > > > > > > > > The coordinator stores the offsets as messages in an internal > > topic > > > > and > > > > > > > also keeps the latest offset values in in-memory. It answers > > > > > > > ConsumerGroupRequest using its cached offset, and upon the > > consumer > > > > > group > > > > > > > being removed since no member is alive already, it removed it > > from > > > > its > > > > > > > in-memory cache and add a "tombstone" to the offset log as > well. > > > But > > > > > the > > > > > > > offsets are still persistent as messages in the log, which will > > > only > > > > be > > > > > > > compacted after a while (this is depend on the log compaction > > > > policy). > > > > > > > > > > > > > > There is a ticket open for improving on this scenario ( > > > > > > > https://issues.apache.org/jira/browse/KAFKA-2720) which lets > the > > > > > > > coordinator to only "purge" dead groups periodically instead of > > > > > > > immediately, and that may partially resolve your case. > > > > > > > > > > > > > > Guozhang > > > > > > > > > > > > > > > > > > > > > On Thu, Jan 28, 2016 at 12:13 PM, Cliff Rhyne < > crh...@signal.co> > > > > > wrote: > > > > > > > > > > > > > > > Just following up on this concern. Is there a constraint > that > > > > > prevents > > > > > > > > ConsumerGroupCommand from reporting offsets on a group if no > > > > members > > > > > > are > > > > > > > > connected, or is this just the current implementation? > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Cliff > > > > > > > > > > > > > > > > On Mon, Jan 25, 2016 at 3:50 PM, Cliff Rhyne < > crh...@signal.co > > > > > > > > wrote: > > > > > > > > > > > > > > > > > I'm running into a few challenges trying to evaluate > offsets > > > and > > > > > lag > > > > > > > > > (pending message count) in the new Java KafkaConsumer. The > > old
Re: kafka-consumer-groups.sh doesn't work when consumers are off
Tao, You are right, ConsumerOffsetChecker can still get offsets from the offset manager in Kafka. Guozhang On Thu, Jan 28, 2016 at 9:36 PM, tao xiao wrote: > it first issues an offsetrequest to broker and check if offset is stored in > broker if not it will queries zk > > > https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/tools/ConsumerOffsetChecker.scala#L171 > > On Fri, 29 Jan 2016 at 13:11 Guozhang Wang wrote: > > > Tao, > > > > Hmm that is a bit wired since ConsumerOffsetChecker itself does not talk > to > > brokers at all, but only through ZK. > > > > Guozhang > > > > On Thu, Jan 28, 2016 at 6:07 PM, tao xiao wrote: > > > > > Guozhang, > > > > > > The old ConsumerOffsetChecker works for new consumer too with offset > > stored > > > in Kafka. I tested it with mirror maker with new consumer enabled. it > is > > > able to show offset during mirror maker running and after its shutdown. > > > > > > On Fri, 29 Jan 2016 at 06:34 Guozhang Wang wrote: > > > > > > > Once the offset is written to the log it is persistent and hence > should > > > > survive broker failures. And its retention policy is configurable. > > > > > > > > It may be a bit misleading in saying "in-memory cache" in my previous > > > > email: the brokers just keep the in-memory map of [group, partition] > -> > > > > latest_offset, while the offset commit history is kept in the log. > When > > > we > > > > delete the group, we remove the corresponding entry from memory map > and > > > put > > > > a tombstone into log as well so that the old offsets will be > compacted > > > > eventually according to compaction policy. > > > > > > > > The old ConsumerOffsetChecker only works for old consumer that stores > > > > offset in ZK. > > > > > > > > Guozhang > > > > > > > > On Thu, Jan 28, 2016 at 1:43 PM, Cliff Rhyne > wrote: > > > > > > > > > Hi Guozhang, > > > > > > > > > > That looks like it might help but feels like there might be some > > gaps. > > > > > Would it be able to survive restarts of the kafka broker? How long > > > would > > > > > it stay in the cache (and is that configurable)? If it expires > from > > > the > > > > > cache, what's the cache-miss operation look like? (yes, a lot of > > this > > > > > depends on the data still being in the logs to recover) > > > > > > > > > > In the mean time, can I rely on the deprecated > ConsumerOffsetChecker > > > > (which > > > > > looks at zookeeper) even though I'm using the new KafkaConsumer? > > > > > > > > > > Thanks, > > > > > Cliff > > > > > > > > > > On Thu, Jan 28, 2016 at 3:30 PM, Guozhang Wang > > > > > wrote: > > > > > > > > > > > Hi Cliff, > > > > > > > > > > > > Short answer to your question is it is just the current > > > implementation. > > > > > > > > > > > > The coordinator stores the offsets as messages in an internal > topic > > > and > > > > > > also keeps the latest offset values in in-memory. It answers > > > > > > ConsumerGroupRequest using its cached offset, and upon the > consumer > > > > group > > > > > > being removed since no member is alive already, it removed it > from > > > its > > > > > > in-memory cache and add a "tombstone" to the offset log as well. > > But > > > > the > > > > > > offsets are still persistent as messages in the log, which will > > only > > > be > > > > > > compacted after a while (this is depend on the log compaction > > > policy). > > > > > > > > > > > > There is a ticket open for improving on this scenario ( > > > > > > https://issues.apache.org/jira/browse/KAFKA-2720) which lets the > > > > > > coordinator to only "purge" dead groups periodically instead of > > > > > > immediately, and that may partially resolve your case. > > > > > > > > > > > > Guozhang > > > > > > > > > > > > > > > > > > On Thu, Jan 28, 2016 at 12:13 PM, Cliff Rhyne > > > > wrote: > > > > > > > > > > > > > Just following up on this concern. Is there a constraint that > > > > prevents > > > > > > > ConsumerGroupCommand from reporting offsets on a group if no > > > members > > > > > are > > > > > > > connected, or is this just the current implementation? > > > > > > > > > > > > > > Thanks, > > > > > > > Cliff > > > > > > > > > > > > > > On Mon, Jan 25, 2016 at 3:50 PM, Cliff Rhyne > > > > > wrote: > > > > > > > > > > > > > > > I'm running into a few challenges trying to evaluate offsets > > and > > > > lag > > > > > > > > (pending message count) in the new Java KafkaConsumer. The > old > > > > > > > > ConsumerOffsetChecker doesn't work anymore since the offsets > > > aren't > > > > > > > stored > > > > > > > > in zookeeper after switching from the old consumer. This > would > > > be > > > > > > fine, > > > > > > > > but the kafka-consumer-groups.sh command doesn't work if the > > > > > consumers > > > > > > > are > > > > > > > > shut off. This seems like an unnecessary limitation and is > > > > > problematic > > > > > > > for > > > > > > > > troubleshooting / monitoring when the application is turned > off > > > (or > > > > > > while > >
Re: kafka-consumer-groups.sh doesn't work when consumers are off
it first issues an offsetrequest to broker and check if offset is stored in broker if not it will queries zk https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/tools/ConsumerOffsetChecker.scala#L171 On Fri, 29 Jan 2016 at 13:11 Guozhang Wang wrote: > Tao, > > Hmm that is a bit wired since ConsumerOffsetChecker itself does not talk to > brokers at all, but only through ZK. > > Guozhang > > On Thu, Jan 28, 2016 at 6:07 PM, tao xiao wrote: > > > Guozhang, > > > > The old ConsumerOffsetChecker works for new consumer too with offset > stored > > in Kafka. I tested it with mirror maker with new consumer enabled. it is > > able to show offset during mirror maker running and after its shutdown. > > > > On Fri, 29 Jan 2016 at 06:34 Guozhang Wang wrote: > > > > > Once the offset is written to the log it is persistent and hence should > > > survive broker failures. And its retention policy is configurable. > > > > > > It may be a bit misleading in saying "in-memory cache" in my previous > > > email: the brokers just keep the in-memory map of [group, partition] -> > > > latest_offset, while the offset commit history is kept in the log. When > > we > > > delete the group, we remove the corresponding entry from memory map and > > put > > > a tombstone into log as well so that the old offsets will be compacted > > > eventually according to compaction policy. > > > > > > The old ConsumerOffsetChecker only works for old consumer that stores > > > offset in ZK. > > > > > > Guozhang > > > > > > On Thu, Jan 28, 2016 at 1:43 PM, Cliff Rhyne wrote: > > > > > > > Hi Guozhang, > > > > > > > > That looks like it might help but feels like there might be some > gaps. > > > > Would it be able to survive restarts of the kafka broker? How long > > would > > > > it stay in the cache (and is that configurable)? If it expires from > > the > > > > cache, what's the cache-miss operation look like? (yes, a lot of > this > > > > depends on the data still being in the logs to recover) > > > > > > > > In the mean time, can I rely on the deprecated ConsumerOffsetChecker > > > (which > > > > looks at zookeeper) even though I'm using the new KafkaConsumer? > > > > > > > > Thanks, > > > > Cliff > > > > > > > > On Thu, Jan 28, 2016 at 3:30 PM, Guozhang Wang > > > wrote: > > > > > > > > > Hi Cliff, > > > > > > > > > > Short answer to your question is it is just the current > > implementation. > > > > > > > > > > The coordinator stores the offsets as messages in an internal topic > > and > > > > > also keeps the latest offset values in in-memory. It answers > > > > > ConsumerGroupRequest using its cached offset, and upon the consumer > > > group > > > > > being removed since no member is alive already, it removed it from > > its > > > > > in-memory cache and add a "tombstone" to the offset log as well. > But > > > the > > > > > offsets are still persistent as messages in the log, which will > only > > be > > > > > compacted after a while (this is depend on the log compaction > > policy). > > > > > > > > > > There is a ticket open for improving on this scenario ( > > > > > https://issues.apache.org/jira/browse/KAFKA-2720) which lets the > > > > > coordinator to only "purge" dead groups periodically instead of > > > > > immediately, and that may partially resolve your case. > > > > > > > > > > Guozhang > > > > > > > > > > > > > > > On Thu, Jan 28, 2016 at 12:13 PM, Cliff Rhyne > > > wrote: > > > > > > > > > > > Just following up on this concern. Is there a constraint that > > > prevents > > > > > > ConsumerGroupCommand from reporting offsets on a group if no > > members > > > > are > > > > > > connected, or is this just the current implementation? > > > > > > > > > > > > Thanks, > > > > > > Cliff > > > > > > > > > > > > On Mon, Jan 25, 2016 at 3:50 PM, Cliff Rhyne > > > wrote: > > > > > > > > > > > > > I'm running into a few challenges trying to evaluate offsets > and > > > lag > > > > > > > (pending message count) in the new Java KafkaConsumer. The old > > > > > > > ConsumerOffsetChecker doesn't work anymore since the offsets > > aren't > > > > > > stored > > > > > > > in zookeeper after switching from the old consumer. This would > > be > > > > > fine, > > > > > > > but the kafka-consumer-groups.sh command doesn't work if the > > > > consumers > > > > > > are > > > > > > > shut off. This seems like an unnecessary limitation and is > > > > problematic > > > > > > for > > > > > > > troubleshooting / monitoring when the application is turned off > > (or > > > > > while > > > > > > > my application is running due to our stopping/starting > > consumers). > > > > > > > > > > > > > > Is there a constraint that I'm not aware of or is this > something > > > that > > > > > > > could be changed? > > > > > > > > > > > > > > Thanks, > > > > > > > Cliff > > > > > > > > > > > > > > -- > > > > > > > Cliff Rhyne > > > > > > > Software Engineering Lead > > > > > > > e: crh...@signal.co > > > > > > > signal.co > > > > > > > ___
Re: kafka-consumer-groups.sh doesn't work when consumers are off
Tao, Hmm that is a bit wired since ConsumerOffsetChecker itself does not talk to brokers at all, but only through ZK. Guozhang On Thu, Jan 28, 2016 at 6:07 PM, tao xiao wrote: > Guozhang, > > The old ConsumerOffsetChecker works for new consumer too with offset stored > in Kafka. I tested it with mirror maker with new consumer enabled. it is > able to show offset during mirror maker running and after its shutdown. > > On Fri, 29 Jan 2016 at 06:34 Guozhang Wang wrote: > > > Once the offset is written to the log it is persistent and hence should > > survive broker failures. And its retention policy is configurable. > > > > It may be a bit misleading in saying "in-memory cache" in my previous > > email: the brokers just keep the in-memory map of [group, partition] -> > > latest_offset, while the offset commit history is kept in the log. When > we > > delete the group, we remove the corresponding entry from memory map and > put > > a tombstone into log as well so that the old offsets will be compacted > > eventually according to compaction policy. > > > > The old ConsumerOffsetChecker only works for old consumer that stores > > offset in ZK. > > > > Guozhang > > > > On Thu, Jan 28, 2016 at 1:43 PM, Cliff Rhyne wrote: > > > > > Hi Guozhang, > > > > > > That looks like it might help but feels like there might be some gaps. > > > Would it be able to survive restarts of the kafka broker? How long > would > > > it stay in the cache (and is that configurable)? If it expires from > the > > > cache, what's the cache-miss operation look like? (yes, a lot of this > > > depends on the data still being in the logs to recover) > > > > > > In the mean time, can I rely on the deprecated ConsumerOffsetChecker > > (which > > > looks at zookeeper) even though I'm using the new KafkaConsumer? > > > > > > Thanks, > > > Cliff > > > > > > On Thu, Jan 28, 2016 at 3:30 PM, Guozhang Wang > > wrote: > > > > > > > Hi Cliff, > > > > > > > > Short answer to your question is it is just the current > implementation. > > > > > > > > The coordinator stores the offsets as messages in an internal topic > and > > > > also keeps the latest offset values in in-memory. It answers > > > > ConsumerGroupRequest using its cached offset, and upon the consumer > > group > > > > being removed since no member is alive already, it removed it from > its > > > > in-memory cache and add a "tombstone" to the offset log as well. But > > the > > > > offsets are still persistent as messages in the log, which will only > be > > > > compacted after a while (this is depend on the log compaction > policy). > > > > > > > > There is a ticket open for improving on this scenario ( > > > > https://issues.apache.org/jira/browse/KAFKA-2720) which lets the > > > > coordinator to only "purge" dead groups periodically instead of > > > > immediately, and that may partially resolve your case. > > > > > > > > Guozhang > > > > > > > > > > > > On Thu, Jan 28, 2016 at 12:13 PM, Cliff Rhyne > > wrote: > > > > > > > > > Just following up on this concern. Is there a constraint that > > prevents > > > > > ConsumerGroupCommand from reporting offsets on a group if no > members > > > are > > > > > connected, or is this just the current implementation? > > > > > > > > > > Thanks, > > > > > Cliff > > > > > > > > > > On Mon, Jan 25, 2016 at 3:50 PM, Cliff Rhyne > > wrote: > > > > > > > > > > > I'm running into a few challenges trying to evaluate offsets and > > lag > > > > > > (pending message count) in the new Java KafkaConsumer. The old > > > > > > ConsumerOffsetChecker doesn't work anymore since the offsets > aren't > > > > > stored > > > > > > in zookeeper after switching from the old consumer. This would > be > > > > fine, > > > > > > but the kafka-consumer-groups.sh command doesn't work if the > > > consumers > > > > > are > > > > > > shut off. This seems like an unnecessary limitation and is > > > problematic > > > > > for > > > > > > troubleshooting / monitoring when the application is turned off > (or > > > > while > > > > > > my application is running due to our stopping/starting > consumers). > > > > > > > > > > > > Is there a constraint that I'm not aware of or is this something > > that > > > > > > could be changed? > > > > > > > > > > > > Thanks, > > > > > > Cliff > > > > > > > > > > > > -- > > > > > > Cliff Rhyne > > > > > > Software Engineering Lead > > > > > > e: crh...@signal.co > > > > > > signal.co > > > > > > > > > > > > > > > > > > Cut Through the Noise > > > > > > > > > > > > This e-mail and any files transmitted with it are for the sole > use > > of > > > > the > > > > > > intended recipient(s) and may contain confidential and privileged > > > > > > information. Any unauthorized use of this email is strictly > > > prohibited. > > > > > > ©2015 Signal. All rights reserved. > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > Cliff Rhyne > > > > > Software Engineering Lead > > > > > e: crh...@signal.co > > > > > signal.
Re: kafka-consumer-groups.sh doesn't work when consumers are off
Guozhang, The old ConsumerOffsetChecker works for new consumer too with offset stored in Kafka. I tested it with mirror maker with new consumer enabled. it is able to show offset during mirror maker running and after its shutdown. On Fri, 29 Jan 2016 at 06:34 Guozhang Wang wrote: > Once the offset is written to the log it is persistent and hence should > survive broker failures. And its retention policy is configurable. > > It may be a bit misleading in saying "in-memory cache" in my previous > email: the brokers just keep the in-memory map of [group, partition] -> > latest_offset, while the offset commit history is kept in the log. When we > delete the group, we remove the corresponding entry from memory map and put > a tombstone into log as well so that the old offsets will be compacted > eventually according to compaction policy. > > The old ConsumerOffsetChecker only works for old consumer that stores > offset in ZK. > > Guozhang > > On Thu, Jan 28, 2016 at 1:43 PM, Cliff Rhyne wrote: > > > Hi Guozhang, > > > > That looks like it might help but feels like there might be some gaps. > > Would it be able to survive restarts of the kafka broker? How long would > > it stay in the cache (and is that configurable)? If it expires from the > > cache, what's the cache-miss operation look like? (yes, a lot of this > > depends on the data still being in the logs to recover) > > > > In the mean time, can I rely on the deprecated ConsumerOffsetChecker > (which > > looks at zookeeper) even though I'm using the new KafkaConsumer? > > > > Thanks, > > Cliff > > > > On Thu, Jan 28, 2016 at 3:30 PM, Guozhang Wang > wrote: > > > > > Hi Cliff, > > > > > > Short answer to your question is it is just the current implementation. > > > > > > The coordinator stores the offsets as messages in an internal topic and > > > also keeps the latest offset values in in-memory. It answers > > > ConsumerGroupRequest using its cached offset, and upon the consumer > group > > > being removed since no member is alive already, it removed it from its > > > in-memory cache and add a "tombstone" to the offset log as well. But > the > > > offsets are still persistent as messages in the log, which will only be > > > compacted after a while (this is depend on the log compaction policy). > > > > > > There is a ticket open for improving on this scenario ( > > > https://issues.apache.org/jira/browse/KAFKA-2720) which lets the > > > coordinator to only "purge" dead groups periodically instead of > > > immediately, and that may partially resolve your case. > > > > > > Guozhang > > > > > > > > > On Thu, Jan 28, 2016 at 12:13 PM, Cliff Rhyne > wrote: > > > > > > > Just following up on this concern. Is there a constraint that > prevents > > > > ConsumerGroupCommand from reporting offsets on a group if no members > > are > > > > connected, or is this just the current implementation? > > > > > > > > Thanks, > > > > Cliff > > > > > > > > On Mon, Jan 25, 2016 at 3:50 PM, Cliff Rhyne > wrote: > > > > > > > > > I'm running into a few challenges trying to evaluate offsets and > lag > > > > > (pending message count) in the new Java KafkaConsumer. The old > > > > > ConsumerOffsetChecker doesn't work anymore since the offsets aren't > > > > stored > > > > > in zookeeper after switching from the old consumer. This would be > > > fine, > > > > > but the kafka-consumer-groups.sh command doesn't work if the > > consumers > > > > are > > > > > shut off. This seems like an unnecessary limitation and is > > problematic > > > > for > > > > > troubleshooting / monitoring when the application is turned off (or > > > while > > > > > my application is running due to our stopping/starting consumers). > > > > > > > > > > Is there a constraint that I'm not aware of or is this something > that > > > > > could be changed? > > > > > > > > > > Thanks, > > > > > Cliff > > > > > > > > > > -- > > > > > Cliff Rhyne > > > > > Software Engineering Lead > > > > > e: crh...@signal.co > > > > > signal.co > > > > > > > > > > > > > > > Cut Through the Noise > > > > > > > > > > This e-mail and any files transmitted with it are for the sole use > of > > > the > > > > > intended recipient(s) and may contain confidential and privileged > > > > > information. Any unauthorized use of this email is strictly > > prohibited. > > > > > ©2015 Signal. All rights reserved. > > > > > > > > > > > > > > > > > > > > > -- > > > > Cliff Rhyne > > > > Software Engineering Lead > > > > e: crh...@signal.co > > > > signal.co > > > > > > > > > > > > Cut Through the Noise > > > > > > > > This e-mail and any files transmitted with it are for the sole use of > > the > > > > intended recipient(s) and may contain confidential and privileged > > > > information. Any unauthorized use of this email is strictly > prohibited. > > > > ©2015 Signal. All rights reserved. > > > > > > > > > > > > > > > > -- > > > -- Guozhang > > > > > > > > > > > -- > > Cliff Rhyne > > S
Re: kafka-consumer-groups.sh doesn't work when consumers are off
Once the offset is written to the log it is persistent and hence should survive broker failures. And its retention policy is configurable. It may be a bit misleading in saying "in-memory cache" in my previous email: the brokers just keep the in-memory map of [group, partition] -> latest_offset, while the offset commit history is kept in the log. When we delete the group, we remove the corresponding entry from memory map and put a tombstone into log as well so that the old offsets will be compacted eventually according to compaction policy. The old ConsumerOffsetChecker only works for old consumer that stores offset in ZK. Guozhang On Thu, Jan 28, 2016 at 1:43 PM, Cliff Rhyne wrote: > Hi Guozhang, > > That looks like it might help but feels like there might be some gaps. > Would it be able to survive restarts of the kafka broker? How long would > it stay in the cache (and is that configurable)? If it expires from the > cache, what's the cache-miss operation look like? (yes, a lot of this > depends on the data still being in the logs to recover) > > In the mean time, can I rely on the deprecated ConsumerOffsetChecker (which > looks at zookeeper) even though I'm using the new KafkaConsumer? > > Thanks, > Cliff > > On Thu, Jan 28, 2016 at 3:30 PM, Guozhang Wang wrote: > > > Hi Cliff, > > > > Short answer to your question is it is just the current implementation. > > > > The coordinator stores the offsets as messages in an internal topic and > > also keeps the latest offset values in in-memory. It answers > > ConsumerGroupRequest using its cached offset, and upon the consumer group > > being removed since no member is alive already, it removed it from its > > in-memory cache and add a "tombstone" to the offset log as well. But the > > offsets are still persistent as messages in the log, which will only be > > compacted after a while (this is depend on the log compaction policy). > > > > There is a ticket open for improving on this scenario ( > > https://issues.apache.org/jira/browse/KAFKA-2720) which lets the > > coordinator to only "purge" dead groups periodically instead of > > immediately, and that may partially resolve your case. > > > > Guozhang > > > > > > On Thu, Jan 28, 2016 at 12:13 PM, Cliff Rhyne wrote: > > > > > Just following up on this concern. Is there a constraint that prevents > > > ConsumerGroupCommand from reporting offsets on a group if no members > are > > > connected, or is this just the current implementation? > > > > > > Thanks, > > > Cliff > > > > > > On Mon, Jan 25, 2016 at 3:50 PM, Cliff Rhyne wrote: > > > > > > > I'm running into a few challenges trying to evaluate offsets and lag > > > > (pending message count) in the new Java KafkaConsumer. The old > > > > ConsumerOffsetChecker doesn't work anymore since the offsets aren't > > > stored > > > > in zookeeper after switching from the old consumer. This would be > > fine, > > > > but the kafka-consumer-groups.sh command doesn't work if the > consumers > > > are > > > > shut off. This seems like an unnecessary limitation and is > problematic > > > for > > > > troubleshooting / monitoring when the application is turned off (or > > while > > > > my application is running due to our stopping/starting consumers). > > > > > > > > Is there a constraint that I'm not aware of or is this something that > > > > could be changed? > > > > > > > > Thanks, > > > > Cliff > > > > > > > > -- > > > > Cliff Rhyne > > > > Software Engineering Lead > > > > e: crh...@signal.co > > > > signal.co > > > > > > > > > > > > Cut Through the Noise > > > > > > > > This e-mail and any files transmitted with it are for the sole use of > > the > > > > intended recipient(s) and may contain confidential and privileged > > > > information. Any unauthorized use of this email is strictly > prohibited. > > > > ©2015 Signal. All rights reserved. > > > > > > > > > > > > > > > > -- > > > Cliff Rhyne > > > Software Engineering Lead > > > e: crh...@signal.co > > > signal.co > > > > > > > > > Cut Through the Noise > > > > > > This e-mail and any files transmitted with it are for the sole use of > the > > > intended recipient(s) and may contain confidential and privileged > > > information. Any unauthorized use of this email is strictly prohibited. > > > ©2015 Signal. All rights reserved. > > > > > > > > > > > -- > > -- Guozhang > > > > > > -- > Cliff Rhyne > Software Engineering Lead > e: crh...@signal.co > signal.co > > > Cut Through the Noise > > This e-mail and any files transmitted with it are for the sole use of the > intended recipient(s) and may contain confidential and privileged > information. Any unauthorized use of this email is strictly prohibited. > ©2015 Signal. All rights reserved. > -- -- Guozhang
Re: kafka-consumer-groups.sh doesn't work when consumers are off
Hi Guozhang, That looks like it might help but feels like there might be some gaps. Would it be able to survive restarts of the kafka broker? How long would it stay in the cache (and is that configurable)? If it expires from the cache, what's the cache-miss operation look like? (yes, a lot of this depends on the data still being in the logs to recover) In the mean time, can I rely on the deprecated ConsumerOffsetChecker (which looks at zookeeper) even though I'm using the new KafkaConsumer? Thanks, Cliff On Thu, Jan 28, 2016 at 3:30 PM, Guozhang Wang wrote: > Hi Cliff, > > Short answer to your question is it is just the current implementation. > > The coordinator stores the offsets as messages in an internal topic and > also keeps the latest offset values in in-memory. It answers > ConsumerGroupRequest using its cached offset, and upon the consumer group > being removed since no member is alive already, it removed it from its > in-memory cache and add a "tombstone" to the offset log as well. But the > offsets are still persistent as messages in the log, which will only be > compacted after a while (this is depend on the log compaction policy). > > There is a ticket open for improving on this scenario ( > https://issues.apache.org/jira/browse/KAFKA-2720) which lets the > coordinator to only "purge" dead groups periodically instead of > immediately, and that may partially resolve your case. > > Guozhang > > > On Thu, Jan 28, 2016 at 12:13 PM, Cliff Rhyne wrote: > > > Just following up on this concern. Is there a constraint that prevents > > ConsumerGroupCommand from reporting offsets on a group if no members are > > connected, or is this just the current implementation? > > > > Thanks, > > Cliff > > > > On Mon, Jan 25, 2016 at 3:50 PM, Cliff Rhyne wrote: > > > > > I'm running into a few challenges trying to evaluate offsets and lag > > > (pending message count) in the new Java KafkaConsumer. The old > > > ConsumerOffsetChecker doesn't work anymore since the offsets aren't > > stored > > > in zookeeper after switching from the old consumer. This would be > fine, > > > but the kafka-consumer-groups.sh command doesn't work if the consumers > > are > > > shut off. This seems like an unnecessary limitation and is problematic > > for > > > troubleshooting / monitoring when the application is turned off (or > while > > > my application is running due to our stopping/starting consumers). > > > > > > Is there a constraint that I'm not aware of or is this something that > > > could be changed? > > > > > > Thanks, > > > Cliff > > > > > > -- > > > Cliff Rhyne > > > Software Engineering Lead > > > e: crh...@signal.co > > > signal.co > > > > > > > > > Cut Through the Noise > > > > > > This e-mail and any files transmitted with it are for the sole use of > the > > > intended recipient(s) and may contain confidential and privileged > > > information. Any unauthorized use of this email is strictly prohibited. > > > ©2015 Signal. All rights reserved. > > > > > > > > > > > -- > > Cliff Rhyne > > Software Engineering Lead > > e: crh...@signal.co > > signal.co > > > > > > Cut Through the Noise > > > > This e-mail and any files transmitted with it are for the sole use of the > > intended recipient(s) and may contain confidential and privileged > > information. Any unauthorized use of this email is strictly prohibited. > > ©2015 Signal. All rights reserved. > > > > > > -- > -- Guozhang > -- Cliff Rhyne Software Engineering Lead e: crh...@signal.co signal.co Cut Through the Noise This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized use of this email is strictly prohibited. ©2015 Signal. All rights reserved.
Re: kafka-consumer-groups.sh doesn't work when consumers are off
Hi Cliff, Short answer to your question is it is just the current implementation. The coordinator stores the offsets as messages in an internal topic and also keeps the latest offset values in in-memory. It answers ConsumerGroupRequest using its cached offset, and upon the consumer group being removed since no member is alive already, it removed it from its in-memory cache and add a "tombstone" to the offset log as well. But the offsets are still persistent as messages in the log, which will only be compacted after a while (this is depend on the log compaction policy). There is a ticket open for improving on this scenario ( https://issues.apache.org/jira/browse/KAFKA-2720) which lets the coordinator to only "purge" dead groups periodically instead of immediately, and that may partially resolve your case. Guozhang On Thu, Jan 28, 2016 at 12:13 PM, Cliff Rhyne wrote: > Just following up on this concern. Is there a constraint that prevents > ConsumerGroupCommand from reporting offsets on a group if no members are > connected, or is this just the current implementation? > > Thanks, > Cliff > > On Mon, Jan 25, 2016 at 3:50 PM, Cliff Rhyne wrote: > > > I'm running into a few challenges trying to evaluate offsets and lag > > (pending message count) in the new Java KafkaConsumer. The old > > ConsumerOffsetChecker doesn't work anymore since the offsets aren't > stored > > in zookeeper after switching from the old consumer. This would be fine, > > but the kafka-consumer-groups.sh command doesn't work if the consumers > are > > shut off. This seems like an unnecessary limitation and is problematic > for > > troubleshooting / monitoring when the application is turned off (or while > > my application is running due to our stopping/starting consumers). > > > > Is there a constraint that I'm not aware of or is this something that > > could be changed? > > > > Thanks, > > Cliff > > > > -- > > Cliff Rhyne > > Software Engineering Lead > > e: crh...@signal.co > > signal.co > > > > > > Cut Through the Noise > > > > This e-mail and any files transmitted with it are for the sole use of the > > intended recipient(s) and may contain confidential and privileged > > information. Any unauthorized use of this email is strictly prohibited. > > ©2015 Signal. All rights reserved. > > > > > > -- > Cliff Rhyne > Software Engineering Lead > e: crh...@signal.co > signal.co > > > Cut Through the Noise > > This e-mail and any files transmitted with it are for the sole use of the > intended recipient(s) and may contain confidential and privileged > information. Any unauthorized use of this email is strictly prohibited. > ©2015 Signal. All rights reserved. > -- -- Guozhang
Re: kafka-consumer-groups.sh doesn't work when consumers are off
Just following up on this concern. Is there a constraint that prevents ConsumerGroupCommand from reporting offsets on a group if no members are connected, or is this just the current implementation? Thanks, Cliff On Mon, Jan 25, 2016 at 3:50 PM, Cliff Rhyne wrote: > I'm running into a few challenges trying to evaluate offsets and lag > (pending message count) in the new Java KafkaConsumer. The old > ConsumerOffsetChecker doesn't work anymore since the offsets aren't stored > in zookeeper after switching from the old consumer. This would be fine, > but the kafka-consumer-groups.sh command doesn't work if the consumers are > shut off. This seems like an unnecessary limitation and is problematic for > troubleshooting / monitoring when the application is turned off (or while > my application is running due to our stopping/starting consumers). > > Is there a constraint that I'm not aware of or is this something that > could be changed? > > Thanks, > Cliff > > -- > Cliff Rhyne > Software Engineering Lead > e: crh...@signal.co > signal.co > > > Cut Through the Noise > > This e-mail and any files transmitted with it are for the sole use of the > intended recipient(s) and may contain confidential and privileged > information. Any unauthorized use of this email is strictly prohibited. > ©2015 Signal. All rights reserved. > -- Cliff Rhyne Software Engineering Lead e: crh...@signal.co signal.co Cut Through the Noise This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized use of this email is strictly prohibited. ©2015 Signal. All rights reserved.
kafka-consumer-groups.sh doesn't work when consumers are off
I'm running into a few challenges trying to evaluate offsets and lag (pending message count) in the new Java KafkaConsumer. The old ConsumerOffsetChecker doesn't work anymore since the offsets aren't stored in zookeeper after switching from the old consumer. This would be fine, but the kafka-consumer-groups.sh command doesn't work if the consumers are shut off. This seems like an unnecessary limitation and is problematic for troubleshooting / monitoring when the application is turned off (or while my application is running due to our stopping/starting consumers). Is there a constraint that I'm not aware of or is this something that could be changed? Thanks, Cliff -- Cliff Rhyne Software Engineering Lead e: crh...@signal.co signal.co Cut Through the Noise This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized use of this email is strictly prohibited. ©2015 Signal. All rights reserved.
