Re: checking consumer lag on KStreams app?

2016-12-18 Thread Jon Yeargers 
The group should be whatever you used for
StreamsConfig.APPLICATION_ID_CONFIG.

Using this value Im able to see offsets using 'kafka-consumer-groups
--new-consumer --bootstrap-server  --group  --describe'

On Sat, Dec 17, 2016 at 10:48 PM, Sachin Mittal  wrote:

> Hi,
> Looks like the kafka-consumer-groups.sh command is still not working.
>
> I ran:
> bin/kafka-consumer-groups.sh --zookeeper 192.168.73.198:2181 --describe
> --group test
> I get the output like
>
> No topic available for consumer group provided
> GROUP  TOPIC  PARTITION
>  CURRENT-OFFSET  LOG-END-OFFSET  LAG OWNER
>
> Note that my stream application id is test and when executing the command,
> the stream application was running.
> Why it is not able to get any topics from that.
> My version is kafka_2.10-0.10.0.1.
>
> Thanks
> Sachin
>
>
>
> On Tue, Dec 13, 2016 at 11:06 PM, Matthias J. Sax 
> wrote:
>
> > There is a workaround:
> >
> > http://stackoverflow.com/questions/41097126/how-to-get-
> > the-group-commit-offset-from-kafka0-10-x
> >
> > I never had time to follow up with this issue. Will put it back on the
> > agenda. Should really get fixed.
> >
> >
> > -Matthias
> >
> > On 12/13/16 2:03 AM, Sachin Mittal wrote:
> > > If this is a bug then it is not fixed because I just build kafka from
> > > source and it gave me the reported error.
> > >
> > > On Tue, Dec 13, 2016 at 3:24 PM, Sam Pegler
>  > com>
> > > wrote:
> > >
> > >>> You can only check the offsets when there is an active member of the
> > >> consumer group.
> > >>
> > >> This was a bug [1] thats been fixed.  Thanks to Vincent Dautremont for
> > >> pointing this out to me a while ago.
> > >>
> > >> http://mail-archives.apache.org/mod_mbox/kafka-users/201611.mbox/%
> > >> 3CCAD2WViSAgwc9i4-9xEw1oz1xzpsbveFt1%3DSZ0qkHRiFEc3fXbw%40mail.
> > >> gmail.com%3E
> > >>
> > >> __
> > >>
> > >> Sam Pegler
> > >>
> > >> PRODUCTION ENGINEER
> > >>
> > >> T. +44(0) 07 562 867 486
> > >>
> > >> 
> > >> 3-7 Herbal Hill / London / EC1R 5EJ
> > >> www.infectiousmedia.com
> > >>
> > >> This email and any attachments are confidential and may also be
> > privileged.
> > >> If you
> > >> are not the intended recipient, please notify the sender immediately,
> > and
> > >> do not
> > >> disclose the contents to another person, use it for any purpose, or
> > store,
> > >> or copy
> > >> the information in any medium. Please also destroy and delete the
> > message
> > >> from
> > >> your computer.
> > >>
> > >>
> > >> On 13 December 2016 at 09:39, Damian Guy 
> wrote:
> > >>
> > >>> Hi Sachin
> > >>>
> > >>> That is correct. You can only check the offsets when there is an
> active
> > >>> member of the consumer group. In this case that would mean that you
> > have
> > >> at
> > >>> least one instance of your streams application running.
> > >>>
> > >>> Thanks,
> > >>> Damian
> > >>>
> > >>> On Tue, 13 Dec 2016 at 06:58 Sachin Mittal 
> wrote:
> > >>>
> >  Hi,
> >  I used the following command
> >  bin\windows\kafka-consumer-groups.bat --bootstrap-server
> > >> localhost:9092
> >  --describe --group test
> >  and I get the following output
> > 
> >  Note: This will only show information about consumers that use the
> > Java
> >  consumer API (non-ZooKeeper-based consumers).
> > 
> >  Error: Consumer group 'test' has no active members.
> > 
> >  What does this mean.
> > 
> >  It means I can check the offset of consumer only when streams
> > >> applictaion
> >  "test" is running.
> > 
> >  Thanks
> >  Sachin
> > 
> > 
> >  On Mon, Dec 12, 2016 at 8:33 PM, Damian Guy 
> > >>> wrote:
> > 
> > > Hi Sachin,
> > >
> > > You should use the kafka-consumer-groups.sh command. The
> > > ConsumerOffsetChecker is deprecated and is only for the old
> consumer.
> > >
> > > Thanks,
> > > Damian
> > >
> > > On Mon, 12 Dec 2016 at 14:32 Sachin Mittal 
> > >> wrote:
> > >
> > >> Hi,
> > >> I have a streams application running with application id test.
> > >> When I try to check consumer lag like you suggested I get the
> > >>> following
> > >> issue:
> > >>
> > >> bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker
> > >> --zookeeper
> > >> localhost:2181 --group test
> > >> [2016-12-12 10:26:01,348] WARN WARNING: ConsumerOffsetChecker is
> > >> deprecated and will be dropped in releases following 0.9.0. Use
> > >> ConsumerGroupCommand instead. (kafka.tools.
> ConsumerOffsetChecker$)
> > >> SLF4J: Class path contains multiple SLF4J bindings.
> > >> SLF4J: Found binding in
> > >>
> > >> [jar:file:/home/testuser/kafka/kafka_2.10-0.10.0.1/
> > >

Re: checking consumer lag on KStreams app?

2016-12-17 Thread Sachin Mittal 
Hi,
Looks like the kafka-consumer-groups.sh command is still not working.

I ran:
bin/kafka-consumer-groups.sh --zookeeper 192.168.73.198:2181 --describe
--group test
I get the output like

No topic available for consumer group provided
GROUP  TOPIC  PARTITION
 CURRENT-OFFSET  LOG-END-OFFSET  LAG OWNER

Note that my stream application id is test and when executing the command,
the stream application was running.
Why it is not able to get any topics from that.
My version is kafka_2.10-0.10.0.1.

Thanks
Sachin



On Tue, Dec 13, 2016 at 11:06 PM, Matthias J. Sax 
wrote:

> There is a workaround:
>
> http://stackoverflow.com/questions/41097126/how-to-get-
> the-group-commit-offset-from-kafka0-10-x
>
> I never had time to follow up with this issue. Will put it back on the
> agenda. Should really get fixed.
>
>
> -Matthias
>
> On 12/13/16 2:03 AM, Sachin Mittal wrote:
> > If this is a bug then it is not fixed because I just build kafka from
> > source and it gave me the reported error.
> >
> > On Tue, Dec 13, 2016 at 3:24 PM, Sam Pegler  com>
> > wrote:
> >
> >>> You can only check the offsets when there is an active member of the
> >> consumer group.
> >>
> >> This was a bug [1] thats been fixed.  Thanks to Vincent Dautremont for
> >> pointing this out to me a while ago.
> >>
> >> http://mail-archives.apache.org/mod_mbox/kafka-users/201611.mbox/%
> >> 3CCAD2WViSAgwc9i4-9xEw1oz1xzpsbveFt1%3DSZ0qkHRiFEc3fXbw%40mail.
> >> gmail.com%3E
> >>
> >> __
> >>
> >> Sam Pegler
> >>
> >> PRODUCTION ENGINEER
> >>
> >> T. +44(0) 07 562 867 486
> >>
> >> 
> >> 3-7 Herbal Hill / London / EC1R 5EJ
> >> www.infectiousmedia.com
> >>
> >> This email and any attachments are confidential and may also be
> privileged.
> >> If you
> >> are not the intended recipient, please notify the sender immediately,
> and
> >> do not
> >> disclose the contents to another person, use it for any purpose, or
> store,
> >> or copy
> >> the information in any medium. Please also destroy and delete the
> message
> >> from
> >> your computer.
> >>
> >>
> >> On 13 December 2016 at 09:39, Damian Guy  wrote:
> >>
> >>> Hi Sachin
> >>>
> >>> That is correct. You can only check the offsets when there is an active
> >>> member of the consumer group. In this case that would mean that you
> have
> >> at
> >>> least one instance of your streams application running.
> >>>
> >>> Thanks,
> >>> Damian
> >>>
> >>> On Tue, 13 Dec 2016 at 06:58 Sachin Mittal  wrote:
> >>>
>  Hi,
>  I used the following command
>  bin\windows\kafka-consumer-groups.bat --bootstrap-server
> >> localhost:9092
>  --describe --group test
>  and I get the following output
> 
>  Note: This will only show information about consumers that use the
> Java
>  consumer API (non-ZooKeeper-based consumers).
> 
>  Error: Consumer group 'test' has no active members.
> 
>  What does this mean.
> 
>  It means I can check the offset of consumer only when streams
> >> applictaion
>  "test" is running.
> 
>  Thanks
>  Sachin
> 
> 
>  On Mon, Dec 12, 2016 at 8:33 PM, Damian Guy 
> >>> wrote:
> 
> > Hi Sachin,
> >
> > You should use the kafka-consumer-groups.sh command. The
> > ConsumerOffsetChecker is deprecated and is only for the old consumer.
> >
> > Thanks,
> > Damian
> >
> > On Mon, 12 Dec 2016 at 14:32 Sachin Mittal 
> >> wrote:
> >
> >> Hi,
> >> I have a streams application running with application id test.
> >> When I try to check consumer lag like you suggested I get the
> >>> following
> >> issue:
> >>
> >> bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker
> >> --zookeeper
> >> localhost:2181 --group test
> >> [2016-12-12 10:26:01,348] WARN WARNING: ConsumerOffsetChecker is
> >> deprecated and will be dropped in releases following 0.9.0. Use
> >> ConsumerGroupCommand instead. (kafka.tools.ConsumerOffsetChecker$)
> >> SLF4J: Class path contains multiple SLF4J bindings.
> >> SLF4J: Found binding in
> >>
> >> [jar:file:/home/testuser/kafka/kafka_2.10-0.10.0.1/
> > libs/logback-classic-1.0.3.jar!/org/slf4j/impl/
> >>> StaticLoggerBinder.class]
> >> SLF4J: Found binding in
> >>
> >> [jar:file:/home/testuser/kafka/kafka_2.10-0.10.0.1/
> > libs/slf4j-log4j12-1.7.21.jar!/org/slf4j/impl/
> >>> StaticLoggerBinder.class]
> >> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for
> >> an
> >> explanation.
> >> SLF4J: Actual binding is of type
> >> [ch.qos.logback.classic.selector.DefaultContextSelector]
> >> Exiting due to: org.apache.zookeeper.KeeperException$
> >>> NoNodeException:
> >> KeeperErrorCode = NoNode for /consumers/test/owners.
>

Re: checking consumer lag on KStreams app?

2016-12-13 Thread Matthias J. Sax 
There is a workaround:

http://stackoverflow.com/questions/41097126/how-to-get-the-group-commit-offset-from-kafka0-10-x

I never had time to follow up with this issue. Will put it back on the
agenda. Should really get fixed.


-Matthias

On 12/13/16 2:03 AM, Sachin Mittal wrote:
> If this is a bug then it is not fixed because I just build kafka from
> source and it gave me the reported error.
> 
> On Tue, Dec 13, 2016 at 3:24 PM, Sam Pegler 
> wrote:
> 
>>> You can only check the offsets when there is an active member of the
>> consumer group.
>>
>> This was a bug [1] thats been fixed.  Thanks to Vincent Dautremont for
>> pointing this out to me a while ago.
>>
>> http://mail-archives.apache.org/mod_mbox/kafka-users/201611.mbox/%
>> 3CCAD2WViSAgwc9i4-9xEw1oz1xzpsbveFt1%3DSZ0qkHRiFEc3fXbw%40mail.
>> gmail.com%3E
>>
>> __
>>
>> Sam Pegler
>>
>> PRODUCTION ENGINEER
>>
>> T. +44(0) 07 562 867 486
>>
>> 
>> 3-7 Herbal Hill / London / EC1R 5EJ
>> www.infectiousmedia.com
>>
>> This email and any attachments are confidential and may also be privileged.
>> If you
>> are not the intended recipient, please notify the sender immediately, and
>> do not
>> disclose the contents to another person, use it for any purpose, or store,
>> or copy
>> the information in any medium. Please also destroy and delete the message
>> from
>> your computer.
>>
>>
>> On 13 December 2016 at 09:39, Damian Guy  wrote:
>>
>>> Hi Sachin
>>>
>>> That is correct. You can only check the offsets when there is an active
>>> member of the consumer group. In this case that would mean that you have
>> at
>>> least one instance of your streams application running.
>>>
>>> Thanks,
>>> Damian
>>>
>>> On Tue, 13 Dec 2016 at 06:58 Sachin Mittal  wrote:
>>>
 Hi,
 I used the following command
 bin\windows\kafka-consumer-groups.bat --bootstrap-server
>> localhost:9092
 --describe --group test
 and I get the following output

 Note: This will only show information about consumers that use the Java
 consumer API (non-ZooKeeper-based consumers).

 Error: Consumer group 'test' has no active members.

 What does this mean.

 It means I can check the offset of consumer only when streams
>> applictaion
 "test" is running.

 Thanks
 Sachin


 On Mon, Dec 12, 2016 at 8:33 PM, Damian Guy 
>>> wrote:

> Hi Sachin,
>
> You should use the kafka-consumer-groups.sh command. The
> ConsumerOffsetChecker is deprecated and is only for the old consumer.
>
> Thanks,
> Damian
>
> On Mon, 12 Dec 2016 at 14:32 Sachin Mittal 
>> wrote:
>
>> Hi,
>> I have a streams application running with application id test.
>> When I try to check consumer lag like you suggested I get the
>>> following
>> issue:
>>
>> bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker
>> --zookeeper
>> localhost:2181 --group test
>> [2016-12-12 10:26:01,348] WARN WARNING: ConsumerOffsetChecker is
>> deprecated and will be dropped in releases following 0.9.0. Use
>> ConsumerGroupCommand instead. (kafka.tools.ConsumerOffsetChecker$)
>> SLF4J: Class path contains multiple SLF4J bindings.
>> SLF4J: Found binding in
>>
>> [jar:file:/home/testuser/kafka/kafka_2.10-0.10.0.1/
> libs/logback-classic-1.0.3.jar!/org/slf4j/impl/
>>> StaticLoggerBinder.class]
>> SLF4J: Found binding in
>>
>> [jar:file:/home/testuser/kafka/kafka_2.10-0.10.0.1/
> libs/slf4j-log4j12-1.7.21.jar!/org/slf4j/impl/
>>> StaticLoggerBinder.class]
>> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for
>> an
>> explanation.
>> SLF4J: Actual binding is of type
>> [ch.qos.logback.classic.selector.DefaultContextSelector]
>> Exiting due to: org.apache.zookeeper.KeeperException$
>>> NoNodeException:
>> KeeperErrorCode = NoNode for /consumers/test/owners.
>>
>> Please let me know where I may be going wrong.
>> I have the kafka logs set in folder
>> /data01/testuser/kafka-logs
>>
>> Under kafka-logs I see many folders with name something like
>> consumer_offsets_*
>>
>> I have the stream dir set in folder
>> /data01/testuser/kafka-streams/test
>>
>> Thanks
>> Sachin
>>
>>
>> On Sun, Dec 11, 2016 at 2:19 AM, Matthias J. Sax <
 matth...@confluent.io>
>> wrote:
>>
>>> It's basically just a consumer as any other. The application.id
>> is
> used
>>> as consumer group.id.
>>>
>>> So just use the available tools you do use to check consumer lag.
>>>
>>>
>>> -Matthias
>>>
>>> On 12/9/16 5:49 PM, Jon Yeargers wrote:
 How would this be done?

>>>
>>>
>>
>

>>>
>>
> 



signature.asc
Description: OpenPGP digital signature

Re: checking consumer lag on KStreams app?

2016-12-13 Thread Sam Pegler 
Is the following PR present https://github.com/apache/kafka/pull/1336?

__

Sam Pegler

PRODUCTION ENGINEER

T. +44(0) 07 562 867 486


3-7 Herbal Hill / London / EC1R 5EJ
www.infectiousmedia.com

This email and any attachments are confidential and may also be privileged.
If you
are not the intended recipient, please notify the sender immediately, and
do not
disclose the contents to another person, use it for any purpose, or store,
or copy
the information in any medium. Please also destroy and delete the message
from
your computer.


On 13 December 2016 at 10:03, Sachin Mittal  wrote:

> If this is a bug then it is not fixed because I just build kafka from
> source and it gave me the reported error.
>
> On Tue, Dec 13, 2016 at 3:24 PM, Sam Pegler  com>
> wrote:
>
> > >You can only check the offsets when there is an active member of the
> > consumer group.
> >
> > This was a bug [1] thats been fixed.  Thanks to Vincent Dautremont for
> > pointing this out to me a while ago.
> >
> > http://mail-archives.apache.org/mod_mbox/kafka-users/201611.mbox/%
> > 3CCAD2WViSAgwc9i4-9xEw1oz1xzpsbveFt1%3DSZ0qkHRiFEc3fXbw%40mail.
> > gmail.com%3E
> >
> > __
> >
> > Sam Pegler
> >
> > PRODUCTION ENGINEER
> >
> > T. +44(0) 07 562 867 486
> >
> > 
> > 3-7 Herbal Hill / London / EC1R 5EJ
> > www.infectiousmedia.com
> >
> > This email and any attachments are confidential and may also be
> privileged.
> > If you
> > are not the intended recipient, please notify the sender immediately, and
> > do not
> > disclose the contents to another person, use it for any purpose, or
> store,
> > or copy
> > the information in any medium. Please also destroy and delete the message
> > from
> > your computer.
> >
> >
> > On 13 December 2016 at 09:39, Damian Guy  wrote:
> >
> > > Hi Sachin
> > >
> > > That is correct. You can only check the offsets when there is an active
> > > member of the consumer group. In this case that would mean that you
> have
> > at
> > > least one instance of your streams application running.
> > >
> > > Thanks,
> > > Damian
> > >
> > > On Tue, 13 Dec 2016 at 06:58 Sachin Mittal  wrote:
> > >
> > > > Hi,
> > > > I used the following command
> > > > bin\windows\kafka-consumer-groups.bat --bootstrap-server
> > localhost:9092
> > > > --describe --group test
> > > > and I get the following output
> > > >
> > > > Note: This will only show information about consumers that use the
> Java
> > > > consumer API (non-ZooKeeper-based consumers).
> > > >
> > > > Error: Consumer group 'test' has no active members.
> > > >
> > > > What does this mean.
> > > >
> > > > It means I can check the offset of consumer only when streams
> > applictaion
> > > > "test" is running.
> > > >
> > > > Thanks
> > > > Sachin
> > > >
> > > >
> > > > On Mon, Dec 12, 2016 at 8:33 PM, Damian Guy 
> > > wrote:
> > > >
> > > > > Hi Sachin,
> > > > >
> > > > > You should use the kafka-consumer-groups.sh command. The
> > > > > ConsumerOffsetChecker is deprecated and is only for the old
> consumer.
> > > > >
> > > > > Thanks,
> > > > > Damian
> > > > >
> > > > > On Mon, 12 Dec 2016 at 14:32 Sachin Mittal 
> > wrote:
> > > > >
> > > > > > Hi,
> > > > > > I have a streams application running with application id test.
> > > > > > When I try to check consumer lag like you suggested I get the
> > > following
> > > > > > issue:
> > > > > >
> > > > > > bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker
> > --zookeeper
> > > > > > localhost:2181 --group test
> > > > > > [2016-12-12 10:26:01,348] WARN WARNING: ConsumerOffsetChecker is
> > > > > > deprecated and will be dropped in releases following 0.9.0. Use
> > > > > > ConsumerGroupCommand instead. (kafka.tools.
> ConsumerOffsetChecker$)
> > > > > > SLF4J: Class path contains multiple SLF4J bindings.
> > > > > > SLF4J: Found binding in
> > > > > >
> > > > > > [jar:file:/home/testuser/kafka/kafka_2.10-0.10.0.1/
> > > > > libs/logback-classic-1.0.3.jar!/org/slf4j/impl/
> > > StaticLoggerBinder.class]
> > > > > > SLF4J: Found binding in
> > > > > >
> > > > > > [jar:file:/home/testuser/kafka/kafka_2.10-0.10.0.1/
> > > > > libs/slf4j-log4j12-1.7.21.jar!/org/slf4j/impl/
> > > StaticLoggerBinder.class]
> > > > > > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for
> > an
> > > > > > explanation.
> > > > > > SLF4J: Actual binding is of type
> > > > > > [ch.qos.logback.classic.selector.DefaultContextSelector]
> > > > > > Exiting due to: org.apache.zookeeper.KeeperException$
> > > NoNodeException:
> > > > > > KeeperErrorCode = NoNode for /consumers/test/owners.
> > > > > >
> > > > > > Please let me know where I may be going wrong.
> > > > > > I have the kafka logs set in folder
> > > > > > /data01/testuser/kafka-logs
> > > > > >
> > > > > > Under kafka-logs I see many folders with name something like
> > > >

Re: checking consumer lag on KStreams app?

2016-12-13 Thread Sachin Mittal 
If this is a bug then it is not fixed because I just build kafka from
source and it gave me the reported error.

On Tue, Dec 13, 2016 at 3:24 PM, Sam Pegler 
wrote:

> >You can only check the offsets when there is an active member of the
> consumer group.
>
> This was a bug [1] thats been fixed.  Thanks to Vincent Dautremont for
> pointing this out to me a while ago.
>
> http://mail-archives.apache.org/mod_mbox/kafka-users/201611.mbox/%
> 3CCAD2WViSAgwc9i4-9xEw1oz1xzpsbveFt1%3DSZ0qkHRiFEc3fXbw%40mail.
> gmail.com%3E
>
> __
>
> Sam Pegler
>
> PRODUCTION ENGINEER
>
> T. +44(0) 07 562 867 486
>
> 
> 3-7 Herbal Hill / London / EC1R 5EJ
> www.infectiousmedia.com
>
> This email and any attachments are confidential and may also be privileged.
> If you
> are not the intended recipient, please notify the sender immediately, and
> do not
> disclose the contents to another person, use it for any purpose, or store,
> or copy
> the information in any medium. Please also destroy and delete the message
> from
> your computer.
>
>
> On 13 December 2016 at 09:39, Damian Guy  wrote:
>
> > Hi Sachin
> >
> > That is correct. You can only check the offsets when there is an active
> > member of the consumer group. In this case that would mean that you have
> at
> > least one instance of your streams application running.
> >
> > Thanks,
> > Damian
> >
> > On Tue, 13 Dec 2016 at 06:58 Sachin Mittal  wrote:
> >
> > > Hi,
> > > I used the following command
> > > bin\windows\kafka-consumer-groups.bat --bootstrap-server
> localhost:9092
> > > --describe --group test
> > > and I get the following output
> > >
> > > Note: This will only show information about consumers that use the Java
> > > consumer API (non-ZooKeeper-based consumers).
> > >
> > > Error: Consumer group 'test' has no active members.
> > >
> > > What does this mean.
> > >
> > > It means I can check the offset of consumer only when streams
> applictaion
> > > "test" is running.
> > >
> > > Thanks
> > > Sachin
> > >
> > >
> > > On Mon, Dec 12, 2016 at 8:33 PM, Damian Guy 
> > wrote:
> > >
> > > > Hi Sachin,
> > > >
> > > > You should use the kafka-consumer-groups.sh command. The
> > > > ConsumerOffsetChecker is deprecated and is only for the old consumer.
> > > >
> > > > Thanks,
> > > > Damian
> > > >
> > > > On Mon, 12 Dec 2016 at 14:32 Sachin Mittal 
> wrote:
> > > >
> > > > > Hi,
> > > > > I have a streams application running with application id test.
> > > > > When I try to check consumer lag like you suggested I get the
> > following
> > > > > issue:
> > > > >
> > > > > bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker
> --zookeeper
> > > > > localhost:2181 --group test
> > > > > [2016-12-12 10:26:01,348] WARN WARNING: ConsumerOffsetChecker is
> > > > > deprecated and will be dropped in releases following 0.9.0. Use
> > > > > ConsumerGroupCommand instead. (kafka.tools.ConsumerOffsetChecker$)
> > > > > SLF4J: Class path contains multiple SLF4J bindings.
> > > > > SLF4J: Found binding in
> > > > >
> > > > > [jar:file:/home/testuser/kafka/kafka_2.10-0.10.0.1/
> > > > libs/logback-classic-1.0.3.jar!/org/slf4j/impl/
> > StaticLoggerBinder.class]
> > > > > SLF4J: Found binding in
> > > > >
> > > > > [jar:file:/home/testuser/kafka/kafka_2.10-0.10.0.1/
> > > > libs/slf4j-log4j12-1.7.21.jar!/org/slf4j/impl/
> > StaticLoggerBinder.class]
> > > > > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for
> an
> > > > > explanation.
> > > > > SLF4J: Actual binding is of type
> > > > > [ch.qos.logback.classic.selector.DefaultContextSelector]
> > > > > Exiting due to: org.apache.zookeeper.KeeperException$
> > NoNodeException:
> > > > > KeeperErrorCode = NoNode for /consumers/test/owners.
> > > > >
> > > > > Please let me know where I may be going wrong.
> > > > > I have the kafka logs set in folder
> > > > > /data01/testuser/kafka-logs
> > > > >
> > > > > Under kafka-logs I see many folders with name something like
> > > > > consumer_offsets_*
> > > > >
> > > > > I have the stream dir set in folder
> > > > > /data01/testuser/kafka-streams/test
> > > > >
> > > > > Thanks
> > > > > Sachin
> > > > >
> > > > >
> > > > > On Sun, Dec 11, 2016 at 2:19 AM, Matthias J. Sax <
> > > matth...@confluent.io>
> > > > > wrote:
> > > > >
> > > > > > It's basically just a consumer as any other. The application.id
> is
> > > > used
> > > > > > as consumer group.id.
> > > > > >
> > > > > > So just use the available tools you do use to check consumer lag.
> > > > > >
> > > > > >
> > > > > > -Matthias
> > > > > >
> > > > > > On 12/9/16 5:49 PM, Jon Yeargers wrote:
> > > > > > > How would this be done?
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: checking consumer lag on KStreams app?

2016-12-13 Thread Sam Pegler 
>You can only check the offsets when there is an active member of the
consumer group.

This was a bug [1] thats been fixed.  Thanks to Vincent Dautremont for
pointing this out to me a while ago.

http://mail-archives.apache.org/mod_mbox/kafka-users/201611.mbox/%3CCAD2WViSAgwc9i4-9xEw1oz1xzpsbveFt1%3DSZ0qkHRiFEc3fXbw%40mail.gmail.com%3E

__

Sam Pegler

PRODUCTION ENGINEER

T. +44(0) 07 562 867 486


3-7 Herbal Hill / London / EC1R 5EJ
www.infectiousmedia.com

This email and any attachments are confidential and may also be privileged.
If you
are not the intended recipient, please notify the sender immediately, and
do not
disclose the contents to another person, use it for any purpose, or store,
or copy
the information in any medium. Please also destroy and delete the message
from
your computer.


On 13 December 2016 at 09:39, Damian Guy  wrote:

> Hi Sachin
>
> That is correct. You can only check the offsets when there is an active
> member of the consumer group. In this case that would mean that you have at
> least one instance of your streams application running.
>
> Thanks,
> Damian
>
> On Tue, 13 Dec 2016 at 06:58 Sachin Mittal  wrote:
>
> > Hi,
> > I used the following command
> > bin\windows\kafka-consumer-groups.bat --bootstrap-server localhost:9092
> > --describe --group test
> > and I get the following output
> >
> > Note: This will only show information about consumers that use the Java
> > consumer API (non-ZooKeeper-based consumers).
> >
> > Error: Consumer group 'test' has no active members.
> >
> > What does this mean.
> >
> > It means I can check the offset of consumer only when streams applictaion
> > "test" is running.
> >
> > Thanks
> > Sachin
> >
> >
> > On Mon, Dec 12, 2016 at 8:33 PM, Damian Guy 
> wrote:
> >
> > > Hi Sachin,
> > >
> > > You should use the kafka-consumer-groups.sh command. The
> > > ConsumerOffsetChecker is deprecated and is only for the old consumer.
> > >
> > > Thanks,
> > > Damian
> > >
> > > On Mon, 12 Dec 2016 at 14:32 Sachin Mittal  wrote:
> > >
> > > > Hi,
> > > > I have a streams application running with application id test.
> > > > When I try to check consumer lag like you suggested I get the
> following
> > > > issue:
> > > >
> > > > bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --zookeeper
> > > > localhost:2181 --group test
> > > > [2016-12-12 10:26:01,348] WARN WARNING: ConsumerOffsetChecker is
> > > > deprecated and will be dropped in releases following 0.9.0. Use
> > > > ConsumerGroupCommand instead. (kafka.tools.ConsumerOffsetChecker$)
> > > > SLF4J: Class path contains multiple SLF4J bindings.
> > > > SLF4J: Found binding in
> > > >
> > > > [jar:file:/home/testuser/kafka/kafka_2.10-0.10.0.1/
> > > libs/logback-classic-1.0.3.jar!/org/slf4j/impl/
> StaticLoggerBinder.class]
> > > > SLF4J: Found binding in
> > > >
> > > > [jar:file:/home/testuser/kafka/kafka_2.10-0.10.0.1/
> > > libs/slf4j-log4j12-1.7.21.jar!/org/slf4j/impl/
> StaticLoggerBinder.class]
> > > > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> > > > explanation.
> > > > SLF4J: Actual binding is of type
> > > > [ch.qos.logback.classic.selector.DefaultContextSelector]
> > > > Exiting due to: org.apache.zookeeper.KeeperException$
> NoNodeException:
> > > > KeeperErrorCode = NoNode for /consumers/test/owners.
> > > >
> > > > Please let me know where I may be going wrong.
> > > > I have the kafka logs set in folder
> > > > /data01/testuser/kafka-logs
> > > >
> > > > Under kafka-logs I see many folders with name something like
> > > > consumer_offsets_*
> > > >
> > > > I have the stream dir set in folder
> > > > /data01/testuser/kafka-streams/test
> > > >
> > > > Thanks
> > > > Sachin
> > > >
> > > >
> > > > On Sun, Dec 11, 2016 at 2:19 AM, Matthias J. Sax <
> > matth...@confluent.io>
> > > > wrote:
> > > >
> > > > > It's basically just a consumer as any other. The application.id is
> > > used
> > > > > as consumer group.id.
> > > > >
> > > > > So just use the available tools you do use to check consumer lag.
> > > > >
> > > > >
> > > > > -Matthias
> > > > >
> > > > > On 12/9/16 5:49 PM, Jon Yeargers wrote:
> > > > > > How would this be done?
> > > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>

Re: checking consumer lag on KStreams app?

2016-12-13 Thread Damian Guy 
Hi Sachin

That is correct. You can only check the offsets when there is an active
member of the consumer group. In this case that would mean that you have at
least one instance of your streams application running.

Thanks,
Damian

On Tue, 13 Dec 2016 at 06:58 Sachin Mittal  wrote:

> Hi,
> I used the following command
> bin\windows\kafka-consumer-groups.bat --bootstrap-server localhost:9092
> --describe --group test
> and I get the following output
>
> Note: This will only show information about consumers that use the Java
> consumer API (non-ZooKeeper-based consumers).
>
> Error: Consumer group 'test' has no active members.
>
> What does this mean.
>
> It means I can check the offset of consumer only when streams applictaion
> "test" is running.
>
> Thanks
> Sachin
>
>
> On Mon, Dec 12, 2016 at 8:33 PM, Damian Guy  wrote:
>
> > Hi Sachin,
> >
> > You should use the kafka-consumer-groups.sh command. The
> > ConsumerOffsetChecker is deprecated and is only for the old consumer.
> >
> > Thanks,
> > Damian
> >
> > On Mon, 12 Dec 2016 at 14:32 Sachin Mittal  wrote:
> >
> > > Hi,
> > > I have a streams application running with application id test.
> > > When I try to check consumer lag like you suggested I get the following
> > > issue:
> > >
> > > bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --zookeeper
> > > localhost:2181 --group test
> > > [2016-12-12 10:26:01,348] WARN WARNING: ConsumerOffsetChecker is
> > > deprecated and will be dropped in releases following 0.9.0. Use
> > > ConsumerGroupCommand instead. (kafka.tools.ConsumerOffsetChecker$)
> > > SLF4J: Class path contains multiple SLF4J bindings.
> > > SLF4J: Found binding in
> > >
> > > [jar:file:/home/testuser/kafka/kafka_2.10-0.10.0.1/
> > libs/logback-classic-1.0.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> > > SLF4J: Found binding in
> > >
> > > [jar:file:/home/testuser/kafka/kafka_2.10-0.10.0.1/
> > libs/slf4j-log4j12-1.7.21.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> > > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> > > explanation.
> > > SLF4J: Actual binding is of type
> > > [ch.qos.logback.classic.selector.DefaultContextSelector]
> > > Exiting due to: org.apache.zookeeper.KeeperException$NoNodeException:
> > > KeeperErrorCode = NoNode for /consumers/test/owners.
> > >
> > > Please let me know where I may be going wrong.
> > > I have the kafka logs set in folder
> > > /data01/testuser/kafka-logs
> > >
> > > Under kafka-logs I see many folders with name something like
> > > consumer_offsets_*
> > >
> > > I have the stream dir set in folder
> > > /data01/testuser/kafka-streams/test
> > >
> > > Thanks
> > > Sachin
> > >
> > >
> > > On Sun, Dec 11, 2016 at 2:19 AM, Matthias J. Sax <
> matth...@confluent.io>
> > > wrote:
> > >
> > > > It's basically just a consumer as any other. The application.id is
> > used
> > > > as consumer group.id.
> > > >
> > > > So just use the available tools you do use to check consumer lag.
> > > >
> > > >
> > > > -Matthias
> > > >
> > > > On 12/9/16 5:49 PM, Jon Yeargers wrote:
> > > > > How would this be done?
> > > > >
> > > >
> > > >
> > >
> >
>

Re: checking consumer lag on KStreams app?

2016-12-12 Thread Sachin Mittal 
Hi,
I used the following command
bin\windows\kafka-consumer-groups.bat --bootstrap-server localhost:9092
--describe --group test
and I get the following output

Note: This will only show information about consumers that use the Java
consumer API (non-ZooKeeper-based consumers).

Error: Consumer group 'test' has no active members.

What does this mean.

It means I can check the offset of consumer only when streams applictaion
"test" is running.

Thanks
Sachin


On Mon, Dec 12, 2016 at 8:33 PM, Damian Guy  wrote:

> Hi Sachin,
>
> You should use the kafka-consumer-groups.sh command. The
> ConsumerOffsetChecker is deprecated and is only for the old consumer.
>
> Thanks,
> Damian
>
> On Mon, 12 Dec 2016 at 14:32 Sachin Mittal  wrote:
>
> > Hi,
> > I have a streams application running with application id test.
> > When I try to check consumer lag like you suggested I get the following
> > issue:
> >
> > bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --zookeeper
> > localhost:2181 --group test
> > [2016-12-12 10:26:01,348] WARN WARNING: ConsumerOffsetChecker is
> > deprecated and will be dropped in releases following 0.9.0. Use
> > ConsumerGroupCommand instead. (kafka.tools.ConsumerOffsetChecker$)
> > SLF4J: Class path contains multiple SLF4J bindings.
> > SLF4J: Found binding in
> >
> > [jar:file:/home/testuser/kafka/kafka_2.10-0.10.0.1/
> libs/logback-classic-1.0.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> > SLF4J: Found binding in
> >
> > [jar:file:/home/testuser/kafka/kafka_2.10-0.10.0.1/
> libs/slf4j-log4j12-1.7.21.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> > explanation.
> > SLF4J: Actual binding is of type
> > [ch.qos.logback.classic.selector.DefaultContextSelector]
> > Exiting due to: org.apache.zookeeper.KeeperException$NoNodeException:
> > KeeperErrorCode = NoNode for /consumers/test/owners.
> >
> > Please let me know where I may be going wrong.
> > I have the kafka logs set in folder
> > /data01/testuser/kafka-logs
> >
> > Under kafka-logs I see many folders with name something like
> > consumer_offsets_*
> >
> > I have the stream dir set in folder
> > /data01/testuser/kafka-streams/test
> >
> > Thanks
> > Sachin
> >
> >
> > On Sun, Dec 11, 2016 at 2:19 AM, Matthias J. Sax 
> > wrote:
> >
> > > It's basically just a consumer as any other. The application.id is
> used
> > > as consumer group.id.
> > >
> > > So just use the available tools you do use to check consumer lag.
> > >
> > >
> > > -Matthias
> > >
> > > On 12/9/16 5:49 PM, Jon Yeargers wrote:
> > > > How would this be done?
> > > >
> > >
> > >
> >
>

Re: checking consumer lag on KStreams app?

2016-12-12 Thread Damian Guy 
Hi Sachin,

You should use the kafka-consumer-groups.sh command. The
ConsumerOffsetChecker is deprecated and is only for the old consumer.

Thanks,
Damian

On Mon, 12 Dec 2016 at 14:32 Sachin Mittal  wrote:

> Hi,
> I have a streams application running with application id test.
> When I try to check consumer lag like you suggested I get the following
> issue:
>
> bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --zookeeper
> localhost:2181 --group test
> [2016-12-12 10:26:01,348] WARN WARNING: ConsumerOffsetChecker is
> deprecated and will be dropped in releases following 0.9.0. Use
> ConsumerGroupCommand instead. (kafka.tools.ConsumerOffsetChecker$)
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
>
> [jar:file:/home/testuser/kafka/kafka_2.10-0.10.0.1/libs/logback-classic-1.0.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
>
> [jar:file:/home/testuser/kafka/kafka_2.10-0.10.0.1/libs/slf4j-log4j12-1.7.21.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type
> [ch.qos.logback.classic.selector.DefaultContextSelector]
> Exiting due to: org.apache.zookeeper.KeeperException$NoNodeException:
> KeeperErrorCode = NoNode for /consumers/test/owners.
>
> Please let me know where I may be going wrong.
> I have the kafka logs set in folder
> /data01/testuser/kafka-logs
>
> Under kafka-logs I see many folders with name something like
> consumer_offsets_*
>
> I have the stream dir set in folder
> /data01/testuser/kafka-streams/test
>
> Thanks
> Sachin
>
>
> On Sun, Dec 11, 2016 at 2:19 AM, Matthias J. Sax 
> wrote:
>
> > It's basically just a consumer as any other. The application.id is used
> > as consumer group.id.
> >
> > So just use the available tools you do use to check consumer lag.
> >
> >
> > -Matthias
> >
> > On 12/9/16 5:49 PM, Jon Yeargers wrote:
> > > How would this be done?
> > >
> >
> >
>

Re: checking consumer lag on KStreams app?

2016-12-12 Thread Sachin Mittal 
Hi,
I have a streams application running with application id test.
When I try to check consumer lag like you suggested I get the following
issue:

bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --zookeeper
localhost:2181 --group test
[2016-12-12 10:26:01,348] WARN WARNING: ConsumerOffsetChecker is
deprecated and will be dropped in releases following 0.9.0. Use
ConsumerGroupCommand instead. (kafka.tools.ConsumerOffsetChecker$)
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/home/testuser/kafka/kafka_2.10-0.10.0.1/libs/logback-classic-1.0.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/home/testuser/kafka/kafka_2.10-0.10.0.1/libs/slf4j-log4j12-1.7.21.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type
[ch.qos.logback.classic.selector.DefaultContextSelector]
Exiting due to: org.apache.zookeeper.KeeperException$NoNodeException:
KeeperErrorCode = NoNode for /consumers/test/owners.

Please let me know where I may be going wrong.
I have the kafka logs set in folder
/data01/testuser/kafka-logs

Under kafka-logs I see many folders with name something like
consumer_offsets_*

I have the stream dir set in folder
/data01/testuser/kafka-streams/test

Thanks
Sachin


On Sun, Dec 11, 2016 at 2:19 AM, Matthias J. Sax 
wrote:

> It's basically just a consumer as any other. The application.id is used
> as consumer group.id.
>
> So just use the available tools you do use to check consumer lag.
>
>
> -Matthias
>
> On 12/9/16 5:49 PM, Jon Yeargers wrote:
> > How would this be done?
> >
>
>

Re: checking consumer lag on KStreams app?

2016-12-10 Thread Matthias J. Sax 
It's basically just a consumer as any other. The application.id is used
as consumer group.id.

So just use the available tools you do use to check consumer lag.


-Matthias

On 12/9/16 5:49 PM, Jon Yeargers wrote:
> How would this be done?
> 



signature.asc
Description: OpenPGP digital signature

checking consumer lag on KStreams app?

2016-12-09 Thread Jon Yeargers 
How would this be done?