Re: Very long consumer rebalances

[Shantanu Deshmukh]

 I am facing too many problems these days. Now one of our consumer groups
is rebalancing every now and then. And rebalance takes very low, more than
5-10 minutes. Even after re-balancing I see that only half of the consumers
are active/receive assignment. Its all going haywire.

I am seeing these logs in kafka consumer logs. Can anyone help me
understand what is going on here? It is a very long piece of log, but
someone please help me. I am desperately looking for any solution since
more than 2 months now. But to no avail.

[2018-08-09 11:39:51] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:39:53] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25465113 for partition bulk-email-8
[2018-08-09 11:39:53] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
metadata=''}} for group bulk-email-consumer
[2018-08-09 11:39:53] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25463566 for partition bulk-email-6
[2018-08-09 11:39:53] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
metadata=''}} for group bulk-email-consumer
[2018-08-09 11:39:53] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 2588 for partition bulk-email-9
[2018-08-09 11:39:53] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=2588,
metadata=''}} for group bulk-email-consumer
[2018-08-09 11:39:54] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:39:54] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:39:54] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:39:54] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:39:54] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:39:54] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:39:54] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:39:54] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:39:56] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25463566 for partition bulk-email-6
[2018-08-09 11:39:56] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 2588 for partition bulk-email-9
[2018-08-09 11:39:56] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25465113 for partition bulk-email-8
[2018-08-09 11:39:56] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
metadata=''}} for group bulk-email-consumer
[2018-08-09 11:39:56] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=2588,
metadata=''}} for group bulk-email-consumer
[2018-08-09 11:39:56] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
metadata=''}} for group bulk-email-consumer
[2018-08-09 11:39:57] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:39:57] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:39:57] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:39:57] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:39:57] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:39:57] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:39:57] :: DEBUG ::
A

Re: Very long consumer rebalances

[M. Manna]

In the simplest way, how have you implemented your consumer?

1) Does your consumers join a designated group, process messages, and then
closes all connection? Or does it stay open perpetually until server
shutdown?
2) Have you configured the session timeouts for client and zookeeper
accordingly?

Regards,

On 9 August 2018 at 08:00, Shantanu Deshmukh  wrote:

>  I am facing too many problems these days. Now one of our consumer groups
> is rebalancing every now and then. And rebalance takes very low, more than
> 5-10 minutes. Even after re-balancing I see that only half of the consumers
> are active/receive assignment. Its all going haywire.
>
> I am seeing these logs in kafka consumer logs. Can anyone help me
> understand what is going on here? It is a very long piece of log, but
> someone please help me. I am desperately looking for any solution since
> more than 2 months now. But to no avail.
>
> [2018-08-09 11:39:51] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:39:53] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25465113 for partition bulk-email-8
> [2018-08-09 11:39:53] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
> metadata=''}} for group bulk-email-consumer
> [2018-08-09 11:39:53] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25463566 for partition bulk-email-6
> [2018-08-09 11:39:53] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
> metadata=''}} for group bulk-email-consumer
> [2018-08-09 11:39:53] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 2588 for partition bulk-email-9
> [2018-08-09 11:39:53] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=2588,
> metadata=''}} for group bulk-email-consumer
> [2018-08-09 11:39:54] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:39:54] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:39:54] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:39:54] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:39:54] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:39:54] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:39:54] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:39:54] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:39:56] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25463566 for partition bulk-email-6
> [2018-08-09 11:39:56] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 2588 for partition bulk-email-9
> [2018-08-09 11:39:56] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25465113 for partition bulk-email-8
> [2018-08-09 11:39:56] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
> metadata=''}} for group bulk-email-consumer
> [2018-08-09 11:39:56] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=2588,
> metadata=''}} for group bulk-email-consumer
> [2018-08-09 11:39:56] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
> metadata=''}} for group bulk-email-consumer
> [2018-08-09 11:39:57] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:39:57] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:39:57] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response fo

(tentative) release date for next 8.5.x

[M. Manna]

Hello,

Would it be possible to get a tentative timeline for next 8.5.x release? Or
is it confidential.

We wanted to move to 8.5.32 but saw a latest bugfix for URLEncoder reported
about a week back, so wondering if it's worth the wait.

Regards,

Re: Very long consumer rebalances

[Kamal Chandraprakash]

In v0.10.0.1, consumer heartbeat background thread feature is not
available.
Lot of users faced similar errors. So, KIP-62

is
proposed. You have to update your
Kafka version.

I highly recommend you to upgrade Kafka to the version where the heartbeat
background
thread feature is implemented (v0.10.1.0). If you don't have any option to
upgrade, you
have to heartbeat the co-ordinator manually from your consumer. You can use
this

code
snippet for reference.




On Thu, Aug 9, 2018 at 3:03 PM M. Manna  wrote:

> In the simplest way, how have you implemented your consumer?
>
> 1) Does your consumers join a designated group, process messages, and then
> closes all connection? Or does it stay open perpetually until server
> shutdown?
> 2) Have you configured the session timeouts for client and zookeeper
> accordingly?
>
> Regards,
>
> On 9 August 2018 at 08:00, Shantanu Deshmukh 
> wrote:
>
> >  I am facing too many problems these days. Now one of our consumer groups
> > is rebalancing every now and then. And rebalance takes very low, more
> than
> > 5-10 minutes. Even after re-balancing I see that only half of the
> consumers
> > are active/receive assignment. Its all going haywire.
> >
> > I am seeing these logs in kafka consumer logs. Can anyone help me
> > understand what is going on here? It is a very long piece of log, but
> > someone please help me. I am desperately looking for any solution since
> > more than 2 months now. But to no avail.
> >
> > [2018-08-09 11:39:51] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:39:53] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25465113 for partition bulk-email-8
> > [2018-08-09 11:39:53] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
> > metadata=''}} for group bulk-email-consumer
> > [2018-08-09 11:39:53] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25463566 for partition bulk-email-6
> > [2018-08-09 11:39:53] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
> > metadata=''}} for group bulk-email-consumer
> > [2018-08-09 11:39:53] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 2588 for partition bulk-email-9
> > [2018-08-09 11:39:53] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=2588,
> > metadata=''}} for group bulk-email-consumer
> > [2018-08-09 11:39:54] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:39:54] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:39:54] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:39:54] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:39:54] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:39:54] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:39:54] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:39:54] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:39:56] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25463566 for partition bulk-email-6
> > [2018-08-09 11:39:56] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 2588 for partition bulk-email-9
> > [2018-08-09 11:39:56] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25465113 for partition bulk-email-8
> > [2018-08-09 11:39:56] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > autocommit of offsets {b

Re: Very long consumer rebalances

[Shantanu Deshmukh]

Hi,

Yes my consumer application works like below

   1. Reads how many workers are required to process each topics from
   properties file
   2. As many threads are spawned as there are workers mentioned in
   properties file, topic name is passed to this thread. FixedThreadPool
   implementation is used.
   3. Each worker thread initializes one consumer object and subscribes to
   given topic. Consumer group is simply -consumer. So if my topic
   bulk-email, then consumer group for all those threads is bulk-email-consumer
   4. Once this is done, inside an infinite while loop consumer.poll(100)
   method keeps running. So this application is a daemon. Shuts down only when
   server shuts down or in case of kill command.

I have configured session.timeout.ms in consumer properties. I haven't done
anything about zookeeper timeout. Is it required now? Since consumer
accesses only the brokers.

On Thu, Aug 9, 2018 at 3:03 PM M. Manna  wrote:

> In the simplest way, how have you implemented your consumer?
>
> 1) Does your consumers join a designated group, process messages, and then
> closes all connection? Or does it stay open perpetually until server
> shutdown?
> 2) Have you configured the session timeouts for client and zookeeper
> accordingly?
>
> Regards,
>
> On 9 August 2018 at 08:00, Shantanu Deshmukh 
> wrote:
>
> >  I am facing too many problems these days. Now one of our consumer groups
> > is rebalancing every now and then. And rebalance takes very low, more
> than
> > 5-10 minutes. Even after re-balancing I see that only half of the
> consumers
> > are active/receive assignment. Its all going haywire.
> >
> > I am seeing these logs in kafka consumer logs. Can anyone help me
> > understand what is going on here? It is a very long piece of log, but
> > someone please help me. I am desperately looking for any solution since
> > more than 2 months now. But to no avail.
> >
> > [2018-08-09 11:39:51] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:39:53] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25465113 for partition bulk-email-8
> > [2018-08-09 11:39:53] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
> > metadata=''}} for group bulk-email-consumer
> > [2018-08-09 11:39:53] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25463566 for partition bulk-email-6
> > [2018-08-09 11:39:53] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
> > metadata=''}} for group bulk-email-consumer
> > [2018-08-09 11:39:53] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 2588 for partition bulk-email-9
> > [2018-08-09 11:39:53] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=2588,
> > metadata=''}} for group bulk-email-consumer
> > [2018-08-09 11:39:54] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:39:54] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:39:54] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:39:54] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:39:54] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:39:54] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:39:54] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:39:54] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:39:56] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25463566 for partition bulk-email-6
> > [2018-08-09 11:39:56] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 2588 for partition bulk-email-9
> > [2018-08-09 11:39:56] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group

Re: Looking for help with a question on the consumer API

[Shantanu Deshmukh]

Consumer gets kicked out if it fails to send heart beat in designated time
period. Every call to poll sends one heart beat to consumer group
coordinator.

You need to look at *how much time is it taking to process your single
record*. *Maybe it is exceeding session.timeout.ms
 value* which you have set as 30s. Try
increasing that. Also *keep max.poll.records to a lower value*. This
setting determines how many records are fetched after the call to poll
method. If you fetch too many records then even if you keep
session.timeout.ms to a large value your consumer might still get kicked
out and group will enter rebalancing stage.

On Thu, Aug 9, 2018 at 2:00 AM Moiz Raja (moraja) 
wrote:

> Hi All,
>
> I have an issue with the consumer getting kicked out of the group possibly
> due to other issues going on in the system. The issue is detailed here
> https://stackoverflow.com/questions/51754794/how-to-reinstate-a-kafka-consumer-which-has-been-kicked-out-of-the-group
>
> Any help with this issue would be appreciated.
>
> Regards,
> -Moiz
>

RE: [External] Re: Java API to read metrics via JMX

[Tauzell, Dave]

We use Jolokia (which has a java agent you can load with kafka to expose 
metrics via HTTP) and Influx/Telegraf which has support for Jolokia.   There is 
a fair bit of configuration but it can be done without any coding.

-Dave

-Original Message-
From: Ted Yu [mailto:yuzhih...@gmail.com]
Sent: Wednesday, August 8, 2018 9:18 PM
To: users@kafka.apache.org
Subject: [External] Re: Java API to read metrics via JMX

Boris:
BrokerWithJMX is referenced but I didn't find the class source after a brief 
search.

FYI

On Wed, Aug 8, 2018 at 7:10 PM Boris Lublinsky < boris.lublin...@lightbend.com> 
wrote:

> Its actually quite simple, unfortunately you have to read, and then
> write to TSDB.
> Enclosed is an example doing this and dumping to InfluxDB
>
>
> Boris Lublinsky
> FDP Architect
> boris.lublin...@lightbend.com
> https://www.lightbend.com/
>
> On Aug 8, 2018, at 8:46 PM, Raghav  wrote:
>
> Hi
>
> Is there any Java API available so that I can enable our Kafka
> cluster's JMX port, and consume metrics via JMX api, and dump to a
> time series database.
>
> I checked out jmxtrans, but currently it does not dump to TSDB (time
> series database).
>
> Thanks.
>
> R
>
>
>
This e-mail and any files transmitted with it are confidential, may contain 
sensitive information, and are intended solely for the use of the individual or 
entity to whom they are addressed. If you have received this e-mail in error, 
please notify the sender by reply e-mail immediately and destroy all copies of 
the e-mail and any attachments.

Kafka Streams - Merge vs. Join

[jheller]

Hi All,

  I am a little confused on the difference between the 
KStreamBuilder merge() function and doing a KStream-to-KStream Join operation. 
I understand the difference between Inner, Left and Outer joins, but I don't 
understand exactly what the difference is between the two. It appears to me 
that both ways would merge two streams into a single stream, but the joins do 
have the ability to remove duplicate data. Is that the only difference? Also, 
on a side note, I am really clueless as to what the difference between Windowed 
and Windowless means when referring to the joins.

  Any help would be greatly appreciated. Thank you.

John Heller

Re: Looking for help with a question on the consumer API

[Manoj Khangaonkar]

Hi,

Yes , if you don'nt call poll within the configured timeouts, the broker
thinks the consumer is gone.

But increasing the timeout is not a sustainable design.

In general the code in the consumer poll loop should be very fast and do
minimal work.

Any heavy duty work should be done by handing of to a thread pool.

What is happening is that the partitions for this consumer are assigned to
another.

If reading too many messages too fast becomes an issue, You can consider
using the pause and resume API to pause the consumer.

In some special cases, you might consider manually assigning partitions to
the consumer.


regards


On Wed, Aug 8, 2018 at 1:30 PM, Moiz Raja (moraja)  wrote:

> Hi All,
>
> I have an issue with the consumer getting kicked out of the group possibly
> due to other issues going on in the system. The issue is detailed here
> https://stackoverflow.com/questions/51754794/how-to-
> reinstate-a-kafka-consumer-which-has-been-kicked-out-of-the-group
>
> Any help with this issue would be appreciated.
>
> Regards,
> -Moiz
>



-- 
http://khangaonkar.blogspot.com/

Kafka Streams - Merge vs. Join

[jheller]

Hi All,



  I am a little confused on the difference between the 
KStreamBuilder merge() function and doing a KStream-to-KStream Join operation. 
I understand the difference between Inner, Left and Outer joins, but I don't 
understand exactly what the difference is between the two. It appears to me 
that both ways would merge two streams into a single stream, but the joins do 
have the ability to remove duplicate data. Is that the only difference? Also, 
on a side note, I am really clueless as to what the difference between Windowed 
and Windowless means when referring to the joins.



  Any help would be greatly appreciated. Thank you.



John Heller

Re: Kafka Streams - Merge vs. Join

[John Roesler]

Hi John,

Sorry for the confusion! I just noticed that we failed to document the
merge operator. I've created
https://issues.apache.org/jira/browse/KAFKA-7269 to fix it.

But in the mean time,
* merge: interleave the records from two streams to produce one collated
stream
* join: compute a new stream by fusing together records from the two inputs
by key

For example:
input-1:
(A, 1)
(B, 2)

input-2:
(A, 500)
(C, 60)

join( (l,r) -> new KeyValue(l, r) ):// simplified API
(A, (1, 500) )
(B, (2, null) )
(C, (null, 60) )

merge:
(A, 1)
(A, 500)
(B, 2)
(C, 60)


Does that make sense?
-John Roesler

On Thu, Aug 9, 2018 at 2:13 PM  wrote:

> Hi All,
>
>
>
>   I am a little confused on the difference between the
> KStreamBuilder merge() function and doing a KStream-to-KStream Join
> operation. I understand the difference between Inner, Left and Outer joins,
> but I don't understand exactly what the difference is between the two. It
> appears to me that both ways would merge two streams into a single stream,
> but the joins do have the ability to remove duplicate data. Is that the
> only difference? Also, on a side note, I am really clueless as to what the
> difference between Windowed and Windowless means when referring to the
> joins.
>
>
>
>   Any help would be greatly appreciated. Thank you.
>
>
>
> John Heller
>
>

Re: Looking for help with a question on the consumer API

[Moiz Raja]

Actually there is very minimal work done by the thread doing the polling. 
However it is possible that the thread doing the polling may not get schedule 
due to other things happening on the system - like a long GC pause.

From what you all are saying it sounds like a poll will do a heartbeat which 
will cause the consumer to be added back to the consumer group. If that is the 
case then I think I either have a problem with session timeout being hit or an 
issue with the thread doing the poll dying.

> On Aug 9, 2018, at 12:06 PM, Manoj Khangaonkar  wrote:
> 
> Hi,
> 
> Yes , if you don'nt call poll within the configured timeouts, the broker
> thinks the consumer is gone.
> 
> But increasing the timeout is not a sustainable design.
> 
> In general the code in the consumer poll loop should be very fast and do
> minimal work.
> 
> Any heavy duty work should be done by handing of to a thread pool.
> 
> What is happening is that the partitions for this consumer are assigned to
> another.
> 
> If reading too many messages too fast becomes an issue, You can consider
> using the pause and resume API to pause the consumer.
> 
> In some special cases, you might consider manually assigning partitions to
> the consumer.
> 
> 
> regards
> 
> 
> On Wed, Aug 8, 2018 at 1:30 PM, Moiz Raja (moraja) > wrote:
> 
>> Hi All,
>> 
>> I have an issue with the consumer getting kicked out of the group possibly
>> due to other issues going on in the system. The issue is detailed here
>> https://stackoverflow.com/questions/51754794/how-to-
>> reinstate-a-kafka-consumer-which-has-been-kicked-out-of-the-group
>> 
>> Any help with this issue would be appreciated.
>> 
>> Regards,
>> -Moiz
>> 
> 
> 
> 
> -- 
> http://khangaonkar.blogspot.com/

Re: Java API to read metrics via JMX

[Raghav]

Hi

I found
https://github.com/kafka-dev/kafka/blob/master/perf/src/main/java/kafka/perf/jmx/BrokerJmxClient.java
code
written by Neha to pull JMX metrics via MBean.

In here:
https://github.com/kafka-dev/kafka/blob/master/perf/src/main/java/kafka/perf/jmx/BrokerJmxClient.java#L37
there
is a mention of object name (kafka:type=kafka.SocketServerStats)  and
subsequently SocketServerStatsMBean.class.

My question is how to get these two for the latest Kafka 1.1.

I want to write code in Java to get Kafka Stats exposed via JMX and then
want to write to DB that our UI can read.

Thanks.

R

On Wed, Aug 8, 2018 at 6:46 PM, Raghav  wrote:

> Hi
>
> Is there any Java API available so that I can enable our Kafka cluster's
> JMX port, and consume metrics via JMX api, and dump to a time series
> database.
>
> I checked out jmxtrans, but currently it does not dump to TSDB (time
> series database).
>
> Thanks.
>
> R
>



-- 
Raghav

Re: Java API to read metrics via JMX

[Ishwor Gurung]

I don’t know of Java-based solution but I have successfully used:

Kafka JMX Exporter for Prometheus <———> Kafka

to collect JMX metrics from Kafka.

> On 10 Aug 2018, at 11:10 am, Raghav  wrote:
> 
> Hi
> 
> I found
> https://github.com/kafka-dev/kafka/blob/master/perf/src/main/java/kafka/perf/jmx/BrokerJmxClient.java
> code
> written by Neha to pull JMX metrics via MBean.
> 
> In here:
> https://github.com/kafka-dev/kafka/blob/master/perf/src/main/java/kafka/perf/jmx/BrokerJmxClient.java#L37
> there
> is a mention of object name (kafka:type=kafka.SocketServerStats)  and
> subsequently SocketServerStatsMBean.class.
> 
> My question is how to get these two for the latest Kafka 1.1.
> 
> I want to write code in Java to get Kafka Stats exposed via JMX and then
> want to write to DB that our UI can read.
> 
> Thanks.
> 
> R
> 
>> On Wed, Aug 8, 2018 at 6:46 PM, Raghav  wrote:
>> 
>> Hi
>> 
>> Is there any Java API available so that I can enable our Kafka cluster's
>> JMX port, and consume metrics via JMX api, and dump to a time series
>> database.
>> 
>> I checked out jmxtrans, but currently it does not dump to TSDB (time
>> series database).
>> 
>> Thanks.
>> 
>> R
>> 
> 
> 
> 
> -- 
> Raghav