Re: how to identify rogue consumer

[Jun Rao]

Yes.

Thanks,

Jun

On Wed, Oct 8, 2014 at 10:53 PM, Steven Wu  wrote:

> Jun, you mean trace level logging for requestAppender?
> log4j.logger.kafka.network.Processor=TRACE, requestAppender
>
> if it happens again, I can try to enable it.
>
> On Wed, Oct 8, 2014 at 9:54 PM, Jun Rao  wrote:
>
> > If enabled request logging, you can find this out.
> >
> > Thanks,
> >
> > Jun
> >
> > On Wed, Oct 8, 2014 at 8:57 PM, Steven Wu  wrote:
> >
> > > I have seen very high "Fetch-Consumer-RequestsPerSec" (like 15K) per
> > broker
> > > in a relatively idle cluster. My hypothesis some misbehaving consumer
> > has a
> > > tight polling loop without any back-off logic with empty fetch.
> > >
> > > Unfortunately, this metric doesn't have per-topic breakdown like
> > > "BytesInPerSec" or "MessagesInPerSec". So I can't really tell which
> > > topic/consumer is pounding the cluster.
> > >
> > > Also the storm already ended. So I can't use tcpdump to capture live
> > > traffic.
> > >
> > > So any suggestion?
> > >
> >
>

Re: how to identify rogue consumer

[Steven Wu]

Jun, you mean trace level logging for requestAppender?
log4j.logger.kafka.network.Processor=TRACE, requestAppender

if it happens again, I can try to enable it.

On Wed, Oct 8, 2014 at 9:54 PM, Jun Rao  wrote:

> If enabled request logging, you can find this out.
>
> Thanks,
>
> Jun
>
> On Wed, Oct 8, 2014 at 8:57 PM, Steven Wu  wrote:
>
> > I have seen very high "Fetch-Consumer-RequestsPerSec" (like 15K) per
> broker
> > in a relatively idle cluster. My hypothesis some misbehaving consumer
> has a
> > tight polling loop without any back-off logic with empty fetch.
> >
> > Unfortunately, this metric doesn't have per-topic breakdown like
> > "BytesInPerSec" or "MessagesInPerSec". So I can't really tell which
> > topic/consumer is pounding the cluster.
> >
> > Also the storm already ended. So I can't use tcpdump to capture live
> > traffic.
> >
> > So any suggestion?
> >
>

Re: how to identify rogue consumer

[Jun Rao]

If enabled request logging, you can find this out.

Thanks,

Jun

On Wed, Oct 8, 2014 at 8:57 PM, Steven Wu  wrote:

> I have seen very high "Fetch-Consumer-RequestsPerSec" (like 15K) per broker
> in a relatively idle cluster. My hypothesis some misbehaving consumer has a
> tight polling loop without any back-off logic with empty fetch.
>
> Unfortunately, this metric doesn't have per-topic breakdown like
> "BytesInPerSec" or "MessagesInPerSec". So I can't really tell which
> topic/consumer is pounding the cluster.
>
> Also the storm already ended. So I can't use tcpdump to capture live
> traffic.
>
> So any suggestion?
>