Re: Consumer has high network bandwidth when consuming no messages

[Kirk True]

Hi Lee,

Some questions:

1. Can you enable metrics for your consumer(s)? See [1] and [2] for links on 
consumer-level and fetch-level metrics, respectively.
2. Can you enable more detailed logging?
3. What do you mean specifically by “concurrency of 2?”
4. Can you verify the name of the fetch.wait.max.ms configuration? I don’t see 
that configuration option. Is that a broker configuration or a consumer 
configuration?

Thanks,
Kirk

[1] https://kafka.apache.org/documentation/#consumer_group_monitoring
[2] https://kafka.apache.org/documentation/#consumer_fetch_monitoring

> On Apr 13, 2023, at 4:52 AM, Lee Carroll 
>  wrote:
> 
> Hi All,
> I've a spring boot kafka consumer group, consuming a topic with 30
> partitions with a concurrency of 2 (staging set up).
> 
> The network traffic in and out was very high  >200k per second even when
> consuming no messages.
> 
> I've adjusted fetch.wait.max.ms to be significantly higher (15,000) than
> the default which has had a significant impact but the network is still
> seeing ~30k a second.
> 
> Is this to be expected?
> 
> Cheers Lee C

Garbage collection suggestions?

[Doug Whitfield]

Are there any garbage collection suggestions specific to Kafka or are the 
standard java suggestions good?


This e-mail may contain information that is privileged or confidential. If you 
are not the intended recipient, please delete the e-mail and any attachments 
and notify us immediately.

Consumer has high network bandwidth when consuming no messages

[Lee Carroll]

Hi All,
I've a spring boot kafka consumer group, consuming a topic with 30
partitions with a concurrency of 2 (staging set up).

The network traffic in and out was very high  >200k per second even when
consuming no messages.

I've adjusted fetch.wait.max.ms to be significantly higher (15,000) than
the default which has had a significant impact but the network is still
seeing ~30k a second.

Is this to be expected?

Cheers Lee C