Re: Some of the Kafka consumers are not getting the messages from the broker

2019-11-07 Thread Karts
unsubscribe On Wed, Aug 21, 2019 at 2:23 AM sampath kumar wrote: > Hi, > > Using Broker 5.3.0, new consumers(Consumers managed by brokers). Brokers > are deployed in a Kubernetes environment > > Number of brokers : 3, Number of 3 Zookeeper setup > > One of the Topic "inventory.request" we have 3

Re: Some of the Kafka consumers are not getting the messages from the broker

2019-08-21 Thread Lisheng Wang
Hi Sampath Maybe need to check log to see if can find any clue, hard to say why those happened. Best, Lisheng sampath kumar 于2019年8月22日周四 上午2:24写道: > Hi Lisheng, > > I guess the issue is not with message.max.bytes, same messages consumer > after just running rebalancing. > > Regards, > Sampat

Re: Some of the Kafka consumers are not getting the messages from the broker

2019-08-21 Thread sampath kumar
Hi Lisheng, I guess the issue is not with message.max.bytes, same messages consumer after just running rebalancing. Regards, Sampath On Wed, Aug 21, 2019 at 7:57 PM Lisheng Wang wrote: > Hi Sampath > > the description of fetch.max.bytes is following from > https://kafka.apache.org/documentatio

Re: Some of the Kafka consumers are not getting the messages from the broker

2019-08-21 Thread Lisheng Wang
Hi Sampath the description of fetch.max.bytes is following from https://kafka.apache.org/documentation/#consumerconfigs The maximum amount of data the server should return for a fetch request. Records are fetched in batches by the consumer, and if the first record batch in the first non-empty par

Re: Some of the Kafka consumers are not getting the messages from the broker

2019-08-21 Thread sampath kumar
Lisheng , Issue not with fetch max bytes as same message start processing after restarting the consumer Regards, Sampath On Wed, Aug 21, 2019 at 4:30 PM Lisheng Wang wrote: > Hi Sampath > > Can you confirm that "fetch.max.bytes" on consumer is not smaller than > "message.max.bytes" on broker?

Re: Some of the Kafka consumers are not getting the messages from the broker

2019-08-21 Thread Lisheng Wang
Hi Sampath Can you confirm that "fetch.max.bytes" on consumer is not smaller than "message.max.bytes" on broker? Maybe need you check consumer log to see if can find any clue once you enable it. if no any error/exception found on consumer side, maybe need change log level to "debug" to get more d

Re: Some of the Kafka consumers are not getting the messages from the broker

2019-08-21 Thread sampath kumar
Hi Lisheng, Thanks for the response. Right now we have enabled info in the broker However logs not enabled for the consumer client will enable it. Yes, when we manually stop and start the consumer in affected microservice instance rebalance triggers and consuming resumes. And in Broker side con

Re: Some of the Kafka consumers are not getting the messages from the broker

2019-08-21 Thread Lisheng Wang
May i know what log level did you configured on consumer and broker? you say it will resume when rebalance happen, so consumer is alive, can you see any heartbeat information in consumer log? Best, Lisheng sampath kumar 于2019年8月21日周三 下午5:23写道: > Hi, > > Using Broker 5.3.0, new consumers(Cons

Some of the Kafka consumers are not getting the messages from the broker

2019-08-21 Thread sampath kumar
Hi, Using Broker 5.3.0, new consumers(Consumers managed by brokers). Brokers are deployed in a Kubernetes environment Number of brokers : 3, Number of 3 Zookeeper setup One of the Topic "inventory.request" we have 3 replication, with insync replicas configured as 2 and partition count is 1024 W