RE: Consumer poll stuck on

2019-03-21 Thread Manu Jacob
ffsets Partition: 9 Leader: 2 Replicas: 2,1 Isr: 1,2 Topic: __consumer_offsets Partition: 10 Leader: 1 Replicas: 1,2 Isr: 1,2 If any partitions is not in Isr, you can fix it. 1095193...@qq.com From: Manu Jacob Date: 2019-03-21 09:39 To: users@kafka.apache.org; d...@kafka.apache.org Subject: Consumer poll

Consumer poll stuck on

2019-03-20 Thread Manu Jacob
Hi, We have a Kafka cluster (version 1.1.1) where one node unexpectedly failed. After that consumers from a couple of consumers are stuck in the poll() API call. Looking at the thread dump, it looks like the consumer is stuck in org.apache.kafka.clients.consumer.internals.AbstractCoordinator.

Log retains beyond the set expiration time for some partitions.

2019-02-26 Thread Manu Jacob
Hi, I have a topic with 50 partitions. This is a development setup so we have 2 days log retention and 1 week offset retention. We had a situation where the consumer service was down for 6 days while the producer was still producing records to the topic. The consumer is configure to "latest" f

RE: Kafka connect FieldPartitioner with scheduled rotation

2019-02-05 Thread Manu Jacob
it is a defect or a genuine requirement. Thanks, -Manu From: Pere Urbón Bayes Sent: Monday, February 04, 2019 11:41 PM To: Manu Jacob Cc: users@kafka.apache.org Subject: Re: Kafka connect FieldPartitioner with scheduled rotation EXTERNAL Hi Manu, if we take master of the connector, I

RE: Kafka connect FieldPartitioner with scheduled rotation

2019-02-03 Thread Manu Jacob
tor config as well the exception you are getting? with only this info, I do need more details to understand your issue. Keep in mind the option you are using "rotate.schedule.interval.ms" from the docs says: > This configuration is useful when you have to commit your data based &

Kafka connect FieldPartitioner with scheduled rotation

2019-02-03 Thread Manu Jacob
Hi, I want to use s3 connect by portioning with FieldPartitioner and partition.field.name set to a non timestamp based field. I want to commit and flush based on both size and time. I am getting an exception when I use the option "rotate.schedule.interval.ms". Is it possible to rotate it with

Single consumer subscribing to multiple topics

2019-01-23 Thread Manu Jacob
Hi Experts, I have a requirement to consume records from multiple topics and process those messages with a similar pattern. The schema of the messages in each topic is different and I am consuming it as Generic records. Both these topics have very high rate of traffic (10-15K messages/second) a