Re: Long start time for consumer

2018-05-29 Thread Shantanu Deshmukh
Thanks for your suggestion. However, this doesn't seem applicable for our Kafka version. We are using 0.10.0.1 On Tue, May 29, 2018 at 7:04 PM Manikumar wrote: > Pls check "group.initial.rebalance.delay.ms" broker config property. This > will be the delay for the initial consumer rebalance. > >

Re: Long start time for consumer

2018-05-29 Thread Manikumar
Pls check "group.initial.rebalance.delay.ms" broker config property. This will be the delay for the initial consumer rebalance. from docs "The rebalance will be further delayed by the value of group.initial.rebalance.delay.ms as new members join the group, up to a maximum of max.poll.interval.m

Re: Long start time for consumer

2018-05-29 Thread Shantanu Deshmukh
I cannot because there are messages which need high priority. Setting poll interval to 4 second means there might be delay of 4 seconds + regular processing time, which is not desirable. Also, will it impact heartbeating? On Tue, May 29, 2018 at 6:17 PM M. Manna wrote: > Have you tried increase

Re: Long start time for consumer

2018-05-29 Thread Shantanu Deshmukh
No, no dynamic topic creation. On Tue, May 29, 2018 at 6:38 PM Jaikiran Pai wrote: > Are your topics dynamically created? If so, see this > threadhttps://www.mail-archive.com/dev@kafka.apache.org/msg67224.html > > -Jaikiran > > > On 29/05/18 5:21 PM, Shantanu Deshmukh wrote: > > Hello, > > > > W

Re: Long start time for consumer

2018-05-29 Thread Jaikiran Pai
Are your topics dynamically created? If so, see this threadhttps://www.mail-archive.com/dev@kafka.apache.org/msg67224.html -Jaikiran On 29/05/18 5:21 PM, Shantanu Deshmukh wrote: Hello, We have 3 broker Kafka 0.10.0.1 cluster. We have 5 topics, each with 10 partitions. I have an application

Re: Long start time for consumer

2018-05-29 Thread M. Manna
Have you tried increase the poll time higher, e.g. 4000 and see if that helps matters? On 29 May 2018 at 13:44, Shantanu Deshmukh wrote: > Here is the code which consuming messages > > > while(true && startShutdown == false) { > Context context = new Context(); > JSONObject noti

Re: Long start time for consumer

2018-05-29 Thread Shantanu Deshmukh
Here is the code which consuming messages while(true && startShutdown == false) { Context context = new Context(); JSONObject notifJSON = new JSONObject(); String notificationMsg = ""; NotificationEvent notifEvent = null; initializeContext(); try { consumer

Re: Long start time for consumer

2018-05-29 Thread M. Manna
Thanks.. Where is your consumer code that is consuming messages? On 29 May 2018 at 13:18, Shantanu Deshmukh wrote: > No problem, here are consumer properties > - > auto.commit.interval.ms = 3000 > auto.offset.reset = latest > bootstrap.servers = [x.x.x.x:9092, x.x.x.x:9092, x.x.x.x:9092

Re: Long start time for consumer

2018-05-29 Thread Shantanu Deshmukh
No problem, here are consumer properties - auto.commit.interval.ms = 3000 auto.offset.reset = latest bootstrap.servers = [x.x.x.x:9092, x.x.x.x:9092, x.x.x.x:9092] check.crcs = true client.id = connections.max.idle.ms = 54 enable.auto.commit = true exclude.internal.topics = true fetch.m

Re: Long start time for consumer

2018-05-29 Thread M. Manna
Hi, It's not possible to answer questions based on text. You need to share your consumer.properties, and server.properties file, and also, what exactly you have changed from default configuration. On 29 May 2018 at 12:51, Shantanu Deshmukh wrote: > Hello, > > We have 3 broker Kafka 0.10.0.1 c

Long start time for consumer

2018-05-29 Thread Shantanu Deshmukh
Hello, We have 3 broker Kafka 0.10.0.1 cluster. We have 5 topics, each with 10 partitions. I have an application which consumes from all these topics by creating multiple consumer processes. All of these consumers are under a same consumer group. I am noticing that every time we restart this appli