Hi Florian, This may be helpful https://github.com/omkreddy/kafka-examples/blob/master/consumer/src/main/java/kafka/examples/consumer/advanced/AdvancedConsumer.java
--Kamal On Fri, Apr 15, 2016 at 2:57 AM, Jason Gustafson <ja...@confluent.io> wrote: > Hi Florian, > > It's actually OK if processing takes longer than the heartbeat interval, > but it does need to finish before the session timeout expires or the > consumer will be kicked out of the group (which typically is revealed by > commit failures). If the problem is just that the consumer is handling too > many messages at once, then Kafka 0.10 has an option to tune the number of > messages returned from poll() (max.poll.records), which may be helpful. We > also have a pause/resume API which allows you to call poll() without > consuming any data. That's the best option at the moment for 0.9 consumers. > > For what it's worth, we've considered several times adding a heartbeat() > API, but the challenge is figuring out how to handle rebalancing. > Underneath the covers, we use heartbeats to find out when the group is > rebalancing, so a heartbeat() option would probably have to return a flag > indicating whether a rebalance was needed. If the group has begun > rebalancing, then you would need to call poll() before the expiration of > the session timeout so that the consumer can join the rebalance. > Alternatively, we could let heartbeat() complete the rebalance itself, but > then you'd have to be prepared to abort processing from the rebalance > callback. That's not really different from calling poll() after pausing > partitions though. The main problem in any case is that once a rebalance > begins, you have the duration of the session timeout to stop processing and > join the rebalance. We're seeing this problem pop up pretty much everywhere > that the consumer is used, so we're trying to think of some better options > to handle it. > > Thanks, > Jason > > > On Thu, Apr 14, 2016 at 12:32 PM, Florian Hussonnois < > fhussonn...@gmail.com> > wrote: > > > Hi everyone, > > > > I have a use case where a message can take longer than ' > > heartbeat.interval.ms' to be processed by my application. As I > understand > > the heartbeats of consumer are done while the poll method is invoked. > > > > I would like to instantiate a worker thread to process the messages but I > > need to wait for the messages completion before polling again. > > > > Is there a way to force the consumer to make an heartbeat without polling > > new messages ? > > > > Thanks, > > > > -- > > Florian HUSSONNOIS > > >
