Thanks Christian, Sending a heartbeat without having to poll would also be useful when using a large max.partition.fetch.bytes.
For now I'm just going to shut the consumer down and restart after x period of time. Thanks for your insights. Michael > On 10 Mar 2016, at 18:33, Christian Posta <christian.po...@gmail.com> wrote: > > Yah that's a good point. That was brought up in another thread. > > The granularity of what poll() needs to be addressed. It tries to do too > many things at once, including heartbeating. Not so sure that's entirely > necessary. > > On Thu, Mar 10, 2016 at 1:40 AM, Michael Freeman <mikfree...@gmail.com> > wrote: > >> Thanks Christian, >> We would want to retry indefinitely. Or at >> least for say x minutes. If we don't poll how do we keep the heart beat >> alive to Kafka. We never want to loose this message and only want to commit >> to Kafka when the message is in Mongo. That's either as a successful >> message in a collection or an unsuccessful message in an error collection. >> >> Right now I let the consumer die and don't create a new one for x minutes. >> This causes a lot of rebalancing. >> >> Michael >> >>>> On 9 Mar 2016, at 21:12, Christian Posta <christian.po...@gmail.com> >>> wrote: >>> >>> So can you have to decide how long you're willing to "wait" for the mongo >>> db to come back, and what you'd like to do with that message. So for >>> example, do you just retry inserting to Mongo for a predefined period of >>> time? Do you try forever? If you try forever, are you okay with the >>> consumer threads blocking indefinitely? Or maybe you implement a "circuit >>> breaker" to shed load to mongo? Or are you willing to stash the message >>> into a DLQ and move on and try the next message? >>> >>> You don't need to "re-consume" the message do you? Can you just retry >>> and/or backoff-retry with the message you have? And just do the "commit" >> of >>> the offset if successfully? >>> >>> >>> >>> On Wed, Mar 9, 2016 at 2:00 PM, Michael Freeman <mikfree...@gmail.com> >>> wrote: >>> >>>> Hey, >>>> My team is new to Kafka and we are using the examples found at. >> http://www.confluent.io/blog/tutorial-getting-started-with-the-new-apache-kafka-0.9-consumer-client >>>> >>>> We process messages from kafka and persist them to Mongo. >>>> If Mongo is unavailable we are wondering how we can re-consume the >> messages >>>> while we wait for Mongo to come back up. >>>> >>>> Right now we commit after the messages for each partition are processed >>>> (Following the example). >>>> I have tried a few approaches. >>>> >>>> 1. Catch the application exception and skip the kafka commit. However >> the >>>> next poll does not re consume the messages. >>>> 2. Allow the consumer to fail and restart the consumer. This works but >>>> causes a rebalance. >>>> >>>> Should I attempt to store the offset and parition (in memory) instead >> and >>>> attempt to reseek in order to re consume the messages? >>>> >>>> Whats the best practice approach in this kind of situation? My priority >> is >>>> to never loose a message and to ensure it makes it to Mongo. >> (Redelivery is >>>> ok) >>>> >>>> Thanks for any help or pointers in the right direction. >>>> >>>> Michael >>> >>> >>> >>> -- >>> *Christian Posta* >>> twitter: @christianposta >>> http://www.christianposta.com/blog >>> http://fabric8.io > > > > -- > *Christian Posta* > twitter: @christianposta > http://www.christianposta.com/blog > http://fabric8.io