In fact, KIP-41 has been implemented in trunk -- see https://issues.apache.org/jira/browse/KAFKA-3007. Testing against a version including that change would be greatly appreciated to ensure it fully addresses the problems you're seeing.
-Ewen On Wed, Mar 2, 2016 at 7:00 AM, Olson,Andrew <aols...@cerner.com> wrote: > This topic is currently being discussed at > https://issues.apache.org/jira/browse/KAFKA-2986 and > https://cwiki.apache.org/confluence/display/KAFKA/KIP-41%3A+KafkaConsumer+Max+Records > > > On 3/2/16, 8:11 AM, "Vanessa Gligor" <vanessagli...@gmail.com> wrote: > > >Hello, > > > >I am using Kafka higher consumer 0.9.0. I am not using the auto commit for > >the offsets, so after I consume the messaged (poll from Kafka) I will have > >to commit the offsets manually. > > > >The issue that I have is actually that the processing of the messages > takes > >longer than 30s (and I cannot call poll again, before these messages are > >processed) and when I try to commit the offset a exception is thrown: > >ERROR o.a.k.c.c.i.ConsumerCoordinator - Error ILLEGAL_GENERATION occurred > >while committing offsets for group MetadataConsumerSpout. > >(I have found on stackoverflow this explanation: so if you wait for longer > >that the timeout request then the coordinator for the topic will kickout > >the consumer because it will think is dead and it will rebalance the > group) > > > >In order to get rid of this I have thought about a couple of solutions: > > > >1. The configuration session.timeout.ms has a maximum value, so if I try > to > >set it to 60 seconds, also I get an exception, because this value is not > in > >the valid interval. > > > >2. I have tried to find a solution to get a paginated request when the > >polling method is called - no success. > > > >3. I have tried to send a heart beat from the outside of the poll (because > >this method sends the heartbeats) - no success. > > > > > >Thank you. > > CONFIDENTIALITY NOTICE This message and any included attachments are from > Cerner Corporation and are intended only for the addressee. The information > contained in this message is confidential and may constitute inside or > non-public information under international, federal, or state securities > laws. Unauthorized forwarding, printing, copying, distribution, or use of > such information is strictly prohibited and may be unlawful. If you are not > the addressee, please promptly delete this message and notify the sender of > the delivery error by e-mail or you may call Cerner's corporate offices in > Kansas City, Missouri, U.S.A at (+1) (816)221-1024. > -- Thanks, Ewen
