Hi Vanessa, The workaround we use in this case is closing the consumer and then opening it again when required. In addition you can increase the heartbeat timeout (heartbeat.interval.ms) - see more details in the documentaton.
Thanks, Ofra 2016-03-02 16:09 GMT+02:00 Gligor Vanessa <gligor.vane...@gmail.com>: > Hello, > > I am using Kafka higher consumer 0.9.0. I am not using the auto commit for > the offsets, so after I consume the messaged (poll from Kafka) I will have > to commit the offsets manually. > > The issue that I have is actually that the processing of the messages takes > longer than 30s (and I cannot call poll again, before these messages are > processed) and when I try to commit the offset a exception is thrown: > ERROR o.a.k.c.c.i.ConsumerCoordinator - Error ILLEGAL_GENERATION occurred > while committing offsets for group MetadataConsumerSpout. > (I have found on stackoverflow this explanation: so if you wait for longer > that the timeout request then the coordinator for the topic will kickout > the consumer because it will think is dead and it will rebalance the group) > > In order to get rid of this I have thought about a couple of solutions: > > 1. The configuration session.timeout.ms has a maximum value, so if I try > to > set it to 60 seconds, also I get an exception, because this value is not in > the valid interval. > > 2. I have tried to find a solution to get a paginated request when the > polling method is called - no success. > > 3. I have tried to send a heart beat from the outside of the poll (because > this method sends the heartbeats) - no success. > > > Thank you. > > Vanessa. >
