Hello, I am using Kafka higher consumer 0.9.0. I am not using the auto commit for the offsets, so after I consume the messaged (poll from Kafka) I will have to commit the offsets manually.
The issue that I have is actually that the processing of the messages takes longer than 30s (and I cannot call poll again, before these messages are processed) and when I try to commit the offset a exception is thrown: ERROR o.a.k.c.c.i.ConsumerCoordinator - Error ILLEGAL_GENERATION occurred while committing offsets for group MetadataConsumerSpout. (I have found on stackoverflow this explanation: so if you wait for longer that the timeout request then the coordinator for the topic will kickout the consumer because it will think is dead and it will rebalance the group) In order to get rid of this I have thought about a couple of solutions: 1. The configuration session.timeout.ms has a maximum value, so if I try to set it to 60 seconds, also I get an exception, because this value is not in the valid interval. 2. I have tried to find a solution to get a paginated request when the polling method is called - no success. 3. I have tried to send a heart beat from the outside of the poll (because this method sends the heartbeats) - no success. Thank you. Vanessa.
