Re: How to handle RebalanceInProgressException?

2020-05-11 Thread Benoit Delbosc
On 29.04.20 09:18, Benoit Delbosc wrote: >> 2) So in Kafka 2.0 we introduced poll(Duration) which would practically be >> more strict in respecting the passed in timeout. It means, it could return >> while we are still in the middle of a rebalance. At the same time we >

Re: How to handle RebalanceInProgressException?

2020-04-30 Thread Benoit Delbosc
On 29.04.20 09:18, Benoit Delbosc wrote: > Hi > > On 28.04.20 21:05, Guozhang Wang wrote: >> Thanks for the explanation Ben. They are very helpful. >> >> Just to clarify on the context here: >> >> 1) Before Kafka 2.0 the poll(long) call make sure that the r

Re: How to handle RebalanceInProgressException?

2020-04-29 Thread Benoit Delbosc
ocation during an applicative checkpoint. This should continue to work, assuming that after a `poll(Duration)` call that doesn't trigger any ConsumerRebalanceListener callbacks, `commitSync` must not raise a RebalanceInProgress exception. Am I right? Thank you, I appreciate your very precise a

Re: How to handle RebalanceInProgressException?

2020-04-28 Thread Benoit Delbosc
would suggest to improve the onPartitionsAssigned Javadoc or poll(Duration) to make it clear, may be part of KAFKA-9882. Regards ben > > Guozhang > > > On Mon, Apr 27, 2020 at 12:45 AM Benoit Delbosc wrote: > >> Hi, >> >> I am having a hard time un

How to handle RebalanceInProgressException?

2020-04-27 Thread Benoit Delbosc
Hi, I am having a hard time understanding why a Consumer#commitSync can randomly fail. Not being able to commit creates duplicates, which I can understand in failure cases: - the poll interval is not respected max.poll.interval.ms (300s) - a consumer is stuck or die -> no heart beat during sess