Hi Jun, I created KAFKA-346 and added some comments to it (to move discussion to JIRA).
Greetings Peter 2012/5/17 Jun Rao <[email protected]>: > Peter, > > We probably shouldn't commit offsets automatically during rebalance, if > auto offset commit is turned off. The caveat is that during a rebalance, > all message since the last manual commit point will be redelivered to the > consumer. Could you create a jira and maybe submit a patch for this? > > Thanks, > > Jun > > On Thu, May 17, 2012 at 8:12 AM, Peter Romianowski > <[email protected]>wrote: > >> Hi, >> >> should have found a better subject... >> >> If I read the source correctly then offsets can be committed at any >> time (whenever there is a change in consumer or broker zk registry). >> Our application doesn't use auto-commit in order to batch some >> messages together, process them and if everything went fine, we call >> commitOffsets(). If, for any reason, the processing of messages does >> not succeed, we rely on Kafka's promise to re-deliver the messages. >> >> But if ZKRebalancerListener triggers a rebalance before our "batch" of >> messages is full, then offsets will be committed even if the messages >> have not been processed yet by our application. So if then processing >> of these messages fails, we basically lost them, right? >> >> Are there any recipes how to handle this? I really would like to stick >> to ZookeeperConsumerConnector, so switching to SimpleConsumer would be >> considered as "last resort". >> >> Regards >> >> Peter >> >> -- >> 404 Signature Not Found >> -- --- 404 Signature Not Found
