I think you have it backwards. If you don't write your consumer offsets, the worst case is that consumers will read some messages a second time. If your messages are idempotent, then you wont lose or corrupt any data. When the ZK cluster comes back up you can start writing offsets again.
However, if your ZK cluster is down, I don't think the brokers will accept new messages from the producers. So your producers could end up losing data during a long outage if you don't have enough memory to buffer and retry indefinitely. You could write those messages to disk or S3, etc. and replay them when the brokers recover. On Fri, Jun 26, 2015 at 6:11 AM bit1...@163.com <bit1...@163.com> wrote: > Can someone explain this ? Thanks! > > > > bit1...@163.com > > From: bit1...@163.com > Date: 2015-06-25 11:57 > To: users > Subject: Message loss due to zookeeper ensemble doesn't work > Hi, > > I have the offset saved in zookeeper. Because zookeeper quorum doesn't > work for a short time(leader is down and new leader election).Then there is > a chance that the offset doesn't write to the Zookeeper, which will lose > data. > > I would ask whether Kafka provides some mechism for this kind of issue. > > > > bit1...@163.com >
