Re: Durably storing messages in Kafka

2014-07-16 Thread Guozhang Wang
Hi Daniel, Your understanding is mostly correct, a few notes to add: 1. fsync is done in a backend thread asynchronously, so event setting log.flush.interval.messages = 1 will not guarantee that when the producer request ack back, the messages is definitely on the disk now. But since you are

Durably storing messages in Kafka

2014-07-15 Thread Daniel Compton
I think I know the answer to this already but I wanted to check my assumptions before proceeding. We are using Kafka as a queueing mechanism for receiving messages from stateless producers. We are operating in a legal framework where we can never lose a committed message, but we can reject a