Hi Hans, Thats a great answer compared to the paragraphs I read online! I am assuming you meant HDFS? what is JSDC ? Any idea on which is more common for this kind of use case? Also can I store offsets to zookeeper using ZAB instead of using external store? I am not sure how zookeeper stores data but I keep reading you can (perhaps zookeeper requires external storage?).
Thanks! On Wed, Dec 21, 2016 at 5:11 PM, Hans Jespersen <[email protected]> wrote: > Exactly once Kafka Sink Connectors typically store the offset externally > in the same atomic write as they store the messages. That way after a > crash, they can check the external store (HSFS, JSDC, etc) retrieve the > last committed offset and seek the the next message and continue processing > with no duplicates and exactly once semantics. > > -hans > > > > > > On Dec 21, 2016, at 4:39 PM, kant kodali <[email protected]> wrote: > > > > How does Kafka emulate exactly once processing currently? Does it require > > the producer to send at least once and consumer to de dupe? > > > > I did do my research but I feel like I am going all over the place so a > > simple short answer would be great! > > > > Thanks! > >
