I meant to say HDFS and JDBC which are two of the Confluent Sink Connectors that do exactly once processing.
The Elasticsearch connector also has exactly once semantics but in a very different way. The Elastic API is idempotent if you update the same record multiple times using the same documentID so the Kafka connector actually can update multiple times without causing duplicates in ES. -hans > On Dec 21, 2016, at 6:12 PM, kant kodali <[email protected]> wrote: > > Hi Hans, > > Thats a great answer compared to the paragraphs I read online! I am > assuming you meant HDFS? what is JSDC ? Any idea on which is more common > for this kind of use case? Also can I store offsets to zookeeper using ZAB > instead of using external store? I am not sure how zookeeper stores data > but I keep reading you can (perhaps zookeeper requires external storage?). > > Thanks! > > On Wed, Dec 21, 2016 at 5:11 PM, Hans Jespersen <[email protected]> wrote: > >> Exactly once Kafka Sink Connectors typically store the offset externally >> in the same atomic write as they store the messages. That way after a >> crash, they can check the external store (HSFS, JSDC, etc) retrieve the >> last committed offset and seek the the next message and continue processing >> with no duplicates and exactly once semantics. >> >> -hans >> >> >> >> >>> On Dec 21, 2016, at 4:39 PM, kant kodali <[email protected]> wrote: >>> >>> How does Kafka emulate exactly once processing currently? Does it require >>> the producer to send at least once and consumer to de dupe? >>> >>> I did do my research but I feel like I am going all over the place so a >>> simple short answer would be great! >>> >>> Thanks! >> >>
