Hi Hans,

Thats a great answer compared to the paragraphs I read online! I am
assuming you meant HDFS? what is JSDC ? Any idea on which is more common
for this kind of use case? Also can I store offsets to zookeeper using ZAB
instead of using external store? I am not sure how zookeeper stores data
but I keep reading you can (perhaps zookeeper requires external storage?).

Thanks!

On Wed, Dec 21, 2016 at 5:11 PM, Hans Jespersen <[email protected]> wrote:

> Exactly once Kafka Sink Connectors typically store the offset externally
> in the same atomic write as they store the messages. That way after a
> crash, they can check the external store (HSFS, JSDC, etc) retrieve the
> last committed offset and seek the the next message and continue processing
> with no duplicates and exactly once semantics.
>
> -hans
>
>
>
>
> > On Dec 21, 2016, at 4:39 PM, kant kodali <[email protected]> wrote:
> >
> > How does Kafka emulate exactly once processing currently? Does it require
> > the producer to send at least once and consumer to de dupe?
> >
> > I did do my research but I feel like I am going all over the place so a
> > simple short answer would be great!
> >
> > Thanks!
>
>

Reply via email to