James Thornton created KAFKA-1436:
-------------------------------------
Summary: Idempotent Producer / Duplicate Detection
Key: KAFKA-1436
URL: https://issues.apache.org/jira/browse/KAFKA-1436
Project: Kafka
Issue Type: Improvement
Components: consumer, producer
Affects Versions: 0.8.0, 0.8.1, 0.8.2, 0.9.0, 0.8.1.1
Reporter: James Thornton
Assignee: Neha Narkhede
Dealing with duplicate messages is one of the major issues for teams using
Kafka, and Jay Kreps posted a page on implementing an Idempotent Producer to
address this issue:
https://cwiki.apache.org/confluence/display/KAFKA/Idempotent+Producer
MapDB 1.0 (https://github.com/jankotek/MapDB) was just released, and either it
or Java Chronicle (https://github.com/OpenHFT/Java-Chronicle/) could be
embedded within each broker to provide a high-performance, random-access,
off-heap store for request IDs.
As Jay points out in his post, global unique request IDs probably aren't
needed, but if that need should arise, Twitter's Snowflake service
(https://github.com/twitter/snowflake/) might be useful.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
