Hi, Idempotence flag will guarantee that the message is produce exactly one time on the topic i.e that running your command a single time will produce a single message. It is not a unique enforcement on the message key, there is no such thing in Kafka.
In Kafka, a topic containing the "history" of values for a given key. That means that a consumer need to consume the whole topic and keep only the last value for a given key. So the uniqueness concept is mean to be done on the consumer side. Additionally to that, Kafka can perform log compaction to keep only the last value and preserve disk space (but consumers will still receive duplicates) Best On Wed, Apr 3, 2019 at 1:28 AM jim.me...@concept-solutions.com < jim.me...@concept-solutions.com> wrote: > > > On 2019/04/02 22:43:31, jim.me...@concept-solutions.com < > jim.me...@concept-solutions.com> wrote: > > > > > > On 2019/04/02 22:25:16, jim.me...@concept-solutions.com < > jim.me...@concept-solutions.com> wrote: > > > > > > > > > On 2019/04/02 21:59:21, Hans Jespersen <h...@confluent.io> wrote: > > > > yes. Idempotent publish uses a unique messageID to discard potential > duplicate messages caused by failure conditions when publishing. > > > > > > > > -hans > > > > > > > > > On Apr 1, 2019, at 9:49 PM, jim.me...@concept-solutions.com < > jim.me...@concept-solutions.com> wrote: > > > > > > > > > > Does Kafka have something that behaves like a unique key so a > producer can’t write the same value to a topic twice? > > > > > > > > > > Hi Hans, > > > > > > Is there some documentation or an example with source code where I > can learn more about this feature and how it is implemented? > > > > > > Thanks, > > > Jim > > > > > > > By the way I tried this... > > echo "key1:value1" | ~/kafka/bin/kafka-console-producer.sh > --broker-list localhost:9092 --topic TestTopic --property "parse.key=true" > --property "key.separator=:" --property "enable.idempotence=true" > > /dev/null > > > > And... that didn't seem to do the trick - after running that command > multiple times I did receive key1 value1 for as many times as I had run the > prior command. > > > > Maybe it is the way I am setting the flags... > > Recently I saw that someone did this... > > bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test > --producer-property enable.idempotence=true --request-required-acks -1 > > > > Also... the reason for my question is that we are going to have two JMS > topics with nearly redundant data in them have the UNION written to Kafka > for further processing. > >
