Re: Kafka SinkConnector for K-V store and Keys

2017-01-10 Thread Gwen Shapira
This may be a bit use-case dependent, but I think simply using the key from the Kafka record as the KV key is a good start. Another option is to use topic-partition-offset as the key. This has the benefit of removing duplicates, but it also means that keys are no longer meaningful for applications

Kafka SinkConnector for K-V store and Keys

2017-01-10 Thread Nilkanth Patel
Hello, I am writing a kafka sink connector for my product that is distributed table (Underneath a distribute K-V store where cluster of nodes are holding a different partitions/buckets of a table. This is based on Hash partitioned on Keys). When i write a SinkTask, I get the SinkRecord that conta