Thanks Adam for your response. I will have a mechanism to handle duplicates on the service consuming the messages. Just curious, if there is a way to identify the cause for receiving duplicates. I mean any log file that could help with this?
Regards, Kris On Wed, Jun 17, 2015 at 8:24 AM, Adam Shannon <adam.shan...@banno.com> wrote: > This is actually an expected consequence of using distributed systems. The > kafka FAQ has a good answer > > > https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-HowdoIgetexactly-oncemessagingfromKafka > ? > > On Tue, Jun 16, 2015 at 11:06 PM, Kris K <squareksc...@gmail.com> wrote: > > > Hi, > > > > While testing message delivery using kafka, I realized that few duplicate > > messages got delivered by the consumers in the same consumer group (two > > consumers got the same message with few milli-seconds difference). > However, > > I do not see any redundancy at the producer or broker. One more > observation > > is that - this is not happening when I use only one consumer thread. > > > > I am running 3 brokers (0.8.2.1) with 3 Zookeeper nodes. There are 3 > > partitions in the topic and replication-factor is 3. For producing, am > > using New Producer with compression.type=none. > > > > On the consumer end, I have 3 High level consumers in the same consumer > > group running with one consumer thread each, on three different hosts. > Auto > > commit is set to true for consumer. > > > > Size of each message would range anywhere between 0.7 KB and 2 MB. The > max > > volume for this test is 100 messages/hr. > > > > I looked at controller log for any possibility of consumer rebalance > during > > this time, but did not find any. In the server log of all the brokers the > > error - java.io.IOException: Connection reset by peer is almost being > > written continuously. > > > > So, is it possible to achieve exactly-once delivery with the current high > > level consumer without needing an extra layer to remove redundancy? > > > > Could you please point me to any settings or logs that would help me tune > > the configuration ? > > > > *PS: I tried searching for similar discussions, but could not find any. > If > > its already been answered, please provide the link. > > > > Thanks, > > Kris > > > > > > -- > Adam Shannon | Software Engineer | Banno | Jack Henry > 206 6th Ave Suite 1020 | Des Moines, IA 50309 | Cell: 515.867.8337 >
