Hi Prabhjot The messages are "Thread1_kafka_1" and "Thread2_kafka_1". Something like that.
For GetOffsetShell report below: [kafka@dn-01 bin]$ ./kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list dn-01:9092 --time -1 --topic kafka-test kafka-test:0:12529261 @Jinxing Can you share your flush example to me? How to avoid lost my messages? Thanks. Best regards Hawin On Thu, Nov 12, 2015 at 1:00 AM, jinxing <jinxing6...@126.com> wrote: > there is a flush api of the producer, you can call this to prevent > messages lost; > > > maybe it can help; > > > > > > > > > At 2015-11-12 16:43:54, "Hawin Jiang" <hawin.ji...@gmail.com> wrote: > >Hi Jinxing > > > >I don't think we can resolve this issue by increasing producers. if I > >increased more producers, it should lost more messages. > > > >I just test two producers. > >Thread Producer 1 has 83064 messages in producer side and 82273 messages > in > >consumer side > >Thread Producer 2 has 89844 messages in producer side and 88892 messages > in > >consumer side. > > > >Thanks. > > > > > > > >Best regards > >Hawin > > > > > >On Thu, Nov 12, 2015 at 12:24 AM, jinxing <jinxing6...@126.com> wrote: > > > >> maybe there some changes in 0.9.0.0; > >> > >> > >> but still you can try increase producer sending rate, and see if there > are > >> message lost but no exception; > >> > >> > >> note that, to increase the producer sending rate, you must have enough > >> producer 'power'; > >> > >> > >> in my case, I have 50 producer sending message at the same time : ) > >> > >> > >> > >> > >> > >> > >> > >> At 2015-11-12 16:16:23, "Hawin Jiang" <hawin.ji...@gmail.com> wrote: > >> >Hi Jinxing > >> > > >> >I am using kafka_2.10-0.9.0.0-SNAPSHOT. I have downloaded source code > and > >> >installed it last week. > >> > > >> >I saw 97446 messages have been sent to kafka successfully. > >> > > >> >So far, I have not found any failed messages. > >> > > >> > > >> > > >> >Best regards > >> >Hawin > >> > > >> >On Thu, Nov 12, 2015 at 12:07 AM, jinxing <jinxing6...@126.com> wrote: > >> > > >> >> Hi, what version are you using ? > >> >> > >> >> > >> >> i am using 0.8.2.0, and I encountered this problem before; > >> >> > >> >> > >> >> it seems that if the message rate of the producer side is to high, > some > >> of > >> >> the messages will lost; > >> >> > >> >> > >> >> also, i found that the callback method of the producer 'send' API is > not > >> >> reliable; > >> >> > >> >> > >> >> only successful sent message will trigger the callback, but the > failed > >> >> ones don't; > >> >> > >> >> > >> >> you saw this? > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> At 2015-11-12 16:01:17, "Hawin Jiang" <hawin.ji...@gmail.com> wrote: > >> >> >Hi All > >> >> > > >> >> >I have sent messages to Kafka for one minute. I found 97446 > messages > >> in > >> >> >producer side and 96896 messages in consumer side for Case 1. > >> >> >I also tried case 2. I have faced the same issues. The number is > not > >> >> match > >> >> >between producer and consumer. > >> >> >Can someone take a look at this issue? > >> >> >Thanks. > >> >> > > >> >> > > >> >> >Case 1: > >> >> > > >> >> >long startTime = System.currentTimeMillis(); > >> >> >long maxDurationInMilliseconds = 1 * 60 * 1000; > >> >> >int messageNo = 0; > >> >> >while (true) { > >> >> >if (System.currentTimeMillis() <= startTime > >> >> >+ maxDurationInMilliseconds) { > >> >> >messageNo++; > >> >> >String messageStr = "KAFKA_"+messageNo; > >> >> >System.out.println("Message: "+messageNo); > >> >> >producer.send(new KeyedMessage<Integer, String>(topic,messageStr)); > >> >> >} else { > >> >> >producer.close(); > >> >> >System.out.println("Total kafka Message: "+messageNo); > >> >> >break; > >> >> >} > >> >> >} > >> >> > > >> >> > > >> >> >Case 2: > >> >> > > >> >> >for (int i=1;i<=12000;i++) > >> >> >String messageStr = "KAFKA_"+i; > >> >> >producer.send(new KeyedMessage<Integer, String>(topic,messageStr)); > >> >> > > >> >> > > >> >> > > >> >> >Best regards > >> >> >Hawin > >> >> > >> >