Hi Guozhang,
At first, thank you answer my question, and give me some suggest.But, I'm sure
I readed some introduction about kafka.
In my producer, My Code is( c code):res = rd_kafka_conf_set(kafka_conf,
"request.required.acks", "-1", NULL, 0);res = rd_kafka_topic_conf_set(
topic_conf, "produce.offset.report", "true", errstr, sizeof(errstr) );
In my consumer, My Code is(kafka-python):
self.__consumer = KafkaConsumer(
bootstrap_servers=self.__kafka_config["hosts"],
group_id=self.__kafka_config["group"],
auto_offset_reset="earliest",
)
self.__consumer.subscribe([self.__kafka_config["data_topic"]])
for message in self.__consumer:
I think these codes is common, What's your suggest about these codes?
In the end, I must explain: Lost message is not often, some time, couple days
can find one or two lost messages.But some day, maybe can find over 20 messages
were losted.Our message over 1,200,000 in one day.
So, do your have any suggest?Bye the way, can you speak Chinese? Thank you very
much & Best wishesJerry
--------------------------------
----- 原始邮件 -----
发件人：Guozhang Wang <[email protected]>
收件人："[email protected]" <[email protected]>, [email protected]
主题：Re: I found kafka lsot message
日期：2016年10月10日 01点25分
Jerry,
Message lost scenarios usually are due to producer and consumer mis-configured.
Have you read about the client configs web docs?
http://kafka.apache.org/documentation#producerconfigs
http://kafka.apache.org/documentation#newconsumerconfigs
If not I'd suggest you reading those first and see if you can tune some of
these configs to have better delivery guarantees.
Guozhang
On Fri, Oct 7, 2016 at 9:48 PM, <[email protected]> wrote:
Hello every body,I build a kafka cluster(include 5 domains) use
kafka_2.11-0.10.0.0 and kafka-python-1.3.1.I create a topic by 100 partitions
and 2 replicate, then use 20 consumers to receive message.
But, I found sometime the kafka lost message! I found some partition's offsite
lost at consumer.After, I make a global index for every message from producer
for confirm this problem, and I also found the global index had been break!
Why the kafka lost message? What can I do to fix the problem?
Thanks!Jerry
--------------------------------
--
-- Guozhang
