I'm seeing similar issues with 0.9.0.1. I'm feeding CSV records (65536 total, 1 record per msg) to the console producer, which are consumed via a sink connector (using connect-standalone and a single partition). The sink occasionally reports flushing less than 65536 msgs via the sink flush(). Restarting the sink connector with a forced reset to offset 0 (ie, replaying all the msgs on the topic) shows that the messages are still missing (ie, no gaps in offsets), so I assume the msgs must be lost by the producer ?
On Wed, Jun 15, 2016 at 1:29 AM, Radu Radutiu <rradu...@gmail.com> wrote: > Hi, > > I was following the Quickstart guide and I have noticed that > ConsoleProducer does not publish all messages (the number of messages > published differs from one run to another) and happens mostly on a fresh > started broker. > version: kafka_2.11-0.10.0.0 > OS: Linux (Ubuntu 14.04, Centos 7.2) > JDK: java version "1.7.0_101" > OpenJDK Runtime Environment (IcedTea 2.6.6) (7u101-2.6.6-0ubuntu0.14.04.1), > openjdk version "1.8.0_91" > OpenJDK Runtime Environment (build 1.8.0_91-b14) > > > How to reproduce: > - start zookeeper: > ~/work/kafka_2.11-0.10.0.0$ bin/zookeeper-server-start.sh > config/zookeeper.properties & > > -start kafka: > ~/work/kafka_2.11-0.10.0.0$ bin/kafka-server-start.sh > config/server.properties & > > -start console consumer (topic test1 is already created): > ~/work/kafka_2.11-0.10.0.0$ bin/kafka-console-consumer.sh > --bootstrap-server localhost:9092 -topic test1 --zookeeper localhost:2181 > > -in another terminal start console producer with the LICENSE file in kafka > directory as input: > ~/work/kafka_2.11-0.10.0.0$ bin/kafka-console-producer.sh --topic test1 > --broker-list localhost:9092 <LICENSE > > The last line in the console consumer output is not the last line in the > LICENSE file for the first few runs of the console producer. If I use the > --old-producer parameter, all the lines in the LICENSE file are published > (and appear in the console consumer output). Different runs of console > producer with the same input file publish different number of lines > (sometimes all, sometimes only 182 lines out of 330). I've noticed that if > the kafka server was started a long time ago the console producer publishes > all lines. > I have checked the kafka binary log file (in my case > /tmp/kafka-logs/test1-0/00000000000000000000.log ) and confirmed that the > messages are not published (the console consumer receives all the > messages). > > Is there an explanation for this behavior? > > Best regards, > Radu >
