Kafka cluster not recovering after zookeeper and nodes failure
Hello Kafka users, We have tested failure scenarios and found the following situation in which the kafka cluster will not automatically recover. Cluster setup: 3 VMs (n1,n2,n3) running Centos 7, each VM runs a zookeper v3.4.13 and kafka v2.1.0 instance, configured as systemd services, OpenJDK 1.8.0_191. Current situation: n1 Kafka controller, n2 and n3 leaders for some partitions: ~/kafka/bin/kafka-topics.sh --zookeeper n1:2181 --describe --topic __consumer_offsets Topic:__consumer_offsets PartitionCount:1 ReplicationFactor:3 Configs:segment.bytes=104857600,cleanup.policy=compact,compression.type=producer Topic: __consumer_offsets Partition: 0 Leader: 3 Replicas: 3,1,2 Isr: 3,1,2 If I reboot simultaneously both n2 and n3, causing zookeeper to lose quorum and topics to lose their leaders, the kafka cluster will never recover. Zookeper regains quorum as soon as n2 and n3 are back up, n1 remains kafka controller, but I get the following error in all 3 kafka logs, repeated forever: [2019-03-20 09:43:02,524] ERROR [KafkaApi-2] Number of alive brokers '0' does not meet the required replication factor '3' for the offsets topic (configured via 'offsets.topic.replication.factor'). This error can be ignored if the cluster is starting up and not all brokers are up yet. (kafka.server.KafkaApis) [2019-03-20 09:43:02,830] ERROR [KafkaApi-2] Number of alive brokers '0' does not meet the required replication factor '3' for the offsets topic (configured via 'offsets.topic.replication.factor'). This error can be ignored if the cluster is starting up and not all brokers are up yet. (kafka.server.KafkaApis) [2019-03-20 09:43:03,486] ERROR [KafkaApi-2] Number of alive brokers '0' does not meet the required replication factor '3' for the offsets topic (configured via 'offsets.topic.replication.factor'). This error can be ignored if the cluster is starting up and not all brokers are up yet. (kafka.server.KafkaApis) If I restart the kafka process on n1 (old controller) - the cluster fully recovers. However the old controller does not shut down gracefully (I see "Retrying controlled shutdown after the previous attempt failed..." in the logs and is eventually killed by systemd). I could not reproduce the problem if one of the rebooted nodes is the controller. It looks to me like a race condition, as I can only reproduce the problem about half the time. Best regards, Radu
Re: Experimenting with Kafka and OpenSSL
If you test with Java 9 please make sure to use an accelerated cipher suite (e.g. one that uses AES GCM such as TLS_RSA_WITH_AES_128_GCM_SHA256). Radu On Mon, Oct 30, 2017 at 1:49 PM, Jaikiran Pai wrote: > I haven't yet had a chance to try out Java 9, but that's definitely on my > TODO list, maybe sometime this weekend. > > Thanks for pointing me to KAFKA-2561. I had missed that. > > -Jaikiran > > > > On 30/10/17 4:17 PM, Mickael Maison wrote: > >> Thanks for sharing, very interesting read. >> >> Did you get a chance to try JDK 9 ? >> >> We also considered using OpenSSL instead of JSSE especially since >> Netty made an easy to re-use package (netty-tcnative). >> >> There was KAFKA-2561 >> (https://issues.apache.org/jira/browse/KAFKA-2561) where people shared >> a few numbers and what would be need to get it working. >> >> On Mon, Oct 30, 2017 at 8:08 AM, Jaikiran Pai >> wrote: >> >>> We have been using Kafka in some of our projects for the past couple of >>> years. Our experience with Kafka and SSL had shown some performance >>> issues >>> when we had seriously tested it (which admittedly was around a year >>> back). >>> Our basic tests did show that things had improved over time with newer >>> versions, but we didn't get a chance to fully test and move to SSL for >>> Kafka. >>> >>> Incidentally, I happened to be looking into some other things related to >>> SSL >>> and decided to experiment with using openssl as the SSL provider for >>> Kafka. >>> I had heard OpenSSL performs better than the engine shipped default in >>> JRE, >>> but hadn't ever got a chance to do any experiments. This past few weeks, >>> I >>> decided to spend some time trying it. I have noted the experimentation >>> and >>> the performance numbers in my blog[1]. The initial basic performance >>> testing >>> (using the scripts shipped in Kafka) does show promising improvements. >>> Like >>> I note in my blog, this was a very basic performance test just to see if >>> OpenSSL can be pursued as an option (both in terms of being functional >>> and >>> performant) if we do decide to. >>> >>> I know some of the members in these lists do extensive performance >>> testing >>> with Kafka (and SSL), so I thought I will bring this to their notice. >>> >>> [1] https://jaitechwriteups.blogspot.com/2017/10/kafka-with-openssl.html >>> >>> -Jaikiran >>> >>> >
Re: ISR churn
es a day in our production cluster. We do > have hundreds of topics in our cluster, most of them with 20 or more > partitions, but most of them see only a few hundred messages per minute. > > > > We're running Kafka 0.10.0.1, and we thought upgrading to the 0.10.1.1 > version would fix the issue, but we've already deployed that version to our > staging cluster, and we're seeing the same problem. We still haven't tried > out the latest 0.10.2.0 version, but I don't see any evidence pointing to a > fix for that. > > > > This ticket seems to have some similar details, but it doesn't seem like > there has been follow-up, and there's no target release for fixing: > > > > https://issues.apache.org/jira/browse/KAFKA-4674 < > https://issues.apache.org/jira/browse/KAFKA-4674> > > > > Jun Ma, what exactly did you do to failover the controller to a new > broker? If that works for you, I'd like to try it in our staging clusters. > > > > Thanks, > > > > Marcos Juarez > > > > > > > > > > > > On Wed, Mar 22, 2017 at 11:55 AM, Jun MA <mailto:mj.saber1...@gmail.com>> wrote: > > I have similar issue with our cluster. We don’t know the root cause but > we have some interesting observation. > > > > 1. We do see correlation between ISR churn and fetcher connection > close/create. > > > > > > 2. We’ve tried to add a broker which doesn’t have any partitions on it > dedicate to the controller (rolling restart existing brokers and failover > the controller to the newly added broker), and that indeed eliminate the > random ISR churn. We have a cluster of 6 brokers (besides the dedicated > controller) and each one has about 300 partitions on it. I suspect that > kafka broker cannot handle running controller + 300 partitions. > > > > Anyway that’s so far what I got, I’d also like to know how to debug this. > > We’re running kafka 0.9.0.1 with heap size 8G. > > > > Thanks, > > Jun > > > >> On Mar 22, 2017, at 7:06 AM, Manikumar <mailto:manikumar.re...@gmail.com>> wrote: > >> > >> Any erros related to zookeeper seesion timeout? We can also check for > >> excesssive GC. > >> Some times this may due to forming multiple controllers due to soft > >> failures. > >> You can check ActiveControllerCount on brokers, only one broker in the > >> cluster should have 1. > >> Also check for network issues/partitions > >> > >> On Wed, Mar 22, 2017 at 7:21 PM, Radu Radutiu <mailto:rradu...@gmail.com>> wrote: > >> > >>> Hello, > >>> > >>> Does anyone know how I can debug high ISR churn on the kafka leader on > a > >>> cluster without traffic? I have 2 topics on a 4 node cluster (replica > 4 > >>> and replica 3) and both show constant changes of the number of insync > >>> replicas: > >>> > >>> [2017-03-22 15:30:10,945] INFO Partition [__consumer_offsets,0] on > broker > >>> 2: Expanding ISR for partition __consumer_offsets-0 from 2,4 to 2,4,5 > >>> (kafka.cluster.Partition) > >>> [2017-03-22 15:31:41,193] INFO Partition [__consumer_offsets,0] on > broker > >>> 2: Shrinking ISR for partition [__consumer_offsets,0] from 2,4,5 to 2,5 > >>> (kafka.cluster.Partition) > >>> [2017-03-22 15:31:41,195] INFO Partition [__consumer_offsets,0] on > broker > >>> 2: Expanding ISR for partition __consumer_offsets-0 from 2,5 to 2,5,4 > >>> (kafka.cluster.Partition) > >>> [2017-03-22 15:35:03,443] INFO Partition [__consumer_offsets,0] on > broker > >>> 2: Shrinking ISR for partition [__consumer_offsets,0] from 2,5,4 to 2,5 > >>> (kafka.cluster.Partition) > >>> [2017-03-22 15:35:03,445] INFO Partition [__consumer_offsets,0] on > broker > >>> 2: Expanding ISR for partition __consumer_offsets-0 from 2,5 to 2,5,4 > >>> (kafka.cluster.Partition) > >>> [2017-03-22 15:37:01,443] INFO Partition [__consumer_offsets,0] on > broker > >>> 2: Shrinking ISR for partition [__consumer_offsets,0] from 2,5,4 to 2,4 > >>> (kafka.cluster.Partition) > >>> [2017-03-22 15:37:01,445] INFO Partition [__consumer_offsets,0] on > broker > >>> 2: Expanding ISR for partition __consumer_offsets-0 from 2,4 to 2,4,5 > >>> (kafka.cluster.Partition) > >>> > >>> and > >>> > >>> [2017-03-22 15:09:52,646] INFO Partition [topic1,0] on broker 5: > Shrinking > >>> ISR for part
ISR churn
Hello, Does anyone know how I can debug high ISR churn on the kafka leader on a cluster without traffic? I have 2 topics on a 4 node cluster (replica 4 and replica 3) and both show constant changes of the number of insync replicas: [2017-03-22 15:30:10,945] INFO Partition [__consumer_offsets,0] on broker 2: Expanding ISR for partition __consumer_offsets-0 from 2,4 to 2,4,5 (kafka.cluster.Partition) [2017-03-22 15:31:41,193] INFO Partition [__consumer_offsets,0] on broker 2: Shrinking ISR for partition [__consumer_offsets,0] from 2,4,5 to 2,5 (kafka.cluster.Partition) [2017-03-22 15:31:41,195] INFO Partition [__consumer_offsets,0] on broker 2: Expanding ISR for partition __consumer_offsets-0 from 2,5 to 2,5,4 (kafka.cluster.Partition) [2017-03-22 15:35:03,443] INFO Partition [__consumer_offsets,0] on broker 2: Shrinking ISR for partition [__consumer_offsets,0] from 2,5,4 to 2,5 (kafka.cluster.Partition) [2017-03-22 15:35:03,445] INFO Partition [__consumer_offsets,0] on broker 2: Expanding ISR for partition __consumer_offsets-0 from 2,5 to 2,5,4 (kafka.cluster.Partition) [2017-03-22 15:37:01,443] INFO Partition [__consumer_offsets,0] on broker 2: Shrinking ISR for partition [__consumer_offsets,0] from 2,5,4 to 2,4 (kafka.cluster.Partition) [2017-03-22 15:37:01,445] INFO Partition [__consumer_offsets,0] on broker 2: Expanding ISR for partition __consumer_offsets-0 from 2,4 to 2,4,5 (kafka.cluster.Partition) and [2017-03-22 15:09:52,646] INFO Partition [topic1,0] on broker 5: Shrinking ISR for partition [topic1,0] from 5,2,4 to 5,4 (kafka.cluster.Partition) [2017-03-22 15:09:52,648] INFO Partition [topic1,0] on broker 5: Expanding ISR for partition topic1-0 from 5,4 to 5,4,2 (kafka.cluster.Partition) [2017-03-22 15:24:05,646] INFO Partition [topic1,0] on broker 5: Shrinking ISR for partition [topic1,0] from 5,4,2 to 5,4 (kafka.cluster.Partition) [2017-03-22 15:24:05,648] INFO Partition [topic1,0] on broker 5: Expanding ISR for partition topic1-0 from 5,4 to 5,4,2 (kafka.cluster.Partition) [2017-03-22 15:26:49,599] INFO Partition [topic1,0] on broker 5: Expanding ISR for partition topic1-0 from 5,4,2 to 5,4,2,1 (kafka.cluster.Partition) [2017-03-22 15:27:46,396] INFO Partition [topic1,0] on broker 5: Shrinking ISR for partition [topic1,0] from 5,4,2,1 to 5,4,1 (kafka.cluster.Partition) [2017-03-22 15:27:46,398] INFO Partition [topic1,0] on broker 5: Expanding ISR for partition topic1-0 from 5,4,1 to 5,4,1,2 (kafka.cluster.Partition) [2017-03-22 15:45:47,896] INFO Partition [topic1,0] on broker 5: Shrinking ISR for partition [topic1,0] from 5,4,1,2 to 5,1,2 (kafka.cluster.Partition) [2017-03-22 15:45:47,898] INFO Partition [topic1,0] on broker 5: Expanding ISR for partition topic1-0 from 5,1,2 to 5,1,2,4 (kafka.cluster.Partition) (END) I have tried increasing the num.network.threads (now 8) and num.replica.fetchers (now 2) but nothing has changed. The kafka server config is: default.replication.factor=4 log.retention.check.interval.ms=30 log.retention.hours=168 log.roll.hours=24 log.segment.bytes=104857600 min.insync.replicas=2 num.io.threads=8 num.network.threads=15 num.partitions=1 num.recovery.threads.per.data.dir=1 num.replica.fetchers=2 offsets.topic.num.partitions=1 offsets.topic.replication.factor=3 replica.lag.time.max.ms=500 socket.receive.buffer.bytes=102400 socket.request.max.bytes=104857600 socket.send.buffer.bytes=102400 unclean.leader.election.enable=false zookeeper.connection.timeout.ms=3000 Best regards, Radu
Re: Kafka broker crash
/tmp is not a good location for storing files. It will get cleaned up periodically, depending on your linux distribution. Radu On 22 June 2016 at 19:33, Misra, Rahul wrote: > Hi Madhukar, > > Thanks for your quick response. The path is "/tmp/kafka-logs/". But the > servers have not been restarted any time lately. The uptime for all the 3 > servers is almost 67 days. > > Regards, > Rahul Misra > > > -Original Message- > From: Madhukar Bharti [mailto:bhartimadhu...@gmail.com] > Sent: Wednesday, June 22, 2016 8:37 PM > To: users@kafka.apache.org > Subject: Re: Kafka broker crash > > Hi Rahul, > > Whether the path is "/tmp/kafka-logs/" or "/temp/kafka-logs" ? > > Mostly if path is set to "/tmp/" then in case machine restart it may > delete the files. So it is throwing FileNotFoundException. > you can change the file location to some other path and restart all broker. > This might fix the issue. > > Regrads, > Madhukar > > On Wed, Jun 22, 2016 at 1:40 PM, Misra, Rahul > wrote: > > > Hi, > > > > I'm facing a strange issue in my Kafka cluster. Could anybody please > > help me with it. The issue is as follows: > > > > We have a 3 node kafka cluster. We installed the zookeeper separately > > and have pointed the brokers to it. The zookeeper is also 3 node, but > > for our POC setup, the zookeeper nodes are on the same machines as the > > Kafka brokers. > > > > While receiving messages from an existing topic using a new groupId, 2 > > of the brokers crashed with same FATAL errors: > > > > > > < [server 2 logs] >>> > > > > [2016-06-21 23:09:14,697] INFO [GroupCoordinator 1]: Stabilized group > > pocTestNew11 generation 1 (kafka.coordinator.Gro > > upCoordinator) > > [2016-06-21 23:09:15,006] INFO [GroupCoordinator 1]: Assignment > > received from leader for group pocTestNew11 for genera tion 1 > > (kafka.coordinator.GroupCoordinator) > > [2016-06-21 23:09:20,335] FATAL [Replica Manager on Broker 1]: Halting > > due to unrecoverable I/O error while handling p roduce request: > > (kafka.server.ReplicaManager) > > kafka.common.KafkaStorageException: I/O exception in append to log > > '__consumer_offsets-4' > > at kafka.log.Log.append(Log.scala:318) > > at kafka.cluster.Partition$$anonfun$9.apply(Partition.scala:442) > > at kafka.cluster.Partition$$anonfun$9.apply(Partition.scala:428) > > at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:262) > > at kafka.utils.CoreUtils$.inReadLock(CoreUtils.scala:268) > > at > > kafka.cluster.Partition.appendMessagesToLeader(Partition.scala:428) > > at > > > kafka.server.ReplicaManager$$anonfun$appendToLocalLog$2.apply(ReplicaManager.scala:401) > > at > > > kafka.server.ReplicaManager$$anonfun$appendToLocalLog$2.apply(ReplicaManager.scala:386) > > at > > > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245) > > at > > > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245) > > at scala.collection.immutable.Map$Map1.foreach(Map.scala:116) > > at > > scala.collection.TraversableLike$class.map(TraversableLike.scala:245) > > at > scala.collection.AbstractTraversable.map(Traversable.scala:104) > > at > > kafka.server.ReplicaManager.appendToLocalLog(ReplicaManager.scala:386) > > at > > kafka.server.ReplicaManager.appendMessages(ReplicaManager.scala:322) > > at > > > kafka.coordinator.GroupMetadataManager.store(GroupMetadataManager.scala:228) > > at > > > kafka.coordinator.GroupCoordinator$$anonfun$handleCommitOffsets$9.apply(GroupCoordinator.scala:429) > > at > > > kafka.coordinator.GroupCoordinator$$anonfun$handleCommitOffsets$9.apply(GroupCoordinator.scala:429) > > at scala.Option.foreach(Option.scala:257) > > at > > > kafka.coordinator.GroupCoordinator.handleCommitOffsets(GroupCoordinator.scala:429) > > at > > kafka.server.KafkaApis.handleOffsetCommitRequest(KafkaApis.scala:280) > > at kafka.server.KafkaApis.handle(KafkaApis.scala:76) > > at > > kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60) > > at java.lang.Thread.run(Thread.java:745) > > Caused by: java.io.FileNotFoundException: > > /tmp/kafka-logs/__consumer_offsets-4/.index (No > > such file or directory) > > at java.io.RandomAccessFile.open0(Native Method) > > at java.io.RandomAccessFile.open(RandomAccessFile.java:316) > > at java.io.RandomAccessFile.(RandomAccessFile.java:243) > > at > > kafka.log.OffsetIndex$$anonfun$resize$1.apply(OffsetIndex.scala:277) > > at > > kafka.log.OffsetIndex$$anonfun$resize$1.apply(OffsetIndex.scala:276) > > at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:262) > > at kafka.log.OffsetIndex.resize(OffsetIndex.scala:276) > > at > > > kafka.log.OffsetIndex$$ano
Re: SSL support for command line tools
I have read the documentation and I can connect the consumer and producer
successfully with SSL. However I have trouble running other scripts like
bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list
{brokerUrl} —topic {topicName} --time -2
if the broker is configured with SSL only.
Regards,
Radu
On 23 June 2016 at 01:46, Harsha wrote:
> Radu,
> Please follow the instructions here
> http://kafka.apache.org/documentation.html#security_ssl . At
> the end of the SSL section we've an example for produce and
> consumer command line tools to pass in ssl configs.
>
> Thanks,
> Harsha
>
> On Wed, Jun 22, 2016, at 07:40 AM, Gerard Klijs wrote:
> > To eleborate:
> > We start the process with --command-config /some/folder/ssl.properties
> > the
> > file we include in the image, and contains the ssl properties it needs,
> > which is a subset of the properties (those specific for ssl) the client
> > uses. In this case the certificate is accessed in a data container,
> > having
> > access to the same certificate as the broker (so we don't need to set
> > acl's
> > to use the tool).
> >
> > On Wed, Jun 22, 2016 at 2:47 PM Gerard Klijs
> > wrote:
> >
> > > You need to pass the correct options, similar to how you would do to a
> > > client. We use the consumer-groups in a docker container, in an
> environment
> > > witch is now only SSL (since the schema registry now supports it).
> > >
> > > On Wed, Jun 22, 2016 at 2:47 PM Radu Radutiu
> wrote:
> > >
> > >> Hi,
> > >>
> > >> Is is possible to configure the command line tools like
> > >> kafka-consumer-groups.sh , kafka-topics.sh and all other command that
> are
> > >> not a consumer or producer to connect to a SSL only kafka cluster ?
> > >>
> > >> Regards,
> > >> Radu
> > >>
> > >
>
SSL support for command line tools
Hi, Is is possible to configure the command line tools like kafka-consumer-groups.sh , kafka-topics.sh and all other command that are not a consumer or producer to connect to a SSL only kafka cluster ? Regards, Radu
Re: ConsoleProducer missing messages (random behavior)
Hi, Sorry for posting twice. It seems the issue I have encountered is indeed https://issues.apache.org/jira/browse/KAFKA-3129 as indicated by Vahid Hashemian in the other thread. If I change request.required.acks to 1 it works fine. As far as the problem going away if the broker is started a long time ago I cannot reproduce any longer. The problem occurs 50% of the time if request.required.acks=0 on my machine. If I use the --old-producer it works fine. Best regards, Radu On 16 June 2016 at 16:15, Dustin Cote wrote: > Hi Radu, > > Could you clarify what you mean by "I've noticed that if the kafka server > was started a long time ago the console producer publishes all lines"? > It's a known issue that if there are no brokers available, the producer > will drop messages (https://issues.apache.org/jira/browse/KAFKA-1955) and > if you've scripted this scenario, it's likely your broker hasn't come up > completely before you start producing messages to it. > > Regards, > > > On Thu, Jun 16, 2016 at 3:19 AM, Radu Radutiu wrote: > > > Hi, > > > > I was following the Quickstart guide and I have noticed that > > ConsoleProducer does not publish all messages (the number of messages > > published differs from one run to another) and happens mostly on a fresh > > started broker. > > version: kafka_2.11-0.10.0.0 > > OS: Linux (Ubuntu 14.04, Centos 7.2) > > JDK: java version "1.7.0_101" > > OpenJDK Runtime Environment (IcedTea 2.6.6) > (7u101-2.6.6-0ubuntu0.14.04.1), > > openjdk version "1.8.0_91" > > OpenJDK Runtime Environment (build 1.8.0_91-b14) > > > > > > How to reproduce: > > - start zookeeper: > > ~/work/kafka_2.11-0.10.0.0$ bin/zookeeper-server-start.sh > > config/zookeeper.properties & > > > > -start kafka: > > ~/work/kafka_2.11-0.10.0.0$ bin/kafka-server-start.sh > > config/server.properties & > > > > -start console consumer (topic test1 is already created): > > ~/work/kafka_2.11-0.10.0.0$ bin/kafka-console-consumer.sh > > --bootstrap-server localhost:9092 -topic test1 --zookeeper localhost:2181 > > > > -in another terminal start console producer with the LICENSE file in > kafka > > directory as input: > > ~/work/kafka_2.11-0.10.0.0$ bin/kafka-console-producer.sh --topic test1 > > --broker-list localhost:9092> > > The last line in the console consumer output is not the last line in the > > LICENSE file for the first few runs of the console producer. If I use the > > --old-producer parameter, all the lines in the LICENSE file are published > > (and appear in the console consumer output). Different runs of console > > producer with the same input file publish different number of lines > > (sometimes all, sometimes only 182 lines out of 330). I've noticed that > if > > the kafka server was started a long time ago the console producer > publishes > > all lines. > > I have checked the kafka binary log file (in my case > > /tmp/kafka-logs/test1-0/.log ) and confirmed that the > > messages are not published (the console consumer receives all the > > messages). > > > > Is there an explanation for this behavior? > > > > Best regards, > > Radu > > > > > > -- > Dustin Cote > confluent.io >
ConsoleProducer missing messages (random behavior)
Hi, I was following the Quickstart guide and I have noticed that ConsoleProducer does not publish all messages (the number of messages published differs from one run to another) and happens mostly on a fresh started broker. version: kafka_2.11-0.10.0.0 OS: Linux (Ubuntu 14.04, Centos 7.2) JDK: java version "1.7.0_101" OpenJDK Runtime Environment (IcedTea 2.6.6) (7u101-2.6.6-0ubuntu0.14.04.1), openjdk version "1.8.0_91" OpenJDK Runtime Environment (build 1.8.0_91-b14) How to reproduce: - start zookeeper: ~/work/kafka_2.11-0.10.0.0$ bin/zookeeper-server-start.sh config/zookeeper.properties & -start kafka: ~/work/kafka_2.11-0.10.0.0$ bin/kafka-server-start.sh config/server.properties & -start console consumer (topic test1 is already created): ~/work/kafka_2.11-0.10.0.0$ bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 -topic test1 --zookeeper localhost:2181 -in another terminal start console producer with the LICENSE file in kafka directory as input: ~/work/kafka_2.11-0.10.0.0$ bin/kafka-console-producer.sh --topic test1 --broker-list localhost:9092
ConsoleProducer missing messages (random behavior)
Hi, I was following the Quickstart guide and I have noticed that ConsoleProducer does not publish all messages (the number of messages published differs from one run to another) and happens mostly on a fresh started broker. version: kafka_2.11-0.10.0.0 OS: Linux (Ubuntu 14.04, Centos 7.2) JDK: java version "1.7.0_101" OpenJDK Runtime Environment (IcedTea 2.6.6) (7u101-2.6.6-0ubuntu0.14.04.1), openjdk version "1.8.0_91" OpenJDK Runtime Environment (build 1.8.0_91-b14) How to reproduce: - start zookeeper: ~/work/kafka_2.11-0.10.0.0$ bin/zookeeper-server-start.sh config/zookeeper.properties & -start kafka: ~/work/kafka_2.11-0.10.0.0$ bin/kafka-server-start.sh config/server.properties & -start console consumer (topic test1 is already created): ~/work/kafka_2.11-0.10.0.0$ bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 -topic test1 --zookeeper localhost:2181 -in another terminal start console producer with the LICENSE file in kafka directory as input: ~/work/kafka_2.11-0.10.0.0$ bin/kafka-console-producer.sh --topic test1 --broker-list localhost:9092
