Thanks, kind of obvious afterwards :-). I used ip addresses in my ansible configuration so everything else worked.
2 -> {"jmx_port":13005,"timestamp":"1417702658359","host":"ubuntu","version":1,"port":9092}... 4 -> {"jmx_port":13005,"timestamp":"1417698861869","host":"ubuntu","version":1,"port":9092} etc... best regards svante 2014-12-04 1:37 GMT+01:00 Jun Rao <j...@confluent.io>: > Could you look at the broker registration in Zookeeper ( > > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+structures+in+Zookeeper > ) > and make sure the 3 registered hosts are unique? > > Thanks, > > Jun > > On Wed, Dec 3, 2014 at 5:54 AM, svante karlsson <s...@csi.se> wrote: > > > I've installed (for ansible scripting testing purposes) 3 VM's each > > containing kafka & zookeeer clustered together > > > > Ubuntu 14.04 > > Zookeepers are 3.4.6 and kafka 2.11-0.8.2-beta > > The kafka servers have broker id's 2, 4, 6 > > > > The zookeepers seems happy. > > The kafka servers start up and seems happy. > > > > I can created two test topics as in the getting started guide > > bin/kafka-topics.sh --create --zookeeper localhost:2181 > > --replication-factor 1 --partitions 1 --topic test > > bin/kafka-topics.sh --create --zookeeper localhost:2181 > > --replication-factor 3 --partitions 32 --topic test3 > > > > I published some messages on each topic and after a while I noticed that > I > > ran out of disk space and there are millions of logs in syslog similar to > > the one below. > > > > Any hints on what can cause this problem? > > > > > > /svante > > > > > > Dec 2 07:40:17 ubuntu supervisord: kafka-broker [2014-12-02 > 07:40:17,120] > > WARN [Replica Manager on Broker 4]: Fetch request with correlation id 21 > > from client ReplicaFetcherThread-0-6 on partition [test3,3] failed due to > > Leader not local for partition [test3, 3] on broker 4 > > (kafka.server.ReplicaManager) > > Dec 2 07:40:17 ubuntu supervisord: kafka-broker [2014-12-02 > 07:40:17,120] > > WARN [Replica Manager on Broker 4]: Fetch request with correlation id 21 > > from client ReplicaFetcherThread-0-6 on partition [test3,4] failed due to > > Leader not local for partition [test3, 4] on broker 4 > > (kafka.server.ReplicaManager) > > Dec 2 07:40:17 ubuntu supervisord: kafka-broker [2014-12-02 > 07:40:17,120] > > WARN [Replica Manager on Broker 4]: Fetch request with correlation id 21 > > from client ReplicaFetcherThread-0-6 on partition [test3,26] failed due > to > > Leader not local for partition [test3,26] on broker 4 > > (kafka.server.ReplicaManager) > > Dec 2 07:40:17 ubuntu supervisord: kafka-broker [2014-12-02 > 07:40:17,121] > > ERROR [KafkaApi-4] error when handling request Name: FetchRequest; > Version: > > 0; CorrelationId: 21; ClientId: ReplicaFetcherThread-0-6; ReplicaId: 4; > > MaxWait: 500 ms; MinBytes: 1 bytes; RequestInfo: [test3,22] -> > > PartitionFetchInfo(0,1048576),[test3,31] -> > > PartitionFetchInfo(0,1048576),[test3,21] > > ->PartitionFetchInfo(0,1048576),[test3,13] -> > > PartitionFetchInfo(0,1048576),[test3,9] -> > > PartitionFetchInfo(0,1048576),[test3,28] -> > > PartitionFetchInfo(0,1048576),[test3,27] -> > > PartitionFetchInfo(0,1048576),[test3,15] -> > > PartitionFetchInfo(0,1048576),[test3,7] -> > > PartitionFetchInfo(0,1048576),[test3,8] -> > > PartitionFetchInfo(0,1048576),[test3,2] -> > > PartitionFetchInfo(0,1048576),[test3,19] -> > > PartitionFetchInfo(0,1048576),[test3,25] -> > > PartitionFetchInfo(0,1048576),[test3,20] -> > > PartitionFetchInfo(0,1048576),[test3,14] -> > > PartitionFetchInfo(0,1048576),[test2,0] -> > > PartitionFetchInfo(0,1048576),[test3,16] -> > > PartitionFetchInfo(0,1048576),[test3,1] -> > > PartitionFetchInfo(0,1048576),[test3,10] -> > > PartitionFetchInfo(0,1048576),[test3,3] -> > > PartitionFetchInfo(0,1048576),[test3,4] -> > > PartitionFetchInfo(0,1048576),[test3,26] -> PartitionFetchInfo(0,1048576) > > (kafka.server.KafkaApis) > > Dec 2 07:40:17 ubuntu supervisord: kafka-broker > > kafka.common.KafkaException: Should not set log end offset on partition > > [test3,22]'s local replica 4 > > Dec 2 07:40:17 ubuntu supervisord: kafka-broker #011at > > kafka.cluster.Replica.logEndOffset_$eq(Replica.scala:52) > > Dec 2 07:40:17 ubuntu supervisord: kafka-broker #011at > > > > > kafka.server.ReplicaManager.updateReplicaLEOAndPartitionHW(ReplicaManager.scala:565) > > Dec 2 07:40:17 ubuntu supervisord: kafka-broker #011at > > > > > kafka.server.KafkaApis$$anonfun$recordFollowerLogEndOffsets$2.apply(KafkaApis.scala:348) > > Dec 2 07:40:17 ubuntu supervisord: kafka-broker #011at > > > > > kafka.server.KafkaApis$$anonfun$recordFollowerLogEndOffsets$2.apply(KafkaApis.scala:346) > > Dec 2 07:40:17 ubuntu supervisord: kafka-broker #011at > > > > > scala.collection.MapLike$MappedValues$$anonfun$foreach$3.apply(MapLike.scala:245) > > Dec 2 07:40:17 ubuntu supervisord: kafka-broker #011at > > > > > scala.collection.MapLike$MappedValues$$anonfun$foreach$3.apply(MapLike.scala:245) > > Dec 2 07:40:17 ubuntu supervisord: kafka-broker #011at > > > > > scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:778) > > Dec 2 07:40:17 ubuntu supervisord: kafka-broker #011at > > scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:221) > > Dec 2 07:40:17 ubuntu supervisord: kafka-broker #011at > > scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:428) > > Dec 2 07:40:17 ubuntu supervisord: kafka-broker #011at > > > > > scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:777) > > Dec 2 07:40:17 ubuntu supervisord: kafka-broker #011at > > scala.collection.MapLike$MappedValues.foreach(MapLike.scala:245) > > Dec 2 07:40:17 ubuntu supervisord: kafka-broker #011at > > kafka.server.KafkaApis.recordFollowerLogEndOffsets(KafkaApis.scala:346) > > Dec 2 07:40:17 ubuntu supervisord: kafka-broker #011at > > kafka.server.KafkaApis.handleFetchRequest(KafkaApis.scala:311) > > Dec 2 07:40:17 ubuntu supervisord: kafka-broker #011at > > kafka.server.KafkaApis.handle(KafkaApis.scala:60) > > Dec 2 07:40:17 ubuntu supervisord: kafka-broker #011at > > kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:59) > > Dec 2 07:40:17 ubuntu supervisord: kafka-broker #011at > > java.lang.Thread.run(Thread.java:745) > > Dec 2 07:40:17 ubuntu supervisord: kafka-broker [2014-12-02 > 07:40:17,124] > > ERROR [ReplicaFetcherThread-0-6], Error for partition [test3,22] to > broker > > 6:class kafka.common.UnknownException(kafka.server.ReplicaFetcherThread) > > Dec 2 07:40:17 ubuntu supervisord: kafka-broker [2014-12-02 > 07:40:17,124] > > ERROR [ReplicaFetcherThread-0-6], Error for partition [test3,31] to > broker > > 6:class kafka.common.UnknownExceptionkafka.server.ReplicaFetcherThread) > > Dec 2 07:40:17 ubuntu supervisord: kafka-broker [2014-12-02 > 07:40:17,124] > > ERROR [ReplicaFetcherThread-0-6], Error for partition [test3,21] to > broker > > 6:class kafka.common.UnknownException(kafka.server.ReplicaFetcherThread) > > Dec 2 07:40:17 ubuntu supervisord: kafka-broker [2014-12-02 > 07:40:17,124] > > ERROR [ReplicaFetcherThread-0-6], Error for partition [test3,13] to > broker > > 6:class kafka.common.UnknownException(kafka.server.ReplicaFetcherThread) > > Dec 2 07:40:17 ubuntu supervisord: kafka-broker [2014-12-02 > 07:40:17,124] > > ERROR [ReplicaFetcherThread-0-6], Error for partition [test3,9] to broker > > 6:class kafka.common.UnknownException(kafka.server.ReplicaFetcherThread) > > >