RE: Remote Zookeeper
Yes. True. Thanks for your help. > Date: Tue, 11 Mar 2014 11:48:20 -0700 > Subject: Re: Remote Zookeeper > From: b...@b3k.us > To: users@kafka.apache.org > > If they are on different physical machines then binding to localhost/using > localhost as the host name is unlikely to be what you want. > > On Tuesday, March 11, 2014, A A wrote: > > > Thanks. I already checked out the wiki and step 6 in particular. > > Just to clarify, Zk, Broker1 and Broker2 are on 3 different physical > > machines. > > > > > From: b...@b3k.us > > > Date: Tue, 11 Mar 2014 10:58:50 -0700 > > > Subject: Re: Remote Zookeeper > > > To: users@kafka.apache.org > > > > > > "I'd suggest deleting the zookeeper and Kafka logs and starting over > > using > > > the getting started tutorial from the wiki." > > > > > >| > > >v > > > > > > "Could this be an issue? > > > INFO Registered broker 2 at path /brokers/ids/2 with address > > localhost:9092. > > > INFO Registered broker 1 at path /brokers/ids/1 with address > > > localhost:9092." > > > > > > yes. see step 6: https://kafka.apache.org/documentation.html#quickstart > > > > > > > > > > > > On Tue, Mar 11, 2014 at 10:54 AM, A A > > > > > > wrote: > > > > > > > Okay thanks. I removed zookeeper and kafka logs > > > > > > > > /tmp/kafka-logs > > > > /tmp/logs > > > > /var/zookeeper/version-2/ > > > > /tmp/zookeeper*.log (on Zk) > > > > > > > > restarted zookeeper > > > > > > > > $KAFKA_HOME/bin/kafka-list-topic.sh --zookeeper 192.168.1.120:2181 > > > > no topics exist! > > > > > > > > Started Broker 2 > > > > $KAFKA/kafka-server-start.sh $KAFKA_CONFIG/server.properties > > > > [2014-03-11 12:05:32,651] INFO Verifying properties > > > > (kafka.utils.VerifiableProperties) > > > > [2014-03-11 12:05:32,681] INFO Property port is overridden to 9092 > > > > (kafka.utils.VerifiableProperties) > > > > [2014-03-11 12:05:32,681] INFO Property socket.request.max.bytes is > > > > overridden to 104857600 (kafka.utils.VerifiableProperties) > > > > [2014-03-11 12:05:32,682] INFO Property num.io.threads is overridden > > to 2 > > > > (kafka.utils.VerifiableProperties) > > > > [2014-03-11 12:05:32,682] INFO Property log.dirs is overridden to > > > > /tmp/kafka-logs (kafka.utils.VerifiableProperties) > > > > [2014-03-11 12:05:32,682] INFO Property log.cleanup.interval.mins is > > > > overridden to 1 (kafka.utils.VerifiableProperties) > > > > [2014-03-11 12:05:32,682] INFO Property socket.send.buffer.bytes is > > > > overridden to 1048576 (kafka.utils.VerifiableProperties) > > > > [2014-03-11 12:05:32,683] INFO Property log.flush.interval.ms is > > > > overridden to 1000 (kafka.utils.VerifiableProperties) > > > > [2014-03-11 12:05:32,683] INFO Property zookeeper.connect is > > overridden to > > > > 192.168.1.120:2181 (kafka.utils.VerifiableProperties) > > > > [2014-03-11 12:05:32,684] INFO Property broker.id is overridden to 2 > > > > (kafka.utils.VerifiableProperties) > > > > [2014-03-11 12:05:32,684] INFO Property log.retention.hours is > > overridden > > > > to 168 (kafka.utils.VerifiableProperties) > > > > [2014-03-11 12:05:32,684] INFO Property num.network.threads is > > overridden > > > > to 2 (kafka.utils.VerifiableProperties) > > > > [2014-03-11 12:05:32,684] INFO Property socket.receive.buffer.bytes is > > > > overridden to 1048576 (kafka.utils.VerifiableProperties) > > > > [2014-03-11 12:05:32,684] INFO Property > > zookeeper.connection.timeout.msis overridden to 100 > > (kafka.utils.VerifiableProperties) > > > > [2014-03-11 12:05:32,685] INFO Property num.partitions is overridden > > to 2 > > > > (kafka.utils.VerifiableProperties) > > > > [2014-03-11 12:05:32,685] INFO Property log.flush.interval.messages is > > > > overridden to 1 (kafka.utils.VerifiableProperties) > > > > [2014-03-11 12:05:32,685] INFO Property log.segment.bytes is > > overridden to > > > > 536870912 (kafka.utils.VerifiableProperties) > > > > [2014-03-11 12:05:32,703] INFO [Kafka Server 2], Starting > > > > (kafka.server.KafkaServer) > > > > [2014-03-11 12:05:32,
Re: Remote Zookeeper
If they are on different physical machines then binding to localhost/using localhost as the host name is unlikely to be what you want. On Tuesday, March 11, 2014, A A wrote: > Thanks. I already checked out the wiki and step 6 in particular. > Just to clarify, Zk, Broker1 and Broker2 are on 3 different physical > machines. > > > From: b...@b3k.us > > Date: Tue, 11 Mar 2014 10:58:50 -0700 > > Subject: Re: Remote Zookeeper > > To: users@kafka.apache.org > > > > "I'd suggest deleting the zookeeper and Kafka logs and starting over > using > > the getting started tutorial from the wiki." > > > >| > >v > > > > "Could this be an issue? > > INFO Registered broker 2 at path /brokers/ids/2 with address > localhost:9092. > > INFO Registered broker 1 at path /brokers/ids/1 with address > > localhost:9092." > > > > yes. see step 6: https://kafka.apache.org/documentation.html#quickstart > > > > > > > > On Tue, Mar 11, 2014 at 10:54 AM, A A > > > > wrote: > > > > > Okay thanks. I removed zookeeper and kafka logs > > > > > > /tmp/kafka-logs > > > /tmp/logs > > > /var/zookeeper/version-2/ > > > /tmp/zookeeper*.log (on Zk) > > > > > > restarted zookeeper > > > > > > $KAFKA_HOME/bin/kafka-list-topic.sh --zookeeper 192.168.1.120:2181 > > > no topics exist! > > > > > > Started Broker 2 > > > $KAFKA/kafka-server-start.sh $KAFKA_CONFIG/server.properties > > > [2014-03-11 12:05:32,651] INFO Verifying properties > > > (kafka.utils.VerifiableProperties) > > > [2014-03-11 12:05:32,681] INFO Property port is overridden to 9092 > > > (kafka.utils.VerifiableProperties) > > > [2014-03-11 12:05:32,681] INFO Property socket.request.max.bytes is > > > overridden to 104857600 (kafka.utils.VerifiableProperties) > > > [2014-03-11 12:05:32,682] INFO Property num.io.threads is overridden > to 2 > > > (kafka.utils.VerifiableProperties) > > > [2014-03-11 12:05:32,682] INFO Property log.dirs is overridden to > > > /tmp/kafka-logs (kafka.utils.VerifiableProperties) > > > [2014-03-11 12:05:32,682] INFO Property log.cleanup.interval.mins is > > > overridden to 1 (kafka.utils.VerifiableProperties) > > > [2014-03-11 12:05:32,682] INFO Property socket.send.buffer.bytes is > > > overridden to 1048576 (kafka.utils.VerifiableProperties) > > > [2014-03-11 12:05:32,683] INFO Property log.flush.interval.ms is > > > overridden to 1000 (kafka.utils.VerifiableProperties) > > > [2014-03-11 12:05:32,683] INFO Property zookeeper.connect is > overridden to > > > 192.168.1.120:2181 (kafka.utils.VerifiableProperties) > > > [2014-03-11 12:05:32,684] INFO Property broker.id is overridden to 2 > > > (kafka.utils.VerifiableProperties) > > > [2014-03-11 12:05:32,684] INFO Property log.retention.hours is > overridden > > > to 168 (kafka.utils.VerifiableProperties) > > > [2014-03-11 12:05:32,684] INFO Property num.network.threads is > overridden > > > to 2 (kafka.utils.VerifiableProperties) > > > [2014-03-11 12:05:32,684] INFO Property socket.receive.buffer.bytes is > > > overridden to 1048576 (kafka.utils.VerifiableProperties) > > > [2014-03-11 12:05:32,684] INFO Property > zookeeper.connection.timeout.msis overridden to 100 > (kafka.utils.VerifiableProperties) > > > [2014-03-11 12:05:32,685] INFO Property num.partitions is overridden > to 2 > > > (kafka.utils.VerifiableProperties) > > > [2014-03-11 12:05:32,685] INFO Property log.flush.interval.messages is > > > overridden to 1 (kafka.utils.VerifiableProperties) > > > [2014-03-11 12:05:32,685] INFO Property log.segment.bytes is > overridden to > > > 536870912 (kafka.utils.VerifiableProperties) > > > [2014-03-11 12:05:32,703] INFO [Kafka Server 2], Starting > > > (kafka.server.KafkaServer) > > > [2014-03-11 12:05:32,739] INFO [Log Manager on Broker 2] Starting log > > > cleaner every 6 ms (kafka.log.LogManager) > > > [2014-03-11 12:05:32,753] INFO [Log Manager on Broker 2] Starting log > > > flusher every 3000 ms with the following overrides Map() > > > (kafka.log.LogManager) > > > [2014-03-11 12:05:32,791] INFO Awaiting socket connections on > 0.0.0.0:9092. > > > (kafka.network.Acceptor) > > > [2014-03-11 12:05:32,794] INFO [Socket Server on Broker 2], Started > > > (kafka.network.SocketServer) > > > [2014-03-11 12:05:32,801] INFO connecting to ZK: 192.168.1.120:
RE: Remote Zookeeper
I have now added host.name property for Broker1 and Broker2 and it works. Maybe someone can modify the wiki and add this for Step 6? Also, I would like to mention, as previous replied to my thread; I didn't HAVE_TO add all the brokers to the brokers-list $KAFKA_HOME/bin/kafka-console-producer.sh --broker-list 192.168.1.124:9092 --topic replicated-topic $KAFKA_HOME/bin/kafka-console-producer.sh --broker-list 192.168.1.123:9092 --topic replicated-topic $KAFKA_HOME/bin/kafka-console-producer.sh --broker-list 192.168.1.123:9092,192.168.1.124:9092 --topic replicated-topic All of the above worked for me. > From: andthereitg...@hotmail.com > To: users@kafka.apache.org > Subject: RE: Remote Zookeeper > Date: Tue, 11 Mar 2014 18:18:31 + > > Thanks. I already checked out the wiki and step 6 in particular. > Just to clarify, Zk, Broker1 and Broker2 are on 3 different physical machines. > > > From: b...@b3k.us > > Date: Tue, 11 Mar 2014 10:58:50 -0700 > > Subject: Re: Remote Zookeeper > > To: users@kafka.apache.org > > > > "I'd suggest deleting the zookeeper and Kafka logs and starting over using > > the getting started tutorial from the wiki." > > > >| > >v > > > > "Could this be an issue? > > INFO Registered broker 2 at path /brokers/ids/2 with address localhost:9092. > > INFO Registered broker 1 at path /brokers/ids/1 with address > > localhost:9092." > > > > yes. see step 6: https://kafka.apache.org/documentation.html#quickstart > > > > > > > > On Tue, Mar 11, 2014 at 10:54 AM, A A wrote: > > > > > Okay thanks. I removed zookeeper and kafka logs > > > > > > /tmp/kafka-logs > > > /tmp/logs > > > /var/zookeeper/version-2/ > > > /tmp/zookeeper*.log (on Zk) > > > > > > restarted zookeeper > > > > > > $KAFKA_HOME/bin/kafka-list-topic.sh --zookeeper 192.168.1.120:2181 > > > no topics exist! > > > > > > Started Broker 2 > > > $KAFKA/kafka-server-start.sh $KAFKA_CONFIG/server.properties > > > [2014-03-11 12:05:32,651] INFO Verifying properties > > > (kafka.utils.VerifiableProperties) > > > [2014-03-11 12:05:32,681] INFO Property port is overridden to 9092 > > > (kafka.utils.VerifiableProperties) > > > [2014-03-11 12:05:32,681] INFO Property socket.request.max.bytes is > > > overridden to 104857600 (kafka.utils.VerifiableProperties) > > > [2014-03-11 12:05:32,682] INFO Property num.io.threads is overridden to 2 > > > (kafka.utils.VerifiableProperties) > > > [2014-03-11 12:05:32,682] INFO Property log.dirs is overridden to > > > /tmp/kafka-logs (kafka.utils.VerifiableProperties) > > > [2014-03-11 12:05:32,682] INFO Property log.cleanup.interval.mins is > > > overridden to 1 (kafka.utils.VerifiableProperties) > > > [2014-03-11 12:05:32,682] INFO Property socket.send.buffer.bytes is > > > overridden to 1048576 (kafka.utils.VerifiableProperties) > > > [2014-03-11 12:05:32,683] INFO Property log.flush.interval.ms is > > > overridden to 1000 (kafka.utils.VerifiableProperties) > > > [2014-03-11 12:05:32,683] INFO Property zookeeper.connect is overridden to > > > 192.168.1.120:2181 (kafka.utils.VerifiableProperties) > > > [2014-03-11 12:05:32,684] INFO Property broker.id is overridden to 2 > > > (kafka.utils.VerifiableProperties) > > > [2014-03-11 12:05:32,684] INFO Property log.retention.hours is overridden > > > to 168 (kafka.utils.VerifiableProperties) > > > [2014-03-11 12:05:32,684] INFO Property num.network.threads is overridden > > > to 2 (kafka.utils.VerifiableProperties) > > > [2014-03-11 12:05:32,684] INFO Property socket.receive.buffer.bytes is > > > overridden to 1048576 (kafka.utils.VerifiableProperties) > > > [2014-03-11 12:05:32,684] INFO Property zookeeper.connection.timeout.msis > > > overridden to 100 (kafka.utils.VerifiableProperties) > > > [2014-03-11 12:05:32,685] INFO Property num.partitions is overridden to 2 > > > (kafka.utils.VerifiableProperties) > > > [2014-03-11 12:05:32,685] INFO Property log.flush.interval.messages is > > > overridden to 1 (kafka.utils.VerifiableProperties) > > > [2014-03-11 12:05:32,685] INFO Property log.segment.bytes is overridden to > > > 536870912 (kafka.utils.VerifiableProperties) > > > [2014-03-11 12:05:32,703] INFO [Kafka Server 2], Starting > > > (kafka.server.KafkaServer) > > > [2014-03-11 12:05:32,739] INFO [Log Manager on Broker 2] Starting log > > > cleaner every 6 ms (kafka.log.LogMan
RE: Remote Zookeeper
Thanks. I already checked out the wiki and step 6 in particular. Just to clarify, Zk, Broker1 and Broker2 are on 3 different physical machines. > From: b...@b3k.us > Date: Tue, 11 Mar 2014 10:58:50 -0700 > Subject: Re: Remote Zookeeper > To: users@kafka.apache.org > > "I'd suggest deleting the zookeeper and Kafka logs and starting over using > the getting started tutorial from the wiki." > >| >v > > "Could this be an issue? > INFO Registered broker 2 at path /brokers/ids/2 with address localhost:9092. > INFO Registered broker 1 at path /brokers/ids/1 with address > localhost:9092." > > yes. see step 6: https://kafka.apache.org/documentation.html#quickstart > > > > On Tue, Mar 11, 2014 at 10:54 AM, A A wrote: > > > Okay thanks. I removed zookeeper and kafka logs > > > > /tmp/kafka-logs > > /tmp/logs > > /var/zookeeper/version-2/ > > /tmp/zookeeper*.log (on Zk) > > > > restarted zookeeper > > > > $KAFKA_HOME/bin/kafka-list-topic.sh --zookeeper 192.168.1.120:2181 > > no topics exist! > > > > Started Broker 2 > > $KAFKA/kafka-server-start.sh $KAFKA_CONFIG/server.properties > > [2014-03-11 12:05:32,651] INFO Verifying properties > > (kafka.utils.VerifiableProperties) > > [2014-03-11 12:05:32,681] INFO Property port is overridden to 9092 > > (kafka.utils.VerifiableProperties) > > [2014-03-11 12:05:32,681] INFO Property socket.request.max.bytes is > > overridden to 104857600 (kafka.utils.VerifiableProperties) > > [2014-03-11 12:05:32,682] INFO Property num.io.threads is overridden to 2 > > (kafka.utils.VerifiableProperties) > > [2014-03-11 12:05:32,682] INFO Property log.dirs is overridden to > > /tmp/kafka-logs (kafka.utils.VerifiableProperties) > > [2014-03-11 12:05:32,682] INFO Property log.cleanup.interval.mins is > > overridden to 1 (kafka.utils.VerifiableProperties) > > [2014-03-11 12:05:32,682] INFO Property socket.send.buffer.bytes is > > overridden to 1048576 (kafka.utils.VerifiableProperties) > > [2014-03-11 12:05:32,683] INFO Property log.flush.interval.ms is > > overridden to 1000 (kafka.utils.VerifiableProperties) > > [2014-03-11 12:05:32,683] INFO Property zookeeper.connect is overridden to > > 192.168.1.120:2181 (kafka.utils.VerifiableProperties) > > [2014-03-11 12:05:32,684] INFO Property broker.id is overridden to 2 > > (kafka.utils.VerifiableProperties) > > [2014-03-11 12:05:32,684] INFO Property log.retention.hours is overridden > > to 168 (kafka.utils.VerifiableProperties) > > [2014-03-11 12:05:32,684] INFO Property num.network.threads is overridden > > to 2 (kafka.utils.VerifiableProperties) > > [2014-03-11 12:05:32,684] INFO Property socket.receive.buffer.bytes is > > overridden to 1048576 (kafka.utils.VerifiableProperties) > > [2014-03-11 12:05:32,684] INFO Property zookeeper.connection.timeout.msis > > overridden to 100 (kafka.utils.VerifiableProperties) > > [2014-03-11 12:05:32,685] INFO Property num.partitions is overridden to 2 > > (kafka.utils.VerifiableProperties) > > [2014-03-11 12:05:32,685] INFO Property log.flush.interval.messages is > > overridden to 1 (kafka.utils.VerifiableProperties) > > [2014-03-11 12:05:32,685] INFO Property log.segment.bytes is overridden to > > 536870912 (kafka.utils.VerifiableProperties) > > [2014-03-11 12:05:32,703] INFO [Kafka Server 2], Starting > > (kafka.server.KafkaServer) > > [2014-03-11 12:05:32,739] INFO [Log Manager on Broker 2] Starting log > > cleaner every 6 ms (kafka.log.LogManager) > > [2014-03-11 12:05:32,753] INFO [Log Manager on Broker 2] Starting log > > flusher every 3000 ms with the following overrides Map() > > (kafka.log.LogManager) > > [2014-03-11 12:05:32,791] INFO Awaiting socket connections on 0.0.0.0:9092. > > (kafka.network.Acceptor) > > [2014-03-11 12:05:32,794] INFO [Socket Server on Broker 2], Started > > (kafka.network.SocketServer) > > [2014-03-11 12:05:32,801] INFO connecting to ZK: > > 192.168.1.120:2181(kafka.server.KafkaZooKeeper) > > [2014-03-11 12:05:32,815] INFO Starting ZkClient event thread. > > (org.I0Itec.zkclient.ZkEventThread) > > [2014-03-11 12:05:32,817] INFO Client > > environment:zookeeper.version=3.3.3-1203054, built on 11/17/2011 05:47 GMT > > (org.apache.zookeeper.ZooKeeper) > > [2014-03-11 12:05:32,817] INFO Client environment:host.name=localhost > > (org.apache.zookeeper.ZooKeeper) > > [2014-03-11 12:05:32,817] INFO Client environment:java.version=1.7.0_51 > > (org.apache.zookeeper.ZooKeeper) > > [2014-03-11 12:05:32,822] INFO Client environment:java.vendo
Re: Remote Zookeeper
log.LogManager)
> [2014-03-11 12:07:20,938] INFO [Log Manager on Broker 1] Starting log
> cleaner every 6 ms (kafka.log.LogManager)
> [2014-03-11 12:07:20,951] INFO [Log Manager on Broker 1] Starting log
> flusher every 3000 ms with the following overrides Map()
> (kafka.log.LogManager)
> [2014-03-11 12:07:20,991] INFO Awaiting socket connections on 0.0.0.0:9092.
> (kafka.network.Acceptor)
> [2014-03-11 12:07:20,993] INFO [Socket Server on Broker 1], Started
> (kafka.network.SocketServer)
> [2014-03-11 12:07:21,002] INFO connecting to ZK:
> 192.168.1.120:2181(kafka.server.KafkaZooKeeper)
> [2014-03-11 12:07:21,018] INFO Starting ZkClient event thread.
> (org.I0Itec.zkclient.ZkEventThread)
> [2014-03-11 12:07:21,029] INFO Client
> environment:zookeeper.version=3.3.3-1203054, built on 11/17/2011 05:47 GMT
> (org.apache.zookeeper.ZooKeeper)
> [2014-03-11 12:07:21,029] INFO Client environment:host.name=localhost
> (org.apache.zookeeper.ZooKeeper)
> [2014-03-11 12:07:21,029] INFO Client environment:java.version=1.7.0_51
> (org.apache.zookeeper.ZooKeeper)
> [2014-03-11 12:07:21,029] INFO Client environment:java.vendor=Oracle
> Corporation (org.apache.zookeeper.ZooKeeper)
> [2014-03-11 12:07:21,029] INFO Client
> environment:java.home=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.51.x86_64/jre
> (org.apache.zookeeper.ZooKeeper)
> [2014-03-11 12:07:21,029] INFO Client
> environment:java.class.path=:/opt/kafka-0.8.0/bin/../core/target/scala-2.8.0/kafka_2.8.0-0.8.0.jar:/opt/kafka-0.8.0/bin/../core/target/scala-2.8.0/kafka-assembly-0.8.0-deps.jar:/opt/kafka-0.8.0/bin/../perf/target/scala-2.8.0/kafka-perf_2.8.0-0.8.0.jar:/opt/kafka-0.8.0/bin/../libs/*.jar:/opt/kafka-0.8.0/bin/../kafka*.jar
> (org.apache.zookeeper.ZooKeeper)
> [2014-03-11 12:07:21,029] INFO Client
> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
> (org.apache.zookeeper.ZooKeeper)
> [2014-03-11 12:07:21,029] INFO Client environment:java.io.tmpdir=/tmp
> (org.apache.zookeeper.ZooKeeper)
> [2014-03-11 12:07:21,030] INFO Client environment:java.compiler=
> (org.apache.zookeeper.ZooKeeper)
> [2014-03-11 12:07:21,030] INFO Client environment:os.name=Linux
> (org.apache.zookeeper.ZooKeeper)
> [2014-03-11 12:07:21,030] INFO Client environment:os.arch=amd64
> (org.apache.zookeeper.ZooKeeper)
> [2014-03-11 12:07:21,030] INFO Client
> environment:os.version=2.6.32-279.el6.x86_64
> (org.apache.zookeeper.ZooKeeper)
> [2014-03-11 12:07:21,030] INFO Client environment:user.name=root
> (org.apache.zookeeper.ZooKeeper)
> [2014-03-11 12:07:21,030] INFO Client environment:user.home=/root
> (org.apache.zookeeper.ZooKeeper)
> [2014-03-11 12:07:21,030] INFO Client environment:user.dir=/var/zookeeper
> (org.apache.zookeeper.ZooKeeper)
> [2014-03-11 12:07:21,031] INFO Initiating client connection, connectString=
> 192.168.1.120:2181 sessionTimeout=6000
> watcher=org.I0Itec.zkclient.ZkClient@45419cee(org.apache.zookeeper.ZooKeeper)
> [2014-03-11 12:07:21,043] INFO Opening socket connection to server /
> 192.168.1.120:2181 (org.apache.zookeeper.ClientCnxn)
> [2014-03-11 12:07:21,049] INFO Socket connection established to
> 192.168.1.120/192.168.1.120:2181, initiating session
> (org.apache.zookeeper.ClientCnxn)
> [2014-03-11 12:07:21,058] INFO Session establishment complete on server
> 192.168.1.120/192.168.1.120:2181, sessionid = 0x144b104e0210005,
> negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn)
> [2014-03-11 12:07:21,060] INFO zookeeper state changed (SyncConnected)
> (org.I0Itec.zkclient.ZkClient)
> [2014-03-11 12:07:21,095] INFO Registered broker 1 at path /brokers/ids/1
> with address localhost:9092. (kafka.utils.ZkUtils$)
> [2014-03-11 12:07:21,096] INFO [Kafka Server 1], Connecting to ZK:
> 192.168.1.120:2181 (kafka.server.KafkaServer)
> [2014-03-11 12:07:21,222] INFO Will not load MX4J, mx4j-tools.jar is not
> in the classpath (kafka.utils.Mx4jLoader$)
> [2014-03-11 12:07:21,249] INFO conflict in /controller data: {
> "brokerid":1, "timestamp":"1394539641229", "version":1 } stored data: {
> "brokerid":2, "timestamp":"1394539533037", "version":1 }
> (kafka.utils.ZkUtils$)
> [2014-03-11 12:07:21,469] INFO [Kafka Server 1], Started
> (kafka.server.KafkaServer)
>
> Could this be an issue?
> INFO Registered broker 2 at path /brokers/ids/2 with address
> localhost:9092.
> INFO Registered broker 1 at path /brokers/ids/1 with address
> localhost:9092.
>
>
>
> > Date: Tue, 11 Mar 2014 09:51:54 -0700
> > Subject: Re: Remote Zookeeper
> > From: b...@b3k.us
> > To: users@kafka.apache.org
> >
> > Seems you my have put your cluster in a very confused state with random
> > add
RE: Remote Zookeeper
:java.home=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.51.x86_64/jre
(org.apache.zookeeper.ZooKeeper)
[2014-03-11 12:07:21,029] INFO Client
environment:java.class.path=:/opt/kafka-0.8.0/bin/../core/target/scala-2.8.0/kafka_2.8.0-0.8.0.jar:/opt/kafka-0.8.0/bin/../core/target/scala-2.8.0/kafka-assembly-0.8.0-deps.jar:/opt/kafka-0.8.0/bin/../perf/target/scala-2.8.0/kafka-perf_2.8.0-0.8.0.jar:/opt/kafka-0.8.0/bin/../libs/*.jar:/opt/kafka-0.8.0/bin/../kafka*.jar
(org.apache.zookeeper.ZooKeeper)
[2014-03-11 12:07:21,029] INFO Client
environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
(org.apache.zookeeper.ZooKeeper)
[2014-03-11 12:07:21,029] INFO Client environment:java.io.tmpdir=/tmp
(org.apache.zookeeper.ZooKeeper)
[2014-03-11 12:07:21,030] INFO Client environment:java.compiler=
(org.apache.zookeeper.ZooKeeper)
[2014-03-11 12:07:21,030] INFO Client environment:os.name=Linux
(org.apache.zookeeper.ZooKeeper)
[2014-03-11 12:07:21,030] INFO Client environment:os.arch=amd64
(org.apache.zookeeper.ZooKeeper)
[2014-03-11 12:07:21,030] INFO Client
environment:os.version=2.6.32-279.el6.x86_64 (org.apache.zookeeper.ZooKeeper)
[2014-03-11 12:07:21,030] INFO Client environment:user.name=root
(org.apache.zookeeper.ZooKeeper)
[2014-03-11 12:07:21,030] INFO Client environment:user.home=/root
(org.apache.zookeeper.ZooKeeper)
[2014-03-11 12:07:21,030] INFO Client environment:user.dir=/var/zookeeper
(org.apache.zookeeper.ZooKeeper)
[2014-03-11 12:07:21,031] INFO Initiating client connection,
connectString=192.168.1.120:2181 sessionTimeout=6000
watcher=org.I0Itec.zkclient.ZkClient@45419cee (org.apache.zookeeper.ZooKeeper)
[2014-03-11 12:07:21,043] INFO Opening socket connection to server
/192.168.1.120:2181 (org.apache.zookeeper.ClientCnxn)
[2014-03-11 12:07:21,049] INFO Socket connection established to
192.168.1.120/192.168.1.120:2181, initiating session
(org.apache.zookeeper.ClientCnxn)
[2014-03-11 12:07:21,058] INFO Session establishment complete on server
192.168.1.120/192.168.1.120:2181, sessionid = 0x144b104e0210005, negotiated
timeout = 6000 (org.apache.zookeeper.ClientCnxn)
[2014-03-11 12:07:21,060] INFO zookeeper state changed (SyncConnected)
(org.I0Itec.zkclient.ZkClient)
[2014-03-11 12:07:21,095] INFO Registered broker 1 at path /brokers/ids/1 with
address localhost:9092. (kafka.utils.ZkUtils$)
[2014-03-11 12:07:21,096] INFO [Kafka Server 1], Connecting to ZK:
192.168.1.120:2181 (kafka.server.KafkaServer)
[2014-03-11 12:07:21,222] INFO Will not load MX4J, mx4j-tools.jar is not in the
classpath (kafka.utils.Mx4jLoader$)
[2014-03-11 12:07:21,249] INFO conflict in /controller data: { "brokerid":1,
"timestamp":"1394539641229", "version":1 } stored data: { "brokerid":2,
"timestamp":"1394539533037", "version":1 } (kafka.utils.ZkUtils$)
[2014-03-11 12:07:21,469] INFO [Kafka Server 1], Started
(kafka.server.KafkaServer)
Could this be an issue?
INFO Registered broker 2 at path /brokers/ids/2 with address localhost:9092.
INFO Registered broker 1 at path /brokers/ids/1 with address localhost:9092.
> Date: Tue, 11 Mar 2014 09:51:54 -0700
> Subject: Re: Remote Zookeeper
> From: b...@b3k.us
> To: users@kafka.apache.org
>
> Seems you my have put your cluster in a very confused state with random
> addition and removal of brokers and topics. I'd suggest deleting the
> zookeeper and Kafka logs and starting over using the getting started
> tutorial from the wiki.
>
> On Tuesday, March 11, 2014, A A wrote:
>
> > I noticed a problem with my topics. The previous two topics would only be
> > present on 1 broker. So I created a new topic
> >
> > $KAFKA_HOME/bin/kafka-create-topic.sh --zookeeper
> > 192.168.1.120:2181--replica 2 --partition 1 --topic test-rep
> > $KAFKA_HOME/bin/kafka-list-topic.sh --zookeeper 192.168.1.120:2181
> > topic: test partition: 0leader: 1 replicas: 1 isr: 1
> > topic: test-rep partition: 0leader: 1 replicas: 1,2 isr: 1
> > topic: test1partition: 0leader: 1 replicas: 1 isr: 1
> >
> > No there is no firewall. Even then I stopped the iptables. And then I
> > tried the following
> > $KAFKA_HOME/bin/kafka-console-producer.sh --broker-list
> > 192.168.1.123:9092,192.168.1.124:9092 --topic test-rep
> > replicated message
> >
> > [2014-03-11 10:59:25,844] WARN Error while fetching metadata
> > [{TopicMetadata for topic test-rep ->
> > No partition metadata for topic test-rep due to
> > kafka.common.LeaderNotAvailableException}] for topic [test-rep]: class
> > kafka.common.LeaderNotAvailableException
> > (kafka.producer.BrokerPartitionInfo)
> > [2014-03-11 10:59:25,867] WARN Error while fetching metadata
> > [{TopicMetadata for topic test-rep -
Re: Remote Zookeeper
Seems you my have put your cluster in a very confused state with random
addition and removal of brokers and topics. I'd suggest deleting the
zookeeper and Kafka logs and starting over using the getting started
tutorial from the wiki.
On Tuesday, March 11, 2014, A A wrote:
> I noticed a problem with my topics. The previous two topics would only be
> present on 1 broker. So I created a new topic
>
> $KAFKA_HOME/bin/kafka-create-topic.sh --zookeeper 192.168.1.120:2181--replica
> 2 --partition 1 --topic test-rep
> $KAFKA_HOME/bin/kafka-list-topic.sh --zookeeper 192.168.1.120:2181
> topic: test partition: 0leader: 1 replicas: 1 isr: 1
> topic: test-rep partition: 0leader: 1 replicas: 1,2 isr: 1
> topic: test1partition: 0leader: 1 replicas: 1 isr: 1
>
> No there is no firewall. Even then I stopped the iptables. And then I
> tried the following
> $KAFKA_HOME/bin/kafka-console-producer.sh --broker-list
> 192.168.1.123:9092,192.168.1.124:9092 --topic test-rep
> replicated message
>
> [2014-03-11 10:59:25,844] WARN Error while fetching metadata
> [{TopicMetadata for topic test-rep ->
> No partition metadata for topic test-rep due to
> kafka.common.LeaderNotAvailableException}] for topic [test-rep]: class
> kafka.common.LeaderNotAvailableException
> (kafka.producer.BrokerPartitionInfo)
> [2014-03-11 10:59:25,867] WARN Error while fetching metadata
> [{TopicMetadata for topic test-rep ->
> No partition metadata for topic test-rep due to
> kafka.common.LeaderNotAvailableException}] for topic [test-rep]: class
> kafka.common.LeaderNotAvailableException
> (kafka.producer.BrokerPartitionInfo)
> [2014-03-11 10:59:25,868] ERROR Failed to collate messages by topic,
> partition due to: Failed to fetch topic metadata for topic: test-rep
> (kafka.producer.async.DefaultEventHandler)
> [2014-03-11 10:59:25,979] WARN Error while fetching metadata
> [{TopicMetadata for topic test-rep ->
> No partition metadata for topic test-rep due to
> kafka.common.LeaderNotAvailableException}] for topic [test-rep]: class
> kafka.common.LeaderNotAvailableException
> (kafka.producer.BrokerPartitionInfo)
> [2014-03-11 10:59:25,986] WARN Error while fetching metadata
> [{TopicMetadata for topic test-rep ->
> No partition metadata for topic test-rep due to
> kafka.common.LeaderNotAvailableException}] for topic [test-rep]: class
> kafka.common.LeaderNotAvailableException
> (kafka.producer.BrokerPartitionInfo)
> [2014-03-11 10:59:25,987] ERROR Failed to collate messages by topic,
> partition due to: Failed to fetch topic metadata for topic: test-rep
> (kafka.producer.async.DefaultEventHandler)
> [2014-03-11 10:59:26,098] WARN Error while fetching metadata
> [{TopicMetadata for topic test-rep ->
> No partition metadata for topic test-rep due to
> kafka.common.LeaderNotAvailableException}] for topic [test-rep]: class
> kafka.common.LeaderNotAvailableException
> (kafka.producer.BrokerPartitionInfo)
>
>
>
>
> > Date: Tue, 11 Mar 2014 09:05:54 -0700
> > Subject: Re: Remote Zookeeper
> > From: pradeep...@gmail.com
> > To: users@kafka.apache.org
> >
> > Is there a firewall thats blocking connections on port 9092? Also, the
> > broker list should be comma separated.
> >
> >
> > On Tue, Mar 11, 2014 at 9:02 AM, A A wrote:
> >
> > > Sorry one of the brokers for was down. Brought it back up. Tried the
> > > following
> > >
> > > $KAFKA_HOME/bin/kafka-console-producer.sh --broker-list localhost:9092
> > > 192.168.1.124:9092 --topic test
> > > hello brokers
> > >
> > > [2014-03-11 10:16:55,547] WARN Error while fetching metadata
> > > [{TopicMetadata for topic test ->
> > > No partition metadata for topic test due to
> > > kafka.common.LeaderNotAvailableException}] for topic [test]: class
> > > kafka.common.LeaderNotAvailableException
> > > (kafka.producer.BrokerPartitionInfo)
> > > [2014-03-11 10:16:55,576] WARN Error while fetching metadata
> > > [{TopicMetadata for topic test ->
> > > No partition metadata for topic test due to
> > > kafka.common.LeaderNotAvailableException}] for topic [test]: class
> > > kafka.common.LeaderNotAvailableException
> > > (kafka.producer.BrokerPartitionInfo)
> > > [2014-03-11 10:16:55,578] ERROR Failed to collate messages by topic,
> > > partition due to: Failed to fetch topic metadata for topic: test
> > > (kafka.producer.async.DefaultEventHandler)
> > > [2014-03-11 10:16:55,693] WARN Error while fetching metadata
> > > [{TopicMetadata for topic test ->
> > &
RE: Remote Zookeeper
I noticed a problem with my topics. The previous two topics would only be
present on 1 broker. So I created a new topic
$KAFKA_HOME/bin/kafka-create-topic.sh --zookeeper 192.168.1.120:2181 --replica
2 --partition 1 --topic test-rep
$KAFKA_HOME/bin/kafka-list-topic.sh --zookeeper 192.168.1.120:2181
topic: test partition: 0leader: 1 replicas: 1 isr: 1
topic: test-rep partition: 0leader: 1 replicas: 1,2 isr: 1
topic: test1partition: 0leader: 1 replicas: 1 isr: 1
No there is no firewall. Even then I stopped the iptables. And then I tried
the following
$KAFKA_HOME/bin/kafka-console-producer.sh --broker-list
192.168.1.123:9092,192.168.1.124:9092 --topic test-rep
replicated message
[2014-03-11 10:59:25,844] WARN Error while fetching metadata [{TopicMetadata
for topic test-rep ->
No partition metadata for topic test-rep due to
kafka.common.LeaderNotAvailableException}] for topic [test-rep]: class
kafka.common.LeaderNotAvailableException (kafka.producer.BrokerPartitionInfo)
[2014-03-11 10:59:25,867] WARN Error while fetching metadata [{TopicMetadata
for topic test-rep ->
No partition metadata for topic test-rep due to
kafka.common.LeaderNotAvailableException}] for topic [test-rep]: class
kafka.common.LeaderNotAvailableException (kafka.producer.BrokerPartitionInfo)
[2014-03-11 10:59:25,868] ERROR Failed to collate messages by topic, partition
due to: Failed to fetch topic metadata for topic: test-rep
(kafka.producer.async.DefaultEventHandler)
[2014-03-11 10:59:25,979] WARN Error while fetching metadata [{TopicMetadata
for topic test-rep ->
No partition metadata for topic test-rep due to
kafka.common.LeaderNotAvailableException}] for topic [test-rep]: class
kafka.common.LeaderNotAvailableException (kafka.producer.BrokerPartitionInfo)
[2014-03-11 10:59:25,986] WARN Error while fetching metadata [{TopicMetadata
for topic test-rep ->
No partition metadata for topic test-rep due to
kafka.common.LeaderNotAvailableException}] for topic [test-rep]: class
kafka.common.LeaderNotAvailableException (kafka.producer.BrokerPartitionInfo)
[2014-03-11 10:59:25,987] ERROR Failed to collate messages by topic, partition
due to: Failed to fetch topic metadata for topic: test-rep
(kafka.producer.async.DefaultEventHandler)
[2014-03-11 10:59:26,098] WARN Error while fetching metadata [{TopicMetadata
for topic test-rep ->
No partition metadata for topic test-rep due to
kafka.common.LeaderNotAvailableException}] for topic [test-rep]: class
kafka.common.LeaderNotAvailableException (kafka.producer.BrokerPartitionInfo)
> Date: Tue, 11 Mar 2014 09:05:54 -0700
> Subject: Re: Remote Zookeeper
> From: pradeep...@gmail.com
> To: users@kafka.apache.org
>
> Is there a firewall thats blocking connections on port 9092? Also, the
> broker list should be comma separated.
>
>
> On Tue, Mar 11, 2014 at 9:02 AM, A A wrote:
>
> > Sorry one of the brokers for was down. Brought it back up. Tried the
> > following
> >
> > $KAFKA_HOME/bin/kafka-console-producer.sh --broker-list localhost:9092
> > 192.168.1.124:9092 --topic test
> > hello brokers
> >
> > [2014-03-11 10:16:55,547] WARN Error while fetching metadata
> > [{TopicMetadata for topic test ->
> > No partition metadata for topic test due to
> > kafka.common.LeaderNotAvailableException}] for topic [test]: class
> > kafka.common.LeaderNotAvailableException
> > (kafka.producer.BrokerPartitionInfo)
> > [2014-03-11 10:16:55,576] WARN Error while fetching metadata
> > [{TopicMetadata for topic test ->
> > No partition metadata for topic test due to
> > kafka.common.LeaderNotAvailableException}] for topic [test]: class
> > kafka.common.LeaderNotAvailableException
> > (kafka.producer.BrokerPartitionInfo)
> > [2014-03-11 10:16:55,578] ERROR Failed to collate messages by topic,
> > partition due to: Failed to fetch topic metadata for topic: test
> > (kafka.producer.async.DefaultEventHandler)
> > [2014-03-11 10:16:55,693] WARN Error while fetching metadata
> > [{TopicMetadata for topic test ->
> > No partition metadata for topic test due to
> > kafka.common.LeaderNotAvailableException}] for topic [test]: class
> > kafka.common.LeaderNotAvailableException
> > (kafka.producer.BrokerPartitionInfo)
> > [2014-03-11 10:16:55,706] WARN Error while fetching metadata
> > [{TopicMetadata for topic test ->
> > No partition metadata for topic test due to
> > kafka.common.LeaderNotAvailableException}] for topic [test]: class
> > kafka.common.LeaderNotAvailableException
> > (kafka.producer.BrokerPartitionInfo)
> > [2014-03-11 10:16:55,706] ERROR Failed to collate messages by topic,
> > partition due to: Failed to fetch topic metadata for topi
Re: Remote Zookeeper
Is there a firewall thats blocking connections on port 9092? Also, the
broker list should be comma separated.
On Tue, Mar 11, 2014 at 9:02 AM, A A wrote:
> Sorry one of the brokers for was down. Brought it back up. Tried the
> following
>
> $KAFKA_HOME/bin/kafka-console-producer.sh --broker-list localhost:9092
> 192.168.1.124:9092 --topic test
> hello brokers
>
> [2014-03-11 10:16:55,547] WARN Error while fetching metadata
> [{TopicMetadata for topic test ->
> No partition metadata for topic test due to
> kafka.common.LeaderNotAvailableException}] for topic [test]: class
> kafka.common.LeaderNotAvailableException
> (kafka.producer.BrokerPartitionInfo)
> [2014-03-11 10:16:55,576] WARN Error while fetching metadata
> [{TopicMetadata for topic test ->
> No partition metadata for topic test due to
> kafka.common.LeaderNotAvailableException}] for topic [test]: class
> kafka.common.LeaderNotAvailableException
> (kafka.producer.BrokerPartitionInfo)
> [2014-03-11 10:16:55,578] ERROR Failed to collate messages by topic,
> partition due to: Failed to fetch topic metadata for topic: test
> (kafka.producer.async.DefaultEventHandler)
> [2014-03-11 10:16:55,693] WARN Error while fetching metadata
> [{TopicMetadata for topic test ->
> No partition metadata for topic test due to
> kafka.common.LeaderNotAvailableException}] for topic [test]: class
> kafka.common.LeaderNotAvailableException
> (kafka.producer.BrokerPartitionInfo)
> [2014-03-11 10:16:55,706] WARN Error while fetching metadata
> [{TopicMetadata for topic test ->
> No partition metadata for topic test due to
> kafka.common.LeaderNotAvailableException}] for topic [test]: class
> kafka.common.LeaderNotAvailableException
> (kafka.producer.BrokerPartitionInfo)
> [2014-03-11 10:16:55,706] ERROR Failed to collate messages by topic,
> partition due to: Failed to fetch topic metadata for topic: test
> (kafka.producer.async.DefaultEventHandler)
> [2014-03-11 10:16:55,815] WARN Error while fetching metadata
> [{TopicMetadata for topic test ->
> No partition metadata for topic test due to
> kafka.common.LeaderNotAvailableException}] for topic [test]: class
> kafka.common.LeaderNotAvailableException
> (kafka.producer.BrokerPartitionInfo)
> [2014-03-11 10:16:55,823] WARN Error while fetching metadata
> [{TopicMetadata for topic test ->
> No partition metadata for topic test due to
> kafka.common.LeaderNotAvailableException}] for topic [test]: class
> kafka.common.LeaderNotAvailableException
> (kafka.producer.BrokerPartitionInfo)
> [2014-03-11 10:16:55,823] ERROR Failed to collate messages by topic,
> partition due to: Failed to fetch topic metadata for topic: test
> (kafka.producer.async.DefaultEventHandler)
> [2014-03-11 10:16:55,934] WARN Error while fetching metadata
> [{TopicMetadata for topic test ->
> No partition metadata for topic test due to
> kafka.common.LeaderNotAvailableException}] for topic [test]: class
> kafka.common.LeaderNotAvailableException
> (kafka.producer.BrokerPartitionInfo)
> [2014-03-11 10:16:55,949] WARN Error while fetching metadata
> [{TopicMetadata for topic test ->
> No partition metadata for topic test due to
> kafka.common.LeaderNotAvailableException}] for topic [test]: class
> kafka.common.LeaderNotAvailableException
> (kafka.producer.BrokerPartitionInfo)
> [2014-03-11 10:16:55,949] ERROR Failed to collate messages by topic,
> partition due to: Failed to fetch topic metadata for topic: test
> (kafka.producer.async.DefaultEventHandler)
> [2014-03-11 10:16:56,057] WARN Error while fetching metadata
> [{TopicMetadata for topic test ->
> No partition metadata for topic test due to
> kafka.common.LeaderNotAvailableException}] for topic [test]: class
> kafka.common.LeaderNotAvailableException
> (kafka.producer.BrokerPartitionInfo)
> [2014-03-11 10:16:56,059] ERROR Failed to send requests for topics test
> with correlation ids in [0,8] (kafka.producer.async.DefaultEventHandler)
> [2014-03-11 10:16:56,061] ERROR Error in handling batch of 1 events
> (kafka.producer.async.ProducerSendThread)
> kafka.common.FailedToSendMessageException: Failed to send messages after 3
> tries.
> at
> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
> at
> kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)
> at
> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87)
> at
> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67)
> at scala.collection.immutable.Stream.foreach(Stream.scala:254)
> at
> kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThre
RE: Remote Zookeeper
Sorry one of the brokers for was down. Brought it back up. Tried the following
$KAFKA_HOME/bin/kafka-console-producer.sh --broker-list localhost:9092
192.168.1.124:9092 --topic test
hello brokers
[2014-03-11 10:16:55,547] WARN Error while fetching metadata [{TopicMetadata
for topic test ->
No partition metadata for topic test due to
kafka.common.LeaderNotAvailableException}] for topic [test]: class
kafka.common.LeaderNotAvailableException (kafka.producer.BrokerPartitionInfo)
[2014-03-11 10:16:55,576] WARN Error while fetching metadata [{TopicMetadata
for topic test ->
No partition metadata for topic test due to
kafka.common.LeaderNotAvailableException}] for topic [test]: class
kafka.common.LeaderNotAvailableException (kafka.producer.BrokerPartitionInfo)
[2014-03-11 10:16:55,578] ERROR Failed to collate messages by topic, partition
due to: Failed to fetch topic metadata for topic: test
(kafka.producer.async.DefaultEventHandler)
[2014-03-11 10:16:55,693] WARN Error while fetching metadata [{TopicMetadata
for topic test ->
No partition metadata for topic test due to
kafka.common.LeaderNotAvailableException}] for topic [test]: class
kafka.common.LeaderNotAvailableException (kafka.producer.BrokerPartitionInfo)
[2014-03-11 10:16:55,706] WARN Error while fetching metadata [{TopicMetadata
for topic test ->
No partition metadata for topic test due to
kafka.common.LeaderNotAvailableException}] for topic [test]: class
kafka.common.LeaderNotAvailableException (kafka.producer.BrokerPartitionInfo)
[2014-03-11 10:16:55,706] ERROR Failed to collate messages by topic, partition
due to: Failed to fetch topic metadata for topic: test
(kafka.producer.async.DefaultEventHandler)
[2014-03-11 10:16:55,815] WARN Error while fetching metadata [{TopicMetadata
for topic test ->
No partition metadata for topic test due to
kafka.common.LeaderNotAvailableException}] for topic [test]: class
kafka.common.LeaderNotAvailableException (kafka.producer.BrokerPartitionInfo)
[2014-03-11 10:16:55,823] WARN Error while fetching metadata [{TopicMetadata
for topic test ->
No partition metadata for topic test due to
kafka.common.LeaderNotAvailableException}] for topic [test]: class
kafka.common.LeaderNotAvailableException (kafka.producer.BrokerPartitionInfo)
[2014-03-11 10:16:55,823] ERROR Failed to collate messages by topic, partition
due to: Failed to fetch topic metadata for topic: test
(kafka.producer.async.DefaultEventHandler)
[2014-03-11 10:16:55,934] WARN Error while fetching metadata [{TopicMetadata
for topic test ->
No partition metadata for topic test due to
kafka.common.LeaderNotAvailableException}] for topic [test]: class
kafka.common.LeaderNotAvailableException (kafka.producer.BrokerPartitionInfo)
[2014-03-11 10:16:55,949] WARN Error while fetching metadata [{TopicMetadata
for topic test ->
No partition metadata for topic test due to
kafka.common.LeaderNotAvailableException}] for topic [test]: class
kafka.common.LeaderNotAvailableException (kafka.producer.BrokerPartitionInfo)
[2014-03-11 10:16:55,949] ERROR Failed to collate messages by topic, partition
due to: Failed to fetch topic metadata for topic: test
(kafka.producer.async.DefaultEventHandler)
[2014-03-11 10:16:56,057] WARN Error while fetching metadata [{TopicMetadata
for topic test ->
No partition metadata for topic test due to
kafka.common.LeaderNotAvailableException}] for topic [test]: class
kafka.common.LeaderNotAvailableException (kafka.producer.BrokerPartitionInfo)
[2014-03-11 10:16:56,059] ERROR Failed to send requests for topics test with
correlation ids in [0,8] (kafka.producer.async.DefaultEventHandler)
[2014-03-11 10:16:56,061] ERROR Error in handling batch of 1 events
(kafka.producer.async.ProducerSendThread)
kafka.common.FailedToSendMessageException: Failed to send messages after 3
tries.
at
kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
at
kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)
at
kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87)
at
kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67)
at scala.collection.immutable.Stream.foreach(Stream.scala:254)
at
kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66)
at
kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44)
From: andthereitg...@hotmail.com
To: users@kafka.apache.org
Subject: RE: Remote Zookeeper
Date: Tue, 11 Mar 2014 15:59:44 +
Okay thanks. Just to verify my setup I tried the following on broker1 (by
publishing to localhost)
$KAFKA_HOME/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic
test
test message
this is a test message
another test message
$KAFKA_HOME/bin/kafka-console-consumer.sh --zookeeper 192.168.1.120:2181
--topic
RE: Remote Zookeeper
Okay thanks. Just to verify my setup I tried the following on broker1 (by
publishing to localhost)
$KAFKA_HOME/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic
test
test message
this is a test message
another test message
$KAFKA_HOME/bin/kafka-console-consumer.sh --zookeeper 192.168.1.120:2181
--topic test --from-beginning
test message
this is a test message
another test message
Consumed 3 messages
This works fine. How do I provide the broker-list?
I tried the following on broker1 (192.168.1.123)
$KAFKA_HOME/bin/kafka-console-producer.sh --broker-list 192.168.1.124:9092
192.168.1.123:9092 --topic test
$KAFKA_HOME/bin/kafka-console-producer.sh --broker-list
192.168.1.124:9092,192.168.1.123:9092 --topic test
$KAFKA_HOME/bin/kafka-console-producer.sh --broker-list localhost:9092
192.168.1.123:9092 --topic test
$KAFKA_HOME/bin/kafka-console-producer.sh --broker-list
localhost:9092,192.168.1.123:9092 --topic test
Didn't work. I either get connection refused or no route to host exceptions.
> Date: Tue, 11 Mar 2014 08:38:22 -0700
> Subject: Re: Remote Zookeeper
> From: b...@b3k.us
> To: users@kafka.apache.org
>
> That's not how Kafka works. You need to pass the full list of brokers.
>
> On Tuesday, March 11, 2014, A A wrote:
>
> > Hi again.
> >
> > Got the setup working. I now have 2 brokers (broker 1 and broker 2) with
> > one remote zk. I was also able to create some topics
> >
> > $KAFKA_HOME/bin/kafka-list-topic.sh --zookeeper 192.168.1.120:2181
> > topic: test partition: 0leader: 1 replicas: 1 isr: 1
> > topic: test1partition: 0leader: 1 replicas: 1 isr: 1
> >
> > I then try to publish a message from broker1 to broker 1 (localhost) while
> > I am consuming on broker2
> > $KAFKA_HOME/bin/kafka-console-producer.sh --broker-list localhost:9092
> > --topic test1
> > hell
> > [2014-03-11 07:36:35,206] WARN Error while fetching metadata
> > [{TopicMetadata for topic test1 ->
> > No partition metadata for topic test1 due to
> > kafka.common.LeaderNotAvailableException}] for topic [test1]: class
> > kafka.common.LeaderNotAvailableException (kafka.pr
> >oducer.BrokerPartitionInfo)
> > [2014-03-11 07:36:35,244] WARN Error while fetching metadata
> > [{TopicMetadata for topic test1 ->
> > No partition metadata for topic test1 due to
> > kafka.common.LeaderNotAvailableException}] for topic [test1]: class
> > kafka.common.LeaderNotAvailableException (kafka.pr
> >oducer.BrokerPartitionInfo)
> > [2014-03-11 07:36:35,248] ERROR Failed to collate messages by topic,
> > partition due to: Failed to fetch topic metadata for topic: test1
> > (kafka.producer.async.Default
> > EventHandler)
> > [2014-03-11 07:36:35,358] WARN Error while fetching metadata
> > [{TopicMetadata for topic test1 ->
> > No partition metadata for topic test1 due to
> > kafka.common.LeaderNotAvailableException}] for topic [test1]: class
> > kafka.common.LeaderNotAvailableException (kafka.pr
> >oducer.BrokerPartitionInfo)
> > [2014-03-11 07:36:35,368] WARN Error while fetching metadata
> > [{TopicMetadata for topic test1 ->
> > No partition metadata for topic test1 due to
> > kafka.common.LeaderNotAvailableException}] for topic [test1]: class
> > kafka.common.LeaderNotAvailableException (kafka.pr
> >oducer.BrokerPartitionInfo)
> > [2014-03-11 07:36:35,368] ERROR Failed to collate messages by topic,
> > partition due to: Failed to fetch topic metadata for topic: test1
> > (kafka.producer.async.Default
> > EventHandler)
> > [2014-03-11 07:36:35,476] WARN Error while fetching metadata
> > [{TopicMetadata for topic test1 ->
> > No partition metadata for topic test1 due to
> > kafka.common.LeaderNotAvailableException}] for topic [test1]: class
> > kafka.common.LeaderNotAvailableException (kafka.pr
> >oducer.BrokerPartitionInfo)
> > [2014-03-11 07:36:35,490] WARN Error while fetching metadata
> > [{TopicMetadata for topic test1 ->
> > No partition metadata for topic test1 due to
> > kafka.common.LeaderNotAvailableException}] for topic [test1]: class
> > kafka.common.LeaderNotAvailableException (kafka.pr
> >oducer.BrokerPartitionInfo)
> > [2014-03-11 07:36:35,490] ERROR Failed to collate messages by topic,
> > partition due to: Failed to fetch topic metadata for topic: test1
> > (kafka.producer.async.Default
> > EventHandler)
> > [2014-03-11 07:36:35,599] WARN Error while fetching metadata
> > [{TopicMetadata for topic test1 ->
> &
Re: Remote Zookeeper
le(DefaultEventHandler.scala:90) > at > kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104) > at > kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87) > at > kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67) > at scala.collection.immutable.Stream.foreach(Stream.scala:254) > at > kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66) > at > kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44) > > What am I missing? > > > From: andthereitg...@hotmail.com > > To: users@kafka.apache.org > > Subject: RE: Remote Zookeeper > > Date: Tue, 11 Mar 2014 02:29:34 + > > > > Thanks. Totally missed that. > > > > > From: b...@b3k.us > > > Date: Mon, 10 Mar 2014 19:18:50 -0700 > > > Subject: Re: Remote Zookeeper > > > To: users@kafka.apache.org > > > > > > zookeeper.connect > > > > > > https://kafka.apache.org/08/configuration.html > > > > > > > > > On Mon, Mar 10, 2014 at 7:17 PM, A A > > > > > wrote: > > > > > > > Hi > > > > > > > > Pretty new to Kafka. Have been successful in installing Kafka 0.8.0. > > > > I am just wondering how should I make my kafka cluster (2 brokers) > connect > > > > to a single remote zookeper server? > > > > > > > > I am using $KAFKA/kafka-server-start.sh > $KAFKA_CONFIG/server.properties > > > > on both the brokers to start them up > > > > and I have a remote zookeeper running at 192.168.1.120 > > > > > > > > Can anyone help? > > > > > > > > A > > > > > > > > > > >
RE: Remote Zookeeper
onfun$processEvents$3.apply(ProducerSendThread.scala:87) at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67) at scala.collection.immutable.Stream.foreach(Stream.scala:254) at kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66) at kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44) What am I missing? > From: andthereitg...@hotmail.com > To: users@kafka.apache.org > Subject: RE: Remote Zookeeper > Date: Tue, 11 Mar 2014 02:29:34 + > > Thanks. Totally missed that. > > > From: b...@b3k.us > > Date: Mon, 10 Mar 2014 19:18:50 -0700 > > Subject: Re: Remote Zookeeper > > To: users@kafka.apache.org > > > > zookeeper.connect > > > > https://kafka.apache.org/08/configuration.html > > > > > > On Mon, Mar 10, 2014 at 7:17 PM, A A wrote: > > > > > Hi > > > > > > Pretty new to Kafka. Have been successful in installing Kafka 0.8.0. > > > I am just wondering how should I make my kafka cluster (2 brokers) connect > > > to a single remote zookeper server? > > > > > > I am using $KAFKA/kafka-server-start.sh $KAFKA_CONFIG/server.properties > > > on both the brokers to start them up > > > and I have a remote zookeeper running at 192.168.1.120 > > > > > > Can anyone help? > > > > > > A > > > > > > >
RE: Remote Zookeeper
Thanks. Totally missed that. > From: b...@b3k.us > Date: Mon, 10 Mar 2014 19:18:50 -0700 > Subject: Re: Remote Zookeeper > To: users@kafka.apache.org > > zookeeper.connect > > https://kafka.apache.org/08/configuration.html > > > On Mon, Mar 10, 2014 at 7:17 PM, A A wrote: > > > Hi > > > > Pretty new to Kafka. Have been successful in installing Kafka 0.8.0. > > I am just wondering how should I make my kafka cluster (2 brokers) connect > > to a single remote zookeper server? > > > > I am using $KAFKA/kafka-server-start.sh $KAFKA_CONFIG/server.properties > > on both the brokers to start them up > > and I have a remote zookeeper running at 192.168.1.120 > > > > Can anyone help? > > > > A > > > >
Re: Remote Zookeeper
zookeeper.connect https://kafka.apache.org/08/configuration.html On Mon, Mar 10, 2014 at 7:17 PM, A A wrote: > Hi > > Pretty new to Kafka. Have been successful in installing Kafka 0.8.0. > I am just wondering how should I make my kafka cluster (2 brokers) connect > to a single remote zookeper server? > > I am using $KAFKA/kafka-server-start.sh $KAFKA_CONFIG/server.properties > on both the brokers to start them up > and I have a remote zookeeper running at 192.168.1.120 > > Can anyone help? > > A > >
