Re: LeaderNotAvailable Exception
Did you create the topic without a # of partitions then try to delete/recreate it? I've had that happen to me before. Try shutting down everything (including zookeeper) and restarting. On Tue, Apr 23, 2013 at 9:08 PM, Jun Rao jun...@gmail.com wrote: Does this happen on every message that you type in producer console? Thanks, Jun On Tue, Apr 23, 2013 at 4:15 PM, Yin Yin yin@outlook.com wrote: I tried to run the kafka 0.8 version as instructed in Quick Start. The kafka server shows the following message when I launch the producer. ERROR Error while fetching metadata for partition [test,0] (kafka.admin.AdminUtils$) kafka.common.LeaderNotAvailableException: No leader exists for partition 0 at kafka.admin.AdminUtils$$anonfun$3.apply(AdminUtils.scala:219) at kafka.admin.AdminUtils$$anonfun$3.apply(AdminUtils.scala:201) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206) at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:61) at scala.collection.immutable.List.foreach(List.scala:45) at scala.collection.TraversableLike$class.map(TraversableLike.scala:206) at scala.collection.immutable.List.map(List.scala:45) at kafka.admin.AdminUtils$.kafka$admin$AdminUtils$$fetchTopicMetadataFromZk(AdminUtils.scala:201) at kafka.admin.AdminUtils$.fetchTopicMetadataFromZk(AdminUtils.scala:190) at kafka.server.KafkaApis$$anonfun$handleTopicMetadataRequest$1.apply(KafkaApis.scala:479) at kafka.server.KafkaApis$$anonfun$handleTopicMetadataRequest$1.apply(KafkaApis.scala:465) at scala.collection.immutable.Set$Set1.foreach(Set.scala:81) at kafka.server.KafkaApis.handleTopicMetadataRequest(KafkaApis.scala:464) at kafka.server.KafkaApis.handle(KafkaApis.scala:67) at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:41) at java.lang.Thread.run(Unknown Source) Then, when I type a message in the producer console, the server console pops out WARN [KafkaApi-0] Leader not local for topic test partition 0 on broker 0 (kafka.server.KafkaApis) Consumer console also didn't get any message. Any help is appreciated. Thanks
Re: How to compile java files in examples directory bundled in kafka 0.8?
Thanks a lot. Regards, Libo
Re: Occasional batch send errors
Thanks Andrew, I'm not seeing the event queue exception but, I'm running my cluster on a set of virtual machines which share the same physical hardware (I know, exactly what I'm not supposed to do) and I'm getting some slow fsync zookeeper warnings in my logs. I imagine that my broker writes are slow as well. It could be that I'm getting write contention because of the shared metal/disk. I know kafka and zookeeper are susceptible to latency spikes... Karl On Apr 23, 2013, at 7:16 PM, Andrew Neilson arsneil...@gmail.com wrote: Hey Karl, I have a very similar setup (3 kafka 0.7.2 brokers, 3 ZK 3.4.3 nodes) that I'm running right now and am getting the same error on the producers. Haven't resolved it yet: ERROR ProducerSendThread--1585663279 kafka.producer.async.ProducerSendThread - Error in handling batch of 200 events java.io.IOException: Connection reset by peer ... For me these errors appear to be coupled with other errors like this: kafka.producer.async.AsyncProducer - Event queue is full of unsent messages, could not send event: As I understand it, this happens when you are producing faster than the brokers can persist the messages. It's possible these are two different issues... Anyway I've been doing a lot of work on this this afternoon so I may have more information later. Someone else probably knows more though. On Tue, Apr 23, 2013 at 4:57 PM, Karl Kirch kki...@wdtinc.com wrote: Hmmm… that didn't seem to help. Anyone else see this sort of errors? Karl On Apr 23, 2013, at 5:58 PM, Karl Kirch kki...@wdtinc.com wrote: I'm going to try bumping up the numRetries key in my producer config. Is this a good option in this case? I am using the zookeeper connect option so I'm aware that I may get stuck retrying to a failed node, but if it's just a temporary network glitch I'll at least get a bit more of a chance to recover. Thanks, Karl On Apr 23, 2013, at 5:35 PM, Karl Kirch kki...@wdtinc.com wrote: I occasionally am getting some batch send errors from the stock async producer. This is on a cluster of 3 kafka (0.7.2) and 3 zookeeper nodes. Is there anyway to check what happens when those batch errors occur? Or bump up the retry count? (looks like it only did a single retry). I need the speed of the async producer, but it needs to be reliable (I see a handful of these a day but in a weather alerting system it only takes missing one let alone 25 or 100/1000). Here's a stack trace of one of the errors that I'm seeing. 22:23:39.405 [ProducerSendThread-1824508747] WARN k.p.a.DefaultEventHandler - Error sending messages, 0 attempts remaining java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcher.writev0(Native Method) ~[na:1.6.0_24] at sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:33) ~[na:1.6.0_24] at sun.nio.ch.IOUtil.write(IOUtil.java:125) ~[na:1.6.0_24] at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:367) ~[na:1.6.0_24] at java.nio.channels.SocketChannel.write(SocketChannel.java:360) ~[na:1.6.0_24] at kafka.network.BoundedByteBufferSend.writeTo(BoundedByteBufferSend.scala:49) ~[apns-consumer-1.0.jar:na] at kafka.network.Send$class.writeCompletely(Transmission.scala:73) ~[apns-consumer-1.0.jar:na] at kafka.network.BoundedByteBufferSend.writeCompletely(BoundedByteBufferSend.scala:25) ~[apns-consumer-1.0.jar:na] at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:95) ~[apns-consumer-1.0.jar:na] at kafka.producer.SyncProducer.send(SyncProducer.scala:94) ~[apns-consumer-1.0.jar:na] at kafka.producer.SyncProducer.multiSend(SyncProducer.scala:135) ~[apns-consumer-1.0.jar:na] at kafka.producer.async.DefaultEventHandler.send(DefaultEventHandler.scala:58) [apns-consumer-1.0.jar:na] at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:44) [apns-consumer-1.0.jar:na] at kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:116) [apns-consumer-1.0.jar:na] at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:95) [apns-consumer-1.0.jar:na] at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:71) [apns-consumer-1.0.jar:na] at scala.collection.immutable.Stream.foreach(Stream.scala:260) [apns-consumer-1.0.jar:na] at kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:70) [apns-consumer-1.0.jar:na] at kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:41) [apns-consumer-1.0.jar:na] 22:23:39.406 [ProducerSendThread-1824508747] ERROR k.p.a.ProducerSendThread - Error in handling batch of 27 events java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcher.writev0(Native Method) ~[na:1.6.0_24] at sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:33)
Node-Kafka Client Review and Question
Hi Everyone, I just wanted to follow-up on a previous thread concerning our investigation of identifying a stable Node-Kafka client. To date we have tested the following: 1. Franz-Kafka (https://github.com/dannycoates/franz-kafka) 2. Node-Kafka (v2.1, https://github.com/radekg/node-kafka) 3. Node-Kafka (v2.3, https://github.com/marcuswestin/node-kafka) 4. Prozess (v0.3.5, https://github.com/cainus/Prozess) Results: 1. Could not get Franz-Kafka and Prozess to work. Requires funky dependencies. 2. Node-Kafka, v2.1 was successfully setup but performed less stable than #3. 3. Node-Kafka, v2.3 was successfully setup, exhibited the best performance profile but the consumer is highly inconsistent - specifically, consumer object remained in-memory regardless what we did (i.e. var consumer = undefined after receiving message). Nothing appears to mitigate this and ALL consumed messaged get replayed on reception of a new message. With this said, is there a Node-Kafka client people are actually using in production that doesn't exhibit the profiles we have seen? We have back-tracked using Node-Kafka (v2.3) to only produce messages and rely on Redis PubSub channels for asynchronous acking of these messages. We would be willing to roll-up our sleeves with the community to develop a much more stable Node-Kafka client. Kind regards, Chris Alexander Chief Technical Architect and Engineer Gravy, Inc.
Re: zookeeper version issue
According to what I tried, kafka 0.8 works with zookeepr 3.4.3. Regards, Libo
Re: Node-Kafka Client Review and Question
Hi Everyone, I have been experimenting with the libraries listed below and experienced the same problems. I have not found any another other node clients. I am interested in finding a node solution as well. Happy to contribute on a common solution. Christian Carollo On Apr 24, 2013, at 10:19 AM, Christopher Alexander calexan...@gravycard.com wrote: Hi Everyone, I just wanted to follow-up on a previous thread concerning our investigation of identifying a stable Node-Kafka client. To date we have tested the following: 1. Franz-Kafka (https://github.com/dannycoates/franz-kafka) 2. Node-Kafka (v2.1, https://github.com/radekg/node-kafka) 3. Node-Kafka (v2.3, https://github.com/marcuswestin/node-kafka) 4. Prozess (v0.3.5, https://github.com/cainus/Prozess) Results: 1. Could not get Franz-Kafka and Prozess to work. Requires funky dependencies. 2. Node-Kafka, v2.1 was successfully setup but performed less stable than #3. 3. Node-Kafka, v2.3 was successfully setup, exhibited the best performance profile but the consumer is highly inconsistent - specifically, consumer object remained in-memory regardless what we did (i.e. var consumer = undefined after receiving message). Nothing appears to mitigate this and ALL consumed messaged get replayed on reception of a new message. With this said, is there a Node-Kafka client people are actually using in production that doesn't exhibit the profiles we have seen? We have back-tracked using Node-Kafka (v2.3) to only produce messages and rely on Redis PubSub channels for asynchronous acking of these messages. We would be willing to roll-up our sleeves with the community to develop a much more stable Node-Kafka client. Kind regards, Chris Alexander Chief Technical Architect and Engineer Gravy, Inc.
Multiple log directories in 0.7.2
Hi, I'm aware of KAFKA-188 which implements multiple log directories in kafka 0.8. Is there a way to backport this functionality in 0.7.2? Thanks, Anand
Re: Occasional batch send errors
So I'm seeing CancelledKeyExceptions cropping up about the time that the connections get reset. Is this a zookeeper error that I'm hitting? Karl On Apr 24, 2013, at 9:55 AM, Karl Kirch kki...@wdtinc.com wrote: Just got logging cranked up. Will let you know when I see it again. Thanks, Karl On Apr 23, 2013, at 8:11 PM, Jun Rao jun...@gmail.com wrote: This means that the broker closed the socket connection for some reason. The broker log around the same time should show the reason. Could you dig that out? Thanks, Jun On Tue, Apr 23, 2013 at 3:35 PM, Karl Kirch kki...@wdtinc.com wrote: I occasionally am getting some batch send errors from the stock async producer. This is on a cluster of 3 kafka (0.7.2) and 3 zookeeper nodes. Is there anyway to check what happens when those batch errors occur? Or bump up the retry count? (looks like it only did a single retry). I need the speed of the async producer, but it needs to be reliable (I see a handful of these a day but in a weather alerting system it only takes missing one let alone 25 or 100/1000). Here's a stack trace of one of the errors that I'm seeing. 22:23:39.405 [ProducerSendThread-1824508747] WARN k.p.a.DefaultEventHandler - Error sending messages, 0 attempts remaining java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcher.writev0(Native Method) ~[na:1.6.0_24] at sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:33) ~[na:1.6.0_24] at sun.nio.ch.IOUtil.write(IOUtil.java:125) ~[na:1.6.0_24] at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:367) ~[na:1.6.0_24] at java.nio.channels.SocketChannel.write(SocketChannel.java:360) ~[na:1.6.0_24] at kafka.network.BoundedByteBufferSend.writeTo(BoundedByteBufferSend.scala:49) ~[apns-consumer-1.0.jar:na] at kafka.network.Send$class.writeCompletely(Transmission.scala:73) ~[apns-consumer-1.0.jar:na] at kafka.network.BoundedByteBufferSend.writeCompletely(BoundedByteBufferSend.scala:25) ~[apns-consumer-1.0.jar:na] at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:95) ~[apns-consumer-1.0.jar:na] at kafka.producer.SyncProducer.send(SyncProducer.scala:94) ~[apns-consumer-1.0.jar:na] at kafka.producer.SyncProducer.multiSend(SyncProducer.scala:135) ~[apns-consumer-1.0.jar:na] at kafka.producer.async.DefaultEventHandler.send(DefaultEventHandler.scala:58) [apns-consumer-1.0.jar:na] at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:44) [apns-consumer-1.0.jar:na] at kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:116) [apns-consumer-1.0.jar:na] at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:95) [apns-consumer-1.0.jar:na] at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:71) [apns-consumer-1.0.jar:na] at scala.collection.immutable.Stream.foreach(Stream.scala:260) [apns-consumer-1.0.jar:na] at kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:70) [apns-consumer-1.0.jar:na] at kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:41) [apns-consumer-1.0.jar:na] 22:23:39.406 [ProducerSendThread-1824508747] ERROR k.p.a.ProducerSendThread - Error in handling batch of 27 events java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcher.writev0(Native Method) ~[na:1.6.0_24] at sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:33) ~[na:1.6.0_24] at sun.nio.ch.IOUtil.write(IOUtil.java:125) ~[na:1.6.0_24] at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:367) ~[na:1.6.0_24] at java.nio.channels.SocketChannel.write(SocketChannel.java:360) ~[na:1.6.0_24] at kafka.network.BoundedByteBufferSend.writeTo(BoundedByteBufferSend.scala:49) ~[apns-consumer-1.0.jar:na] at kafka.network.Send$class.writeCompletely(Transmission.scala:73) ~[apns-consumer-1.0.jar:na] at kafka.network.BoundedByteBufferSend.writeCompletely(BoundedByteBufferSend.scala:25) ~[apns-consumer-1.0.jar:na] at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:95) ~[apns-consumer-1.0.jar:na] at kafka.producer.SyncProducer.send(SyncProducer.scala:94) ~[apns-consumer-1.0.jar:na] at kafka.producer.SyncProducer.multiSend(SyncProducer.scala:135) ~[apns-consumer-1.0.jar:na] at kafka.producer.async.DefaultEventHandler.send(DefaultEventHandler.scala:58) ~[apns-consumer-1.0.jar:na] at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:44) ~[apns-consumer-1.0.jar:na] at kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:116) [apns-consumer-1.0.jar:na] at
Re: Occasional batch send errors
So switched to sync producer to see what would happen. I still get the connection reset by peer error randomly (I say randomly, but seems to be connected to some zookeeper CancelledKeyExceptions), but unfortunately it throws an error on the message after the one that didn't get sent. Is that the way its supposed to work? Karl On Apr 23, 2013, at 7:18 PM, Xavier Stevens xav...@gaikai.com wrote: Usually when these types of errors are because you're not connecting to the proper host:port. Double check your configs, make sure everything is running and listening on the host:port you think they are. Have you tried using the sync producer to work out your bugs? My guess is the sync producer would fail on the first message rather than failing when the batch is submitted. On Tue, Apr 23, 2013 at 4:57 PM, Karl Kirch kki...@wdtinc.com wrote: Hmmm… that didn't seem to help. Anyone else see this sort of errors? Karl On Apr 23, 2013, at 5:58 PM, Karl Kirch kki...@wdtinc.com wrote: I'm going to try bumping up the numRetries key in my producer config. Is this a good option in this case? I am using the zookeeper connect option so I'm aware that I may get stuck retrying to a failed node, but if it's just a temporary network glitch I'll at least get a bit more of a chance to recover. Thanks, Karl On Apr 23, 2013, at 5:35 PM, Karl Kirch kki...@wdtinc.com wrote: I occasionally am getting some batch send errors from the stock async producer. This is on a cluster of 3 kafka (0.7.2) and 3 zookeeper nodes. Is there anyway to check what happens when those batch errors occur? Or bump up the retry count? (looks like it only did a single retry). I need the speed of the async producer, but it needs to be reliable (I see a handful of these a day but in a weather alerting system it only takes missing one let alone 25 or 100/1000). Here's a stack trace of one of the errors that I'm seeing. 22:23:39.405 [ProducerSendThread-1824508747] WARN k.p.a.DefaultEventHandler - Error sending messages, 0 attempts remaining java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcher.writev0(Native Method) ~[na:1.6.0_24] at sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:33) ~[na:1.6.0_24] at sun.nio.ch.IOUtil.write(IOUtil.java:125) ~[na:1.6.0_24] at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:367) ~[na:1.6.0_24] at java.nio.channels.SocketChannel.write(SocketChannel.java:360) ~[na:1.6.0_24] at kafka.network.BoundedByteBufferSend.writeTo(BoundedByteBufferSend.scala:49) ~[apns-consumer-1.0.jar:na] at kafka.network.Send$class.writeCompletely(Transmission.scala:73) ~[apns-consumer-1.0.jar:na] at kafka.network.BoundedByteBufferSend.writeCompletely(BoundedByteBufferSend.scala:25) ~[apns-consumer-1.0.jar:na] at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:95) ~[apns-consumer-1.0.jar:na] at kafka.producer.SyncProducer.send(SyncProducer.scala:94) ~[apns-consumer-1.0.jar:na] at kafka.producer.SyncProducer.multiSend(SyncProducer.scala:135) ~[apns-consumer-1.0.jar:na] at kafka.producer.async.DefaultEventHandler.send(DefaultEventHandler.scala:58) [apns-consumer-1.0.jar:na] at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:44) [apns-consumer-1.0.jar:na] at kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:116) [apns-consumer-1.0.jar:na] at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:95) [apns-consumer-1.0.jar:na] at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:71) [apns-consumer-1.0.jar:na] at scala.collection.immutable.Stream.foreach(Stream.scala:260) [apns-consumer-1.0.jar:na] at kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:70) [apns-consumer-1.0.jar:na] at kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:41) [apns-consumer-1.0.jar:na] 22:23:39.406 [ProducerSendThread-1824508747] ERROR k.p.a.ProducerSendThread - Error in handling batch of 27 events java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcher.writev0(Native Method) ~[na:1.6.0_24] at sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:33) ~[na:1.6.0_24] at sun.nio.ch.IOUtil.write(IOUtil.java:125) ~[na:1.6.0_24] at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:367) ~[na:1.6.0_24] at java.nio.channels.SocketChannel.write(SocketChannel.java:360) ~[na:1.6.0_24] at kafka.network.BoundedByteBufferSend.writeTo(BoundedByteBufferSend.scala:49) ~[apns-consumer-1.0.jar:na] at kafka.network.Send$class.writeCompletely(Transmission.scala:73) ~[apns-consumer-1.0.jar:na] at kafka.network.BoundedByteBufferSend.writeCompletely(BoundedByteBufferSend.scala:25) ~[apns-consumer-1.0.jar:na]
Broker List vs Zookeeper
Any advice on using a static broker list vs using zookeeper? I keep having fits with keeping things stable with zookeeper involved (i.e. dropped connections). If I use a static broker list do I still get failover if a broker goes down? (i.e. 1 broker goes down, will my producers still try to send message to it?) Thanks, Karl
Advice on cacheing producers
Is there a best practice on how to handle producer objects for long running apps? Right now I have an app that is long running and will sit for large stretches of time (days/weeks) with next to none load and then get slammed. In that case I'd like to cache the producer so I don't incur a hit in creating a producer connection. Is it a problem to keep that producer cached? Or should I create a new producer every batch of messages I send? Thanks, Karl
Exception from kafka server 0.8
I got this error while running produce performance test. This is from server: [2013-04-24 15:23:19,082] ERROR Error while fetching metadata for partition [test5,2] (kafka.admin.AdminUtils$) kafka.common.ReplicaNotAvailableException at kafka.admin.AdminUtils$$anonfun$3.apply(AdminUtils.scala:228) at kafka.admin.AdminUtils$$anonfun$3.apply(AdminUtils.scala:201) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206) at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:61) at scala.collection.immutable.List.foreach(List.scala:45) at scala.collection.TraversableLike$class.map(TraversableLike.scala:206) at scala.collection.immutable.List.map(List.scala:45) at kafka.admin.AdminUtils$.kafka$admin$AdminUtils$$fetchTopicMetadataFromZk(AdminUtils.scala:201) at kafka.admin.AdminUtils$$anonfun$fetchTopicMetadataFromZk$1.apply(AdminUtils.scala:194) at kafka.admin.AdminUtils$$anonfun$fetchTopicMetadataFromZk$1.apply(AdminUtils.scala:194) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206) at scala.collection.immutable.Set$Set1.foreach(Set.scala:81) at scala.collection.TraversableLike$class.map(TraversableLike.scala:206) at scala.collection.immutable.Set$Set1.map(Set.scala:68) at kafka.admin.AdminUtils$.fetchTopicMetadataFromZk(AdminUtils.scala:194) at kafka.server.KafkaApis.handleTopicMetadataRequest(KafkaApis.scala:463) at kafka.server.KafkaApis.handle(KafkaApis.scala:67) at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:41) at java.lang.Thread.run(Thread.java:722) This is from producer: [2013-04-24 15:23:28,297] WARN Failed to send producer request with correlation id 5531 to broker 2 with data for partitions [test5,0] (kafka.producer.async.DefaultEventHandler) java.net.SocketTimeoutException at sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:226) at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103) at java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385) at kafka.utils.Utils$.read(Utils.scala:374) at kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54) at kafka.network.Receive$class.readCompletely(Transmission.scala:56) at kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29) at kafka.network.BlockingChannel.receive(BlockingChannel.scala:100) at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:74) at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:71) at kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.a Regards, Libo
Re: Broker List vs Zookeeper
Typically, if you use broker list, you will set up a VIP in a load balancer in front of all brokers. Thanks, Jun On Wed, Apr 24, 2013 at 11:41 AM, Karl Kirch kki...@wdtinc.com wrote: Any advice on using a static broker list vs using zookeeper? I keep having fits with keeping things stable with zookeeper involved (i.e. dropped connections). If I use a static broker list do I still get failover if a broker goes down? (i.e. 1 broker goes down, will my producers still try to send message to it?) Thanks, Karl
Re: Exception from kafka server 0.8
Actually, replication factor 2 means a total of 2 replicas (a leader and a follower). If the leader is down, another replica should automatically take over as the leader. There could be some transient errors in the producer, but they shouldn't last long. Thanks, Jun On Wed, Apr 24, 2013 at 12:46 PM, Chris Curtin curtin.ch...@gmail.comwrote: I'm pretty sure a replication factor of 2 means leader and 2 slaves. Shutting down one means the 2 slaves requirement isn't met. On Wed, Apr 24, 2013 at 3:42 PM, Yu, Libo libo...@citi.com wrote: I figured out the scenario. I have three machines, one server on each of them. I created a topic with three partitions and replication factor 2. After using the Topic for some time, I shut down one server. When producer sent data to the same topic, the error occurred. I still don't know what is the cause of it. Regards, Libo From: Yu, Libo [ICG-IT] Sent: Wednesday, April 24, 2013 3:29 PM To: 'users@kafka.apache.org' Subject: Exception from kafka server 0.8 I got this error while running produce performance test. This is from server: [2013-04-24 15:23:19,082] ERROR Error while fetching metadata for partition [test5,2] (kafka.admin.AdminUtils$) kafka.common.ReplicaNotAvailableException at kafka.admin.AdminUtils$$anonfun$3.apply(AdminUtils.scala:228) at kafka.admin.AdminUtils$$anonfun$3.apply(AdminUtils.scala:201) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206) at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:61) at scala.collection.immutable.List.foreach(List.scala:45) at scala.collection.TraversableLike$class.map(TraversableLike.scala:206) at scala.collection.immutable.List.map(List.scala:45) at kafka.admin.AdminUtils$.kafka$admin$AdminUtils$$fetchTopicMetadataFromZk(AdminUtils.scala:201) at kafka.admin.AdminUtils$$anonfun$fetchTopicMetadataFromZk$1.apply(AdminUtils.scala:194) at kafka.admin.AdminUtils$$anonfun$fetchTopicMetadataFromZk$1.apply(AdminUtils.scala:194) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206) at scala.collection.immutable.Set$Set1.foreach(Set.scala:81) at scala.collection.TraversableLike$class.map(TraversableLike.scala:206) at scala.collection.immutable.Set$Set1.map(Set.scala:68) at kafka.admin.AdminUtils$.fetchTopicMetadataFromZk(AdminUtils.scala:194) at kafka.server.KafkaApis.handleTopicMetadataRequest(KafkaApis.scala:463) at kafka.server.KafkaApis.handle(KafkaApis.scala:67) at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:41) at java.lang.Thread.run(Thread.java:722) This is from producer: [2013-04-24 15:23:28,297] WARN Failed to send producer request with correlation id 5531 to broker 2 with data for partitions [test5,0] (kafka.producer.async.DefaultEventHandler) java.net.SocketTimeoutException at sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:226) at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103) at java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385) at kafka.utils.Utils$.read(Utils.scala:374) at kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54) at kafka.network.Receive$class.readCompletely(Transmission.scala:56) at kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29) at kafka.network.BlockingChannel.receive(BlockingChannel.scala:100) at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:74) at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:71) at kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.a Regards, Libo
Re: acknowledged mode for performance test
request.required.acks=-1 in fact is the strongest durability guarantee on the producer. It means the producer waits for all replicas to write the data before receiving an ack. Thanks, Neha On Wednesday, April 24, 2013, Yu, Libo wrote: Hi, I am running kafka-producer-perf-test.sh for performance test. I notice this line from the log: INFO Property request.required.acks is overridden to -1 So what does -1 mean in this case? Is acknowledgement enabled? In producer.properties, I set request.required.acks to 1 and started the job. I still saw: INFO Property request.required.acks is overridden to -1 Then I I set request.required.acks to 0 and started the job. I got the same log: INFO Property request.required.acks is overridden to -1 Is it possible to turn on acknowledgement for this producer Performance test? BTW: For our application, we cannot afford losing any message and we are only interested in acknowledged mode. Regards, Libo
Re: Occasional batch send errors
It is highly recommended that Kafka and Zookeeper be deployed on different boxes. Also make sure they get dedicated disks, separate from log4j and the OS. Thanks, Neha On Wednesday, April 24, 2013, Karl Kirch wrote: So switched to sync producer to see what would happen. I still get the connection reset by peer error randomly (I say randomly, but seems to be connected to some zookeeper CancelledKeyExceptions), but unfortunately it throws an error on the message after the one that didn't get sent. Is that the way its supposed to work? Karl On Apr 23, 2013, at 7:18 PM, Xavier Stevens xav...@gaikai.comjavascript:; wrote: Usually when these types of errors are because you're not connecting to the proper host:port. Double check your configs, make sure everything is running and listening on the host:port you think they are. Have you tried using the sync producer to work out your bugs? My guess is the sync producer would fail on the first message rather than failing when the batch is submitted. On Tue, Apr 23, 2013 at 4:57 PM, Karl Kirch kki...@wdtinc.com wrote: Hmmm… that didn't seem to help. Anyone else see this sort of errors? Karl On Apr 23, 2013, at 5:58 PM, Karl Kirch kki...@wdtinc.com wrote: I'm going to try bumping up the numRetries key in my producer config. Is this a good option in this case? I am using the zookeeper connect option so I'm aware that I may get stuck retrying to a failed node, but if it's just a temporary network glitch I'll at least get a bit more of a chance to recover. Thanks, Karl On Apr 23, 2013, at 5:35 PM, Karl Kirch kki...@wdtinc.com wrote: I occasionally am getting some batch send errors from the stock async producer. This is on a cluster of 3 kafka (0.7.2) and 3 zookeeper nodes. Is there anyway to check what happens when those batch errors occur? Or bump up the retry count? (looks like it only did a single retry). I need the speed of the async producer, but it needs to be reliable (I see a handful of these a day but in a weather alerting system it only takes missing one let alone 25 or 100/1000). Here's a stack trace of one of the errors that I'm seeing. 22:23:39.405 [ProducerSendThread-1824508747] WARN k.p.a.DefaultEventHandler - Error sending messages, 0 attempts remaining java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcher.writev0(Native Method) ~[na:1.6.0_24] at sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:33) ~[na:1.6.0_24] at sun.nio.ch.IOUtil.write(IOUtil.java:125) ~[na:1.6.0_24] at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:367) ~[na:1.6.0_24] at java.nio.channels.SocketChannel.write(SocketChannel.java:360) ~[na:1.6.0_24] at kafka.network.BoundedByteBufferSend.writeTo(BoundedByteBufferSend.scala:49) ~[apns-consumer-1.0.jar:na] at kafka.network.Send$class.writeCompletely(Transmission.scala:73) ~[apns-consumer-1.0.jar:na] at kafka.network.BoundedByteBufferSend.writeCompletely(BoundedByteBufferSend.scala:25) ~[apns-consumer-1.0.jar:na] at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:95) ~[apns-consumer-1.0.jar:na] at kafka.producer.SyncProducer.send(SyncProducer.scala:94) ~[apns-consumer-1.0.jar:na] at kafka.producer.SyncProducer.multiSend(SyncProducer.scala:135) ~[apns-consumer-1.0.jar:na] at kafka.producer.async.DefaultEventHandler.send(DefaultEventHandler.scala:58) [apns-consumer-1.0.jar:na] at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:44) [apns-consumer-1.0.jar:na] at kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:116) [apns-consumer-1.0.jar:na] at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(Prod
