Re: Kafka connection loss with high volume of messages

2014-06-26 Thread Ahmed H.
, Jun 25, 2014 at 6:02 PM, Neha Narkhede neha.narkh...@gmail.com wrote: If rebalance succeeded, then those error messages are harmless. Though I agree we shouldn't log those in the first place. On Wed, Jun 25, 2014 at 2:12 PM, Ahmed H. ahmed.ham...@gmail.com wrote: Unfortunately I do

Kafka connection loss with high volume of messages

2014-06-25 Thread Ahmed H.
Hello All, I am seeing this issue very frequently when running a high volume of messages through Kafka. It starts off well, and it can go on for minutes that way, but eventually it reaches a point where the connection to Kafka dies, then it reconnects and carries on. This repeats more frequently

Re: Kafka connection loss with high volume of messages

2014-06-25 Thread Ahmed H.
exceptions on the broker logs? Guozhang On Wed, Jun 25, 2014 at 7:47 AM, Ahmed H. ahmed.ham...@gmail.com wrote: Hello All, I am seeing this issue very frequently when running a high volume of messages through Kafka. It starts off well, and it can go on for minutes that way

Re: Kafka connection loss with high volume of messages

2014-06-25 Thread Ahmed H.
at 8:19 AM, Ahmed H. ahmed.ham...@gmail.com wrote: Are you referring to the zookeeper logs? If so, I am seeing a lot of those: 2014-06-25 11:15:02 NIOServerCnxn [WARN] caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid

Re: Kafka rebalancing causes Zookeeper to fail

2014-01-23 Thread Ahmed H.
expiration, which were fixed in 3.3.4. Not sure if 3.4.5 exposes any new issues. The easiest thing is probably to avoid GC-induced ZK session timeout in the first place or use a larger session timeout. Thanks, Jun On Wed, Jan 22, 2014 at 8:29 AM, Ahmed H. ahmed.ham...@gmail.com wrote: Hello

Kafka rebalancing causes Zookeeper to fail

2014-01-22 Thread Ahmed H.
I have a basic Zookeeper/Kafka setup. I am still on Kafka 0.8 beta 1, and Zookeeper 3.4.5. The activity on this machine isn't massive...I would say the Kafka queues get a consistent 1 message every 2-3 seconds, as well as occasional spikes, but still nothing large enough to push the limits. Both

Re: Kafka rebalancing causes Zookeeper to fail

2014-01-22 Thread Ahmed H.
, 2014 at 6:24 AM, Ahmed H. ahmed.ham...@gmail.com wrote: I have a basic Zookeeper/Kafka setup. I am still on Kafka 0.8 beta 1, and Zookeeper 3.4.5. The activity on this machine isn't massive...I would say the Kafka queues get a consistent 1 message every 2-3 seconds, as well as occasional

Re: Kafka client dies after rebalancing attempt

2013-11-08 Thread Ahmed H.
On Thu, Nov 07, 2013 at 02:51:41PM -0500, Ahmed H. wrote: Hello all, I am not sure if this is a Kafka issue, or an issue with the client that I am using. We have a fairly small setup, where everything sits on one server (Kafka 0.8, and Zookeeper). The message frequency is not too high (1

Re: Kafka client dies after rebalancing attempt

2013-11-08 Thread Ahmed H.
, you may want to try with the latest HEAD as opposed to the beta. Thanks, Joel On Fri, Nov 08, 2013 at 01:18:07PM -0500, Ahmed H. wrote: Hello, I am using the beta right now. I'm not sure if it's GC or something else at this point. To be honest I've never really fiddled with any GC

Kafka client dies after rebalancing attempt

2013-11-07 Thread Ahmed H.
Hello all, I am not sure if this is a Kafka issue, or an issue with the client that I am using. We have a fairly small setup, where everything sits on one server (Kafka 0.8, and Zookeeper). The message frequency is not too high (1-2 per second). The setup works fine for a certain period of time

Wrong request type 768

2013-11-04 Thread Ahmed H.
Hello All, I am getting this error and I am not sure what that means. Specifically the Wrong request type 768 [2013-11-04 16:56:47,540] ERROR Closing socket for /0:0:0:0:0:0:0:1 because of error (kafka.network.Processor) kafka.common.KafkaException: Wrong request type 768 at