Kafka log4j appender infinite logging cycle question

2013-08-01 Thread David Black
Hi, I have the following scenario: - using the Kafka log4j appender (in fact its a variant on the OOTB appender) - configuring the appender for the root log4j category - appender is using the async producer I am wondering about what happens when e.g. the producer cannot connect to any broker

Re: reprocessing messages in kafka

2013-08-01 Thread Oleg Ruchovets
I try to resolve such behavior: suppose storm consumes messages from kafka. In case part of it's consumers crashed for any reasons and as a result didn't succeed to process the consumed messages. But if it is impossible after recover to reprocess these messages the system will not be robust

Consumer pauses when running many threads

2013-08-01 Thread Sybrandy, Casey
Hello, We're seeing an issue running 0.7.0 where one or more of our consumers are pausing after about an hour when we have a lot of threads configured. Our setup is as follows: * 4 brokers configured for 32 threads and 32 partitions on each broker. * 2 consumers each

RE: class kafka.common.LeaderNotAvailableException (kafka.producer.BrokerPartitionInfo)

2013-08-01 Thread Nandigam, Sujitha
Sorry..I saw that error after killing brokers only...but before that no error -Original Message- From: Jun Rao [mailto:jun...@gmail.com] Sent: Thursday, August 01, 2013 11:10 AM To: users@kafka.apache.org Subject: Re: class kafka.common.LeaderNotAvailableException

java.net.SocketException: Too many open files

2013-08-01 Thread Nandigam, Sujitha
Hi, In producer I was continuously getting this exception java.net.SocketException: Too many open files even though I added the below line to /etc/security/limits.conf kafka-0.8.0-beta1-src-nofile983040 ERROR Producer connection to localhost:9093 unsuccessful

Re: java.net.SocketException: Too many open files

2013-08-01 Thread Jun Rao
If you do netstat, what hosts are those connections for and what state are those connections in? Thanks, Jun On Thu, Aug 1, 2013 at 9:04 AM, Nandigam, Sujitha snandi...@verisign.comwrote: Hi, In producer I was continuously getting this exception java.net.SocketException: Too many open

Re: Topic stuck with leader: none

2013-08-01 Thread Benjamin Black
If you shutdown all brokers, then bring them up one at a time does it change? Might have to do it several times. On Aug 1, 2013 8:28 PM, Vinicius Carvalho viniciusccarva...@gmail.com wrote: Hi there! So, I setup a demo and forgot to create the topic. But since my producer is set to create