I had this error before and corrected by increasing nofile limit
add to file an entry for the user running the broker.
/etc/security/limits.conf
kafka - nofile 98304
On Thu, Apr 24, 2014 at 1:46 PM, Yashika Gupta
<yashika.gu...@impetus.co.in>wrote:
> Jun,
>
> The detailed logs are as follows:
>
> 24.04.2014 13:37:31812 INFO main kafka.producer.SyncProducer -
> Disconnecting from localhost:9092
> 24.04.2014 13:37:38612 WARN main kafka.producer.BrokerPartitionInfo -
> Error while fetching metadata [{TopicMetadata for topic LOGFILE04 ->
> No partition metadata for topic LOGFILE04 due to
> kafka.common.LeaderNotAvailableException}] for topic [LOGFILE04]: class
> kafka.common.LeaderNotAvailableException
> 24.04.2014 13:37:40712 INFO main kafka.client.ClientUtils$ - Fetching
> metadata from broker id:0,host:localhost,port:9092 with correlation id 1
> for 1 topic(s) Set(LOGFILE04)
> 24.04.2014 13:37:41212 INFO main kafka.producer.SyncProducer - Connected
> to localhost:9092 for producing
> 24.04.2014 13:37:48812 INFO main kafka.producer.SyncProducer -
> Disconnecting from localhost:9092
> 24.04.2014 13:37:48912 WARN main kafka.producer.BrokerPartitionInfo -
> Error while fetching metadata [{TopicMetadata for topic LOGFILE04 ->
> No partition metadata for topic LOGFILE04 due to
> kafka.common.LeaderNotAvailableException}] for topic [LOGFILE04]: class
> kafka.common.LeaderNotAvailableException
> 24.04.2014 13:37:49012 ERROR main kafka.producer.async.DefaultEventHandler
> - Failed to collate messages by topic, partition due to: Failed to fetch
> topic metadata for topic: LOGFILE04
>
>
> 24.04.2014 13:39:96513 WARN
> ConsumerFetcherThread-produceLogLine2_vcmd-devanshu-1398361030812-8a0c706e-0-0
> kafka.consumer.ConsumerFetcherThread -
> [ConsumerFetcherThread-produceLogLine2_vcmd-devanshu-1398361030812-8a0c706e-0-0],
> Error in fetch Name: FetchRequest; Version: 0; CorrelationId: 4; ClientId:
> produceLogLine2-ConsumerFetcherThread-produceLogLine2_vcmd-devanshu-1398361030812-8a0c706e-0-0;
> ReplicaId: -1; MaxWait: 60000 ms; MinBytes: 1 bytes; RequestInfo:
> [LOGFILE04,0] -> PartitionFetchInfo(2,1048576)
> java.net.SocketTimeoutException
> at
> sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:229)
> at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
> at
> java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385)
> at kafka.utils.Utils$.read(Unknown Source)
> at kafka.network.BoundedByteBufferReceive.readFrom(Unknown Source)
> at kafka.network.Receive$class.readCompletely(Unknown Source)
> at kafka.network.BoundedByteBufferReceive.readCompletely(Unknown
> Source)
> at kafka.network.BlockingChannel.receive(Unknown Source)
> at kafka.consumer.SimpleConsumer.liftedTree1$1(Unknown Source)
> at
> kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(Unknown
> Source)
> at
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(Unknown
> Source)
> at
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(Unknown
> Source)
> at
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(Unknown
> Source)
> at kafka.metrics.KafkaTimer.time(Unknown Source)
> at
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply$mcV$sp(Unknown Source)
> at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(Unknown
> Source)
> at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(Unknown
> Source)
> at kafka.metrics.KafkaTimer.time(Unknown Source)
> at kafka.consumer.SimpleConsumer.fetch(Unknown Source)
> at kafka.server.AbstractFetcherThread.processFetchRequest(Unknown
> Source)
> at kafka.server.AbstractFetcherThread.doWork(Unknown Source)
> at kafka.utils.ShutdownableThread.run(Unknown Source)
>
>
> Regards,
> Yashika
> ________________________________________
> From: Jun Rao <jun...@gmail.com>
> Sent: Thursday, April 24, 2014 10:49 PM
> To: users@kafka.apache.org
> Subject: Re: Kafka Performance Tuning
>
> Before that error messge, the log should tell you the cause of the error.
> Could you dig that out?
>
> Thanks,
>
> Jun
>
>
> On Thu, Apr 24, 2014 at 10:12 AM, Yashika Gupta <
> yashika.gu...@impetus.co.in
> > wrote:
>
> > Hi,
> >
> > I am working on a POC where I have 1 Zookeeper and 2 Kafka Brokers on my
> > local machine. I am running 8 sets of Kafka consumers and producers
> running
> > in parallel.
> >
> > Below are my configurations:
> > Consumer Configs:
> > zookeeper.session.timeout.ms=120000
> > zookeeper.sync.time.ms=2000
> > zookeeper.connection.timeout.ms=120000
> > auto.commit.interval.ms=60000
> > rebalance.backoff.ms=2000
> > fetch.wait.max.ms=60000
> > auto.offset.reset=smallest
> >
> > Producer configs:
> > key.serializer.class=kafka.serializer.StringEncoder
> > request.required.acks=-1
> > message.send.max.retries=3
> > request.timeout.ms=60000
> >
> > I have tried various more configuration changes but I am running into the
> > same exception again and again.
> >
> > 17.04.2014 06:31:83216 ERROR pool-5-thread-1
> > kafka.producer.async.DefaultEventHandler - Failed to collate messages by
> > topic, partition due to: Failed to fetch topic metadata for topic:
> TOPIC99
> > 17.04.2014 06:31:85616 ERROR pool-4-thread-1
> > kafka.producer.async.DefaultEventHandler - Failed to collate messages by
> > topic, partition due to: Failed to fetch topic metadata for topic:
> TOPIC99
> >
> > I am not able to get to the root cause of the issue.
> > Appreciate your help.
> >
> > Regards,
> > Yashika
> >
> > ________________________________
> >
> >
> >
> >
> >
> >
> > NOTE: This message may contain information that is confidential,
> > proprietary, privileged or otherwise protected by law. The message is
> > intended solely for the named addressee. If received in error, please
> > destroy and notify the sender. Any use of this email is prohibited when
> > received in error. Impetus does not represent, warrant and/or guarantee,
> > that the integrity of this communication has been maintained nor that the
> > communication is free of errors, virus, interception or interference.
> >
>
> ________________________________
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>
