One possibility is that broker 0 has exhausted it's available file descriptors. If this is the case it will be able to maintain it's existing connections, giving off the appearance that it is operating normally while refusing new ones.
I don't recall the exact exception message but something along the lines of 'too many files open' may be present within the logs. On Tue, Sep 6, 2016 at 2:36 PM, Tommy Becker <tobec...@tivo.com> wrote: > We had a hardware failure on broker 1 of a 3 broker cluster over the > weekend. The broker was replaced, and when the replacement broker came up > it started to replicate partitions from the other 2 brokers as you'd > expect. But while broker 1 (the replacement) was able to fetch properly > from broker 2, it continually disconnects trying to do so from broker 0, > resulting in the following error message every 30s: > > [2016-09-06 14:45:50,103] WARN [ReplicaFetcherThread-0-0], Error in fetch > kafka.server.ReplicaFetcherThread$FetchRequest@73e7afd8 > (kafka.server.ReplicaFetcherThread) > java.io.IOException: Connection to 0 was disconnected before the response > was read > at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAn > dReceive$extension$1$$anonfun$apply$1.apply(NetworkClientBlo > ckingOps.scala:87) > at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAn > dReceive$extension$1$$anonfun$apply$1.apply(NetworkClientBlo > ckingOps.scala:84) > at scala.Option.foreach(Option.scala:257) > at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAn > dReceive$extension$1.apply(NetworkClientBlockingOps.scala:84) > at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAn > dReceive$extension$1.apply(NetworkClientBlockingOps.scala:80) > at kafka.utils.NetworkClientBlockingOps$.recursivePoll$2(Networ > kClientBlockingOps.scala:137) > at kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkCli > entBlockingOps$$pollContinuously$extension(NetworkClientBloc > kingOps.scala:143) > at kafka.utils.NetworkClientBlockingOps$.blockingSendAndReceive > $extension(NetworkClientBlockingOps.scala:80) > at kafka.server.ReplicaFetcherThread.sendRequest(ReplicaFetcher > Thread.scala:244) > at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread > .scala:229) > at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread > .scala:42) > at kafka.server.AbstractFetcherThread.processFetchRequest(Abstr > actFetcherThread.scala:107) > at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThr > ead.scala:98) > at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63) > > > We are running Kafka version 0.10.0.0. Any ideas of what to check for? > Network connectivity between all brokers is fine. > > -- > Tommy Becker > Senior Software Engineer > > Digitalsmiths > A TiVo Company > > www.digitalsmiths.com<http://www.digitalsmiths.com> > tobec...@tivo.com<mailto:tobec...@tivo.com> > > ________________________________ > > This email and any attachments may contain confidential and privileged > material for the sole use of the intended recipient. Any review, copying, > or distribution of this email (or any attachments) by others is prohibited. > If you are not the intended recipient, please contact the sender > immediately and permanently delete this email and any attachments. No > employee or agent of TiVo Inc. is authorized to conclude any binding > agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo > Inc. may only be made by a signed written agreement. >
