Thanks for the response; we found the issue. We run in AWS, and inexplicably, 
the new instance we launched to replace the dead one had exceedingly low 
network bandwidth to exactly one of the remaining brokers, resulting in 
timeouts. After rolling the dice again things are replicating normally.

On 09/07/2016 03:56 PM, Ryan Pridgeon wrote:

One possibility is that broker 0 has exhausted it's available file
descriptors. If this is the case it will be able to maintain it's existing
connections, giving off the appearance that it is operating normally while
refusing new ones.

I don't recall the exact exception message but something along the lines of
'too many files open' may be present within the logs.


On Tue, Sep 6, 2016 at 2:36 PM, Tommy Becker 
<[email protected]><mailto:[email protected]> wrote:



We had a hardware failure on broker 1 of a 3 broker cluster over the
weekend. The broker was replaced, and when the replacement broker came up
it started to replicate partitions from the other 2 brokers as you'd
expect. But while broker 1 (the replacement) was able to fetch properly
from broker 2, it continually disconnects trying to do so from broker 0,
resulting in the following error message every 30s:

[2016-09-06 14:45:50,103] WARN [ReplicaFetcherThread-0-0], Error in fetch
kafka.server.ReplicaFetcherThread$FetchRequest@73e7afd8
(kafka.server.ReplicaFetcherThread)
java.io.IOException: Connection to 0 was disconnected before the response
was read
      at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAn
dReceive$extension$1$$anonfun$apply$1.apply(NetworkClientBlo
ckingOps.scala:87)
      at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAn
dReceive$extension$1$$anonfun$apply$1.apply(NetworkClientBlo
ckingOps.scala:84)
      at scala.Option.foreach(Option.scala:257)
      at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAn
dReceive$extension$1.apply(NetworkClientBlockingOps.scala:84)
      at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAn
dReceive$extension$1.apply(NetworkClientBlockingOps.scala:80)
      at kafka.utils.NetworkClientBlockingOps$.recursivePoll$2(Networ
kClientBlockingOps.scala:137)
      at kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkCli
entBlockingOps$$pollContinuously$extension(NetworkClientBloc
kingOps.scala:143)
      at kafka.utils.NetworkClientBlockingOps$.blockingSendAndReceive
$extension(NetworkClientBlockingOps.scala:80)
      at kafka.server.ReplicaFetcherThread.sendRequest(ReplicaFetcher
Thread.scala:244)
      at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread
.scala:229)
      at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread
.scala:42)
      at kafka.server.AbstractFetcherThread.processFetchRequest(Abstr
actFetcherThread.scala:107)
      at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThr
ead.scala:98)
      at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)


We are running Kafka version 0.10.0.0. Any ideas of what to check for?
Network connectivity between all brokers is fine.

--
Tommy Becker
Senior Software Engineer

Digitalsmiths
A TiVo Company

www.digitalsmiths.com<http://www.digitalsmiths.com><http://www.digitalsmiths.com><http://www.digitalsmiths.com>
[email protected]<mailto:[email protected]><mailto:[email protected]><mailto:[email protected]>

________________________________

This email and any attachments may contain confidential and privileged
material for the sole use of the intended recipient. Any review, copying,
or distribution of this email (or any attachments) by others is prohibited.
If you are not the intended recipient, please contact the sender
immediately and permanently delete this email and any attachments. No
employee or agent of TiVo Inc. is authorized to conclude any binding
agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo
Inc. may only be made by a signed written agreement.






--
Tommy Becker
Senior Software Engineer

Digitalsmiths
A TiVo Company

www.digitalsmiths.com<http://www.digitalsmiths.com>
[email protected]<mailto:[email protected]>

________________________________

This email and any attachments may contain confidential and privileged material 
for the sole use of the intended recipient. Any review, copying, or 
distribution of this email (or any attachments) by others is prohibited. If you 
are not the intended recipient, please contact the sender immediately and 
permanently delete this email and any attachments. No employee or agent of TiVo 
Inc. is authorized to conclude any binding agreement on behalf of TiVo Inc. by 
email. Binding agreements with TiVo Inc. may only be made by a signed written 
agreement.

Reply via email to