ERROR Processor got uncaught exception. (kafka.network.Processor)

[Ramanan, Buvana (Nokia - US)]

Hello,

We are trying to understand the following error in one of the brokers in our 
cluster. It's a 3 broker cluster (all 3 servers are identical -  Intel Xeon 
E5-2670 @2.6GHz, 8cores, 16 threads 64 GB RAM & 1 TB Disk),  serving about 8000 
topics, single partitioned & replication factor = 3. Each topic gets data at a 
low rate  - 200 bytes per sec.  Cluster is managed by 3 server ZK cluster 
(these servers are different from Kafka broker servers). All 6 servers are 
connected via 10G switch. Producers run from external servers.

Leaders are balanced across the topics.

This error message occurs repeatedly in one of the servers. Between the hours 
of 10:30am and 1:30pm today, there were about 10 Million such occurrences.

We tuned Linux network params (somaxconn, rmem_max, wmem_max, etc). I can share 
the current settings if needed.

Please let me know what is causing this error and a way to overcome it.

Regards,
Buvana

[2016-05-09 10:38:43,932] ERROR Processor got uncaught exception. 
(kafka.network.Processor)
java.lang.IllegalArgumentException: Attempted to decrease connection count for 
address with no connections, address: /X.Y.Z.144 (actual network address masked)
        at 
kafka.network.ConnectionQuotas$$anonfun$9.apply(SocketServer.scala:565)
        at 
kafka.network.ConnectionQuotas$$anonfun$9.apply(SocketServer.scala:565)
        at scala.collection.MapLike$class.getOrElse(MapLike.scala:128)
        at scala.collection.AbstractMap.getOrElse(Map.scala:59)
        at kafka.network.ConnectionQuotas.dec(SocketServer.scala:564)
        at kafka.network.Processor$$anonfun$run$13.apply(SocketServer.scala:450)
        at kafka.network.Processor$$anonfun$run$13.apply(SocketServer.scala:445)
        at scala.collection.Iterator$class.foreach(Iterator.scala:742)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1194)
        at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
        at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
        at kafka.network.Processor.run(SocketServer.scala:445)
        at java.lang.Thread.run(Thread.java:745)
[2016-05-09 10:38:43,932] ERROR Processor got uncaught exception. 
(kafka.network.Processor)
java.lang.IllegalArgumentException: Attempted to decrease connection count for 
address with no connections, address: /X.Y.Z.144
        at 
kafka.network.ConnectionQuotas$$anonfun$9.apply(SocketServer.scala:565)
        at 
kafka.network.ConnectionQuotas$$anonfun$9.apply(SocketServer.scala:565)
        at scala.collection.MapLike$class.getOrElse(MapLike.scala:128)
        at scala.collection.AbstractMap.getOrElse(Map.scala:59)
        at kafka.network.ConnectionQuotas.dec(SocketServer.scala:564)
        at kafka.network.Processor$$anonfun$run$13.apply(SocketServer.scala:450)
        at kafka.network.Processor$$anonfun$run$13.apply(SocketServer.scala:445)
        at scala.collection.Iterator$class.foreach(Iterator.scala:742)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1194)
        at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
        at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
        at kafka.network.Processor.run(SocketServer.scala:445)
        at java.lang.Thread.run(Thread.java:745)