Hi,
We're testing Kafka and have found that the process dies often. There's little
to no indication of why. We're running the latest code from Git, which we built
using the instructions there. We're also running Zookeeper 3.3.5. Our setup has
three brokers, producers running on the same network and consumers in another
data center, a 30ms Internet ping away.
Does anyone have some intuition about why this is happening?
The only stack trace we're seeing is coming from Zookeeper:
1193285089 [CommitProcessor:2] ERROR org.apache.zookeeper.server.NIOServerCnxn
- Unexpected Exception:
java.nio.channels.CancelledKeyException
at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:55)
at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:59)
at
org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:418)
at
org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1509)
at
org.apache.zookeeper.server.NIOServerCnxn.process(NIOServerCnxn.java:1545)
at
org.apache.zookeeper.server.WatchManager.triggerWatch(WatchManager.java:115)
at
org.apache.zookeeper.server.WatchManager.triggerWatch(WatchManager.java:87)
at org.apache.zookeeper.server.DataTree.deleteNode(DataTree.java:577)
at org.apache.zookeeper.server.DataTree.killSession(DataTree.java:829)
at org.apache.zookeeper.server.DataTree.processTxn(DataTree.java:804)
at
org.apache.zookeeper.server.ZKDatabase.processTxn(ZKDatabase.java:328)
at
org.apache.zookeeper.server.ZooKeeperServer.processTxn(ZooKeeperServer.java:715)
at
org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:107)
at
org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73)
Also, we constantly are seeing these in the logs:
1193365748 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] INFO
org.apache.zookeeper.server.NIOServerCnxn - Closed socket connection for
client /127.0.0.1:53426 (no session established for client)
1193425755 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] INFO
org.apache.zookeeper.server.NIOServerCnxn - Accepted socket connection from
/127.0.0.1:53428
1193425755 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] WARN
org.apache.zookeeper.server.NIOServerCnxn - EndOfStreamException: Unable to
read additional data from client sessionid 0x0, likely client has closed socket
