ZooKeeper client is getting "java.io.IOException: Nothing in the queue, but got "

[Rakesh R]

Hi,

Following exception is coming always for the client session 0xf49d7f31e4c0000 
in zookeeper client logs.
Client & server is using kerberoes based sasl authentication mechanism.

I'm using 3.4.6 release.

2014-11-25 22:06:19,114 INFO  [Thread-1-SendThread(vm-216:24002)] Client will 
use GSSAPI as SASL mechanism. 
org.apache.zookeeper.client.ZooKeeperSaslClient$1.run(ZooKeeperSaslClient.java:285)
2014-11-25 22:06:19,114 INFO  [Thread-1-SendThread(vm-216:24002)] Opening 
socket connection to server vm-01/**.**.**.**:24002. Will attempt to 
SASL-authenticate using Login Context section 'Client' 
org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.java:1000)
2014-11-25 22:06:19,115 INFO  [Thread-1-SendThread(vm-216:24002)] Socket 
connection established to vm-216/**.**.**.**:24002, initiating session 
org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:855)
2014-11-25 22:06:19,116 INFO  [Thread-1-SendThread(vm-216:24002)] Session 
establishment complete on server vm-216/**.**.**.**:24002, sessionid = 
0xf49d7f31e4c0000, negotiated timeout = 90000 
org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:1260)
2014-11-25 22:06:19,118 WARN  [Thread-1-SendThread(vm-216:24002)] Session 
0xf49d7f31e4c0000 for server vm-216/**.**.**.**:24002, unexpected error, 
closing socket connection and attempting reconnect 
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1127)
java.io.IOException: Nothing in the queue, but got 425467
                at 
org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:788)
                at 
org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:94)
                at 
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366)
                at 
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1106)


This exception has occured in the following logic.

CilentCnxn#readResponse()

            synchronized (pendingQueue) {
                if (pendingQueue.size() == 0) {
                    throw new IOException("Nothing in the queue, but got "
                            + replyHdr.getXid());
                }
                packet = pendingQueue.remove();
            }


I failed to find the reason why 'pendingQueue' becomes empty. Only possibility 
I could see is the following condition fails and missed to add it to the 
"pendingQueue"

ClientCnxnSocketNIO.java

                    sock.write(p.bb);
                    if (!p.bb.hasRemaining()) {
                        .....
                        .....
                        .....
                        pendingQueue.add(p);
                    }

Does anyone phase similar issue and would appreciate any help. Thanks!

Thanks & Regards,
Rakesh