Thrift is not happy on some replication ITs I've run lately. I had one test timeout after 40 minutes... and it never finished. The symptom is lots of client side messages about failure to open transport, and the server side messages were (and both were occurring a *lot*, indicating indefinite retries):
2016-08-30 19:48:13,476 [rpc.CustomNonBlockingServer$CustomFrameBuffer] WARN : Got an IOException in internalRead! java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcherImpl.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) at sun.nio.ch.IOUtil.read(IOUtil.java:197) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:384) at org.apache.thrift.transport.TNonblockingSocket.read(TNonblockingSocket.java:142) at org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.internalRead(AbstractNonblockingServer.java:539) at org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.read(AbstractNonblockingServer.java:338) at org.apache.thrift.server.AbstractNonblockingServer$AbstractSelectThread.handleRead(AbstractNonblockingServer.java:203) at org.apache.thrift.server.TNonblockingServer$SelectAcceptThread.select(TNonblockingServer.java:203) at org.apache.thrift.server.TNonblockingServer$SelectAcceptThread.run(TNonblockingServer.java:154) I saw one comment on a mailing list somewhere that indicated this might be caused by a client side handling of a custom Thrift Exception, not properly closing the connection. It's possible we're doing something badly before we retry. I think more investigation is needed before I file a JIRA (not even sure what to file it against, right now... because I'm not sure what component is even at fault). In the meantime, has anybody seen this? Does anybody have any insight into this? This is all on a single node, running ITs. There really shouldn't be any "network" problems which would cause a TCP reset from external to the test and Accumulo itself, since it's all localhost.