Hi,I am testing large Ignite Cache of 900GB, on 4 node VM(96GB RAM, 8CPU and 500GB SAN Storage) Spark Ignite Cluster .It happened tow times after reaching 350GB plus one or two nodes not processing data load and the data load is stopped. Please advise, the CLuster , Server and Client Logs below.
<http://apache-ignite-users.70518.x6.nabble.com/file/t1842/IgniteClusterSnapshot.png> Server Logs: [11:59:34] Topology snapshot [ver=121, servers=4, clients=9, CPUs=32, offheap=1000.0GB, heap=78.0GB] [11:59:34] ^-- Node [id=F6605E96-47C9-479B-A840-03316500C9A3, clusterState=ACTIVE] [11:59:34] ^-- Baseline [id=0, size=4, online=4, offline=0] [11:59:34] Data Regions Configured: [11:59:34] ^-- default_mem_region [initSize=256.0 MiB, maxSize=20.0 GiB, persistenceEnabled=true] [11:59:34] ^-- q_major [initSize=10.0 GiB, maxSize=30.0 GiB, persistenceEnabled=true] [11:59:34] ^-- q_minor [initSize=10.0 GiB, maxSize=30.0 GiB, persistenceEnabled=true] [14:33:15,872][SEVERE][grid-nio-worker-client-listener-3-#33][ClientListenerProcessor] Failed to process selector key [ses=GridSelectorNioSessionImpl [worker=ByteBufferNioClientWorker [readBuf=java.nio.HeapByteBuffer[pos=0 lim=8192 cap=8192], super=AbstractNioClientWorker [idx=3, bytesRcvd=0, bytesSent=0, bytesRcvd0=0, bytesSent0=0, select=true, super=GridWorker [name=grid-nio-worker-client-listener-3, igniteInstanceName=null, finished=false, hashCode=254322881, interrupted=false, runner=grid-nio-worker-client-listener-3-#33]]], writeBuf=null, readBuf=null, inRecovery=null, outRecovery=null, super=GridNioSessionImpl [locAddr=/64.102.213.190:10800, rmtAddr=/10.82.249.225:51449, createTime=1538740798912, closeTime=0, bytesSent=397, bytesRcvd=302, bytesSent0=0, bytesRcvd0=0, sndSchedTime=1538742789216, lastSndTime=1538742789216, lastRcvTime=1538742789216, readsPaused=false, filterChain=FilterChain[filters=[GridNioAsyncNotifyFilter, GridNioCodecFilter [parser=ClientListenerBufferedParser, directMode=false]], accepted=true]]] java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcherImpl.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) at sun.nio.ch.IOUtil.read(IOUtil.java:197) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) at org.apache.ignite.internal.util.nio.GridNioServer$ByteBufferNioClientWorker.processRead(GridNioServer.java:1085) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2339) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2110) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.lang.Thread.run(Thread.java:748) [21:43:26,312][SEVERE][grid-nio-worker-client-listener-0-#30][ClientListenerProcessor] Failed to process selector key [ses=GridSelectorNioSessionImpl [worker=ByteBufferNioClientWorker [readBuf=java.nio.HeapByteBuffer[pos=0 lim=8192 cap=8192], super=AbstractNioClientWorker [idx=0, bytesRcvd=0, bytesSent=0, bytesRcvd0=0, bytesSent0=0, select=true, super=GridWorker [name=grid-nio-worker-client-listener-0, igniteInstanceName=null, finished=false, hashCode=2211598, interrupted=false, runner=grid-nio-worker-client-listener-0-#30]]], writeBuf=null, readBuf=null, inRecovery=null, outRecovery=null, super=GridNioSessionImpl [locAddr=/64.102.213.190:10800, rmtAddr=/10.82.32.114:59525, createTime=1538746249024, closeTime=0, bytesSent=2035, bytesRcvd=1532, bytesSent0=0, bytesRcvd0=0, sndSchedTime=1538767916701, lastSndTime=1538767916701, lastRcvTime=1538767916701, readsPaused=false, filterChain=FilterChain[filters=[GridNioAsyncNotifyFilter, GridNioCodecFilter [parser=ClientListenerBufferedParser, directMode=false]], accepted=true]]] java.io.IOException: Connection timed out at sun.nio.ch.FileDispatcherImpl.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) at sun.nio.ch.IOUtil.read(IOUtil.java:197) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) at org.apache.ignite.internal.util.nio.GridNioServer$ByteBufferNioClientWorker.processRead(GridNioServer.java:1085) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2339) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2110) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.lang.Thread.run(Thread.java:748) [23:02:32,031][SEVERE][grid-nio-worker-client-listener-1-#31][ClientListenerProcessor] Failed to process selector key [ses=GridSelectorNioSessionImpl [worker=ByteBufferNioClientWorker [readBuf=java.nio.HeapByteBuffer[pos=0 lim=8192 cap=8192], super=AbstractNioClientWorker [idx=1, bytesRcvd=0, bytesSent=0, bytesRcvd0=0, bytesSent0=0, select=true, super=GridWorker [name=grid-nio-worker-client-listener-1, igniteInstanceName=null, finished=false, hashCode=1626735999, interrupted=false, runner=grid-nio-worker-client-listener-1-#31]]], writeBuf=null, readBuf=null, inRecovery=null, outRecovery=null, super=GridNioSessionImpl [locAddr=/64.102.213.190:10800, rmtAddr=/10.82.249.225:51882, createTime=1538769618223, closeTime=0, bytesSent=397, bytesRcvd=302, bytesSent0=0, bytesRcvd0=0, sndSchedTime=1538773344029, lastSndTime=1538773344029, lastRcvTime=1538773344029, readsPaused=false, filterChain=FilterChain[filters=[GridNioAsyncNotifyFilter, GridNioCodecFilter [parser=ClientListenerBufferedParser, directMode=false]], accepted=true]]] java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcherImpl.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) at sun.nio.ch.IOUtil.read(IOUtil.java:197) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) at org.apache.ignite.internal.util.nio.GridNioServer$ByteBufferNioClientWorker.processRead(GridNioServer.java:1085) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2339) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2110) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.lang.Thread.run(Thread.java:748) [05:52:08,034][SEVERE][grid-nio-worker-client-listener-2-#32][ClientListenerProcessor] Failed to process selector key [ses=GridSelectorNioSessionImpl [worker=ByteBufferNioClientWorker [readBuf=java.nio.HeapByteBuffer[pos=0 lim=8192 cap=8192], super=AbstractNioClientWorker [idx=2, bytesRcvd=0, bytesSent=0, bytesRcvd0=0, bytesSent0=0, select=true, super=GridWorker [name=grid-nio-worker-client-listener-2, igniteInstanceName=null, finished=false, hashCode=1810870884, interrupted=false, runner=grid-nio-worker-client-listener-2-#32]]], writeBuf=null, readBuf=null, inRecovery=null, outRecovery=null, super=GridNioSessionImpl [locAddr=/64.102.213.190:10800, rmtAddr=/10.82.177.186:54754, createTime=1538797913271, closeTime=0, bytesSent=163, bytesRcvd=128, bytesSent0=0, bytesRcvd0=0, sndSchedTime=1538797924460, lastSndTime=1538797924460, lastRcvTime=1538797924460, readsPaused=false, filterChain=FilterChain[filters=[GridNioAsyncNotifyFilter, GridNioCodecFilter [parser=ClientListenerBufferedParser, directMode=false]], accepted=true]]] java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcherImpl.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) at sun.nio.ch.IOUtil.read(IOUtil.java:197) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) at org.apache.ignite.internal.util.nio.GridNioServer$ByteBufferNioClientWorker.processRead(GridNioServer.java:1085) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2339) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2110) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.lang.Thread.run(Thread.java:748) [13:16:10,473][SEVERE][grid-nio-worker-client-listener-3-#33][ClientListenerProcessor] Failed to process selector key [ses=GridSelectorNioSessionImpl [worker=ByteBufferNioClientWorker [readBuf=java.nio.HeapByteBuffer[pos=0 lim=8192 cap=8192], super=AbstractNioClientWorker [idx=3, bytesRcvd=0, bytesSent=0, bytesRcvd0=0, bytesSent0=0, select=true, super=GridWorker [name=grid-nio-worker-client-listener-3, igniteInstanceName=null, finished=false, hashCode=254322881, interrupted=false, runner=grid-nio-worker-client-listener-3-#33]]], writeBuf=null, readBuf=null, inRecovery=null, outRecovery=null, super=GridNioSessionImpl [locAddr=/64.102.213.190:10800, rmtAddr=/10.82.224.11:60529, createTime=1538824143152, closeTime=0, bytesSent=280, bytesRcvd=215, bytesSent0=0, bytesRcvd0=0, sndSchedTime=1538824568991, lastSndTime=1538824568991, lastRcvTime=1538824568991, readsPaused=false, filterChain=FilterChain[filters=[GridNioAsyncNotifyFilter, GridNioCodecFilter [parser=ClientListenerBufferedParser, directMode=false]], accepted=true]]] java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcherImpl.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) at sun.nio.ch.IOUtil.read(IOUtil.java:197) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) at org.apache.ignite.internal.util.nio.GridNioServer$ByteBufferNioClientWorker.processRead(GridNioServer.java:1085) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2339) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2110) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.lang.Thread.run(Thread.java:748) [16:43:22,848][SEVERE][grid-nio-worker-client-listener-0-#30][ClientListenerProcessor] Failed to process selector key [ses=GridSelectorNioSessionImpl [worker=ByteBufferNioClientWorker [readBuf=java.nio.HeapByteBuffer[pos=0 lim=8192 cap=8192], super=AbstractNioClientWorker [idx=0, bytesRcvd=0, bytesSent=0, bytesRcvd0=0, bytesSent0=0, select=true, super=GridWorker [name=grid-nio-worker-client-listener-0, igniteInstanceName=null, finished=false, hashCode=2211598, interrupted=false, runner=grid-nio-worker-client-listener-0-#30]]], writeBuf=null, readBuf=null, inRecovery=null, outRecovery=null, super=GridNioSessionImpl [locAddr=/64.102.213.190:10800, rmtAddr=/10.82.224.11:57693, createTime=1538836966711, closeTime=0, bytesSent=163, bytesRcvd=128, bytesSent0=0, bytesRcvd0=0, sndSchedTime=1538837001780, lastSndTime=1538837001780, lastRcvTime=1538837001780, readsPaused=false, filterChain=FilterChain[filters=[GridNioAsyncNotifyFilter, GridNioCodecFilter [parser=ClientListenerBufferedParser, directMode=false]], accepted=true]]] java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcherImpl.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) at sun.nio.ch.IOUtil.read(IOUtil.java:197) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) at org.apache.ignite.internal.util.nio.GridNioServer$ByteBufferNioClientWorker.processRead(GridNioServer.java:1085) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2339) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2110) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.lang.Thread.run(Thread.java:748) [19:11:56,770][SEVERE][grid-nio-worker-client-listener-1-#31][ClientListenerProcessor] Failed to process selector key [ses=GridSelectorNioSessionImpl [worker=ByteBufferNioClientWorker [readBuf=java.nio.HeapByteBuffer[pos=0 lim=8192 cap=8192], super=AbstractNioClientWorker [idx=1, bytesRcvd=0, bytesSent=0, bytesRcvd0=0, bytesSent0=0, select=true, super=GridWorker [name=grid-nio-worker-client-listener-1, igniteInstanceName=null, finished=false, hashCode=1626735999, interrupted=false, runner=grid-nio-worker-client-listener-1-#31]]], writeBuf=null, readBuf=null, inRecovery=null, outRecovery=null, super=GridNioSessionImpl [locAddr=/64.102.213.190:10800, rmtAddr=/10.82.224.11:49209, createTime=1538845894215, closeTime=0, bytesSent=163, bytesRcvd=128, bytesSent0=0, bytesRcvd0=0, sndSchedTime=1538845911149, lastSndTime=1538845911149, lastRcvTime=1538845911149, readsPaused=false, filterChain=FilterChain[filters=[GridNioAsyncNotifyFilter, GridNioCodecFilter [parser=ClientListenerBufferedParser, directMode=false]], accepted=true]]] java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcherImpl.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) at sun.nio.ch.IOUtil.read(IOUtil.java:197) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) at org.apache.ignite.internal.util.nio.GridNioServer$ByteBufferNioClientWorker.processRead(GridNioServer.java:1085) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2339) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2110) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.lang.Thread.run(Thread.java:748) [21:32:26,339][SEVERE][grid-nio-worker-client-listener-2-#32][ClientListenerProcessor] Failed to process selector key [ses=GridSelectorNioSessionImpl [worker=ByteBufferNioClientWorker [readBuf=java.nio.HeapByteBuffer[pos=0 lim=8192 cap=8192], super=AbstractNioClientWorker [idx=2, bytesRcvd=0, bytesSent=0, bytesRcvd0=0, bytesSent0=0, select=true, super=GridWorker [name=grid-nio-worker-client-listener-2, igniteInstanceName=null, finished=false, hashCode=1810870884, interrupted=false, runner=grid-nio-worker-client-listener-2-#32]]], writeBuf=null, readBuf=null, inRecovery=null, outRecovery=null, super=GridNioSessionImpl [locAddr=/64.102.213.190:10800, rmtAddr=/10.82.224.11:65323, createTime=1538852004067, closeTime=0, bytesSent=280, bytesRcvd=215, bytesSent0=0, bytesRcvd0=0, sndSchedTime=1538854342759, lastSndTime=1538854342759, lastRcvTime=1538854342759, readsPaused=false, filterChain=FilterChain[filters=[GridNioAsyncNotifyFilter, GridNioCodecFilter [parser=ClientListenerBufferedParser, directMode=false]], accepted=true]]] java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcherImpl.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) at sun.nio.ch.IOUtil.read(IOUtil.java:197) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) at org.apache.ignite.internal.util.nio.GridNioServer$ByteBufferNioClientWorker.processRead(GridNioServer.java:1085) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2339) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2110) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.lang.Thread.run(Thread.java:748) [09:27:11,456][SEVERE][grid-nio-worker-client-listener-3-#33][ClientListenerProcessor] Failed to process selector key [ses=GridSelectorNioSessionImpl [worker=ByteBufferNioClientWorker [readBuf=java.nio.HeapByteBuffer[pos=0 lim=8192 cap=8192], super=AbstractNioClientWorker [idx=3, bytesRcvd=0, bytesSent=0, bytesRcvd0=0, bytesSent0=0, select=true, super=GridWorker [name=grid-nio-worker-client-listener-3, igniteInstanceName=null, finished=false, hashCode=254322881, interrupted=false, runner=grid-nio-worker-client-listener-3-#33]]], writeBuf=null, readBuf=null, inRecovery=null, outRecovery=null, super=GridNioSessionImpl [locAddr=/64.102.213.190:10800, rmtAddr=/10.82.224.11:53182, createTime=1538897206161, closeTime=0, bytesSent=163, bytesRcvd=128, bytesSent0=0, bytesRcvd0=0, sndSchedTime=1538897228457, lastSndTime=1538897228457, lastRcvTime=1538897228457, readsPaused=false, filterChain=FilterChain[filters=[GridNioAsyncNotifyFilter, GridNioCodecFilter [parser=ClientListenerBufferedParser, directMode=false]], accepted=true]]] java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcherImpl.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) at sun.nio.ch.IOUtil.read(IOUtil.java:197) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) at org.apache.ignite.internal.util.nio.GridNioServer$ByteBufferNioClientWorker.processRead(GridNioServer.java:1085) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2339) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2110) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.lang.Thread.run(Thread.java:748) [16:27:51,008][SEVERE][grid-nio-worker-client-listener-0-#30][ClientListenerProcessor] Failed to process selector key [ses=GridSelectorNioSessionImpl [worker=ByteBufferNioClientWorker [readBuf=java.nio.HeapByteBuffer[pos=0 lim=8192 cap=8192], super=AbstractNioClientWorker [idx=0, bytesRcvd=0, bytesSent=0, bytesRcvd0=0, bytesSent0=0, select=true, super=GridWorker [name=grid-nio-worker-client-listener-0, igniteInstanceName=null, finished=false, hashCode=2211598, interrupted=false, runner=grid-nio-worker-client-listener-0-#30]]], writeBuf=null, readBuf=null, inRecovery=null, outRecovery=null, super=GridNioSessionImpl [locAddr=/64.102.213.190:10800, rmtAddr=/10.82.224.11:58799, createTime=1538920292729, closeTime=0, bytesSent=397, bytesRcvd=302, bytesSent0=0, bytesRcvd0=0, sndSchedTime=1538922468102, lastSndTime=1538922468102, lastRcvTime=1538922468102, readsPaused=false, filterChain=FilterChain[filters=[GridNioAsyncNotifyFilter, GridNioCodecFilter [parser=ClientListenerBufferedParser, directMode=false]], accepted=true]]] java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcherImpl.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) at sun.nio.ch.IOUtil.read(IOUtil.java:197) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) at org.apache.ignite.internal.util.nio.GridNioServer$ByteBufferNioClientWorker.processRead(GridNioServer.java:1085) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2339) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2110) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.lang.Thread.run(Thread.java:748) [23:43:07,105][SEVERE][grid-nio-worker-client-listener-1-#31][ClientListenerProcessor] Failed to process selector key [ses=GridSelectorNioSessionImpl [worker=ByteBufferNioClientWorker [readBuf=java.nio.HeapByteBuffer[pos=0 lim=8192 cap=8192], super=AbstractNioClientWorker [idx=1, bytesRcvd=0, bytesSent=0, bytesRcvd0=0, bytesSent0=0, select=true, super=GridWorker [name=grid-nio-worker-client-listener-1, igniteInstanceName=null, finished=false, hashCode=1626735999, interrupted=false, runner=grid-nio-worker-client-listener-1-#31]]], writeBuf=null, readBuf=null, inRecovery=null, outRecovery=null, super=GridNioSessionImpl [locAddr=/64.102.213.190:10800, rmtAddr=/10.82.224.11:57332, createTime=1538947042237, closeTime=0, bytesSent=631, bytesRcvd=476, bytesSent0=0, bytesRcvd0=0, sndSchedTime=1538948581568, lastSndTime=1538948581568, lastRcvTime=1538948581568, readsPaused=false, filterChain=FilterChain[filters=[GridNioAsyncNotifyFilter, GridNioCodecFilter [parser=ClientListenerBufferedParser, directMode=false]], accepted=true]]] java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcherImpl.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) at sun.nio.ch.IOUtil.read(IOUtil.java:197) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) at org.apache.ignite.internal.util.nio.GridNioServer$ByteBufferNioClientWorker.processRead(GridNioServer.java:1085) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2339) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2110) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.lang.Thread.run(Thread.java:748) Client Logs: 2018-10-08 14:03:06 INFO IgniteKernal:566 - Metrics for local node (to disable set 'metricsLogFrequency' to 0) ^-- Node [id=2760b50c, uptime=14:03:31.475] ^-- H/N/C [hosts=4, nodes=13, CPUs=32] ^-- CPU [cur=0.07%, avg=0.1%, GC=0%] ^-- PageMemory [pages=0] ^-- Heap [used=2279MB, free=68.69%, comm=3143MB] ^-- Non heap [used=138MB, free=-1%, comm=142MB] ^-- Outbound messages queue [size=0] ^-- Public thread pool [active=0, idle=0, qSize=0] ^-- System thread pool [active=3, idle=0, qSize=0] 2018-10-08 14:03:55 WARN diagnostic:571 - Failed to wait for partition map exchange [topVer=AffinityTopologyVersion [topVer=123, minorTopVer=0], node=2760b50c-0617-4dfd-bbba-23e842b362f5]. Consider changing TransactionConfiguration.txTimeoutOnPartitionMapSynchronization to non default value to avoid this message. Dumping pending objects that might be the cause: 2018-10-08 14:03:55 WARN diagnostic:571 - Ready affinity version: AffinityTopologyVersion [topVer=122, minorTopVer=0] 2018-10-08 14:03:55 WARN diagnostic:571 - Last exchange future: GridDhtPartitionsExchangeFuture [firstDiscoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=80d74493-b1c2-48ee-a998-98a927692a0d, addrs=[0:0:0:0:0:0:0:1%lo, 127.0.0.1, 64.102.213.190], sockAddrs=[host-64-102-213-190.cisco.com/64.102.213.190:47501, /0:0:0:0:0:0:0:1%lo:47501, /127.0.0.1:47501], discPort=47501, order=122, intOrder=68, lastExchangeTime=1538930290162, loc=false, ver=2.6.0#20180710-sha1:669feacc, isClient=false], topVer=123, nodeId8=2760b50c, msg=Node left: TcpDiscoveryNode [id=80d74493-b1c2-48ee-a998-98a927692a0d, addrs=[0:0:0:0:0:0:0:1%lo, 127.0.0.1, 64.102.213.190], sockAddrs=[host-64-102-213-190.cisco.com/64.102.213.190:47501, /0:0:0:0:0:0:0:1%lo:47501, /127.0.0.1:47501], discPort=47501, order=122, intOrder=68, lastExchangeTime=1538930290162, loc=false, ver=2.6.0#20180710-sha1:669feacc, isClient=false], type=NODE_LEFT, tstamp=1538931072415], crd=TcpDiscoveryNode [id=512609ab-1fcf-4a51-bb7c-3965abc6b386, addrs=[0:0:0:0:0:0:0:1%lo, 127.0.0.1, 64.102.212.151], sockAddrs=[ccrc-rptignite-stg1-01.cisco.com/64.102.212.151:47500, /0:0:0:0:0:0:0:1%lo:47500, /127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1538740774375, loc=false, ver=2.6.0#20180710-sha1:669feacc, isClient=false], exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=123, minorTopVer=0], discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=80d74493-b1c2-48ee-a998-98a927692a0d, addrs=[0:0:0:0:0:0:0:1%lo, 127.0.0.1, 64.102.213.190], sockAddrs=[host-64-102-213-190.cisco.com/64.102.213.190:47501, /0:0:0:0:0:0:0:1%lo:47501, /127.0.0.1:47501], discPort=47501, order=122, intOrder=68, lastExchangeTime=1538930290162, loc=false, ver=2.6.0#20180710-sha1:669feacc, isClient=false], topVer=123, nodeId8=2760b50c, msg=Node left: TcpDiscoveryNode [id=80d74493-b1c2-48ee-a998-98a927692a0d, addrs=[0:0:0:0:0:0:0:1%lo, 127.0.0.1, 64.102.213.190], sockAddrs=[host-64-102-213-190.cisco.com/64.102.213.190:47501, /0:0:0:0:0:0:0:1%lo:47501, /127.0.0.1:47501], discPort=47501, order=122, intOrder=68, lastExchangeTime=1538930290162, loc=false, ver=2.6.0#20180710-sha1:669feacc, isClient=false], type=NODE_LEFT, tstamp=1538931072415], nodeId=80d74493, evt=NODE_LEFT], added=true, initFut=GridFutureAdapter [ignoreInterrupts=false, state=DONE, res=true, hash=1553040840], init=true, lastVer=null, partReleaseFut=null, exchActions=null, affChangeMsg=null, initTs=1538931072455, centralizedAff=false, forceAffReassignment=false, changeGlobalStateE=null, done=false, state=CLIENT, evtLatch=0, remaining=[f44497fe-3f02-453d-8407-078807e74288, f6605e96-47c9-479b-a840-03316500c9a3, 512609ab-1fcf-4a51-bb7c-3965abc6b386, 4470553b-4f25-48cc-abb6-ac260f4d6301], super=GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=992063964]] 2018-10-08 14:03:55 WARN GridCachePartitionExchangeManager:571 - First 10 pending exchange futures [total=2] 2018-10-08 14:03:55 WARN diagnostic:571 - Last 10 exchange futures (total: 4): 2018-10-08 14:03:55 WARN diagnostic:571 - >>> GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=123, minorTopVer=0], evt=NODE_LEFT, evtNode=TcpDiscoveryNode [id=80d74493-b1c2-48ee-a998-98a927692a0d, addrs=[0:0:0:0:0:0:0:1%lo, 127.0.0.1, 64.102.213.190], sockAddrs=[host-64-102-213-190.cisco.com/64.102.213.190:47501, /0:0:0:0:0:0:0:1%lo:47501, /127.0.0.1:47501], discPort=47501, order=122, intOrder=68, lastExchangeTime=1538930290162, loc=false, ver=2.6.0#20180710-sha1:669feacc, isClient=false], done=false] 2018-10-08 14:03:55 WARN diagnostic:571 - >>> GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=122, minorTopVer=0], evt=NODE_JOINED, evtNode=TcpDiscoveryNode [id=80d74493-b1c2-48ee-a998-98a927692a0d, addrs=[0:0:0:0:0:0:0:1%lo, 127.0.0.1, 64.102.213.190], sockAddrs=[host-64-102-213-190.cisco.com/64.102.213.190:47501, /0:0:0:0:0:0:0:1%lo:47501, /127.0.0.1:47501], discPort=47501, order=122, intOrder=68, lastExchangeTime=1538930290162, loc=false, ver=2.6.0#20180710-sha1:669feacc, isClient=false], done=true] 2018-10-08 14:03:55 WARN diagnostic:571 - >>> GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=121, minorTopVer=0], evt=NODE_JOINED, evtNode=TcpDiscoveryNode [id=81855ff0-46db-4284-ade4-3823667cc194, addrs=[0:0:0:0:0:0:0:1%lo, 127.0.0.1, 64.102.213.190], sockAddrs=[host-64-102-213-190.cisco.com/64.102.213.190:0, /0:0:0:0:0:0:0:1%lo:0, /127.0.0.1:0], discPort=0, order=121, intOrder=67, lastExchangeTime=1538740774476, loc=false, ver=2.6.0#20180710-sha1:669feacc, isClient=true], done=true] 2018-10-08 14:03:55 WARN diagnostic:571 - >>> GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=120, minorTopVer=0], evt=NODE_JOINED, evtNode=TcpDiscoveryNode [id=2760b50c-0617-4dfd-bbba-23e842b362f5, addrs=[0:0:0:0:0:0:0:1%lo, 127.0.0.1, 64.102.213.190], sockAddrs=[host-64-102-213-190.cisco.com/64.102.213.190:0, /0:0:0:0:0:0:0:1%lo:0, /127.0.0.1:0], discPort=0, order=120, intOrder=0, lastExchangeTime=1538740772995, loc=true, ver=2.6.0#20180710-sha1:669feacc, isClient=true], done=true] 2018-10-08 14:03:55 WARN diagnostic:571 - Latch manager state: ExchangeLatchManager [serverLatches={}, clientLatches={}] 2018-10-08 14:03:55 WARN diagnostic:571 - Pending transactions: 2018-10-08 14:03:55 WARN diagnostic:571 - Pending explicit locks: 2018-10-08 14:03:55 WARN diagnostic:571 - Pending cache futures: 2018-10-08 14:03:55 WARN diagnostic:571 - Pending atomic cache futures: 2018-10-08 14:03:55 WARN diagnostic:571 - Pending data streamer futures: 2018-10-08 14:03:55 WARN diagnostic:571 - Pending transaction deadlock detection futures: 2018-10-08 14:04:06 INFO IgniteKernal:566 - Metrics for local node (to disable set 'metricsLogFrequency' to 0) ^-- Node [id=2760b50c, uptime=14:04:31.475] ^-- H/N/C [hosts=4, nodes=13, CPUs=32] ^-- CPU [cur=0.13%, avg=0.1%, GC=0%] ^-- PageMemory [pages=0] ^-- Heap [used=2295MB, free=68.48%, comm=3143MB] ^-- Non heap [used=138MB, free=-1%, comm=142MB] ^-- Outbound messages queue [size=0] ^-- Public thread pool [active=0, idle=0, qSize=0] ^-- System thread pool [active=3, idle=0, qSize=0] 2018-10-08 14:05:06 INFO IgniteKernal:566 - Thanks -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/