Also, this is ignite 2.3.0, please let me know if there's any more information I can provide.
On Tue, Feb 27, 2018 at 9:59 AM, Bryan Rosander <[email protected]> wrote: > We're using ignite in a 3 node grid with SSL just hit an issue where after > a period of time (hours after starting), 2 of the 3 nodes seem to have lost > connectivity and we see the following stack trace over and over. > > The cluster starts up fine so I doubt it's an issue with the certificates > or keystores. Also bouncing the ignite instances seems to have "fixed" > it. Any ideas as to what could have happened? > > Thanks, > Bryan > > 2018-02-27 14:52:36,071 INFO [grid-nio-worker-tcp-comm-2-#27] > o.a.i.s.c.tcp.TcpCommunicationSpi - Accepted incoming communication > connection [locAddr=/100.96.3.72:47100, rmtAddr=/100.96.6.183:45484] > 2018-02-27 14:52:37,072 ERROR [grid-nio-worker-tcp-comm-2-#27] > o.a.i.s.c.tcp.TcpCommunicationSpi - Failed to process selector key > [ses=GridSelectorNioSessionImpl > [worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=2, > bytesRcvd=17479234, bytesSent=0, bytesRcvd0=2536, bytesSent0=0, > select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-2, > igniteInstanceName=null, finished=false, hashCode=1854311052, > interrupted=false, runner=grid-nio-worker-tcp-comm-2-#27]]], > writeBuf=java.nio.DirectByteBuffer[pos=0 lim=10 cap=32768], > readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], > inRecovery=null, outRecovery=null, super=GridNioSessionImpl [locAddr=/ > 100.96.3.72:47100, rmtAddr=/100.96.6.183:45484, createTime=1519743156030, > closeTime=0, bytesSent=2448, bytesRcvd=2536, bytesSent0=2448, > bytesRcvd0=2536, sndSchedTime=1519743156071, lastSndTime=1519743156071, > lastRcvTime=1519743156071, readsPaused=false, > filterChain=FilterChain[filters=[GridNioCodecFilter > [parser=o.a.i.i.util.nio.GridDirectParser@497350a6, directMode=true], > GridConnectionBytesVerifyFilter, SSL filter], accepted=true]]] > javax.net.ssl.SSLException: Failed to encrypt data (SSL engine error) > [status=CLOSED, handshakeStatus=NEED_UNWRAP, ses=GridSelectorNioSessionImpl > [worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=2, > bytesRcvd=17479234, bytesSent=0, bytesRcvd0=2536, bytesSent0=0, > select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-2, > igniteInstanceName=null, finished=false, hashCode=1854311052, > interrupted=false, runner=grid-nio-worker-tcp-comm-2-#27]]], > writeBuf=java.nio.DirectByteBuffer[pos=0 lim=10 cap=32768], > readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], > inRecovery=null, outRecovery=null, super=GridNioSessionImpl [locAddr=/ > 100.96.3.72:47100, rmtAddr=/100.96.6.183:45484, createTime=1519743156030, > closeTime=0, bytesSent=2448, bytesRcvd=2536, bytesSent0=2448, > bytesRcvd0=2536, sndSchedTime=1519743156071, lastSndTime=1519743156071, > lastRcvTime=1519743156071, readsPaused=false, > filterChain=FilterChain[filters=[GridNioCodecFilter > [parser=org.apache.ignite.internal.util.nio.GridDirectParser@497350a6, > directMode=true], GridConnectionBytesVerifyFilter, SSL filter], > accepted=true]]] > at org.apache.ignite.internal.util.nio.ssl. > GridNioSslHandler.encrypt(GridNioSslHandler.java:379) > at org.apache.ignite.internal.util.nio.ssl.GridNioSslFilter. > encrypt(GridNioSslFilter.java:270) > at org.apache.ignite.internal.util.nio.GridNioServer$ > DirectNioClientWorker.processWriteSsl(GridNioServer.java:1418) > at org.apache.ignite.internal.util.nio.GridNioServer$ > DirectNioClientWorker.processWrite(GridNioServer.java:1287) > at org.apache.ignite.internal.util.nio.GridNioServer$ > AbstractNioClientWorker.processSelectedKeysOptimized( > GridNioServer.java:2275) > at org.apache.ignite.internal.util.nio.GridNioServer$ > AbstractNioClientWorker.bodyInternal(GridNioServer.java:2048) > at org.apache.ignite.internal.util.nio.GridNioServer$ > AbstractNioClientWorker.body(GridNioServer.java:1717) > at org.apache.ignite.internal.util.worker.GridWorker.run( > GridWorker.java:110) > at java.lang.Thread.run(Thread.java:748) > 2018-02-27 14:52:37,072 WARN [grid-nio-worker-tcp-comm-2-#27] > o.a.i.s.c.tcp.TcpCommunicationSpi - Closing NIO session because of > unhandled exception [cls=class o.a.i.i.util.nio.GridNioException, > msg=Failed to encrypt data (SSL engine error) [status=CLOSED, > handshakeStatus=NEED_UNWRAP, ses=GridSelectorNioSessionImpl > [worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=2, > bytesRcvd=17479234, bytesSent=0, bytesRcvd0=2536, bytesSent0=0, > select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-2, > igniteInstanceName=null, finished=false, hashCode=1854311052, > interrupted=false, runner=grid-nio-worker-tcp-comm-2-#27]]], > writeBuf=java.nio.DirectByteBuffer[pos=0 lim=10 cap=32768], > readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], > inRecovery=null, outRecovery=null, super=GridNioSessionImpl [locAddr=/ > 100.96.3.72:47100, rmtAddr=/100.96.6.183:45484, createTime=1519743156030, > closeTime=0, bytesSent=2448, bytesRcvd=2536, bytesSent0=2448, > bytesRcvd0=2536, sndSchedTime=1519743156071, lastSndTime=1519743156071, > lastRcvTime=1519743156071, readsPaused=false, > filterChain=FilterChain[filters=[GridNioCodecFilter > [parser=o.a.i.i.util.nio.GridDirectParser@497350a6, directMode=true], > GridConnectionBytesVerifyFilter, SSL filter], accepted=true]]]] > 2018-02-27 14:52:37,321 INFO [grid-nio-worker-tcp-comm-3-#28] > o.a.i.s.c.tcp.TcpCommunicationSpi - Accepted incoming communication > connection [locAddr=/100.96.3.72:47100, rmtAddr=/100.96.6.183:45490] >
