Donot think so. As below the remote ip 10.254.13.83 is the
another server node.
--- log
Accepted incoming communication connection [locAddr=/10.254.32.162:47100,
rmtAddr=/10.254.13.83:35160]
super=GridNioSessionImpl [locAddr=/10.254.32.162:52542,
rmtAddr=/10.254.13.83:47100
---
so the multiple connections kept being rejected were between two server nodes.
Any scenarios could cause that ? it appears to be that the original
connection was shutdown or interrupted fast by one node but the other node was
not aware of the connection close event, or was not informed ? Any
configuration can help on that ?
Thanks,
-MJ
Original Email
From:"Pavel Tupitsyn"< [email protected] >;
Sent Time:2024/9/16 12:58
To:"user"< [email protected] >;
Subject:Re: rejecting communication connection & Failed to process selector
key
Looks like some non-Ignite application connects to the Ignite server, then
sends unexpected data or disconnects quickly.
Could it be some kind of a security tool, port scanner, or a misconfigured
service somewhere on the network?
On Mon, Sep 16, 2024 at 3:59 AM MJ <[email protected]> wrote:
Hi Igniters,
I am experiencing the “Failed to process selector key” error once every one or
two days. Every time it appears received and rejected multiple communication
connections and then threw the exception.
Below logging is about “Broken pipe” original exception but not only “Broken
pipe”, occasionally the “Failed to process selector key” wraps “Connection
Reset”, “javax.net.ssl.SSLException: Failed to encrypt data (SSL engine error)
[status=CLOSED, handshakeStatus=NOT_HANDSHAKING”.
Is there any solution to fix it ? or its configuration can be improved ?
Ignite 2.16.0 / 4 data nodes, running in openshift 4
---- config of communicationSpi
<property name="communicationSpi">
<bean class="org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi">
<!-- <property name="localPort" value="48100" /> -->
<property name="messageQueueLimit" value="1024"/>
<property name="socketWriteTimeout" value="25000"/>
<property name="idleConnectionTimeout" value="60000"/>
</bean>
</property>
24-09-15 17:18:35.146 [INFO ]
grid-nio-worker-tcp-comm-2-#25%TcpCommunicationSpi%
o.a.i.s.c.t.TcpCommunicationSpi:117 - Accepted incoming communication
connection [locAddr=/10.254.32.162:47100, rmtAddr=/10.254.13.83:35160]
24-09-15 17:18:35.147 [INFO ]
grid-nio-worker-tcp-comm-2-#25%TcpCommunicationSpi%
o.a.i.s.c.t.TcpCommunicationSpi:117 - Received incoming connection when already
connected to this node, rejecting
[locNode=52437bc3-3dfe-4f76-bec6-d2f22f8a5d40,
rmtNode=7c28b6bc-8991-47a2-b69c-6adba0482713]
24-09-15 17:18:35.357 [INFO ]
grid-nio-worker-tcp-comm-3-#26%TcpCommunicationSpi%
o.a.i.s.c.t.TcpCommunicationSpi:117 - Accepted incoming communication
connection [locAddr=/10.254.32.162:47100, rmtAddr=/10.254.13.83:35162]
24-09-15 17:18:35.358 [INFO ]
grid-nio-worker-tcp-comm-3-#26%TcpCommunicationSpi%
o.a.i.s.c.t.TcpCommunicationSpi:117 - Received incoming connection when already
connected to this node, rejecting
[locNode=52437bc3-3dfe-4f76-bec6-d2f22f8a5d40,
rmtNode=7c28b6bc-8991-47a2-b69c-6adba0482713]
24-09-15 17:18:35.568 [INFO ]
grid-nio-worker-tcp-comm-0-#23%TcpCommunicationSpi%
o.a.i.s.c.t.TcpCommunicationSpi:117 - Accepted incoming communication
connection [locAddr=/10.254.32.162:47100, rmtAddr=/10.254.13.83:35164]
24-09-15 17:18:35.569 [INFO ]
grid-nio-worker-tcp-comm-0-#23%TcpCommunicationSpi%
o.a.i.s.c.t.TcpCommunicationSpi:117 - Received incoming connection when already
connected to this node, rejecting
[locNode=52437bc3-3dfe-4f76-bec6-d2f22f8a5d40,
rmtNode=7c28b6bc-8991-47a2-b69c-6adba0482713]
24-09-15 17:18:35.975 [ERROR]
grid-nio-worker-tcp-comm-1-#24%TcpCommunicationSpi%
o.a.i.s.c.t.TcpCommunicationSpi:137 - Failed to process selector key
[ses=GridSelectorNioSessionImpl [worker=DirectNioClientWorker
[super=AbstractNioClientWorker [idx=1, bytesRcvd=29406013584, bytesSent=0,
bytesRcvd0=0, bytesSent0=0, select=true, super=GridWorker
[name=grid-nio-worker-tcp-comm-1, igniteInstanceName=TcpCommunicationSpi,
finished=false, heartbeatTs=1726435114873, hashCode=1144648384,
interrupted=false,
runner=grid-nio-worker-tcp-comm-1-#24%TcpCommunicationSpi%]]],
writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
inRecovery=GridNioRecoveryDescriptor [acked=20129536, resendCnt=0,
rcvCnt=19533551, sentCnt=20129879, reserved=true, lastAck=19533551,
nodeLeft=false, node=TcpDiscoveryNode [id=7c28b6bc-8991-47a2-b69c-6adba0482713,
consistentId=10.254.13.83,127.0.0.1:47500, addrs=ArrayList [10.254.13.83,
127.0.0.1], sockAddrs=HashSet [/10.254.13.83:47500, /127.0.0.1:47500],
discPort=47500, order=3, intOrder=3, lastExchangeTime=1724822271382, loc=false,
ver=2.16.0#20231215-sha1:7bde6a42, isClient=false], connected=false,
connectCnt=205, queueLimit=131072, reserveCnt=260, pairedConnections=false],
outRecovery=GridNioRecoveryDescriptor [acked=20129536, resendCnt=0,
rcvCnt=19533551, sentCnt=20129879, reserved=true, lastAck=19533551,
nodeLeft=false, node=TcpDiscoveryNode [id=7c28b6bc-8991-47a2-b69c-6adba0482713,
consistentId=10.254.13.83,127.0.0.1:47500, addrs=ArrayList [10.254.13.83,
127.0.0.1], sockAddrs=HashSet [/10.254.13.83:47500, /127.0.0.1:47500],
discPort=47500, order=3, intOrder=3, lastExchangeTime=1724822271382, loc=false,
ver=2.16.0#20231215-sha1:7bde6a42, isClient=false], connected=false,
connectCnt=205, queueLimit=131072, reserveCnt=260, pairedConnections=false],
closeSocket=true,
outboundMessagesQueueSizeMetric=o.a.i.i.processors.metric.impl.LongAdderMetric@69a257d1,
super=GridNioSessionImpl [locAddr=/10.254.32.162:52542,
rmtAddr=/10.254.13.83:47100, createTime=1726435114863, closeTime=0,
bytesSent=164200, bytesRcvd=468, bytesSent0=0, bytesRcvd0=0,
sndSchedTime=1726435114863, lastSndTime=1726435114972,
lastRcvTime=1726435114972, readsPaused=false,
filterChain=FilterChain[filters=[GridNioCodecFilter
[parser=o.a.i.i.util.nio.GridDirectParser@5196c6f7, directMode=true],
GridConnectionBytesVerifyFilter, SSL filter], accepted=false,
markedForClose=true]]] java.io.IOException: Broken pipe
at java.base/sun.nio.ch.FileDispatcherImpl.write0(Native Method) ~[?:?]
at java.base/sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:62) ~[?:?]
at java.base/sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:132) ~[?:?]
at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:76) ~[?:?]
at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:53) ~[?:?]
at java.base/sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:532)
~[?:?]
at
org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processWriteSsl(GridNioServer.java:1524)
~[ignite-core-2.16.0.jar:2.16.0]
at
org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processWrite(GridNioServer.java:1405)
~[ignite-core-2.16.0.jar:2.16.0]
at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2530)
[ignite-core-2.16.0.jar:2.16.0]
at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2282)
[ignite-core-2.16.0.jar:2.16.0]
at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1911)
[ignite-core-2.16.0.jar:2.16.0]
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:125)
[ignite-core-2.16.0.jar:2.16.0]
at java.base/java.lang.Thread.run(Thread.java:842) [?:?]
Thanks,
-MJ
