Re: Failed to process selector key
These errors look like something which does not speak Ignite binary protocol is connecting and sending useless stuff to your Ignite cluster. IgniteException: Invalid message type: 2057 Check the configuration of the client if the host generating this traffic is known, and check firewalls or monitoring tools if not. On Mon, Nov 13, 2023 at 8:04 AM Humphrey Lopez wrote: > Other errors we are seeing: > > Failed to read message [msg=null, buf=java.nio.DirectByteBuffer[pos=2 > lim=162 cap=32768], reader=DirectMessageReader [state=DirectMessageState > [pos=0, stack=[StateItem [stream=DirectByteBufferStreamImplV2 > [baseOff=140381476511056, arrOff=-1, tmpArrOff=0, valReadBytes=0, > tmpArrBytes=0, msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, > keyDone=false, readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, > uuidMost=0, uuidLeast=0, uuidLocId=0], state=0], StateItem > [stream=DirectByteBufferStreamImplV2 [baseOff=140381476511056, arrOff=-1, > tmpArrOff=0, valReadBytes=0, tmpArrBytes=0, msgTypeDone=false, msg=null, > mapIt=null, it=null, arrPos=-1, keyDone=false, readSize=-1, readItems=0, > prim=0, primShift=0, uuidState=0, uuidMost=0, uuidLeast=0, uuidLocId=0], > state=0], StateItem [stream=DirectByteBufferStreamImplV2 > [baseOff=140381476511056, arrOff=-1, tmpArrOff=0, valReadBytes=0, > tmpArrBytes=0, msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, > keyDone=false, readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, > uuidMost=0, uuidLeast=0, uuidLocId=0], state=0], null, null, null, null, > null, null, null]], protoVer=3, lastRead=true], > ses=GridSelectorNioSessionImpl [worker=DirectNioClientWorker > [super=AbstractNioClientWorker [idx=1, bytesRcvd=6506344847, > bytesSent=5800573007, bytesRcvd0=5461705, bytesSent0=197830, select=true, > super=GridWorker [name=grid-nio-worker-tcp-comm-1, > igniteInstanceName=TcpCommunicationSpi, finished=false, > heartbeatTs=1699706651957, hashCode=2094994491, interrupted=false, > runner=grid-nio-worker-tcp-comm-1-#48%TcpCommunicationSpi%]]], > writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], > readBuf=java.nio.DirectByteBuffer[pos=2 lim=162 cap=32768], > inRecovery=GridNioRecoveryDescriptor [acked=47232, resendCnt=0, > rcvCnt=53951, sentCnt=47247, reserved=true, lastAck=53920, nodeLeft=false, > node=TcpDiscoveryNode [id=34cfcc64-d369-415b-b14f-6ac222087232, > consistentId=34cfcc64-d369-415b-b14f-6ac222087232, addrs=ArrayList > [xx.xxx.xx.xxx, 127.0.0.1], sockAddrs=null, discPort=0, order=24, > intOrder=24, lastExchangeTime=1699691906215, loc=false, > ver=2.15.0#20230425-sha1:f98f7f35, isClient=true], connected=true, > connectCnt=0, queueLimit=4096, reserveCnt=1, pairedConnections=false], > outRecovery=GridNioRecoveryDescriptor [acked=47232, resendCnt=0, > rcvCnt=53951, sentCnt=47247, reserved=true, lastAck=53920, nodeLeft=false, > node=TcpDiscoveryNode [id=34cfcc64-d369-415b-b14f-6ac222087232, > consistentId=34cfcc64-d369-415b-b14f-6ac222087232, addrs=ArrayList > [xx.xxx.xx.xxx, 127.0.0.1], sockAddrs=null, discPort=0, order=24, > intOrder=24, lastExchangeTime=1699691906215, loc=false, > ver=2.15.0#20230425-sha1:f98f7f35, isClient=true], connected=true, > connectCnt=0, queueLimit=4096, reserveCnt=1, pairedConnections=false], > closeSocket=true, > outboundMessagesQueueSizeMetric=o.a.i.i.processors.metric.impl.LongAdderMetric@69a257d1, > super=GridNioSessionImpl [locAddr=/xx.xxx.xx.xx:47100, > rmtAddr=/xx.xxx.xx.xxx:35492, createTime=1699700043744, closeTime=0, > bytesSent=74190856, bytesRcvd=167712723, bytesSent0=0, bytesRcvd0=5260541, > sndSchedTime=1699700043744, lastSndTime=1699706650143, > lastRcvTime=1699706651957, readsPaused=false, > filterChain=FilterChain[filters=[GridNioCodecFilter > [parser=o.a.i.i.util.nio.GridDirectParser@6c311b05, directMode=true], > GridConnectionBytesVerifyFilter], accepted=true, markedForClose=false]]] > <#fb58e00b> o.a.i.IgniteException: Invalid message type: 2057 at > o.a.i.i.m.c.IgniteMessageFactoryImpl.create(IgniteMessageFactoryImpl.java:133) > at > o.a.i.s.c.t.i.GridNioServerWrapper$2.create(GridNioServerWrapper.java:813) > at o.a.i.i.u.n.GridDirectParser.decode(GridDirectParser.java:81) at > o.a.i.i.u.n.GridNioCodecFilter.onMessageReceived(GridNioCodecFilter.java:113) > at > o.a.i.i.u.n.GridNioFilterAdapter.proceedMessageReceived(GridNioFilterAdapter.java:109) > at > o.a.i.i.u.n.GridConnectionBytesVerifyFilter.onMessageReceived(GridConnectionBytesVerifyFilter.java:133) > at > o.a.i.i.u.n.GridNioFilterAdapter.proceedMessageReceived(GridNioFilterAdapter.java:109) > at > o.a.i.i.u.n.GridNioServer$HeadFilter.onMessageReceived(GridNioServer.java:3752) > at > o.a.i.i.u.n.GridNioFilterChain.onMessageReceived(GridNioFilterChain.java:175) > at > o.a.i.i.u.n.GridNioServer$DirectNioClientWorker.processRead(GridNioServer.java:1379) > at > o.a.i.i.u.n.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2526) > at > o.a.i.i.u.n.GridNioServer$AbstractNio
Re: Failed to process selector key
Other errors we are seeing: Failed to read message [msg=null, buf=java.nio.DirectByteBuffer[pos=2 lim=162 cap=32768], reader=DirectMessageReader [state=DirectMessageState [pos=0, stack=[StateItem [stream=DirectByteBufferStreamImplV2 [baseOff=140381476511056, arrOff=-1, tmpArrOff=0, valReadBytes=0, tmpArrBytes=0, msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false, readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0, uuidLeast=0, uuidLocId=0], state=0], StateItem [stream=DirectByteBufferStreamImplV2 [baseOff=140381476511056, arrOff=-1, tmpArrOff=0, valReadBytes=0, tmpArrBytes=0, msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false, readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0, uuidLeast=0, uuidLocId=0], state=0], StateItem [stream=DirectByteBufferStreamImplV2 [baseOff=140381476511056, arrOff=-1, tmpArrOff=0, valReadBytes=0, tmpArrBytes=0, msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false, readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0, uuidLeast=0, uuidLocId=0], state=0], null, null, null, null, null, null, null]], protoVer=3, lastRead=true], ses=GridSelectorNioSessionImpl [worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=1, bytesRcvd=6506344847, bytesSent=5800573007, bytesRcvd0=5461705, bytesSent0=197830, select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-1, igniteInstanceName=TcpCommunicationSpi, finished=false, heartbeatTs=1699706651957, hashCode=2094994491, interrupted=false, runner=grid-nio-worker-tcp-comm-1-#48%TcpCommunicationSpi%]]], writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], readBuf=java.nio.DirectByteBuffer[pos=2 lim=162 cap=32768], inRecovery=GridNioRecoveryDescriptor [acked=47232, resendCnt=0, rcvCnt=53951, sentCnt=47247, reserved=true, lastAck=53920, nodeLeft=false, node=TcpDiscoveryNode [id=34cfcc64-d369-415b-b14f-6ac222087232, consistentId=34cfcc64-d369-415b-b14f-6ac222087232, addrs=ArrayList [xx.xxx.xx.xxx, 127.0.0.1], sockAddrs=null, discPort=0, order=24, intOrder=24, lastExchangeTime=1699691906215, loc=false, ver=2.15.0#20230425-sha1:f98f7f35, isClient=true], connected=true, connectCnt=0, queueLimit=4096, reserveCnt=1, pairedConnections=false], outRecovery=GridNioRecoveryDescriptor [acked=47232, resendCnt=0, rcvCnt=53951, sentCnt=47247, reserved=true, lastAck=53920, nodeLeft=false, node=TcpDiscoveryNode [id=34cfcc64-d369-415b-b14f-6ac222087232, consistentId=34cfcc64-d369-415b-b14f-6ac222087232, addrs=ArrayList [xx.xxx.xx.xxx, 127.0.0.1], sockAddrs=null, discPort=0, order=24, intOrder=24, lastExchangeTime=1699691906215, loc=false, ver=2.15.0#20230425-sha1:f98f7f35, isClient=true], connected=true, connectCnt=0, queueLimit=4096, reserveCnt=1, pairedConnections=false], closeSocket=true, outboundMessagesQueueSizeMetric=o.a.i.i.processors.metric.impl.LongAdderMetric@69a257d1, super=GridNioSessionImpl [locAddr=/xx.xxx.xx.xx:47100, rmtAddr=/xx.xxx.xx.xxx:35492, createTime=1699700043744, closeTime=0, bytesSent=74190856, bytesRcvd=167712723, bytesSent0=0, bytesRcvd0=5260541, sndSchedTime=1699700043744, lastSndTime=1699706650143, lastRcvTime=1699706651957, readsPaused=false, filterChain=FilterChain[filters=[GridNioCodecFilter [parser=o.a.i.i.util.nio.GridDirectParser@6c311b05, directMode=true], GridConnectionBytesVerifyFilter], accepted=true, markedForClose=false]]] <#fb58e00b> o.a.i.IgniteException: Invalid message type: 2057 at o.a.i.i.m.c.IgniteMessageFactoryImpl.create(IgniteMessageFactoryImpl.java:133) at o.a.i.s.c.t.i.GridNioServerWrapper$2.create(GridNioServerWrapper.java:813) at o.a.i.i.u.n.GridDirectParser.decode(GridDirectParser.java:81) at o.a.i.i.u.n.GridNioCodecFilter.onMessageReceived(GridNioCodecFilter.java:113) at o.a.i.i.u.n.GridNioFilterAdapter.proceedMessageReceived(GridNioFilterAdapter.java:109) at o.a.i.i.u.n.GridConnectionBytesVerifyFilter.onMessageReceived(GridConnectionBytesVerifyFilter.java:133) at o.a.i.i.u.n.GridNioFilterAdapter.proceedMessageReceived(GridNioFilterAdapter.java:109) at o.a.i.i.u.n.GridNioServer$HeadFilter.onMessageReceived(GridNioServer.java:3752) at o.a.i.i.u.n.GridNioFilterChain.onMessageReceived(GridNioFilterChain.java:175) at o.a.i.i.u.n.GridNioServer$DirectNioClientWorker.processRead(GridNioServer.java:1379) at o.a.i.i.u.n.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2526) at o.a.i.i.u.n.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2281) at o.a.i.i.u.n.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1910) at o.a.i.i.u.w.GridWorker.run(GridWorker.java:125) at java.lang.Thread.run(Thread.java:833) Op ma 13 nov 2023 om 14:33 schreef Humphrey Lopez : > Hello Ignite community. > > We are running Ignite 2.15 in production with JDK 17. We are seeing the > following errors and have no idea what is causing it. > > Failed to process selector key [ses=GridSelectorNioSessionImpl > [worker=DirectNioClie
Failed to process selector key
Hello Ignite community. We are running Ignite 2.15 in production with JDK 17. We are seeing the following errors and have no idea what is causing it. Failed to process selector key [ses=GridSelectorNioSessionImpl [worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=3, bytesRcvd=64732383766, bytesSent=30081901336, bytesRcvd0=0, bytesSent0=0, select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-3, igniteInstanceName=TcpCommunicationSpi, finished=false, heartbeatTs=1699879571052, hashCode=475467093, interrupted=false, runner=grid-nio-worker-tcp-comm-3-#50%TcpCommunicationSpi%]]], writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], readBuf=java.nio.DirectByteBuffer[pos=10 lim=282 cap=32768], inRecovery=GridNioRecoveryDescriptor [acked=209920, resendCnt=0, rcvCnt=229599, sentCnt=209938, reserved=true, lastAck=229568, nodeLeft=false, node=TcpDiscoveryNode [id=c8353de9-9cd2-4ae5-bc48-3271c47fffae, consistentId=c8353de9-9cd2-4ae5-bc48-3271c47fffae, addrs=ArrayList [machine1, 127.0.0.1], sockAddrs=null, discPort=0, order=26, intOrder=26, lastExchangeTime=1699691907837, loc=false, ver=2.15.0#20230425-sha1:f98f7f35, isClient=true], connected=true, connectCnt=0, queueLimit=4096, reserveCnt=1, pairedConnections=false], outRecovery=GridNioRecoveryDescriptor [acked=209920, resendCnt=0, rcvCnt=229599, sentCnt=209938, reserved=true, lastAck=229568, nodeLeft=false, node=TcpDiscoveryNode [id=c8353de9-9cd2-4ae5-bc48-3271c47fffae, consistentId=c8353de9-9cd2-4ae5-bc48-3271c47fffae, addrs=ArrayList [machine1, 127.0.0.1], sockAddrs=null, discPort=0, order=26, intOrder=26, lastExchangeTime=1699691907837, loc=false, ver=2.15.0#20230425-sha1:f98f7f35, isClient=true], connected=true, connectCnt=0, queueLimit=4096, reserveCnt=1, pairedConnections=false], closeSocket=true, outboundMessagesQueueSizeMetric=o.a.i.i.processors.metric.impl.LongAdderMetric@69a257d1, super=GridNioSessionImpl [locAddr=/10.129.34.235:47100, rmtAddr=/machine1:42492, createTime=1699700047871, closeTime=0, bytesSent=3311788770, bytesRcvd=23387281236, bytesSent0=0, bytesRcvd0=0, sndSchedTime=1699852982273, lastSndTime=1699879565624, lastRcvTime=1699879571052, readsPaused=false, filterChain=FilterChain[filters=[GridNioCodecFilter [parser=o.a.i.i.util.nio.GridDirectParser@f6e5016, directMode=true], GridConnectionBytesVerifyFilter], accepted=true, markedForClose=false]]] And there is also this stacktrace: j.l.NullPointerException: Cannot invoke "Object.hashCode()" because "key" is null at j.u.c.ConcurrentHashMap.get(ConcurrentHashMap.java:936) at o.a.i.i.m.c.GridIoManager.processOrderedMessage(GridIoManager.java:1707) at o.a.i.i.m.c.GridIoManager.onMessage0(GridIoManager.java:1328) at o.a.i.i.m.c.GridIoManager.access$300(GridIoManager.java:243) at o.a.i.i.m.c.GridIoManager$2.onMessage(GridIoManager.java:509) at o.a.i.s.c.t.TcpCommunicationSpi.notifyListener(TcpCommunicationSpi.java:1220) at o.a.i.s.c.t.TcpCommunicationSpi$1.onMessage(TcpCommunicationSpi.java:689) at o.a.i.s.c.t.TcpCommunicationSpi$1.onMessage(TcpCommunicationSpi.java:687) at o.a.i.s.c.t.i.InboundConnectionHandler.onMessage(InboundConnectionHandler.java:392) at o.a.i.s.c.t.i.InboundConnectionHandler.onMessage(InboundConnectionHandler.java:78) at o.a.i.i.u.n.GridNioFilterChain$TailFilter.onMessageReceived(GridNioFilterChain.java:279) at o.a.i.i.u.n.GridNioFilterAdapter.proceedMessageReceived(GridNioFilterAdapter.java:109) at o.a.i.i.u.n.GridNioCodecFilter.onMessageReceived(GridNioCodecFilter.java:116) at o.a.i.i.u.n.GridNioFilterAdapter.proceedMessageReceived(GridNioFilterAdapter.java:109) at o.a.i.i.u.n.GridConnectionBytesVerifyFilter.onMessageReceived(GridConnectionBytesVerifyFilter.java:88) at o.a.i.i.u.n.GridNioFilterAdapter.proceedMessageReceived(GridNioFilterAdapter.java:109) at o.a.i.i.u.n.GridNioServer$HeadFilter.onMessageReceived(GridNioServer.java:3752) at o.a.i.i.u.n.GridNioFilterChain.onMessageReceived(GridNioFilterChain.java:175) at o.a.i.i.u.n.GridNioServer$DirectNioClientWorker.processRead(GridNioServer.java:1379) at o.a.i.i.u.n.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2526) at o.a.i.i.u.n.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2281) at o.a.i.i.u.n.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1910) at o.a.i.i.u.w.GridWorker.run(GridWorker.java:125) at java.lang.Thread.run(Thread.java:833) We are using the following flags: "-XX:+AlwaysPreTouch", "-XX:+UseG1GC", "-XX:+ScavengeBeforeFullGC", "-XX:+DisableExplicitGC", "-XX:MaxMetaspaceSize=640m", "-Djava.net.preferIPv4Stack=true", "-DIGNITE_QUIET=false", "-DIGNITE_UPDATE_NOTIFIER=false", "-DIGNITE_WAIT_FOR_BACKUPS_ON_SHUTDOWN=true", "--add-opens=jdk.management/com.sun.management.internal=ALL-UNNAMED", "--add-opens=java.base/jdk.internal.misc=ALL-UNNAMED", "--add-opens=java.base/sun.nio.ch=ALL-UNNAMED", "--add-opens=java.management/com.sun.jmx.mbeanserver=ALL-UNNAMED", "--add-opens=jdk.internal.jvmstat/sun.jvmstat.m
