Hello! Here is my full logs with Stephen's image :) I think I have enabled INFO logging but I'm not sure tell me if you need more.
Ignite Logs wrote > [15:55:08] __________ ________________ > [15:55:08] / _/ ___/ |/ / _/_ __/ __/ > [15:55:08] _/ // (7 7 // / / / / _/ > [15:55:08] /___/\___/_/|_/___/ /_/ /___/ > [15:55:08] > [15:55:08] ver. 2.8.1#20200521-sha1:86422096 > [15:55:08] 2020 Copyright(C) Apache Software Foundation > [15:55:08] > [15:55:08] Ignite documentation: http://ignite.apache.org > [15:55:08] > [15:55:08] Quiet mode. > [15:55:08] ^-- Logging to file > '/opt/ignite/apache-ignite/work/log/ignite-2b0807c4.0.log' > [15:55:08] ^-- Logging by 'JavaLogger [quiet=true, config=null]' > [15:55:08] ^-- To see **FULL** console log here add -DIGNITE_QUIET=false > or "-v" to ignite.{sh|bat} > [15:55:08] > [15:55:09] OS: Linux 5.4.79-v7+ arm > [15:55:09] VM information: OpenJDK Runtime Environment 1.8.0_212-b04 > IcedTea OpenJDK Zero VM 25.212-b04 > [15:55:09] Please set system property '-Djava.net.preferIPv4Stack=true' to > avoid possible problems in mixed environments. > [15:55:09] Initial heap size is 16MB (should be no less than 512MB, use > -Xms512m -Xmx512m). > [15:55:11] Configured plugins: > [15:55:11] ^-- None > [15:55:11] > [15:55:11] Configured failure handler: [hnd=StopNodeOrHaltFailureHandler > [tryStop=false, timeout=0, super=AbstractFailureHandler > [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED, > SYSTEM_CRITICAL_OPERATION_TIMEOUT]]]] > [15:55:14] Message queue limit is set to 0 which may lead to potential > OOMEs when running cache operations in FULL_ASYNC or PRIMARY_SYNC modes > due to message queues growth on sender and receiver sides. > [15:55:15] Security status [authentication=off, tls/ssl=off] > [15:56:30] Nodes started on local machine require more than 80% of > physical RAM what can lead to significant slowdown due to swapping (please > decrease JVM heap size, data region size or checkpoint buffer size) > [required=574MB, available=924MB] > [15:56:46,041][SEVERE][tcp-disco-msg-worker-[crd]-#2][G] Blocked > system-critical thread has been detected. This can lead to cluster-wide > undefined behaviour [workerName=disco-notifier-worker, > threadName=disco-notifier-worker-#43, blockedFor=15s] > [15:56:46] Possible failure suppressed accordingly to a configured handler > [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, > super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet > [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], > failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class > o.a.i.IgniteException: GridWorker [name=disco-notifier-worker, > igniteInstanceName=null, finished=false, heartbeatTs=1611417390753]]] > [15:56:58,064][SEVERE][tcp-disco-msg-worker-[crd]-#2][G] Blocked > system-critical thread has been detected. This can lead to cluster-wide > undefined behaviour [workerName=disco-notifier-worker, > threadName=disco-notifier-worker-#43, blockedFor=27s] > [15:56:58] Possible failure suppressed accordingly to a configured handler > [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, > super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet > [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], > failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class > o.a.i.IgniteException: GridWorker [name=disco-notifier-worker, > igniteInstanceName=null, finished=false, heartbeatTs=1611417390753]]] > [15:57:09] Performance suggestions for grid (fix if possible) > [15:57:09] To disable, set -DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true > [15:57:09] ^-- Enable server mode for JVM (add '-server' to JVM options) > [15:57:09] ^-- Enable G1 Garbage Collector (add '-XX:+UseG1GC' to JVM > options) > [15:57:09] ^-- Specify JVM heap max size (add '-Xmx > <size> > [g|G|m|M|k|K]' to JVM options) > [15:57:09] ^-- Set max direct memory size if getting 'OOME: Direct > buffer memory' (add '-XX:MaxDirectMemorySize= > <size> > [g|G|m|M|k|K]' to JVM options) > [15:57:09] ^-- Disable processing of calls to System.gc() (add > '-XX:+DisableExplicitGC' to JVM options) > [15:57:09] Refer to this page for more performance suggestions: > https://apacheignite.readme.io/docs/jvm-and-system-tuning > [15:57:09] > [15:57:09] To start Console Management & Monitoring run > ignitevisorcmd.{sh|bat} > [15:57:09] Data Regions Configured: > [15:57:09] ^-- default_region [initSize=50.0 MiB, maxSize=50.0 MiB, > persistence=false, lazyMemoryAllocation=true] > [15:57:09] ^-- persistence_region [initSize=100.0 MiB, maxSize=100.0 > MiB, persistence=true, lazyMemoryAllocation=true] > [15:57:09] > [15:57:09] Ignite node started OK (id=2b0807c4) > [15:57:09] >>> Ignite cluster is not active (limited functionality > available). Use control.(sh|bat) script or IgniteCluster interface to > activate. > [15:57:09] Topology snapshot [ver=1, locNode=2b0807c4, servers=1, > clients=0, state=INACTIVE, CPUs=4, offheap=0.15GB, heap=0.22GB] > [15:57:16] New version is available at ignite.apache.org: 2.9.1 > [16:05:22,842][SEVERE][rest-#109][GridTcpRestProtocol] Failed to process > client request [ses=GridSelectorNioSessionImpl > [worker=ByteBufferNioClientWorker [readBuf=java.nio.HeapByteBuffer[pos=0 > lim=8192 cap=8192], super=AbstractNioClientWorker [idx=1, bytesRcvd=0, > bytesSent=0, bytesRcvd0=0, bytesSent0=0, select=true, super=GridWorker > [name=grid-nio-worker-tcp-rest-1, igniteInstanceName=null, finished=false, > heartbeatTs=1611417921389, hashCode=4009713, interrupted=false, > runner=grid-nio-worker-tcp-rest-1-#36]]], writeBuf=null, readBuf=null, > inRecovery=null, outRecovery=null, closeSocket=true, > outboundMessagesQueueSizeMetric=null, super=GridNioSessionImpl > [locAddr=/127.0.0.1:11211, rmtAddr=/127.0.0.1:33872, > createTime=1611417824153, closeTime=1611417905352, bytesSent=633, > bytesRcvd=519, bytesSent0=5, bytesRcvd0=5, sndSchedTime=1611417922803, > lastSndTime=1611417901655, lastRcvTime=1611417901655, readsPaused=false, > filterChain=FilterChain[filters=[GridNioCodecFilter > [parser=GridTcpRestParser [marsh=JdkMarshaller > [clsFilter=o.a.i.marshaller.MarshallerUtils$1@1864f0b], > routerClient=false], directMode=false]], accepted=true, > markedForClose=false]], msg=GridClientAuthenticationRequest > [super=GridClientAbstractMessage [reqId=3, > id=227a26f2-3ada-4be3-8150-5879fdfb48c1, destId=null, > super=o.a.i.i.processors.rest.client.message.GridClientAuthenticationRequest@177bade]]] > class org.apache.ignite.IgniteCheckedException: Failed to send message > (connection was closed): GridSelectorNioSessionImpl > [worker=ByteBufferNioClientWorker [readBuf=java.nio.HeapByteBuffer[pos=0 > lim=8192 cap=8192], super=AbstractNioClientWorker [idx=1, bytesRcvd=0, > bytesSent=0, bytesRcvd0=0, bytesSent0=0, select=true, super=GridWorker > [name=grid-nio-worker-tcp-rest-1, igniteInstanceName=null, finished=false, > heartbeatTs=1611417921389, hashCode=4009713, interrupted=false, > runner=grid-nio-worker-tcp-rest-1-#36]]], writeBuf=null, readBuf=null, > inRecovery=null, outRecovery=null, closeSocket=true, > outboundMessagesQueueSizeMetric=null, super=GridNioSessionImpl > [locAddr=/127.0.0.1:11211, rmtAddr=/127.0.0.1:33872, > createTime=1611417824153, closeTime=1611417905352, bytesSent=633, > bytesRcvd=519, bytesSent0=5, bytesRcvd0=5, sndSchedTime=1611417922803, > lastSndTime=1611417901655, lastRcvTime=1611417901655, readsPaused=false, > filterChain=FilterChain[filters=[GridNioCodecFilter > [parser=GridTcpRestParser [marsh=JdkMarshaller > [clsFilter=org.apache.ignite.marshaller.MarshallerUtils$1@1864f0b], > routerClient=false], directMode=false]], accepted=true, > markedForClose=false]] > at > org.apache.ignite.internal.util.IgniteUtils.cast(IgniteUtils.java:7507) > at > org.apache.ignite.internal.util.future.GridFutureAdapter.resolve(GridFutureAdapter.java:260) > at > org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:172) > at > org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:141) > at > org.apache.ignite.internal.processors.rest.protocols.tcp.GridTcpRestNioListener$1$1.apply(GridTcpRestNioListener.java:277) > at > org.apache.ignite.internal.processors.rest.protocols.tcp.GridTcpRestNioListener$1$1.apply(GridTcpRestNioListener.java:274) > at > org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:399) > at > org.apache.ignite.internal.util.future.GridFutureAdapter.listen(GridFutureAdapter.java:354) > at > org.apache.ignite.internal.processors.rest.protocols.tcp.GridTcpRestNioListener$1.apply(GridTcpRestNioListener.java:274) > at > org.apache.ignite.internal.processors.rest.protocols.tcp.GridTcpRestNioListener$1.apply(GridTcpRestNioListener.java:242) > at > org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:399) > at > org.apache.ignite.internal.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:347) > at > org.apache.ignite.internal.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:335) > at > org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:511) > at > org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:490) > at > org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:467) > at > org.apache.ignite.internal.processors.rest.GridRestProcessor$2$1.apply(GridRestProcessor.java:184) > at > org.apache.ignite.internal.processors.rest.GridRestProcessor$2$1.apply(GridRestProcessor.java:181) > at > org.apache.ignite.internal.util.future.GridFinishedFuture.listen(GridFinishedFuture.java:124) > at > org.apache.ignite.internal.processors.rest.GridRestProcessor$2.body(GridRestProcessor.java:181) > at > org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.io.IOException: Failed to send message (connection was > closed): GridSelectorNioSessionImpl [worker=ByteBufferNioClientWorker > [readBuf=java.nio.HeapByteBuffer[pos=0 lim=8192 cap=8192], > super=AbstractNioClientWorker [idx=1, bytesRcvd=0, bytesSent=0, > bytesRcvd0=0, bytesSent0=0, select=true, super=GridWorker > [name=grid-nio-worker-tcp-rest-1, igniteInstanceName=null, finished=false, > heartbeatTs=1611417921389, hashCode=4009713, interrupted=false, > runner=grid-nio-worker-tcp-rest-1-#36]]], writeBuf=null, readBuf=null, > inRecovery=null, outRecovery=null, closeSocket=true, > outboundMessagesQueueSizeMetric=null, super=GridNioSessionImpl > [locAddr=/127.0.0.1:11211, rmtAddr=/127.0.0.1:33872, > createTime=1611417824153, closeTime=1611417905352, bytesSent=633, > bytesRcvd=519, bytesSent0=5, bytesRcvd0=5, sndSchedTime=1611417922803, > lastSndTime=1611417901655, lastRcvTime=1611417901655, readsPaused=false, > filterChain=FilterChain[filters=[GridNioCodecFilter > [parser=GridTcpRestParser [marsh=JdkMarshaller > [clsFilter=org.apache.ignite.marshaller.MarshallerUtils$1@1864f0b], > routerClient=false], directMode=false]], accepted=true, > markedForClose=false]] > at > org.apache.ignite.internal.util.nio.GridNioServer.send0(GridNioServer.java:638) > at > org.apache.ignite.internal.util.nio.GridNioServer.send(GridNioServer.java:579) > at > org.apache.ignite.internal.util.nio.GridNioServer$HeadFilter.onSessionWrite(GridNioServer.java:3635) > at > org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedSessionWrite(GridNioFilterAdapter.java:121) > at > org.apache.ignite.internal.util.nio.GridNioCodecFilter.onSessionWrite(GridNioCodecFilter.java:96) > at > org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedSessionWrite(GridNioFilterAdapter.java:121) > at > org.apache.ignite.internal.util.nio.GridNioFilterChain$TailFilter.onSessionWrite(GridNioFilterChain.java:269) > at > org.apache.ignite.internal.util.nio.GridNioFilterChain.onSessionWrite(GridNioFilterChain.java:192) > at > org.apache.ignite.internal.util.nio.GridNioSessionImpl.send(GridNioSessionImpl.java:113) > at > org.apache.ignite.internal.processors.rest.protocols.tcp.GridTcpRestNioListener$1.apply(GridTcpRestNioListener.java:271) > ... 15 more > [16:57:10] New version is available at ignite.apache.org: 2.9.1 -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
