Re: Local node seems to be disconnected from topology (failure detection timeout is reached)

[luqmanahmad]

See [1] for free network segmentation plugin

[1]  https://github.com/luqmanahmad/ignite-plugins
  



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Local node seems to be disconnected from topology (failure detection timeout is reached)

[vkulichenko]

Yuci,

RESTART_JVM policy restarts the whole JVM (surprisingly :) ), so it works
only for standalone nodes started with ignite.sh script.

In your case you can use NOOP policy, listen to EVT_NODE_SEGMENTED event and
restart the node in a listener.

-Val



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Local-node-seems-to-be-disconnected-from-topology-failure-detection-timeout-is-reached-tp6797p10459.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Local node seems to be disconnected from topology (failure detection timeout is reached)

[yucigou]

Hi Val,

Thanks for the explanation for not allowing re-connection of server nodes. 

Regarding the RESTART_JVM policy, this policy does not work for our web
application. The reason is that in our web application Ignite is started by
the servlet context listener
org.apache.ignite.startup.servlet.ServletContextListenerStartup.

And according to the Ignite documentation, the RESTART_JVM policy will work
only if Ignite is started with CommandLineStartup via standard
ignite.{sh|bat} shell script.

https://ignite.apache.org/releases/mobile/org/apache/ignite/plugin/segmentation/SegmentationPolicy.html#RESTART_JVM

Wonder if it is possible to make the RESTART_JVM policy also work for Ignite
being started by the servlet context listener?

Thanks,
Yuci



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Local-node-seems-to-be-disconnected-from-topology-failure-detection-timeout-is-reached-tp6797p10452.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Local node seems to be disconnected from topology (failure detection timeout is reached)

[vkulichenko]

Hi Yuci,

We do not allow reconnection of server nodes because it's dangerous for data
consistency. Segmented node must rejoin topology as a new one. There is
RESTART_JVM policy which restarts node immediately.

-Val



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Local-node-seems-to-be-disconnected-from-topology-failure-detection-timeout-is-reached-tp6797p10401.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Local node seems to be disconnected from topology (failure detection timeout is reached)

[vkulichenko]

Hi Yuci,

It's one or another - you either use failureDetectionTimeout or fine grained
settings that you mentioned. In case you use the latter,
failureDetectionTimeout is ignored.

Having said that I would recommend to try failureDetectionTimeout first and
see how it works. Then try everything else only if needed.

-Val



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Local-node-seems-to-be-disconnected-from-topology-failure-detection-timeout-is-reached-tp6797p7417.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Local node seems to be disconnected from topology (failure detection timeout is reached)

[yucigou]

Hi vdpyatkov,

Thanks for the advice. I've just looked at the documentation
(http://apacheignite.gridgain.org/docs/cluster-config), there are actually
four timeouts to set for TcpDiscoverySpi:

1. setNetworkTimeout(long)
2. setSocketTimeout(long)
3. setAckTimeout(long)
4. setJoinTimeout(long)

Wonder if I should set up the other timeouts accordingly. In my case, set
socketTimeout 6000L, ackTimeout 6000L, networkTimeout 15000L with
joinTimeout staying at 0 as default.

What's your opinion?

Thanks,
Yuci





--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Local-node-seems-to-be-disconnected-from-topology-failure-detection-timeout-is-reached-tp6797p7409.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Local node seems to be disconnected from topology (failure detection timeout is reached)

[Vladislav Pyatkov]

Hello,

I recoment to use failureDetectionTimeout only for test, because this
timeout determines all timeout for all SPI (Communication, Dicovery).
Use specific timeout of DiscoverySPI for prevention of segmentation
(org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi#setSocketTimeout).

The configuration lead to slow change of topology in case failure of node.

On Fri, Aug 26, 2016 at 2:49 PM, yucigou <yuci@gmail.com> wrote:

> The root cause of the problem just found is that the VMs are frozen
> sometimes.
>
> Our service team takes backup of the VMs once per day. During the backup,
> the VMs that our application servers are running on would be frozen for a
> few seconds usually, but sometimes more than 40 seconds! When I say a VM is
> frozen here, I mean it is frozen literally, and nothing is going to run
> during this period of time.
>
> So when one VM is frozen, the other Ignite node will consider it is down,
> and as a result, the node on the frozen VM is disconnected with topology
> segmented, etc.
>
> So the solution seems to be set the failureDetectionTimeout property to 60
> seconds, to tolerate the VM being frozen in its worst cases.
>
> My question is, would there be some side effects to set
> failureDetectionTimeout 60 seconds? Any advice in such a situation? Thank
> you.
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Local-node-seems-to-be-disconnected-
> from-topology-failure-detection-timeout-is-reached-tp6797p7347.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>



-- 
Vladislav Pyatkov

Re: Local node seems to be disconnected from topology (failure detection timeout is reached)

[yucigou]

The root cause of the problem just found is that the VMs are frozen
sometimes.

Our service team takes backup of the VMs once per day. During the backup,
the VMs that our application servers are running on would be frozen for a
few seconds usually, but sometimes more than 40 seconds! When I say a VM is
frozen here, I mean it is frozen literally, and nothing is going to run
during this period of time.

So when one VM is frozen, the other Ignite node will consider it is down,
and as a result, the node on the frozen VM is disconnected with topology
segmented, etc.

So the solution seems to be set the failureDetectionTimeout property to 60
seconds, to tolerate the VM being frozen in its worst cases. 

My question is, would there be some side effects to set
failureDetectionTimeout 60 seconds? Any advice in such a situation? Thank
you.



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Local-node-seems-to-be-disconnected-from-topology-failure-detection-timeout-is-reached-tp6797p7347.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Local node seems to be disconnected from topology (failure detection timeout is reached)

[yucigou]

OK, I'll upload all the logs when they are ready. 

By the way, I've just upgraded to Ignite 1.7.0, but it does not help. The
problem persists.

Just in case if useful, please find below my Ignite configuration:



http://www.springframework.org/schema/beans;
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
xsi:schemaLocation="
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd;>



















 






  









10.3.0.64:48100..48110

10.3.0.65:48100..48110

172.22.70.110:48100..48110


















 






--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Local-node-seems-to-be-disconnected-from-topology-failure-detection-timeout-is-reached-tp6797p7175.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Local node seems to be disconnected from topology (failure detection timeout is reached)

[Vladislav Pyatkov]

Please, attach all logs as files.
In additional, can you provide system statistics (using dstat -t --top-mem
-m -s -g -d --fs --top-io-adv)?

All logs must be collected at the moment of  topology segmentation, hence
dstat must  works parallel with the application.

The actions will help us discover system state when grid is segmented.

On Thu, Aug 11, 2016 at 4:37 PM, yucigou <yuci@gmail.com> wrote:

> Another question is: What is the reason that Ignite shall stop the node in
> the case of big GC pause or temporary network issue? (By the way, I'm not
> saying big GC pause or temporary network issue is the case for this topic
> though.)
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Local-node-seems-to-be-disconnected-
> from-topology-failure-detection-timeout-is-reached-tp6797p6983.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>



-- 
Vladislav Pyatkov

Re: Local node seems to be disconnected from topology (failure detection timeout is reached)

[yucigou]

Another question is: What is the reason that Ignite shall stop the node in
the case of big GC pause or temporary network issue? (By the way, I'm not
saying big GC pause or temporary network issue is the case for this topic
though.)



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Local-node-seems-to-be-disconnected-from-topology-failure-detection-timeout-is-reached-tp6797p6983.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Local node seems to be disconnected from topology (failure detection timeout is reached)

[yucigou]

Ever since the web application has not be usable. Whenever I try to open a
web page, I get the following error message in my application log file:

2016-08-10 15:39:43,052 ERROR root/error 495 - Failed to update web session:
null
java.lang.IllegalStateException: Grid is in invalid state to perform this
operation. It either not started yet or has already being or have stopped
[gridName=null, state=STOPPED]
at
org.apache.ignite.internal.GridKernalGatewayImpl.illegalState(GridKernalGatewayImpl.java:190)
at
org.apache.ignite.internal.GridKernalGatewayImpl.readLock(GridKernalGatewayImpl.java:90)
at
org.apache.ignite.internal.IgniteKernal.guard(IgniteKernal.java:3107)
at
org.apache.ignite.internal.IgniteKernal.cache(IgniteKernal.java:2436)
at
org.apache.ignite.cache.websession.WebSessionFilter.initCache(WebSessionFilter.java:333)
at
org.apache.ignite.cache.websession.WebSessionFilter.handleCacheOperationException(WebSessionFilter.java:877)
at
org.apache.ignite.cache.websession.WebSessionFilter.handleLoadSessionException(WebSessionFilter.java:597)
at
org.apache.ignite.cache.websession.WebSessionFilter.doFilterV2(WebSessionFilter.java:523)
at
org.apache.ignite.cache.websession.WebSessionFilter.doFilterDispatch(WebSessionFilter.java:407)
at
org.apache.ignite.cache.websession.WebSessionFilter.doFilter(WebSessionFilter.java:383)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
at
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1070)
at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611)
at
org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:314)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Local-node-seems-to-be-disconnected-from-topology-failure-detection-timeout-is-reached-tp6797p6928.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Local node seems to be disconnected from topology (failure detection timeout is reached)

[yucigou]

Happened again. 

(1) Ignite logs: 

[08:14:01,990][INFO ][grid-timeout-worker-#33%null%][IgniteKernal] 
Metrics for local node (to disable set 'metricsLogFrequency' to 0) 
^-- Node [id=7faa417f, name=null, uptime=14:52:13:438] 
^-- H/N/C [hosts=2, nodes=2, CPUs=4] 
^-- CPU [cur=16.5%, avg=17.81%, GC=1.6%] 
^-- Heap [used=625MB, free=89.82%, comm=2047MB] 
^-- Non heap [used=93MB, free=-1%, comm=96MB] 
^-- Public thread pool [active=0, idle=16, qSize=0] 
^-- System thread pool [active=0, idle=16, qSize=0] 
^-- Outbound messages queue [size=0] 
[08:15:02,008][INFO ][grid-timeout-worker-#33%null%][IgniteKernal] 
Metrics for local node (to disable set 'metricsLogFrequency' to 0) 
^-- Node [id=7faa417f, name=null, uptime=14:53:13:457] 
^-- H/N/C [hosts=2, nodes=2, CPUs=4] 
^-- CPU [cur=18.67%, avg=17.81%, GC=1.63%] 
^-- Heap [used=373MB, free=93.92%, comm=2047MB] 
^-- Non heap [used=93MB, free=-1%, comm=96MB] 
^-- Public thread pool [active=0, idle=16, qSize=0] 
^-- System thread pool [active=0, idle=16, qSize=0] 
^-- Outbound messages queue [size=0] 
[08:16:00,861][INFO ][tcp-disco-msg-worker-#2%null%][TcpDiscoverySpi] Local
node seems to be disconnected from topology (failure detection timeout is
reached) [failureDetectionTimeout=1, connCheckFreq=] 
[08:16:00,900][WARN ][grid-nio-worker-0-#36%null%][TcpCommunicationSpi]
Failed to process selector key (will close): GridSelectorNioSessionImpl
[selectorIdx=0, queueSize=0, writeBuf=java.nio.DirectByteBuffer[pos=0
lim=32768 cap=32768], readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768
cap=32768], recovery=GridNioRecoveryDescriptor [acked=17892, resendCnt=0,
rcvCnt=18348, sentCnt=17901, reserved=true, lastAck=18348, nodeLeft=false,
node=TcpDiscoveryNode [id=358fcad3-1e47-45df-8cd4-875bdcd7011a,
addrs=[10.3.0.65, 127.0.0.1],
sockAddrs=[ves-hx-41.ebi.ac.uk/10.3.0.65:47500, /10.3.0.65:47500,
/127.0.0.1:47500], discPort=47500, order=4, intOrder=3,
lastExchangeTime=1470759707287, loc=false, ver=1.6.0#20160518-sha1:0b22c45b,
isClient=false], connected=true, connectCnt=0, queueLimit=5120],
super=GridNioSessionImpl [locAddr=/10.3.0.64:47100,
rmtAddr=/10.3.0.65:36219, createTime=1470759707458, closeTime=0,
bytesSent=9992093, bytesRcvd=5477037971, sndSchedTime=1470813320058,
lastSndTime=1470813320341, lastRcvTime=1470813320068, readsPaused=false,
filterChain=FilterChain[filters=[GridNioCodecFilter
[parser=o.a.i.i.util.nio.GridDirectParser@276d0d91, directMode=true],
GridConnectionBytesVerifyFilter], accepted=true]] 
[08:16:00,904][WARN ][grid-nio-worker-0-#36%null%][TcpCommunicationSpi]
Closing NIO session because of unhandled exception [cls=class
o.a.i.i.util.nio.GridNioException, msg=Connection reset by peer] 
[08:16:00,919][WARN ][tcp-disco-msg-worker-#2%null%][TcpDiscoverySpi] Node
is out of topology (probably, due to short-time network problems). 
[08:16:00,926][WARN ][disco-event-worker-#45%null%][GridDiscoveryManager]
Local node SEGMENTED: TcpDiscoveryNode
[id=7faa417f-9760-434d-bc66-6551611ebf42, addrs=[10.3.0.64, 127.0.0.1],
sockAddrs=[ves-hx-40.ebi.ac.uk/10.3.0.64:47500, /10.3.0.64:47500,
/127.0.0.1:47500], discPort=47500, order=6, intOrder=4,
lastExchangeTime=1470813360913, loc=true, ver=1.6.0#20160518-sha1:0b22c45b,
isClient=false] 
[08:16:01,104][WARN ][disco-event-worker-#45%null%][GridDiscoveryManager]
Stopping local node according to configured segmentation policy. 
[08:16:01,109][WARN ][disco-event-worker-#45%null%][GridDiscoveryManager]
Node FAILED: TcpDiscoveryNode [id=358fcad3-1e47-45df-8cd4-875bdcd7011a,
addrs=[10.3.0.65, 127.0.0.1],
sockAddrs=[ves-hx-41.ebi.ac.uk/10.3.0.65:47500, /10.3.0.65:47500,
/127.0.0.1:47500], discPort=47500, order=4, intOrder=3,
lastExchangeTime=1470759707287, loc=false, ver=1.6.0#20160518-sha1:0b22c45b,
isClient=false] 
[08:16:01,113][INFO ][disco-event-worker-#45%null%][GridDiscoveryManager]
Topology snapshot [ver=7, servers=1, clients=0, CPUs=2, heap=6.0GB] 
[08:16:01,150][INFO ][Thread-31][GridTcpRestProtocol] Command protocol
successfully stopped: TCP binary 
[08:16:02,641][INFO ][Thread-31][GridJettyRestProtocol] Command protocol
successfully stopped: Jetty REST 
[08:16:02,685][INFO ][Thread-31][GridCacheProcessor] Stopped cache:
session-cache 
[08:16:02,687][INFO ][Thread-31][GridCacheProcessor] Stopped cache:
ignite-marshaller-sys-cache 
[08:16:02,687][INFO ][Thread-31][GridCacheProcessor] Stopped cache:
ignite-sys-cache 
[08:16:02,688][INFO ][Thread-31][GridCacheProcessor] Stopped cache:
ignite-atomics-sys-cache 

(2) GC logs 

2016-08-10T08:14:02.391+0100: 53542.801: [GC (Allocation Failure)
2016-08-10T08:14:02.391+0100: 53542.801: [ParNew: 130944K->0K(131008K),
0.0423711 secs] 741795K->633860K(2097088K), 0.0425532 secs] [Times:
user=0.08 sys=0.00, real=0.04 secs] 
2016-08-10T08:14:02.672+0100: 53543.081: [GC (Allocation Failure)
2016-08-10T08:14:02.672+0100: 53543.081: [ParNew: 130944K->0K(131008K),
0.0390475 secs] 764804K->6446

Re: Local node seems to be disconnected from topology (failure detection timeout is reached)

[yucigou]

Happened again. 

(1) Ignite logs: 

[08:14:01,990][INFO ][grid-timeout-worker-#33%null%][IgniteKernal] 
Metrics for local node (to disable set 'metricsLogFrequency' to 0) 
^-- Node [id=7faa417f, name=null, uptime=14:52:13:438] 
^-- H/N/C [hosts=2, nodes=2, CPUs=4] 
^-- CPU [cur=16.5%, avg=17.81%, GC=1.6%] 
^-- Heap [used=625MB, free=89.82%, comm=2047MB] 
^-- Non heap [used=93MB, free=-1%, comm=96MB] 
^-- Public thread pool [active=0, idle=16, qSize=0] 
^-- System thread pool [active=0, idle=16, qSize=0] 
^-- Outbound messages queue [size=0] 
[08:15:02,008][INFO ][grid-timeout-worker-#33%null%][IgniteKernal] 
Metrics for local node (to disable set 'metricsLogFrequency' to 0) 
^-- Node [id=7faa417f, name=null, uptime=14:53:13:457] 
^-- H/N/C [hosts=2, nodes=2, CPUs=4] 
^-- CPU [cur=18.67%, avg=17.81%, GC=1.63%] 
^-- Heap [used=373MB, free=93.92%, comm=2047MB] 
^-- Non heap [used=93MB, free=-1%, comm=96MB] 
^-- Public thread pool [active=0, idle=16, qSize=0] 
^-- System thread pool [active=0, idle=16, qSize=0] 
^-- Outbound messages queue [size=0] 
[08:16:00,861][INFO ][tcp-disco-msg-worker-#2%null%][TcpDiscoverySpi] Local
node seems to be disconnected from topology (failure detection timeout is
reached) [failureDetectionTimeout=1, connCheckFreq=] 
[08:16:00,900][WARN ][grid-nio-worker-0-#36%null%][TcpCommunicationSpi]
Failed to process selector key (will close): GridSelectorNioSessionImpl
[selectorIdx=0, queueSize=0, writeBuf=java.nio.DirectByteBuffer[pos=0
lim=32768 cap=32768], readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768
cap=32768], recovery=GridNioRecoveryDescriptor [acked=17892, resendCnt=0,
rcvCnt=18348, sentCnt=17901, reserved=true, lastAck=18348, nodeLeft=false,
node=TcpDiscoveryNode [id=358fcad3-1e47-45df-8cd4-875bdcd7011a,
addrs=[10.3.0.65, 127.0.0.1],
sockAddrs=[ves-hx-41.ebi.ac.uk/10.3.0.65:47500, /10.3.0.65:47500,
/127.0.0.1:47500], discPort=47500, order=4, intOrder=3,
lastExchangeTime=1470759707287, loc=false, ver=1.6.0#20160518-sha1:0b22c45b,
isClient=false], connected=true, connectCnt=0, queueLimit=5120],
super=GridNioSessionImpl [locAddr=/10.3.0.64:47100,
rmtAddr=/10.3.0.65:36219, createTime=1470759707458, closeTime=0,
bytesSent=9992093, bytesRcvd=5477037971, sndSchedTime=1470813320058,
lastSndTime=1470813320341, lastRcvTime=1470813320068, readsPaused=false,
filterChain=FilterChain[filters=[GridNioCodecFilter
[parser=o.a.i.i.util.nio.GridDirectParser@276d0d91, directMode=true],
GridConnectionBytesVerifyFilter], accepted=true]] 
[08:16:00,904][WARN ][grid-nio-worker-0-#36%null%][TcpCommunicationSpi]
Closing NIO session because of unhandled exception [cls=class
o.a.i.i.util.nio.GridNioException, msg=Connection reset by peer] 
[08:16:00,919][WARN ][tcp-disco-msg-worker-#2%null%][TcpDiscoverySpi] Node
is out of topology (probably, due to short-time network problems). 
[08:16:00,926][WARN ][disco-event-worker-#45%null%][GridDiscoveryManager]
Local node SEGMENTED: TcpDiscoveryNode
[id=7faa417f-9760-434d-bc66-6551611ebf42, addrs=[10.3.0.64, 127.0.0.1],
sockAddrs=[ves-hx-40.ebi.ac.uk/10.3.0.64:47500, /10.3.0.64:47500,
/127.0.0.1:47500], discPort=47500, order=6, intOrder=4,
lastExchangeTime=1470813360913, loc=true, ver=1.6.0#20160518-sha1:0b22c45b,
isClient=false] 
[08:16:01,104][WARN ][disco-event-worker-#45%null%][GridDiscoveryManager]
Stopping local node according to configured segmentation policy. 
[08:16:01,109][WARN ][disco-event-worker-#45%null%][GridDiscoveryManager]
Node FAILED: TcpDiscoveryNode [id=358fcad3-1e47-45df-8cd4-875bdcd7011a,
addrs=[10.3.0.65, 127.0.0.1],
sockAddrs=[ves-hx-41.ebi.ac.uk/10.3.0.65:47500, /10.3.0.65:47500,
/127.0.0.1:47500], discPort=47500, order=4, intOrder=3,
lastExchangeTime=1470759707287, loc=false, ver=1.6.0#20160518-sha1:0b22c45b,
isClient=false] 
[08:16:01,113][INFO ][disco-event-worker-#45%null%][GridDiscoveryManager]
Topology snapshot [ver=7, servers=1, clients=0, CPUs=2, heap=6.0GB] 
[08:16:01,150][INFO ][Thread-31][GridTcpRestProtocol] Command protocol
successfully stopped: TCP binary 
[08:16:02,641][INFO ][Thread-31][GridJettyRestProtocol] Command protocol
successfully stopped: Jetty REST 
[08:16:02,685][INFO ][Thread-31][GridCacheProcessor] Stopped cache:
session-cache 
[08:16:02,687][INFO ][Thread-31][GridCacheProcessor] Stopped cache:
ignite-marshaller-sys-cache 
[08:16:02,687][INFO ][Thread-31][GridCacheProcessor] Stopped cache:
ignite-sys-cache 
[08:16:02,688][INFO ][Thread-31][GridCacheProcessor] Stopped cache:
ignite-atomics-sys-cache 

(2) GC logs 

2016-08-10T08:14:02.391+0100: 53542.801: [GC (Allocation Failure)
2016-08-10T08:14:02.391+0100: 53542.801: [ParNew: 130944K->0K(131008K),
0.0423711 secs] 741795K->633860K(2097088K), 0.0425532 secs] [Times:
user=0.08 sys=0.00, real=0.04 secs] 
2016-08-10T08:14:02.672+0100: 53543.081: [GC (Allocation Failure)
2016-08-10T08:14:02.672+0100: 53543.081: [ParNew: 130944K->0K(131008K),
0.0390475 secs] 764804K->6446

Re: Local node seems to be disconnected from topology (failure detection timeout is reached)

[yucigou]

Happened again.

(1) Ignite logs:

[08:14:01,990][INFO ][grid-timeout-worker-#33%null%][IgniteKernal] 
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=7faa417f, name=null, uptime=14:52:13:438]
^-- H/N/C [hosts=2, nodes=2, CPUs=4]
^-- CPU [cur=16.5%, avg=17.81%, GC=1.6%]
^-- Heap [used=625MB, free=89.82%, comm=2047MB]
^-- Non heap [used=93MB, free=-1%, comm=96MB]
^-- Public thread pool [active=0, idle=16, qSize=0]
^-- System thread pool [active=0, idle=16, qSize=0]
^-- Outbound messages queue [size=0]
[08:15:02,008][INFO ][grid-timeout-worker-#33%null%][IgniteKernal] 
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=7faa417f, name=null, uptime=14:53:13:457]
^-- H/N/C [hosts=2, nodes=2, CPUs=4]
^-- CPU [cur=18.67%, avg=17.81%, GC=1.63%]
^-- Heap [used=373MB, free=93.92%, comm=2047MB]
^-- Non heap [used=93MB, free=-1%, comm=96MB]
^-- Public thread pool [active=0, idle=16, qSize=0]
^-- System thread pool [active=0, idle=16, qSize=0]
^-- Outbound messages queue [size=0]
[08:16:00,861][INFO ][tcp-disco-msg-worker-#2%null%][TcpDiscoverySpi] Local
node seems to be disconnected from topology (failure detection timeout is
reached) [failureDetectionTimeout=1, connCheckFreq=]
[08:16:00,900][WARN ][grid-nio-worker-0-#36%null%][TcpCommunicationSpi]
Failed to process selector key (will close): GridSelectorNioSessionImpl
[selectorIdx=0, queueSize=0, writeBuf=java.nio.DirectByteBuffer[pos=0
lim=32768 cap=32768], readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768
cap=32768], recovery=GridNioRecoveryDescriptor [acked=17892, resendCnt=0,
rcvCnt=18348, sentCnt=17901, reserved=true, lastAck=18348, nodeLeft=false,
node=TcpDiscoveryNode [id=358fcad3-1e47-45df-8cd4-875bdcd7011a,
addrs=[10.3.0.65, 127.0.0.1],
sockAddrs=[ves-hx-41.ebi.ac.uk/10.3.0.65:47500, /10.3.0.65:47500,
/127.0.0.1:47500], discPort=47500, order=4, intOrder=3,
lastExchangeTime=1470759707287, loc=false, ver=1.6.0#20160518-sha1:0b22c45b,
isClient=false], connected=true, connectCnt=0, queueLimit=5120],
super=GridNioSessionImpl [locAddr=/10.3.0.64:47100,
rmtAddr=/10.3.0.65:36219, createTime=1470759707458, closeTime=0,
bytesSent=9992093, bytesRcvd=5477037971, sndSchedTime=1470813320058,
lastSndTime=1470813320341, lastRcvTime=1470813320068, readsPaused=false,
filterChain=FilterChain[filters=[GridNioCodecFilter
[parser=o.a.i.i.util.nio.GridDirectParser@276d0d91, directMode=true],
GridConnectionBytesVerifyFilter], accepted=true]]
[08:16:00,904][WARN ][grid-nio-worker-0-#36%null%][TcpCommunicationSpi]
Closing NIO session because of unhandled exception [cls=class
o.a.i.i.util.nio.GridNioException, msg=Connection reset by peer]
[08:16:00,919][WARN ][tcp-disco-msg-worker-#2%null%][TcpDiscoverySpi] Node
is out of topology (probably, due to short-time network problems).
[08:16:00,926][WARN ][disco-event-worker-#45%null%][GridDiscoveryManager]
Local node SEGMENTED: TcpDiscoveryNode
[id=7faa417f-9760-434d-bc66-6551611ebf42, addrs=[10.3.0.64, 127.0.0.1],
sockAddrs=[ves-hx-40.ebi.ac.uk/10.3.0.64:47500, /10.3.0.64:47500,
/127.0.0.1:47500], discPort=47500, order=6, intOrder=4,
lastExchangeTime=1470813360913, loc=true, ver=1.6.0#20160518-sha1:0b22c45b,
isClient=false]
[08:16:01,104][WARN ][disco-event-worker-#45%null%][GridDiscoveryManager]
Stopping local node according to configured segmentation policy.
[08:16:01,109][WARN ][disco-event-worker-#45%null%][GridDiscoveryManager]
Node FAILED: TcpDiscoveryNode [id=358fcad3-1e47-45df-8cd4-875bdcd7011a,
addrs=[10.3.0.65, 127.0.0.1],
sockAddrs=[ves-hx-41.ebi.ac.uk/10.3.0.65:47500, /10.3.0.65:47500,
/127.0.0.1:47500], discPort=47500, order=4, intOrder=3,
lastExchangeTime=1470759707287, loc=false, ver=1.6.0#20160518-sha1:0b22c45b,
isClient=false]
[08:16:01,113][INFO ][disco-event-worker-#45%null%][GridDiscoveryManager]
Topology snapshot [ver=7, servers=1, clients=0, CPUs=2, heap=6.0GB]
[08:16:01,150][INFO ][Thread-31][GridTcpRestProtocol] Command protocol
successfully stopped: TCP binary
[08:16:02,641][INFO ][Thread-31][GridJettyRestProtocol] Command protocol
successfully stopped: Jetty REST
[08:16:02,685][INFO ][Thread-31][GridCacheProcessor] Stopped cache:
session-cache
[08:16:02,687][INFO ][Thread-31][GridCacheProcessor] Stopped cache:
ignite-marshaller-sys-cache
[08:16:02,687][INFO ][Thread-31][GridCacheProcessor] Stopped cache:
ignite-sys-cache
[08:16:02,688][INFO ][Thread-31][GridCacheProcessor] Stopped cache:
ignite-atomics-sys-cache

(2) GC logs

2016-08-10T08:08:11.335+0100: 53191.745: [GC (Allocation Failure)
2016-08-10T08:08:11.335+0100: 53191.745: [ParNew: 130944K->0K(131008K),
0.0430220 secs] 507259K->395398K(2097088K), 0.0432639 secs] [Times:
user=0.08 sys=0.00, real=0.04 secs] 
2016-08-10T08:08:11.760+0100: 53192.169: [GC (Allocation Failure)
2016-08-10T08:08:11.760+0100: 53192.169: [ParNew: 130944K->0K(131008K),
0.0496240 secs] 526342K->408993K(2097088K), 0.0498060 secs] [Times:
u

Re: Local node seems to be disconnected from topology (failure detection timeout is reached)

[Vladislav Pyatkov]

Hi

Unfortunately, this charts do not reflect full picture of GC works. I need
to understand how long continueing pause of garbage collect accurate to the
milliseconds.
Could you please look at the article[1] and provide log file of GC (which
will be by path loggc:/path/to/gc/logs/log.txt)?

[1]:
https://apacheignite.readme.io/docs/jvm-and-system-tuning#section-detailed-garbage-collection-stats

On Fri, Aug 5, 2016 at 5:59 PM, yucigou <yuci@gmail.com> wrote:

> Here is what looked like for the other node:
>
> <http://apache-ignite-users.70518.x6.nabble.com/file/n6804/usedMemory.png>
> <http://apache-ignite-users.70518.x6.nabble.com/file/n6804/cpu.png>
> <http://apache-ignite-users.70518.x6.nabble.com/file/n6804/gc.png>
>
> By the way, there were no HTTP traffic to the clustered servers.
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Local-node-seems-to-be-disconnected-
> from-topology-failure-detection-timeout-is-reached-tp6797p6804.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>



-- 
Vladislav Pyatkov

Re: Local node seems to be disconnected from topology (failure detection timeout is reached)

[yucigou]

Here is what looked like for the other node:

<http://apache-ignite-users.70518.x6.nabble.com/file/n6804/usedMemory.png> 
<http://apache-ignite-users.70518.x6.nabble.com/file/n6804/cpu.png> 
<http://apache-ignite-users.70518.x6.nabble.com/file/n6804/gc.png> 

By the way, there were no HTTP traffic to the clustered servers.



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Local-node-seems-to-be-disconnected-from-topology-failure-detection-timeout-is-reached-tp6797p6804.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Local node seems to be disconnected from topology (failure detection timeout is reached)

[Vladislav Pyatkov]

If in topology contains only two nodes, need to view logs from both.
A pause can occur as a single node and the second. Please collect log from
both nodes and attach files to the topic.

[1]
https://apacheignite.readme.io/docs/jvm-and-system-tuning#section-detailed-garbage-collection-stats

On Fri, Aug 5, 2016 at 4:58 PM, yucigou <yuci@gmail.com> wrote:

> Thanks for your reply. Indeed the CPU and memory was a bit intense during
> the
> period of time when I ran the Apache Ignite Web Console Agent, so as to use
> the Apache Ignite Web Console to monitoring my Ignite nodes. But it seemed
> under control (notes: figures below are provided by JavaMelody):
> <http://apache-ignite-users.70518.x6.nabble.com/file/n6802/usedMemory.png>
> <http://apache-ignite-users.70518.x6.nabble.com/file/n6802/cpu.png>
> Also the garbage collector time was minimal:
> <http://apache-ignite-users.70518.x6.nabble.com/file/n6802/gc.png>
> Could there be something wrong caused by the Apache Ignite Web Console
> Agent? Or any other possibilities?
>
> Cheers,
> Yuci
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Local-node-seems-to-be-disconnected-
> from-topology-failure-detection-timeout-is-reached-tp6797p6802.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>



-- 
Vladislav Pyatkov

Re: Local node seems to be disconnected from topology (failure detection timeout is reached)

[yucigou]

Thanks for your reply. Indeed the CPU and memory was a bit intense during the
period of time when I ran the Apache Ignite Web Console Agent, so as to use
the Apache Ignite Web Console to monitoring my Ignite nodes. But it seemed
under control (notes: figures below are provided by JavaMelody): 
<http://apache-ignite-users.70518.x6.nabble.com/file/n6802/usedMemory.png> 
<http://apache-ignite-users.70518.x6.nabble.com/file/n6802/cpu.png> 
Also the garbage collector time was minimal:
<http://apache-ignite-users.70518.x6.nabble.com/file/n6802/gc.png> 
Could there be something wrong caused by the Apache Ignite Web Console
Agent? Or any other possibilities? 

Cheers, 
Yuci



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Local-node-seems-to-be-disconnected-from-topology-failure-detection-timeout-is-reached-tp6797p6802.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Local node seems to be disconnected from topology (failure detection timeout is reached)

[yucigou]

Thanks for your reply. Indeed the CPU and memory was a bit intense during the
period of time when I ran the Apache Ignite Web Console Agent, so as to use
the Apache Ignite Web Console to monitoring my Ignite nodes. But it seemed
under control (notes: figures below are provided by JavaMelody):

<http://apache-ignite-users.70518.x6.nabble.com/file/n6801/cpu.png> 
<http://apache-ignite-users.70518.x6.nabble.com/file/n6801/cpu.png> 

Also the garbage collector time was minimal.

<http://apache-ignite-users.70518.x6.nabble.com/file/n6801/gc.png> 

Could there be something wrong caused by the Apache Ignite Web Console
Agent? Or any other possibilities?

Cheers,
Yuci



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Local-node-seems-to-be-disconnected-from-topology-failure-detection-timeout-is-reached-tp6797p6801.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Local node seems to be disconnected from topology (failure detection timeout is reached)

[Vladislav Pyatkov]

It look like as segmentation by the reasong pause of garbage collector.
You need attentively look the article [1] and collect GC logs, if GC works
with long pauses (around 10 seconds) try to tune JVM.

If the issue persists, please provide GC log and Ignite log for analyze.

[1] https://apacheignite.readme.io/docs/jvm-and-system-tuning

On Fri, Aug 5, 2016 at 2:57 PM, yucigou <yuci@gmail.com> wrote:

> Hello,
>
> One of my Ignite nodes was stopped and the logs were appended as below. It
> seems that grid-timeout-worker checks the health of the cluster every
> minute. But then in my case, before the due time 23:34:19, at 23:34:03 it
> reported "Local node seems to be disconnected from topology (failure
> detection timeout is reached)", and the Ignite node got stopped. In turn,
> the web session clustering, and so on, stopped working.
>
> Just wonder what could cause this to happen? There should be no network
> issue etc with the host machine then. It is a bit scary to us, as it can
> happen to our production servers in the near future.
>
> Thank you for your help.
>
> Yuci
>
> ===Ignite logs==
> [23:31:19,896][INFO ][grid-timeout-worker-#33%null%][IgniteKernal]
> Metrics for local node (to disable set 'metricsLogFrequency' to 0)
> ^-- Node [id=9a069f70, name=null, uptime=10:37:03:793]
> ^-- H/N/C [hosts=2, nodes=2, CPUs=4]
> ^-- CPU [cur=43.17%, avg=12.83%, GC=1.1%]
> ^-- Heap [used=2115MB, free=61.26%, comm=3955MB]
> ^-- Non heap [used=138MB, free=-1%, comm=143MB]
> ^-- Public thread pool [active=0, idle=16, qSize=0]
> ^-- System thread pool [active=0, idle=16, qSize=0]
> ^-- Outbound messages queue [size=0]
> [23:32:19,904][INFO ][grid-timeout-worker-#33%null%][IgniteKernal]
> Metrics for local node (to disable set 'metricsLogFrequency' to 0)
> ^-- Node [id=9a069f70, name=null, uptime=10:38:03:801]
> ^-- H/N/C [hosts=2, nodes=2, CPUs=4]
> ^-- CPU [cur=0.83%, avg=12.87%, GC=0%]
> ^-- Heap [used=2638MB, free=51.69%, comm=3957MB]
> ^-- Non heap [used=138MB, free=-1%, comm=143MB]
> ^-- Public thread pool [active=0, idle=16, qSize=0]
> ^-- System thread pool [active=0, idle=16, qSize=0]
> ^-- Outbound messages queue [size=0]
> [23:33:19,913][INFO ][grid-timeout-worker-#33%null%][IgniteKernal]
> Metrics for local node (to disable set 'metricsLogFrequency' to 0)
> ^-- Node [id=9a069f70, name=null, uptime=10:39:03:808]
> ^-- H/N/C [hosts=2, nodes=2, CPUs=4]
> ^-- CPU [cur=0.5%, avg=12.86%, GC=0%]
> ^-- Heap [used=796MB, free=85.41%, comm=3921MB]
> ^-- Non heap [used=138MB, free=-1%, comm=143MB]
> ^-- Public thread pool [active=0, idle=16, qSize=0]
> ^-- System thread pool [active=0, idle=16, qSize=0]
> ^-- Outbound messages queue [size=0]
> [23:34:03,752][INFO ][tcp-disco-msg-worker-#2%null%][TcpDiscoverySpi]
> Local
> node seems to be disconnected from topology (failure detection timeout is
> reached) [failureDetectionTimeout=1, connCheckFreq=]
> [23:34:03,783][WARN ][tcp-disco-msg-worker-#2%null%][TcpDiscoverySpi] Node
> is out of topology (probably, due to short-time network problems).
> [23:34:03,786][WARN ][disco-event-worker-#44%null%][GridDiscoveryManager]
> Local node SEGMENTED: TcpDiscoveryNode
> [id=9a069f70-d49d-472e-9771-7ac2353e751f, addrs=[10.3.0.64, 127.0.0.1],
> sockAddrs=[ves-hx-40.ebi.ac.uk/10.3.0.64:47500, /10.3.0.64:47500,
> /127.0.0.1:47500], discPort=47500, order=56, intOrder=29,
> lastExchangeTime=1470350043783, loc=true, ver=1.6.0#20160518-sha1:
> 0b22c45b,
> isClient=false]
> [23:34:03,819][WARN ][disco-event-worker-#44%null%][GridDiscoveryManager]
> Stopping local node according to configured segmentation policy.
> [23:34:03,825][WARN ][disco-event-worker-#44%null%][GridDiscoveryManager]
> Node FAILED: TcpDiscoveryNode [id=cef7fc5e-b854-4072-8e16-396a87d5d556,
> addrs=[10.3.0.65, 127.0.0.1],
> sockAddrs=[ves-hx-41.ebi.ac.uk/10.3.0.65:47500, /10.3.0.65:47500,
> /127.0.0.1:47500], discPort=47500, order=58, intOrder=30,
> lastExchangeTime=1470311808664, loc=false, ver=1.6.0#20160518-sha1:
> 0b22c45b,
> isClient=false]
> [23:34:03,827][INFO ][disco-event-worker-#44%null%][GridDiscoveryManager]
> Topology snapshot [ver=59, servers=1, clients=0, CPUs=2, heap=5.3GB]
> [23:34:03,874][INFO ][Thread-32][GridTcpRestProtocol] Command protocol
> successfully stopped: TCP binary
> [23:34:03,902][INFO ][Thread-32][GridJettyRestProtocol] Command protocol
> successfully stopped: Jetty REST
> [23:34:04,571][INFO ][Thread-32][GridCacheProcessor] Stopped cache:
> session-cache
> [23:34:04,572][INFO ][Thread-32][GridCacheProcessor] Stopped cache:
> ignite-marshaller-sys-cache
> [23:34:04,572][INFO ][Th

Local node seems to be disconnected from topology (failure detection timeout is reached)

[yucigou]

Hello,

One of my Ignite nodes was stopped and the logs were appended as below. It
seems that grid-timeout-worker checks the health of the cluster every
minute. But then in my case, before the due time 23:34:19, at 23:34:03 it
reported "Local node seems to be disconnected from topology (failure
detection timeout is reached)", and the Ignite node got stopped. In turn,
the web session clustering, and so on, stopped working.

Just wonder what could cause this to happen? There should be no network
issue etc with the host machine then. It is a bit scary to us, as it can
happen to our production servers in the near future.

Thank you for your help.

Yuci

===Ignite logs==
[23:31:19,896][INFO ][grid-timeout-worker-#33%null%][IgniteKernal] 
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=9a069f70, name=null, uptime=10:37:03:793]
^-- H/N/C [hosts=2, nodes=2, CPUs=4]
^-- CPU [cur=43.17%, avg=12.83%, GC=1.1%]
^-- Heap [used=2115MB, free=61.26%, comm=3955MB]
^-- Non heap [used=138MB, free=-1%, comm=143MB]
^-- Public thread pool [active=0, idle=16, qSize=0]
^-- System thread pool [active=0, idle=16, qSize=0]
^-- Outbound messages queue [size=0]
[23:32:19,904][INFO ][grid-timeout-worker-#33%null%][IgniteKernal] 
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=9a069f70, name=null, uptime=10:38:03:801]
^-- H/N/C [hosts=2, nodes=2, CPUs=4]
^-- CPU [cur=0.83%, avg=12.87%, GC=0%]
^-- Heap [used=2638MB, free=51.69%, comm=3957MB]
^-- Non heap [used=138MB, free=-1%, comm=143MB]
^-- Public thread pool [active=0, idle=16, qSize=0]
^-- System thread pool [active=0, idle=16, qSize=0]
^-- Outbound messages queue [size=0]
[23:33:19,913][INFO ][grid-timeout-worker-#33%null%][IgniteKernal] 
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=9a069f70, name=null, uptime=10:39:03:808]
^-- H/N/C [hosts=2, nodes=2, CPUs=4]
^-- CPU [cur=0.5%, avg=12.86%, GC=0%]
^-- Heap [used=796MB, free=85.41%, comm=3921MB]
^-- Non heap [used=138MB, free=-1%, comm=143MB]
^-- Public thread pool [active=0, idle=16, qSize=0]
^-- System thread pool [active=0, idle=16, qSize=0]
^-- Outbound messages queue [size=0]
[23:34:03,752][INFO ][tcp-disco-msg-worker-#2%null%][TcpDiscoverySpi] Local
node seems to be disconnected from topology (failure detection timeout is
reached) [failureDetectionTimeout=1, connCheckFreq=]
[23:34:03,783][WARN ][tcp-disco-msg-worker-#2%null%][TcpDiscoverySpi] Node
is out of topology (probably, due to short-time network problems).
[23:34:03,786][WARN ][disco-event-worker-#44%null%][GridDiscoveryManager]
Local node SEGMENTED: TcpDiscoveryNode
[id=9a069f70-d49d-472e-9771-7ac2353e751f, addrs=[10.3.0.64, 127.0.0.1],
sockAddrs=[ves-hx-40.ebi.ac.uk/10.3.0.64:47500, /10.3.0.64:47500,
/127.0.0.1:47500], discPort=47500, order=56, intOrder=29,
lastExchangeTime=1470350043783, loc=true, ver=1.6.0#20160518-sha1:0b22c45b,
isClient=false]
[23:34:03,819][WARN ][disco-event-worker-#44%null%][GridDiscoveryManager]
Stopping local node according to configured segmentation policy.
[23:34:03,825][WARN ][disco-event-worker-#44%null%][GridDiscoveryManager]
Node FAILED: TcpDiscoveryNode [id=cef7fc5e-b854-4072-8e16-396a87d5d556,
addrs=[10.3.0.65, 127.0.0.1],
sockAddrs=[ves-hx-41.ebi.ac.uk/10.3.0.65:47500, /10.3.0.65:47500,
/127.0.0.1:47500], discPort=47500, order=58, intOrder=30,
lastExchangeTime=1470311808664, loc=false, ver=1.6.0#20160518-sha1:0b22c45b,
isClient=false]
[23:34:03,827][INFO ][disco-event-worker-#44%null%][GridDiscoveryManager]
Topology snapshot [ver=59, servers=1, clients=0, CPUs=2, heap=5.3GB]
[23:34:03,874][INFO ][Thread-32][GridTcpRestProtocol] Command protocol
successfully stopped: TCP binary
[23:34:03,902][INFO ][Thread-32][GridJettyRestProtocol] Command protocol
successfully stopped: Jetty REST
[23:34:04,571][INFO ][Thread-32][GridCacheProcessor] Stopped cache:
session-cache
[23:34:04,572][INFO ][Thread-32][GridCacheProcessor] Stopped cache:
ignite-marshaller-sys-cache
[23:34:04,572][INFO ][Thread-32][GridCacheProcessor] Stopped cache:
ignite-sys-cache
[23:34:04,573][INFO ][Thread-32][GridCacheProcessor] Stopped cache:
ignite-atomics-sys-cache
[23:34:04,583][INFO ][Thread-32][GridCacheProcessor] Stopped cache:
wicket-data-store
[23:34:04,623][INFO ][Thread-32][IgniteKernal] 

>>> +-+
>>> Ignite ver. 1.6.0#20160518-sha1:0b22c45bb9b97692208fd0705ddf8045ff34a031
>>> stopped OK
>>> +-+
>>> Grid uptime: 10:39:48:518





--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Local-node-seems-to-be-disconnected-from-topology-failure-detection-timeout-is-reached-tp6797.html