Happened again.
(1) Ignite logs:
[08:14:01,990][INFO ][grid-timeout-worker-#33%null%][IgniteKernal]
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=7faa417f, name=null, uptime=14:52:13:438]
^-- H/N/C [hosts=2, nodes=2, CPUs=4]
^-- CPU [cur=16.5%, avg=17.81%, GC=1.6%]
^-- Heap [used=625MB, free=89.82%, comm=2047MB]
^-- Non heap [used=93MB, free=-1%, comm=96MB]
^-- Public thread pool [active=0, idle=16, qSize=0]
^-- System thread pool [active=0, idle=16, qSize=0]
^-- Outbound messages queue [size=0]
[08:15:02,008][INFO ][grid-timeout-worker-#33%null%][IgniteKernal]
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=7faa417f, name=null, uptime=14:53:13:457]
^-- H/N/C [hosts=2, nodes=2, CPUs=4]
^-- CPU [cur=18.67%, avg=17.81%, GC=1.63%]
^-- Heap [used=373MB, free=93.92%, comm=2047MB]
^-- Non heap [used=93MB, free=-1%, comm=96MB]
^-- Public thread pool [active=0, idle=16, qSize=0]
^-- System thread pool [active=0, idle=16, qSize=0]
^-- Outbound messages queue [size=0]
[08:16:00,861][INFO ][tcp-disco-msg-worker-#2%null%][TcpDiscoverySpi] Local
node seems to be disconnected from topology (failure detection timeout is
reached) [failureDetectionTimeout=10000, connCheckFreq=3333]
[08:16:00,900][WARN ][grid-nio-worker-0-#36%null%][TcpCommunicationSpi]
Failed to process selector key (will close): GridSelectorNioSessionImpl
[selectorIdx=0, queueSize=0, writeBuf=java.nio.DirectByteBuffer[pos=0
lim=32768 cap=32768], readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768
cap=32768], recovery=GridNioRecoveryDescriptor [acked=17892, resendCnt=0,
rcvCnt=18348, sentCnt=17901, reserved=true, lastAck=18348, nodeLeft=false,
node=TcpDiscoveryNode [id=358fcad3-1e47-45df-8cd4-875bdcd7011a,
addrs=[10.3.0.65, 127.0.0.1],
sockAddrs=[ves-hx-41.ebi.ac.uk/10.3.0.65:47500, /10.3.0.65:47500,
/127.0.0.1:47500], discPort=47500, order=4, intOrder=3,
lastExchangeTime=1470759707287, loc=false, ver=1.6.0#20160518-sha1:0b22c45b,
isClient=false], connected=true, connectCnt=0, queueLimit=5120],
super=GridNioSessionImpl [locAddr=/10.3.0.64:47100,
rmtAddr=/10.3.0.65:36219, createTime=1470759707458, closeTime=0,
bytesSent=9992093, bytesRcvd=5477037971, sndSchedTime=1470813320058,
lastSndTime=1470813320341, lastRcvTime=1470813320068, readsPaused=false,
filterChain=FilterChain[filters=[GridNioCodecFilter
[parser=o.a.i.i.util.nio.GridDirectParser@276d0d91, directMode=true],
GridConnectionBytesVerifyFilter], accepted=true]]
[08:16:00,904][WARN ][grid-nio-worker-0-#36%null%][TcpCommunicationSpi]
Closing NIO session because of unhandled exception [cls=class
o.a.i.i.util.nio.GridNioException, msg=Connection reset by peer]
[08:16:00,919][WARN ][tcp-disco-msg-worker-#2%null%][TcpDiscoverySpi] Node
is out of topology (probably, due to short-time network problems).
[08:16:00,926][WARN ][disco-event-worker-#45%null%][GridDiscoveryManager]
Local node SEGMENTED: TcpDiscoveryNode
[id=7faa417f-9760-434d-bc66-6551611ebf42, addrs=[10.3.0.64, 127.0.0.1],
sockAddrs=[ves-hx-40.ebi.ac.uk/10.3.0.64:47500, /10.3.0.64:47500,
/127.0.0.1:47500], discPort=47500, order=6, intOrder=4,
lastExchangeTime=1470813360913, loc=true, ver=1.6.0#20160518-sha1:0b22c45b,
isClient=false]
[08:16:01,104][WARN ][disco-event-worker-#45%null%][GridDiscoveryManager]
Stopping local node according to configured segmentation policy.
[08:16:01,109][WARN ][disco-event-worker-#45%null%][GridDiscoveryManager]
Node FAILED: TcpDiscoveryNode [id=358fcad3-1e47-45df-8cd4-875bdcd7011a,
addrs=[10.3.0.65, 127.0.0.1],
sockAddrs=[ves-hx-41.ebi.ac.uk/10.3.0.65:47500, /10.3.0.65:47500,
/127.0.0.1:47500], discPort=47500, order=4, intOrder=3,
lastExchangeTime=1470759707287, loc=false, ver=1.6.0#20160518-sha1:0b22c45b,
isClient=false]
[08:16:01,113][INFO ][disco-event-worker-#45%null%][GridDiscoveryManager]
Topology snapshot [ver=7, servers=1, clients=0, CPUs=2, heap=6.0GB]
[08:16:01,150][INFO ][Thread-31][GridTcpRestProtocol] Command protocol
successfully stopped: TCP binary
[08:16:02,641][INFO ][Thread-31][GridJettyRestProtocol] Command protocol
successfully stopped: Jetty REST
[08:16:02,685][INFO ][Thread-31][GridCacheProcessor] Stopped cache:
session-cache
[08:16:02,687][INFO ][Thread-31][GridCacheProcessor] Stopped cache:
ignite-marshaller-sys-cache
[08:16:02,687][INFO ][Thread-31][GridCacheProcessor] Stopped cache:
ignite-sys-cache
[08:16:02,688][INFO ][Thread-31][GridCacheProcessor] Stopped cache:
ignite-atomics-sys-cache
(2) GC logs
2016-08-10T08:14:02.391+0100: 53542.801: [GC (Allocation Failure)
2016-08-10T08:14:02.391+0100: 53542.801: [ParNew: 130944K->0K(131008K),
0.0423711 secs] 741795K->633860K(2097088K), 0.0425532 secs] [Times:
user=0.08 sys=0.00, real=0.04 secs]
2016-08-10T08:14:02.672+0100: 53543.081: [GC (Allocation Failure)
2016-08-10T08:14:02.672+0100: 53543.081: [ParNew: 130944K->0K(131008K),
0.0390475 secs] 764804K->644675K(2097088K), 0.0392145 secs] [Times:
user=0.08 sys=0.00, real=0.04 secs]
2016-08-10T08:14:02.757+0100: 53543.166: [GC (Allocation Failure)
2016-08-10T08:14:02.757+0100: 53543.166: [ParNew: 130944K->0K(131008K),
0.0121756 secs] 775619K->649138K(2097088K), 0.0123266 secs] [Times:
user=0.02 sys=0.00, real=0.02 secs]
2016-08-10T08:14:05.892+0100: 53546.302: [GC (Allocation Failure)
2016-08-10T08:14:05.892+0100: 53546.302: [ParNew: 130944K->0K(131008K),
0.0435628 secs] 780082K->672552K(2097088K), 0.0437536 secs] [Times:
user=0.08 sys=0.00, real=0.04 secs]
2016-08-10T08:14:06.261+0100: 53546.670: [GC (Allocation Failure)
2016-08-10T08:14:06.261+0100: 53546.670: [ParNew: 130944K->0K(131008K),
0.0369493 secs] 803496K->682986K(2097088K), 0.0371186 secs] [Times:
user=0.07 sys=0.00, real=0.04 secs]
2016-08-10T08:14:06.344+0100: 53546.753: [GC (Allocation Failure)
2016-08-10T08:14:06.344+0100: 53546.754: [ParNew: 130900K->0K(131008K),
0.0129105 secs] 813886K->686672K(2097088K), 0.0130476 secs] [Times:
user=0.02 sys=0.00, real=0.01 secs]
2016-08-10T08:14:08.915+0100: 53549.325: [GC (Allocation Failure)
2016-08-10T08:14:08.915+0100: 53549.325: [ParNew: 130944K->0K(131008K),
0.0445546 secs] 817616K->708798K(2097088K), 0.0447388 secs] [Times:
user=0.08 sys=0.00, real=0.05 secs]
2016-08-10T08:14:09.177+0100: 53549.586: [GC (Allocation Failure)
2016-08-10T08:14:09.177+0100: 53549.586: [ParNew: 130944K->0K(131008K),
0.0355325 secs] 839742K->718747K(2097088K), 0.0356986 secs] [Times:
user=0.07 sys=0.00, real=0.04 secs]
2016-08-10T08:14:09.276+0100: 53549.685: [GC (Allocation Failure)
2016-08-10T08:14:09.276+0100: 53549.685: [ParNew: 130280K->0K(131008K),
0.0130139 secs] 849028K->725765K(2097088K), 0.0131672 secs] [Times:
user=0.03 sys=0.00, real=0.02 secs]
2016-08-10T08:14:11.732+0100: 53552.142: [GC (Allocation Failure)
2016-08-10T08:14:11.732+0100: 53552.142: [ParNew: 130944K->0K(131008K),
0.0438233 secs] 856709K->744928K(2097088K), 0.0439934 secs] [Times:
user=0.09 sys=0.00, real=0.04 secs]
2016-08-10T08:14:12.092+0100: 53552.502: [GC (Allocation Failure)
2016-08-10T08:14:12.092+0100: 53552.502: [ParNew: 130944K->0K(131008K),
0.0356766 secs] 875872K->754500K(2097088K), 0.0358664 secs] [Times:
user=0.06 sys=0.00, real=0.03 secs]
2016-08-10T08:14:12.197+0100: 53552.607: [GC (Allocation Failure)
2016-08-10T08:14:12.197+0100: 53552.607: [ParNew: 130141K->0K(131008K),
0.0163417 secs] 884641K->764814K(2097088K), 0.0165273 secs] [Times:
user=0.04 sys=0.00, real=0.02 secs]
2016-08-10T08:14:14.733+0100: 53555.143: [GC (Allocation Failure)
2016-08-10T08:14:14.733+0100: 53555.143: [ParNew: 130944K->0K(131008K),
0.0441146 secs] 895758K->780588K(2097088K), 0.0443328 secs] [Times:
user=0.09 sys=0.00, real=0.04 secs]
2016-08-10T08:14:14.992+0100: 53555.401: [GC (Allocation Failure)
2016-08-10T08:14:14.992+0100: 53555.402: [ParNew: 130944K->0K(131008K),
0.0332392 secs] 911532K->789827K(2097088K), 0.0334145 secs] [Times:
user=0.06 sys=0.00, real=0.03 secs]
2016-08-10T08:14:15.087+0100: 53555.497: [GC (Allocation Failure)
2016-08-10T08:14:15.087+0100: 53555.497: [ParNew: 129838K->0K(131008K),
0.0170639 secs] 919665K->803490K(2097088K), 0.0172879 secs] [Times:
user=0.03 sys=0.00, real=0.02 secs]
2016-08-10T08:14:17.582+0100: 53557.991: [GC (Allocation Failure)
2016-08-10T08:14:17.582+0100: 53557.992: [ParNew: 130944K->0K(131008K),
0.0459274 secs] 934434K->815938K(2097088K), 0.0461121 secs] [Times:
user=0.09 sys=0.00, real=0.04 secs]
2016-08-10T08:14:17.880+0100: 53558.289: [GC (Allocation Failure)
2016-08-10T08:14:17.880+0100: 53558.289: [ParNew: 130944K->0K(131008K),
0.0482629 secs] 946882K->825204K(2097088K), 0.0484556 secs] [Times:
user=0.07 sys=0.00, real=0.05 secs]
2016-08-10T08:14:19.811+0100: 53560.221: [GC (Allocation Failure)
2016-08-10T08:14:19.811+0100: 53560.221: [ParNew: 130944K->0K(131008K),
0.0180128 secs] 956148K->840033K(2097088K), 0.0182454 secs] [Times:
user=0.04 sys=0.00, real=0.01 secs]
2016-08-10T08:14:20.651+0100: 53561.060: [GC (Allocation Failure)
2016-08-10T08:14:20.651+0100: 53561.061: [ParNew: 130944K->0K(131008K),
0.0369033 secs] 970977K->851956K(2097088K), 0.0370763 secs] [Times:
user=0.07 sys=0.00, real=0.03 secs]
2016-08-10T08:14:20.864+0100: 53561.274: [GC (Allocation Failure)
2016-08-10T08:14:20.864+0100: 53561.274: [ParNew: 130944K->0K(131008K),
0.0321971 secs] 982900K->861367K(2097088K), 0.0323569 secs] [Times:
user=0.06 sys=0.00, real=0.03 secs]
2016-08-10T08:14:23.123+0100: 53563.533: [GC (Allocation Failure)
2016-08-10T08:14:23.124+0100: 53563.533: [ParNew: 130944K->0K(131008K),
0.0213990 secs] 992311K->875859K(2097088K), 0.0216225 secs] [Times:
user=0.05 sys=0.00, real=0.02 secs]
2016-08-10T08:14:23.743+0100: 53564.152: [GC (Allocation Failure)
2016-08-10T08:14:23.743+0100: 53564.152: [ParNew: 130944K->0K(131008K),
0.0520434 secs] 1006803K->887511K(2097088K), 0.0522115 secs] [Times:
user=0.10 sys=0.00, real=0.05 secs]
2016-08-10T08:14:23.969+0100: 53564.378: [GC (Allocation Failure)
2016-08-10T08:14:23.969+0100: 53564.378: [ParNew: 130944K->0K(131008K),
0.0327217 secs] 1018455K->896405K(2097088K), 0.0328889 secs] [Times:
user=0.07 sys=0.00, real=0.04 secs]
2016-08-10T08:14:26.132+0100: 53566.542: [GC (Allocation Failure)
2016-08-10T08:14:26.132+0100: 53566.542: [ParNew: 130944K->0K(131008K),
0.0273679 secs] 1027349K->912677K(2097088K), 0.0276098 secs] [Times:
user=0.05 sys=0.00, real=0.02 secs]
2016-08-10T08:14:26.653+0100: 53567.062: [GC (Allocation Failure)
2016-08-10T08:14:26.653+0100: 53567.062: [ParNew: 130944K->0K(131008K),
0.0434405 secs] 1043621K->923816K(2097088K), 0.0436319 secs] [Times:
user=0.08 sys=0.00, real=0.04 secs]
2016-08-10T08:14:26.948+0100: 53567.358: [GC (Allocation Failure)
2016-08-10T08:14:26.948+0100: 53567.358: [ParNew: 130944K->0K(131008K),
0.0286392 secs] 1054760K->931796K(2097088K), 0.0288045 secs] [Times:
user=0.05 sys=0.00, real=0.03 secs]
2016-08-10T08:14:29.227+0100: 53569.637: [GC (Allocation Failure)
2016-08-10T08:14:29.227+0100: 53569.637: [ParNew: 130944K->0K(131008K),
0.0247875 secs] 1062740K->948545K(2097088K), 0.0249828 secs] [Times:
user=0.05 sys=0.00, real=0.03 secs]
2016-08-10T08:14:29.710+0100: 53570.120: [GC (Allocation Failure)
2016-08-10T08:14:29.710+0100: 53570.120: [ParNew: 130944K->0K(131008K),
0.0421544 secs] 1079489K->960092K(2097088K), 0.0423354 secs] [Times:
user=0.08 sys=0.00, real=0.05 secs]
2016-08-10T08:14:29.962+0100: 53570.372: [GC (Allocation Failure)
2016-08-10T08:14:29.962+0100: 53570.372: [ParNew: 130944K->0K(131008K),
0.0285003 secs] 1091036K->967147K(2097088K), 0.0286752 secs] [Times:
user=0.05 sys=0.00, real=0.02 secs]
2016-08-10T08:14:32.367+0100: 53572.776: [GC (Allocation Failure)
2016-08-10T08:14:32.367+0100: 53572.777: [ParNew: 130944K->0K(131008K),
0.0324680 secs] 1098091K->984433K(2097088K), 0.0327239 secs] [Times:
user=0.06 sys=0.00, real=0.03 secs]
2016-08-10T08:14:32.831+0100: 53573.241: [GC (Allocation Failure)
2016-08-10T08:14:32.831+0100: 53573.241: [ParNew: 130944K->0K(131008K),
0.0433405 secs] 1115377K->996268K(2097088K), 0.0435092 secs] [Times:
user=0.08 sys=0.00, real=0.05 secs]
2016-08-10T08:14:33.015+0100: 53573.424: [GC (Allocation Failure)
2016-08-10T08:14:33.015+0100: 53573.424: [ParNew: 130944K->0K(131008K),
0.0244849 secs] 1127212K->1002321K(2097088K), 0.0246967 secs] [Times:
user=0.05 sys=0.00, real=0.02 secs]
2016-08-10T08:14:35.461+0100: 53575.871: [GC (Allocation Failure)
2016-08-10T08:14:35.461+0100: 53575.871: [ParNew: 130944K->0K(131008K),
0.0295759 secs] 1133265K->1019270K(2097088K), 0.0297637 secs] [Times:
user=0.06 sys=0.00, real=0.03 secs]
2016-08-10T08:14:35.873+0100: 53576.282: [GC (Allocation Failure)
2016-08-10T08:14:35.873+0100: 53576.282: [ParNew: 130944K->0K(131008K),
0.0435538 secs] 1150214K->1031487K(2097088K), 0.0437339 secs] [Times:
user=0.09 sys=0.00, real=0.05 secs]
2016-08-10T08:14:36.044+0100: 53576.453: [GC (Allocation Failure)
2016-08-10T08:14:36.044+0100: 53576.453: [ParNew: 130944K->0K(131008K),
0.0226148 secs] 1162431K->1037326K(2097088K), 0.0228325 secs] [Times:
user=0.04 sys=0.00, real=0.02 secs]
2016-08-10T08:14:38.420+0100: 53578.830: [GC (Allocation Failure)
2016-08-10T08:14:38.420+0100: 53578.830: [ParNew: 130944K->0K(131008K),
0.0367163 secs] 1168270K->1055107K(2097088K), 0.0369606 secs] [Times:
user=0.07 sys=0.00, real=0.04 secs]
2016-08-10T08:14:38.862+0100: 53579.271: [GC (Allocation Failure)
2016-08-10T08:14:38.862+0100: 53579.272: [ParNew: 130944K->0K(131008K),
0.0468229 secs] 1186051K->1067113K(2097088K), 0.0470378 secs] [Times:
user=0.09 sys=0.00, real=0.05 secs]
2016-08-10T08:14:39.072+0100: 53579.482: [GC (Allocation Failure)
2016-08-10T08:14:39.072+0100: 53579.482: [ParNew: 130944K->0K(131008K),
0.0225087 secs] 1198057K->1072718K(2097088K), 0.0226874 secs] [Times:
user=0.04 sys=0.00, real=0.03 secs]
2016-08-10T08:14:41.457+0100: 53581.867: [GC (Allocation Failure)
2016-08-10T08:14:41.458+0100: 53581.867: [ParNew: 130944K->0K(131008K),
0.0363914 secs] 1203662K->1090562K(2097088K), 0.0366144 secs] [Times:
user=0.07 sys=0.00, real=0.04 secs]
2016-08-10T08:14:41.949+0100: 53582.359: [GC (Allocation Failure)
2016-08-10T08:14:41.949+0100: 53582.359: [ParNew: 130944K->0K(131008K),
0.0445878 secs] 1221506K->1103285K(2097088K), 0.0447325 secs] [Times:
user=0.09 sys=0.00, real=0.05 secs]
2016-08-10T08:14:42.098+0100: 53582.508: [GC (Allocation Failure)
2016-08-10T08:14:42.098+0100: 53582.508: [ParNew: 130944K->0K(131008K),
0.0190605 secs] 1234229K->1108111K(2097088K), 0.0192336 secs] [Times:
user=0.03 sys=0.00, real=0.02 secs]
2016-08-10T08:14:44.517+0100: 53584.926: [GC (Allocation Failure)
2016-08-10T08:14:44.517+0100: 53584.926: [ParNew: 130944K->0K(131008K),
0.0407204 secs] 1239055K->1126424K(2097088K), 0.0409601 secs] [Times:
user=0.08 sys=0.00, real=0.04 secs]
2016-08-10T08:14:44.973+0100: 53585.382: [GC (Allocation Failure)
2016-08-10T08:14:44.973+0100: 53585.383: [ParNew: 130944K->0K(131008K),
0.0496677 secs] 1257368K->1139637K(2097088K), 0.0498551 secs] [Times:
user=0.10 sys=0.00, real=0.05 secs]
2016-08-10T08:14:45.171+0100: 53585.581: [GC (Allocation Failure)
2016-08-10T08:14:45.171+0100: 53585.581: [ParNew: 130944K->0K(131008K),
0.0184316 secs] 1270581K->1143661K(2097088K), 0.0186625 secs] [Times:
user=0.03 sys=0.00, real=0.02 secs]
2016-08-10T08:14:47.603+0100: 53588.013: [GC (Allocation Failure)
2016-08-10T08:14:47.603+0100: 53588.013: [ParNew: 130944K->0K(131008K),
0.0389169 secs] 1274605K->1162333K(2097088K), 0.0391510 secs] [Times:
user=0.07 sys=0.00, real=0.04 secs]
2016-08-10T08:14:48.020+0100: 53588.430: [GC (Allocation Failure)
2016-08-10T08:14:48.020+0100: 53588.430: [ParNew: 130944K->0K(131008K),
0.0495658 secs] 1293277K->1175922K(2097088K), 0.0497736 secs] [Times:
user=0.10 sys=0.00, real=0.05 secs]
2016-08-10T08:14:48.156+0100: 53588.566: [GC (Allocation Failure)
2016-08-10T08:14:48.157+0100: 53588.566: [ParNew: 130944K->0K(131008K),
0.0163738 secs] 1306866K->1179170K(2097088K), 0.0165479 secs] [Times:
user=0.04 sys=0.00, real=0.01 secs]
2016-08-10T08:14:50.392+0100: 53590.801: [GC (Allocation Failure)
2016-08-10T08:14:50.392+0100: 53590.801: [ParNew: 130944K->0K(131008K),
0.0359011 secs] 1310114K->1198502K(2097088K), 0.0361635 secs] [Times:
user=0.06 sys=0.00, real=0.04 secs]
2016-08-10T08:14:50.436+0100: 53590.846: [GC (CMS Initial Mark) [1
CMS-initial-mark: 1198502K(1966080K)] 1201865K(2097088K), 0.0043556 secs]
[Times: user=0.01 sys=0.00, real=0.00 secs]
2016-08-10T08:14:50.440+0100: 53590.850: [CMS-concurrent-mark-start]
2016-08-10T08:14:50.716+0100: 53591.126: [CMS-concurrent-mark: 0.269/0.276
secs] [Times: user=0.54 sys=0.01, real=0.28 secs]
2016-08-10T08:14:50.716+0100: 53591.126: [CMS-concurrent-preclean-start]
2016-08-10T08:14:50.721+0100: 53591.130: [CMS-concurrent-preclean:
0.004/0.005 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]
2016-08-10T08:14:50.721+0100: 53591.130:
[CMS-concurrent-abortable-preclean-start]
2016-08-10T08:14:50.857+0100: 53591.266: [GC (Allocation Failure)
2016-08-10T08:14:50.857+0100: 53591.266: [ParNew: 130944K->0K(131008K),
0.0559454 secs] 1329446K->1212314K(2097088K), 0.0561386 secs] [Times:
user=0.10 sys=0.00, real=0.05 secs]
2016-08-10T08:14:51.049+0100: 53591.459: [GC (Allocation Failure)
2016-08-10T08:14:51.049+0100: 53591.459: [ParNew: 130944K->0K(131008K),
0.0131197 secs] 1343258K->1214690K(2097088K), 0.0132465 secs] [Times:
user=0.03 sys=0.00, real=0.01 secs]
2016-08-10T08:14:51.292+0100: 53591.701: [CMS-concurrent-abortable-preclean:
0.078/0.571 secs] [Times: user=0.51 sys=0.00, real=0.57 secs]
2016-08-10T08:14:51.292+0100: 53591.702: [GC (CMS Final Remark) [YG
occupancy: 72425 K (131008 K)]2016-08-10T08:14:51.292+0100: 53591.702:
[Rescan (parallel) , 0.0131918 secs]2016-08-10T08:14:51.306+0100: 53591.715:
[weak refs processing, 0.0002923 secs]2016-08-10T08:14:51.306+0100:
53591.716: [class unloading, 0.0282178 secs]2016-08-10T08:14:51.334+0100:
53591.744: [scrub symbol table, 0.0086011 secs]2016-08-10T08:14:51.343+0100:
53591.752: [scrub string table, 0.0018392 secs][1 CMS-remark:
1214690K(1966080K)] 1287116K(2097088K), 0.0523555 secs] [Times: user=0.06
sys=0.00, real=0.06 secs]
2016-08-10T08:14:51.345+0100: 53591.755: [CMS-concurrent-sweep-start]
2016-08-10T08:14:52.236+0100: 53592.645: [CMS-concurrent-sweep: 0.882/0.891
secs] [Times: user=0.90 sys=0.01, real=0.89 secs]
2016-08-10T08:14:52.236+0100: 53592.645: [CMS-concurrent-reset-start]
2016-08-10T08:14:52.253+0100: 53592.663: [CMS-concurrent-reset: 0.017/0.017
secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-08-10T08:14:53.407+0100: 53593.817: [GC (Allocation Failure)
2016-08-10T08:14:53.407+0100: 53593.817: [ParNew: 130944K->0K(131008K),
0.0330565 secs] 330252K->219163K(2097088K), 0.0332613 secs] [Times:
user=0.06 sys=0.00, real=0.03 secs]
2016-08-10T08:14:53.826+0100: 53594.236: [GC (Allocation Failure)
2016-08-10T08:14:53.826+0100: 53594.236: [ParNew: 130944K->0K(131008K),
0.0509833 secs] 350107K->233352K(2097088K), 0.0511630 secs] [Times:
user=0.10 sys=0.00, real=0.05 secs]
2016-08-10T08:14:53.986+0100: 53594.396: [GC (Allocation Failure)
2016-08-10T08:14:53.986+0100: 53594.396: [ParNew: 130944K->0K(131008K),
0.0140883 secs] 364296K->234932K(2097088K), 0.0142692 secs] [Times:
user=0.02 sys=0.00, real=0.01 secs]
2016-08-10T08:14:56.302+0100: 53596.711: [GC (Allocation Failure)
2016-08-10T08:14:56.302+0100: 53596.711: [ParNew: 130944K->0K(131008K),
0.0335848 secs] 365876K->255257K(2097088K), 0.0340425 secs] [Times:
user=0.06 sys=0.00, real=0.04 secs]
2016-08-10T08:14:56.829+0100: 53597.239: [GC (Allocation Failure)
2016-08-10T08:14:56.829+0100: 53597.239: [ParNew: 130944K->0K(131008K),
0.0519926 secs] 386201K->269242K(2097088K), 0.0521780 secs] [Times:
user=0.10 sys=0.00, real=0.05 secs]
2016-08-10T08:14:56.971+0100: 53597.381: [GC (Allocation Failure)
2016-08-10T08:14:56.972+0100: 53597.381: [ParNew: 130944K->0K(131008K),
0.0134896 secs] 400186K->270766K(2097088K), 0.0136625 secs] [Times:
user=0.03 sys=0.00, real=0.01 secs]
2016-08-10T08:14:59.314+0100: 53599.724: [GC (Allocation Failure)
2016-08-10T08:14:59.314+0100: 53599.724: [ParNew: 130944K->0K(131008K),
0.0335347 secs] 401710K->291474K(2097088K), 0.0337116 secs] [Times:
user=0.07 sys=0.00, real=0.04 secs]
2016-08-10T08:14:59.682+0100: 53600.091: [GC (Allocation Failure)
2016-08-10T08:14:59.682+0100: 53600.091: [ParNew: 130944K->0K(131008K),
0.0494621 secs] 422418K->304965K(2097088K), 0.0496462 secs] [Times:
user=0.09 sys=0.00, real=0.05 secs]
2016-08-10T08:14:59.779+0100: 53600.189: [GC (Allocation Failure)
2016-08-10T08:14:59.779+0100: 53600.189: [ParNew: 130944K->0K(131008K),
0.0111250 secs] 435909K->306471K(2097088K), 0.0112882 secs] [Times:
user=0.02 sys=0.00, real=0.01 secs]
2016-08-10T08:15:02.265+0100: 53602.674: [GC (Allocation Failure)
2016-08-10T08:15:02.265+0100: 53602.674: [ParNew: 130944K->0K(131008K),
0.0276427 secs] 437415K->327724K(2097088K), 0.0278357 secs] [Times:
user=0.05 sys=0.00, real=0.03 secs]
2016-08-10T08:15:02.615+0100: 53603.025: [GC (Allocation Failure)
2016-08-10T08:15:02.616+0100: 53603.025: [ParNew: 130944K->0K(131008K),
0.0387177 secs] 458668K->340715K(2097088K), 0.0389586 secs] [Times:
user=0.08 sys=0.00, real=0.04 secs]
2016-08-10T08:15:02.702+0100: 53603.111: [GC (Allocation Failure)
2016-08-10T08:15:02.702+0100: 53603.111: [ParNew: 130944K->0K(131008K),
0.0119337 secs] 471659K->344790K(2097088K), 0.0121084 secs] [Times:
user=0.02 sys=0.00, real=0.01 secs]
2016-08-10T08:15:05.217+0100: 53605.627: [GC (Allocation Failure)
2016-08-10T08:15:05.217+0100: 53605.627: [ParNew: 130944K->0K(131008K),
0.0371829 secs] 475734K->366549K(2097088K), 0.0373651 secs] [Times:
user=0.08 sys=0.00, real=0.03 secs]
2016-08-10T08:15:05.607+0100: 53606.017: [GC (Allocation Failure)
2016-08-10T08:15:05.607+0100: 53606.017: [ParNew: 130944K->0K(131008K),
0.0619285 secs] 497493K->379133K(2097088K), 0.0621140 secs] [Times:
user=0.12 sys=0.00, real=0.06 secs]
2016-08-10T08:15:05.736+0100: 53606.146: [GC (Allocation Failure)
2016-08-10T08:15:05.737+0100: 53606.146: [ParNew: 130944K->0K(131008K),
0.0144612 secs] 510077K->383232K(2097088K), 0.0146605 secs] [Times:
user=0.03 sys=0.00, real=0.02 secs]
2016-08-10T08:15:08.312+0100: 53608.721: [GC (Allocation Failure)
2016-08-10T08:15:08.312+0100: 53608.721: [ParNew: 130944K->0K(131008K),
0.0372281 secs] 514176K->405043K(2097088K), 0.0373988 secs] [Times:
user=0.07 sys=0.00, real=0.04 secs]
2016-08-10T08:15:08.642+0100: 53609.051: [GC (Allocation Failure)
2016-08-10T08:15:08.642+0100: 53609.051: [ParNew: 130944K->0K(131008K),
0.0451608 secs] 535987K->417587K(2097088K), 0.0453694 secs] [Times:
user=0.08 sys=0.00, real=0.04 secs]
2016-08-10T08:15:08.756+0100: 53609.166: [GC (Allocation Failure)
2016-08-10T08:15:08.756+0100: 53609.166: [ParNew: 130944K->0K(131008K),
0.0131418 secs] 548531K->421687K(2097088K), 0.0133133 secs] [Times:
user=0.03 sys=0.00, real=0.02 secs]
2016-08-10T08:15:11.397+0100: 53611.806: [GC (Allocation Failure)
2016-08-10T08:15:11.397+0100: 53611.806: [ParNew: 130944K->0K(131008K),
0.0419434 secs] 552631K->444073K(2097088K), 0.0421394 secs] [Times:
user=0.08 sys=0.00, real=0.05 secs]
2016-08-10T08:15:11.819+0100: 53612.229: [GC (Allocation Failure)
2016-08-10T08:15:11.820+0100: 53612.229: [ParNew: 130944K->0K(131008K),
0.0442636 secs] 575017K->456082K(2097088K), 0.0444590 secs] [Times:
user=0.08 sys=0.00, real=0.04 secs]
2016-08-10T08:15:11.932+0100: 53612.341: [GC (Allocation Failure)
2016-08-10T08:15:11.932+0100: 53612.341: [ParNew: 130944K->0K(131008K),
0.0129157 secs] 587026K->460534K(2097088K), 0.0130707 secs] [Times:
user=0.02 sys=0.00, real=0.01 secs]
2016-08-10T08:15:14.465+0100: 53614.874: [GC (Allocation Failure)
2016-08-10T08:15:14.465+0100: 53614.874: [ParNew: 130944K->0K(131008K),
0.0399574 secs] 591478K->482586K(2097088K), 0.0401414 secs] [Times:
user=0.08 sys=0.00, real=0.04 secs]
2016-08-10T08:15:14.890+0100: 53615.300: [GC (Allocation Failure)
2016-08-10T08:15:14.890+0100: 53615.300: [ParNew: 130944K->0K(131008K),
0.0425668 secs] 613530K->494223K(2097088K), 0.0427569 secs] [Times:
user=0.08 sys=0.00, real=0.04 secs]
2016-08-10T08:15:15.002+0100: 53615.411: [GC (Allocation Failure)
2016-08-10T08:15:15.002+0100: 53615.411: [ParNew: 130944K->0K(131008K),
0.0124776 secs] 625167K->498661K(2097088K), 0.0126650 secs] [Times:
user=0.02 sys=0.00, real=0.01 secs]
2016-08-10T08:15:17.487+0100: 53617.897: [GC (Allocation Failure)
2016-08-10T08:15:17.488+0100: 53617.897: [ParNew: 130944K->0K(131008K),
0.0397801 secs] 629605K->521272K(2097088K), 0.0399852 secs] [Times:
user=0.07 sys=0.00, real=0.04 secs]
2016-08-10T08:15:17.776+0100: 53618.186: [GC (Allocation Failure)
2016-08-10T08:15:17.776+0100: 53618.186: [ParNew: 130944K->0K(131008K),
0.0364991 secs] 652216K->532353K(2097088K), 0.0366642 secs] [Times:
user=0.07 sys=0.00, real=0.04 secs]
2016-08-10T08:15:17.875+0100: 53618.285: [GC (Allocation Failure)
2016-08-10T08:15:17.875+0100: 53618.285: [ParNew: 130944K->0K(131008K),
0.0119357 secs] 663297K->536837K(2097088K), 0.0120945 secs] [Times:
user=0.03 sys=0.00, real=0.01 secs]
2016-08-10T08:16:00.956+0100: 53661.365: [GC (Allocation Failure)
2016-08-10T08:16:00.956+0100: 53661.365: [ParNew: 130931K->0K(131008K),
0.0574650 secs] 667768K->558574K(2097088K), 0.0577257 secs] [Times:
user=0.11 sys=0.00, real=0.06 secs]
2016-08-10T08:16:02.368+0100: 53662.777: [GC (Allocation Failure)
2016-08-10T08:16:02.368+0100: 53662.777: [ParNew: 130944K->0K(131008K),
0.0588387 secs] 689518K->573412K(2097088K), 0.0589743 secs] [Times:
user=0.11 sys=0.00, real=0.06 secs]
2016-08-10T08:16:02.577+0100: 53662.987: [GC (Allocation Failure)
2016-08-10T08:16:02.578+0100: 53662.987: [ParNew: 130944K->0K(131008K),
0.0199449 secs] 704356K->576049K(2097088K), 0.0201792 secs] [Times:
user=0.03 sys=0.00, real=0.02 secs]
2016-08-10T08:24:22.773+0100: 54163.182: [GC (Allocation Failure)
2016-08-10T08:24:22.773+0100: 54163.182: [ParNew: 130938K->0K(131008K),
0.0344864 secs] 706988K->591374K(2097088K), 0.0346710 secs] [Times:
user=0.07 sys=0.00, real=0.03 secs]
2016-08-10T08:58:20.910+0100: 56201.319: [GC (Allocation Failure)
2016-08-10T08:58:20.910+0100: 56201.320: [ParNew: 130944K->0K(131008K),
0.0619050 secs] 722318K->598191K(2097088K), 0.0621083 secs] [Times:
user=0.12 sys=0.00, real=0.07 secs]
2016-08-10T09:32:16.079+0100: 58236.488: [GC (Allocation Failure)
2016-08-10T09:32:16.079+0100: 58236.489: [ParNew: 130944K->0K(131008K),
0.0613423 secs] 729135K->605005K(2097088K), 0.0615247 secs] [Times:
user=0.12 sys=0.00, real=0.06 secs]
(3) JVM arguments
-Djava.awt.headless=true
-Dtomcat.hostname=ves-hx-40
-Dhttp.port=8100
-Dsecure.port=8101
-Dshutdown.port=8105
-DIGNITE_HOME=/nfs/public/rw/webadmin/tomcat/bases/3rd-party/apache-ignite-fabric-1.6.0-bin
-DIGNITE_JETTY_PORT=8574
-Dtomcat.major.version=7
-Dwebadmin.path=/nfs/public/rw/webadmin
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
-Xms2g
-Xmx6g
-XX:MaxPermSize=128m
-Doracle.net.tns_admin=
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=8107
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-DIGNITE_QUIET=false
-XX:+UseParNewGC
-XX:+UseConcMarkSweepGC
-XX:+UseTLAB
-XX:NewSize=128m
-XX:MaxNewSize=128m
-XX:MaxTenuringThreshold=0
-XX:SurvivorRatio=1024
-XX:+UseCMSInitiatingOccupancyOnly
-XX:CMSInitiatingOccupancyFraction=60
-XX:+DisableExplicitGC
-XX:+PrintGCTimeStamps
-XX:+PrintGCDateStamps
-verbose:gc
-XX:+PrintGCDetails
-XX:+UseGCLogFileRotation
-XX:NumberOfGCLogFiles=10
-XX:GCLogFileSize=100M
-Xloggc:/nfs/public/rw/webadmin/tomcat/bases/data/ves-hx-40/gc/gc_events.log
-Djava.endorsed.dirs=/nfs/public/rw/webadmin/tomcat/homes/apache-tomcat-7.0.55/endorsed
-Dcatalina.base=/nfs/public/rw/webadmin/tomcat/bases
-Dcatalina.home=/nfs/public/rw/webadmin/tomcat/homes/apache-tomcat-7.0.55
-Djava.io.tmpdir=/nfs/public/rw/webadmin/tomcat/bases/temp
Thanks for your kind help!
Yuci
--
View this message in context:
http://apache-ignite-users.70518.x6.nabble.com/Local-node-seems-to-be-disconnected-from-topology-failure-detection-timeout-is-reached-tp6797p6926.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
