[
https://issues.apache.org/jira/browse/IGNITE-10589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Kosarev updated IGNITE-10589:
------------------------------------
Description:
after stopping a client
we see topology change and pme finish on the coordinator,
and at soon on another nodes we still don't see new topology, but have
Critical error resulting nodes failure
crd log
{code}
2018-12-06 15:55:23.660 [WARN
][disco-event-worker-#159%DPL_GRID%DplGridNodeName%][o.a.i.i.m.d.GridDiscoveryManager]
Node FAILED: ZookeeperClusterNode [id=979f03db-f858-44f6-8646-12034dfd5c93,
addrs=[10.116.206.1], order=129, loc=false, client=true]
2018-12-06 15:55:23.660 [INFO
][disco-event-worker-#159%DPL_GRID%DplGridNodeName%][o.a.i.i.m.d.GridDiscoveryManager]
Topology snapshot [ver=162, servers=128, clients=0, CPUs=7168,
offheap=140000.0GB, heap=4000.0GB]
2018-12-06 15:55:23.660 [INFO
][disco-event-worker-#159%DPL_GRID%DplGridNodeName%][o.a.i.i.m.d.GridDiscoveryManager]
^-- Node [id=44D27930-80E5-4EB7-B377-8B07C02C2033, clusterState=ACTIVE]
2018-12-06 15:55:23.660 [INFO
][zk-DPL_GRID%DplGridNodeName-EventThread][o.a.i.s.d.z.i.ZookeeperDiscoveryImpl]
Process alive nodes change [alives=128]
2018-12-06 15:55:23.661 [INFO
][disco-event-worker-#159%DPL_GRID%DplGridNodeName%][o.a.i.i.m.d.GridDiscoveryManager]
^-- Baseline [id=0, size=128, online=128, offline=0]
2018-12-06 15:55:23.661 [INFO
][disco-event-worker-#159%DPL_GRID%DplGridNodeName%][o.a.i.i.m.d.GridDiscoveryManager]
Data Regions Configured:
2018-12-06 15:55:23.661 [INFO
][disco-event-worker-#159%DPL_GRID%DplGridNodeName%][o.a.i.i.m.d.GridDiscoveryManager]
^-- dpl_mem_plc [initSize=256.0 MiB, maxSize=556.6 GiB,
persistenceEnabled=true]
2018-12-06 15:55:23.661 [INFO
][disco-event-worker-#159%DPL_GRID%DplGridNodeName%][o.a.i.i.m.d.GridDiscoveryManager]
^-- not-persisted [initSize=256.0 MiB, maxSize=556.6 GiB,
persistenceEnabled=false]
2018-12-06 15:55:23.670
[DEBUG][sys-#564%DPL_GRID%DplGridNodeName%][o.a.i.i.p.c.d.d.p.l.ExchangeLatchManager]
Process node left 979f03db-f858-44f6-8646-12034dfd5c93
2018-12-06 15:55:23.670 [INFO
][exchange-worker-#160%DPL_GRID%DplGridNodeName%][o.a.ignite.internal.exchange.time]
Started exchange init [topVer=AffinityTopologyVersion [topVer=162,
minorTopVer=0], crd=true, evt=NODE_FAILED,
evtNode=979f03db-f858-44f6-8646-12034dfd5c93, customEvt=null, allowMerge=true]
2018-12-06 15:55:23.712 [INFO
][exchange-worker-#160%DPL_GRID%DplGridNodeName%][o.a.ignite.internal.exchange.time]
Finished exchange init [topVer=AffinityTopologyVersion [topVer=162,
minorTopVer=0], crd=true]
2018-12-06 15:55:23.699 [INFO
][exchange-worker-#160%DPL_GRID%DplGridNodeName%][o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture]
Finish exchange future [startVer=AffinityTopologyVersion [topVer=162,
minorTopVer=0], resVer=AffinityTopologyVersion [topVer=162, minorTopVer=0],
err=null]
{code}
on one we have critical error(1)
{code}
2018-12-06 15:55:23.727
[ERROR][utility-#432%DPL_GRID%DplGridNodeName%][o.a.i.i.p.cache.GridCacheIoManager]
Failed processing message [senderId=1e17c56a-5213-4a1b-b94b-4575a95a2c81,
msg=GridDhtTxPrepareRequest [nearNodeId=44d27930-80e5-4eb7-b377-8b07c02c2033,
futId=1d225238761-05eea259-5c25-4a4b-8469-9dd8980e218c, miniId=105,
topVer=AffinityTopologyVersion [topVer=162, minorTopVer=0],
invalidateNearEntries={}, nearWrites=null, owned=null,
nearXidVer=GridCacheVersion [topVer=155571374, order=1545423626166, nodeOrd
er=1], subjId=44d27930-80e5-4eb7-b377-8b07c02c2033, taskNameHash=0,
preloadKeys=null, skipCompletedVers=false,
super=GridDistributedTxPrepareRequest [threadId=1281, concurrency=PESSIMISTIC,
isolation=REPEATABLE_READ, writeVer=GridCacheVersion [topVer=15557137
4, order=1545423626614, nodeOrder=96], timeout=0, reads=null, writes=ArrayList
[IgniteTxEntry [key=KeyCacheObjectImpl [part=65, val=GridServiceAssignmentsKey
[name=DPLThreadManager_service], hasValBytes=true], cacheId=-2100569601,
txKey=IgniteTxKey [key=KeyCa
cheObjectImpl [part=65, val=GridServiceAssignmentsKey
[name=DPLThreadManager_service], hasValBytes=true], cacheId=-2100569601],
val=CacheObjectImpl [val=GridServiceAssignments
[nodeId=426a4a51-1af3-4019-9769-4a58d8ece426, topVer=162,
cfg=LazyServiceConfigurat
ion [srvcClsName=com.sbt.dpl.gridgain.thread.DPLThreadManager, svcCls=,
nodeFilterCls=IgniteAllNodesPredicate], assigns=HashMap
{74979bc7-e4c3-424a-8347-2f3a589aca3e=1,
32afc0a1-156d-4998-9b64-2336b86fb1c2=1, 6b34e59e-924c-404b-9451-ddf4b8b935b5=1,
c2af8947-1
a8d-48d5-93ca-165f97399519=1, 12f327e6-0f5f-44e2-be2c-f7bf99d22eec=1,
5638cb8c-4abb-49e3-8eef-edb3d5ccad77=1, 20349aca-ecf9-4a5f-bce9-2640e54fbbb2=1,
e83f5de6-deaf-4f60-af5c-5a13d4f251a7=1, 2b4261be-feb5-4d59-be2a-6c9fcbe2fa4a=1,
c3f0bff0-08fc-4601-b9f9-33192
9065c0a=1, bac0156c-a56b-4ab8-aa7c-7d9878151e9f=1,
ccc2d442-8df4-402e-8589-d8ec3c6ec243=1, e34256a2-1bb9-4a17-86d1-21532833dded=1,
50bae4c5-16a5-48bf-a3f6-aa1b123074af=1, c5ecce59-cf6f-4be3-9861-e5c2622480a5=1,
d95ad91e-abd6-4c59-bc02-6298278f84c5=1, 035787c0
-4497-4682-9488-9be55e875175=1, e17bd18f-4a71-47f5-bdd6-64199a9bfb3a=1,
e3565de2-04a3-4107-95d9-01cdd790838b=1, dd372a51-8239-4f8f-8eef-5d6f206e971e=1,
c7ff660d-a003-493b-9aac-a6f73ad46561=1, 426a4a51-1af3-4019-9769-4a58d8ece426=1,
107810a2-c04d-452c-b4b5-d61
abf16272c=1, 99df5f9f-5bc9-4f6b-a538-25c097124f38=1,
93baebe8-c8fc-4e25-8b6a-17f925e67dce=1, 44f063cf-9ad5-4095-96a3-54e554ab9ca3=1,
d4f7a539-cc66-4d76-afd6-2d41f533c44e=1, 0be09f47-0a70-4589-a78c-6c9fb7393d43=1,
6094f19a-6754-4ce9-8892-a540a52cf775=1, 17ad38
b2-1ec5-4531-9fca-397acbfa4a98=1, 23f220b1-cddf-4ac8-8987-05ec33569855=1,
e7b59e5d-3102-4dda-9d84-95783d80940d=1, cd9fc4bb-b488-40f5-8a72-24ecf633e3b0=1,
76fd7993-7c53-4a59-b199-ece9ce6f1b32=1, 8aef872d-83cd-42f1-9641-612206f1d026=1,
158087d4-2ca2-4ee2-83be-7
a61e52c9aac=1, 59b4bfeb-7690-4844-b3b9-d7939d72c098=1,
16bd8778-33a1-4a06-a904-8792f9991921=1, 5198403c-ffb1-4b64-b523-202cc76aee59=1,
fd636af8-4dea-4df1-b6b1-099bd14ec8aa=1, aa2471f8-f5cd-445f-8b79-fa58e08783a7=1,
22b3ebec-8a3e-4320-8ccc-a0c968362222=1, b72f
a1b9-7547-452c-99d2-5b61b6f8ebec=1, c56babe1-1071-4d8e-9499-3fc211d375a0=1,
43ff2a2f-db9c-4f5e-b721-ef350451ca0a=1, 20a6ff76-53f6-4508-8e4c-c87359e625a8=1,
07377a77-24da-4de3-8993-595b6f77e199=1, 0923e9e8-b412-40ae-bbb6-c75e1923cbd8=1,
416e7265-7507-4d05-b444
-0118999146a1=1, cd12c8c4-d209-4e5b-b34e-9cf25523ee7d=1,
32552ec6-5a88-4ae6-a069-0d86594b7031=1, 89a4c1a0-33a1-40b2-a443-f15533bd13d7=1,
b6545e2e-6ee5-4c0f-8fd8-3448bc2ab546=1, ab1dedb8-919b-4373-af45-250f12c7a8af=1,
944d0054-4b92-4eae-80dc-b6a45fe415c8=1, b7
147304-51c3-4c2d-9fb7-45853b27b79f=1, 6e334bfa-05d7-4909-94fb-e812d0fd4c76=1,
a91104a0-e902-46e4-8725-47438b48b102=1, 5e5fb902-1c22-488a-bb25-87d10c8ccc0e=1,
297d27cc-4cd1-486f-a922-0a27808d8304=1, 5e069562-4ac7-48c6-8ce2-e6feb6be3d44=1,
7db65bfd-5478-4b3d-8d
2f-7f35696fe6fe=1, 66dbd7b0-c7b2-421e-bc11-edc3919e4a0c=1,
c83a8121-38e8-4409-8cdb-d4929bf4d0cf=1, f9b0d868-0a8e-41a5-9c45-73ae688ffc1e=1,
be79ee19-8cc4-467c-bb15-2b066c27d667=1, 9e54bf88-b439-4e2f-9f14-4dee1bb66a0d=1,
9eee3d1d-702e-4e02-916f-f21b4b5dc27a=1,
e347eecb-3489-4e0d-a7dc-420854b8b3e9=1, 9cdf5210-6624-4f2f-9314-3e6bf3b23587=1,
1e17c56a-5213-4a1b-b94b-4575a95a2c81=1, 1988168d-839c-471b-9a83-3c48f0a7447f=1,
835ed2f5-6fd4-43e4-84bd-504f5df0e301=1, 8852b77b-17ee-47a7-af2f-4d63babe970e=1,
d7b33e74-5683-40c4-
983d-6e2661183022=1, 16997ebc-723f-4b26-a14e-c98c40191646=1,
00dd20b3-12b4-46fe-800c-3ec405d11d98=1, b08008de-2ad0-4777-8365-2c7db6b470f3=1,
2bba7299-41f5-4635-8896-c6995425796e=1, 9d9234fe-3a77-4623-b83f-4297193ddf04=1,
fbd450b0-b545-467c-acd6-2d4acf53f239=1
, c36dc0f8-7e04-43c0-b1af-8883c42128f2=1,
8da84ab8-1d64-41c1-b904-03348fec36c6=1, fce7c69c-60c7-4006-8d2e-7363309c05d8=1,
d7412fac-3f80-4c92-806d-cf10ae545a63=1, 7bafe6ad-d5fa-45d0-8c86-cef3d3d1bd3a=1,
ce614913-d061-4e6b-ab6d-58df42fdab80=1, d1b9e739-b957-43e
9-80d2-c2a984c48639=1, af32cdf5-01f7-4196-aea6-e07ed36de5aa=1,
6e64c0d8-2bcb-428c-b64b-bd3391763d4e=1, 9b980efd-edee-462b-adad-c667e4b4ee65=1,
205dddb8-defe-4717-93a3-05953ceb406d=1, dabfb14f-5b83-469b-9798-ab89b440f379=1,
b05053e8-dc96-42b8-99bd-a4918c950aed
=1, 2645af19-64fe-4b9d-bbba-a3d9202105be=1,
3bf0cb89-bdb4-47ff-81e7-ce67377d750b=1, 667bd106-cb20-429c-adec-07293f794db9=1,
4e7d90fb-b536-4ac0-83ca-036e151bf707=1, a0d5cf3b-7c1e-4ae6-a39c-b1c98a04ea8f=1,
625eccc2-d955-4bf9-a293-f53b945b0f09=1... and 28 more}]
, hasValBytes=true][op=UPDATE, val=], prevVal=[op=NOOP, val=null],
oldVal=CacheObjectImpl [val=null, hasValBytes=true][op=UPDATE, val=],
entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null,
explicitVer=null, dhtVer=null, filters=CacheEntr
yPredicate[] [], filtersPassed=false, filtersSet=false, entry=GridCacheMapEntry
[key=KeyCacheObjectImpl [part=65, val=GridServiceAssignmentsKey
[name=DPLThreadManager_service], hasValBytes=true], val=CacheObjectImpl
[val=null, hasValBytes=true], startVer=1545
426660336, ver=GridCacheVersion [topVer=155571373, order=1544175423890,
nodeOrder=96], hash=1900127065, extras=GridCacheMvccEntryExtras
[mvcc=GridCacheMvcc [locs=null, rmts=LinkedList [GridCacheMvccCandidate
[nodeId=1e17c56a-5213-4a1b-b94b-4575a95a2c81, ver=G
ridCacheVersion [topVer=155571374, order=1545423626612, nodeOrder=96],
threadId=1942, id=219868, topVer=AffinityTopologyVersion [topVer=-1,
minorTopVer=0], reentry=null, otherNodeId=44d27930-80e5-4eb7-b377-8b07c02c2033,
otherVer=null, mappedDhtNodes=null, map
pedNearNodes=null, ownerVer=null, serOrder=null, key=KeyCacheObjectImpl
[part=65, val=GridServiceAssignmentsKey [name=DPLThreadManager_service],
hasValBytes=true],
masks=local=0|owner=0|ready=0|reentry=0|used=0|tx=1|single_implicit=0|dht_local=0|near_local=0|
removed=0|read=0, prevVer=null, nextVer=null]]]],
flags=2]GridDistributedCacheEntry [super=]GridDhtCacheEntry [rdrs=ReaderId[]
[], part=65, super=], prepared=1, locked=false, nodeId=null, locMapped=false,
expiryPlc=null, transferExpiryPlc=false, flags=0, part
UpdateCntr=0, serReadVer=null, xidVer=null]], dhtVers=null, txSize=0, plc=5,
txState=null, flags=last|sys, super=GridDistributedBaseMessage
[ver=GridCacheVersion [topVer=155571374, order=1545423626612, nodeOrder=96],
committedVers=null, rolledbackVers=null, c
nt=0, super=GridCacheIdMessage [cacheId=0]]]]]
org.apache.ignite.IgniteException: Failed to resolve nodes topology
[cacheGrp=N/A, topVer=AffinityTopologyVersion [topVer=162, minorTopVer=0],
history=[AffinityTopologyVersion [topVer=161, minorTopVer=0]], snap=Snapshot
[topVer=AffinityTopologyVersion [topVer
=161, minorTopVer=0]], locNode=ZookeeperClusterNode
[id=51dc74ab-c989-4268-b850-ed69a24cca30, addrs=[10.116.206.28], order=161,
loc=true, client=false]]
at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.resolveDiscoCache(GridDiscoveryManager.java:2111)
at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.consistentId(GridDiscoveryManager.java:1950)
at
org.apache.ignite.internal.managers.discovery.ConsistentIdMapper.mapToCompactIds(ConsistentIdMapper.java:104)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxAdapter.state(IgniteTxAdapter.java:1142)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxAdapter.state(IgniteTxAdapter.java:993)
at
org.apache.ignite.internal.processors.cache.distributed.GridDistributedTxRemoteAdapter.prepareRemoteTx(GridDistributedTxRemoteAdapter.java:407)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.startRemoteTx(IgniteTxHandler.java:1759)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processDhtTxPrepareRequest(IgniteTxHandler.java:1121)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.access$400(IgniteTxHandler.java:101)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$5.apply(IgniteTxHandler.java:205)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$5.apply(IgniteTxHandler.java:203)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1061)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:586)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:385)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:311)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:300)
at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1556)
at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1184)
at
org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:125)
at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1091)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{code}
on many other nodes we have Critical error (2) with
*nodeId=51dc74ab-c989-4268-b850-ed69a24cca30* of the node above and nearNodeid
is CRD:
{code}
2018-12-06 15:55:23.730
[ERROR][utility-#386%DPL_GRID%DplGridNodeName%][o.a.i.i.p.cache.GridCacheIoManager]
Failed processing message [senderId=1e17c56a-5213-4a1b-b94b-4575a95a2c81,
msg=GridDhtTxPrepareRequest [nearNodeId=44d27930-80e5-4eb7-b377-8b07c02c2033,
futId=1d225238761-05eea259-5c25-4a4b-8469-9dd8980e218c, miniId=79,
topVer=AffinityTopologyVersion [topVer=162, minorTopVer=0],
invalidateNearEntries={}, nearWrites=null, owned=null,
nearXidVer=GridCacheVersion [topVer=155571374, order=1545423626166, nodeOrde
r=1], subjId=44d27930-80e5-4eb7-b377-8b07c02c2033, taskNameHash=0,
preloadKeys=null, skipCompletedVers=false,
super=GridDistributedTxPrepareRequest [threadId=1281, concurrency=PESSIMISTIC,
isolation=REPEATABLE_READ, writeVer=GridCacheVersion [topVer=155571374
, order=1545423626614, nodeOrder=96], timeout=0, reads=null, writes=ArrayList
[IgniteTxEntry [key=KeyCacheObjectImpl [part=65, val=GridServiceAssignmentsKey
[name=DPLThreadManager_service], hasValBytes=true], cacheId=-2100569601,
txKey=IgniteTxKey [key=KeyCac
heObjectImpl [part=65, val=GridServiceAssignmentsKey
[name=DPLThreadManager_service], hasValBytes=true], cacheId=-2100569601],
val=CacheObjectImpl [val=GridServiceAssignments
[nodeId=426a4a51-1af3-4019-9769-4a58d8ece426, topVer=162,
cfg=LazyServiceConfigurati
on [srvcClsName=com.sbt.dpl.gridgain.thread.DPLThreadManager, svcCls=,
nodeFilterCls=IgniteAllNodesPredicate], assigns=HashMap
{74979bc7-e4c3-424a-8347-2f3a589aca3e=1,
32afc0a1-156d-4998-9b64-2336b86fb1c2=1, 6b34e59e-924c-404b-9451-ddf4b8b935b5=1,
c2af8947-1a
8d-48d5-93ca-165f97399519=1, 12f327e6-0f5f-44e2-be2c-f7bf99d22eec=1,
5638cb8c-4abb-49e3-8eef-edb3d5ccad77=1, 20349aca-ecf9-4a5f-bce9-2640e54fbbb2=1,
e83f5de6-deaf-4f60-af5c-5a13d4f251a7=1, 2b4261be-feb5-4d59-be2a-6c9fcbe2fa4a=1,
c3f0bff0-08fc-4601-b9f9-331929
065c0a=1, bac0156c-a56b-4ab8-aa7c-7d9878151e9f=1,
ccc2d442-8df4-402e-8589-d8ec3c6ec243=1, e34256a2-1bb9-4a17-86d1-21532833dded=1,
50bae4c5-16a5-48bf-a3f6-aa1b123074af=1, c5ecce59-cf6f-4be3-9861-e5c2622480a5=1,
d95ad91e-abd6-4c59-bc02-6298278f84c5=1, 035787c0-
4497-4682-9488-9be55e875175=1, e17bd18f-4a71-47f5-bdd6-64199a9bfb3a=1,
e3565de2-04a3-4107-95d9-01cdd790838b=1, dd372a51-8239-4f8f-8eef-5d6f206e971e=1,
c7ff660d-a003-493b-9aac-a6f73ad46561=1, 426a4a51-1af3-4019-9769-4a58d8ece426=1,
107810a2-c04d-452c-b4b5-d61a
bf16272c=1, 99df5f9f-5bc9-4f6b-a538-25c097124f38=1,
93baebe8-c8fc-4e25-8b6a-17f925e67dce=1, 44f063cf-9ad5-4095-96a3-54e554ab9ca3=1,
d4f7a539-cc66-4d76-afd6-2d41f533c44e=1, 0be09f47-0a70-4589-a78c-6c9fb7393d43=1,
6094f19a-6754-4ce9-8892-a540a52cf775=1, 17ad38b
2-1ec5-4531-9fca-397acbfa4a98=1, 23f220b1-cddf-4ac8-8987-05ec33569855=1,
e7b59e5d-3102-4dda-9d84-95783d80940d=1, cd9fc4bb-b488-40f5-8a72-24ecf633e3b0=1,
76fd7993-7c53-4a59-b199-ece9ce6f1b32=1, 8aef872d-83cd-42f1-9641-612206f1d026=1,
158087d4-2ca2-4ee2-83be-7a
61e52c9aac=1, 59b4bfeb-7690-4844-b3b9-d7939d72c098=1,
16bd8778-33a1-4a06-a904-8792f9991921=1, 5198403c-ffb1-4b64-b523-202cc76aee59=1,
fd636af8-4dea-4df1-b6b1-099bd14ec8aa=1, aa2471f8-f5cd-445f-8b79-fa58e08783a7=1,
22b3ebec-8a3e-4320-8ccc-a0c968362222=1, b72fa
1b9-7547-452c-99d2-5b61b6f8ebec=1, c56babe1-1071-4d8e-9499-3fc211d375a0=1,
43ff2a2f-db9c-4f5e-b721-ef350451ca0a=1, 20a6ff76-53f6-4508-8e4c-c87359e625a8=1,
07377a77-24da-4de3-8993-595b6f77e199=1, 0923e9e8-b412-40ae-bbb6-c75e1923cbd8=1,
416e7265-7507-4d05-b444-
0118999146a1=1, cd12c8c4-d209-4e5b-b34e-9cf25523ee7d=1,
32552ec6-5a88-4ae6-a069-0d86594b7031=1, 89a4c1a0-33a1-40b2-a443-f15533bd13d7=1,
b6545e2e-6ee5-4c0f-8fd8-3448bc2ab546=1, ab1dedb8-919b-4373-af45-250f12c7a8af=1,
944d0054-4b92-4eae-80dc-b6a45fe415c8=1, b71
47304-51c3-4c2d-9fb7-45853b27b79f=1, 6e334bfa-05d7-4909-94fb-e812d0fd4c76=1,
a91104a0-e902-46e4-8725-47438b48b102=1, 5e5fb902-1c22-488a-bb25-87d10c8ccc0e=1,
297d27cc-4cd1-486f-a922-0a27808d8304=1, 5e069562-4ac7-48c6-8ce2-e6feb6be3d44=1,
7db65bfd-5478-4b3d-8d2
f-7f35696fe6fe=1, 66dbd7b0-c7b2-421e-bc11-edc3919e4a0c=1,
c83a8121-38e8-4409-8cdb-d4929bf4d0cf=1, f9b0d868-0a8e-41a5-9c45-73ae688ffc1e=1,
be79ee19-8cc4-467c-bb15-2b066c27d667=1, 9e54bf88-b439-4e2f-9f14-4dee1bb66a0d=1,
9eee3d1d-702e-4e02-916f-f21b4b5dc27a=1, e
347eecb-3489-4e0d-a7dc-420854b8b3e9=1, 9cdf5210-6624-4f2f-9314-3e6bf3b23587=1,
1e17c56a-5213-4a1b-b94b-4575a95a2c81=1, 1988168d-839c-471b-9a83-3c48f0a7447f=1,
835ed2f5-6fd4-43e4-84bd-504f5df0e301=1, 8852b77b-17ee-47a7-af2f-4d63babe970e=1,
d7b33e74-5683-40c4-9
83d-6e2661183022=1, 16997ebc-723f-4b26-a14e-c98c40191646=1,
00dd20b3-12b4-46fe-800c-3ec405d11d98=1, b08008de-2ad0-4777-8365-2c7db6b470f3=1,
2bba7299-41f5-4635-8896-c6995425796e=1, 9d9234fe-3a77-4623-b83f-4297193ddf04=1,
fbd450b0-b545-467c-acd6-2d4acf53f239=1,
c36dc0f8-7e04-43c0-b1af-8883c42128f2=1,
8da84ab8-1d64-41c1-b904-03348fec36c6=1, fce7c69c-60c7-4006-8d2e-7363309c05d8=1,
d7412fac-3f80-4c92-806d-cf10ae545a63=1, 7bafe6ad-d5fa-45d0-8c86-cef3d3d1bd3a=1,
ce614913-d061-4e6b-ab6d-58df42fdab80=1, d1b9e739-b957-43e9
-80d2-c2a984c48639=1, af32cdf5-01f7-4196-aea6-e07ed36de5aa=1,
6e64c0d8-2bcb-428c-b64b-bd3391763d4e=1, 9b980efd-edee-462b-adad-c667e4b4ee65=1,
205dddb8-defe-4717-93a3-05953ceb406d=1, dabfb14f-5b83-469b-9798-ab89b440f379=1,
b05053e8-dc96-42b8-99bd-a4918c950aed=
1, 2645af19-64fe-4b9d-bbba-a3d9202105be=1,
3bf0cb89-bdb4-47ff-81e7-ce67377d750b=1, 667bd106-cb20-429c-adec-07293f794db9=1,
4e7d90fb-b536-4ac0-83ca-036e151bf707=1, a0d5cf3b-7c1e-4ae6-a39c-b1c98a04ea8f=1,
625eccc2-d955-4bf9-a293-f53b945b0f09=1... and 28 more}],
hasValBytes=true][op=UPDATE, val=], prevVal=[op=NOOP, val=null],
oldVal=CacheObjectImpl [val=null, hasValBytes=true][op=UPDATE, val=],
entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null,
explicitVer=null, dhtVer=null, filters=CacheEntry
Predicate[] [], filtersPassed=false, filtersSet=false, entry=GridCacheMapEntry
[key=KeyCacheObjectImpl [part=65, val=GridServiceAssignmentsKey
[name=DPLThreadManager_service], hasValBytes=true], val=CacheObjectImpl
[val=null, hasValBytes=true], startVer=15454
44833545, ver=GridCacheVersion [topVer=155571373, order=1544175423890,
nodeOrder=96], hash=1900127065, extras=GridCacheMvccEntryExtras
[mvcc=GridCacheMvcc [locs=null, rmts=LinkedList [GridCacheMvccCandidate
[nodeId=1e17c56a-5213-4a1b-b94b-4575a95a2c81, ver=Gr
idCacheVersion [topVer=155571374, order=1545423626612, nodeOrder=96],
threadId=1937, id=210410, topVer=AffinityTopologyVersion [topVer=-1,
minorTopVer=0], reentry=null, otherNodeId=44d27930-80e5-4eb7-b377-8b07c02c2033,
otherVer=null, mappedDhtNodes=null, mapp
edNearNodes=null, ownerVer=null, serOrder=null, key=KeyCacheObjectImpl
[part=65, val=GridServiceAssignmentsKey [name=DPLThreadManager_service],
hasValBytes=true],
masks=local=0|owner=0|ready=0|reentry=0|used=0|tx=1|single_implicit=0|dht_local=0|near_local=0|r
emoved=0|read=0, prevVer=null, nextVer=null]]]],
flags=2]GridDistributedCacheEntry [super=]GridDhtCacheEntry [rdrs=ReaderId[]
[], part=65, super=], prepared=1, locked=false, nodeId=null, locMapped=false,
expiryPlc=null, transferExpiryPlc=false, flags=0, partU
pdateCntr=0, serReadVer=null, xidVer=null]], dhtVers=null, txSize=0, plc=5,
txState=null, flags=last|sys, super=GridDistributedBaseMessage
[ver=GridCacheVersion [topVer=155571374, order=1545423626612, nodeOrder=96],
committedVers=null, rolledbackVers=null, cn
t=0, super=GridCacheIdMessage [cacheId=0]]]]]
java.lang.IllegalStateException: Unable to find consistentId by UUID
[nodeId=51dc74ab-c989-4268-b850-ed69a24cca30, topVer=AffinityTopologyVersion
[topVer=162, minorTopVer=0]]
at
org.apache.ignite.internal.managers.discovery.ConsistentIdMapper.mapToCompactId(ConsistentIdMapper.java:62)
at
org.apache.ignite.internal.managers.discovery.ConsistentIdMapper.mapToCompactIds(ConsistentIdMapper.java:123)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxAdapter.state(IgniteTxAdapter.java:1142)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxAdapter.state(IgniteTxAdapter.java:993)
at
org.apache.ignite.internal.processors.cache.distributed.GridDistributedTxRemoteAdapter.prepareRemoteTx(GridDistributedTxRemoteAdapter.java:407)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.startRemoteTx(IgniteTxHandler.java:1759)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processDhtTxPrepareRequest(IgniteTxHandler.java:1121)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.access$400(IgniteTxHandler.java:101)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$5.apply(IgniteTxHandler.java:205)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$5.apply(IgniteTxHandler.java:203)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1061)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:586)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:385)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:311)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:300)
at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1556)
at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1184)
at
org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:125)
at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1091)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{code}
was:
after stopping a client
we see topology change and pme finish on the coordinator,
and at soon on another nodes we still don't see new topology, but have
Critical error resulting nodes failure
crd log
{code}
2018-12-06 15:55:23.660 [WARN
][disco-event-worker-#159%DPL_GRID%DplGridNodeName%][o.a.i.i.m.d.GridDiscoveryManager]
Node FAILED: ZookeeperClusterNode [id=979f03db-f858-44f6-8646-12034dfd5c93,
addrs=[10.116.206.1], order=129, loc=false, client=true]
2018-12-06 15:55:23.660 [INFO
][disco-event-worker-#159%DPL_GRID%DplGridNodeName%][o.a.i.i.m.d.GridDiscoveryManager]
Topology snapshot [ver=162, servers=128, clients=0, CPUs=7168,
offheap=140000.0GB, heap=4000.0GB]
2018-12-06 15:55:23.660 [INFO
][disco-event-worker-#159%DPL_GRID%DplGridNodeName%][o.a.i.i.m.d.GridDiscoveryManager]
^-- Node [id=44D27930-80E5-4EB7-B377-8B07C02C2033, clusterState=ACTIVE]
2018-12-06 15:55:23.660 [INFO
][zk-DPL_GRID%DplGridNodeName-EventThread][o.a.i.s.d.z.i.ZookeeperDiscoveryImpl]
Process alive nodes change [alives=128]
2018-12-06 15:55:23.661 [INFO
][disco-event-worker-#159%DPL_GRID%DplGridNodeName%][o.a.i.i.m.d.GridDiscoveryManager]
^-- Baseline [id=0, size=128, online=128, offline=0]
2018-12-06 15:55:23.661 [INFO
][disco-event-worker-#159%DPL_GRID%DplGridNodeName%][o.a.i.i.m.d.GridDiscoveryManager]
Data Regions Configured:
2018-12-06 15:55:23.661 [INFO
][disco-event-worker-#159%DPL_GRID%DplGridNodeName%][o.a.i.i.m.d.GridDiscoveryManager]
^-- dpl_mem_plc [initSize=256.0 MiB, maxSize=556.6 GiB,
persistenceEnabled=true]
2018-12-06 15:55:23.661 [INFO
][disco-event-worker-#159%DPL_GRID%DplGridNodeName%][o.a.i.i.m.d.GridDiscoveryManager]
^-- not-persisted [initSize=256.0 MiB, maxSize=556.6 GiB,
persistenceEnabled=false]
2018-12-06 15:55:23.670
[DEBUG][sys-#564%DPL_GRID%DplGridNodeName%][o.a.i.i.p.c.d.d.p.l.ExchangeLatchManager]
Process node left 979f03db-f858-44f6-8646-12034dfd5c93
2018-12-06 15:55:23.670 [INFO
][exchange-worker-#160%DPL_GRID%DplGridNodeName%][o.a.ignite.internal.exchange.time]
Started exchange init [topVer=AffinityTopologyVersion [topVer=162,
minorTopVer=0], crd=true, evt=NODE_FAILED,
evtNode=979f03db-f858-44f6-8646-12034dfd5c93, customEvt=null, allowMerge=true]
2018-12-06 15:55:23.712 [INFO
][exchange-worker-#160%DPL_GRID%DplGridNodeName%][o.a.ignite.internal.exchange.time]
Finished exchange init [topVer=AffinityTopologyVersion [topVer=162,
minorTopVer=0], crd=true]
2018-12-06 15:55:23.699 [INFO
][exchange-worker-#160%DPL_GRID%DplGridNodeName%][o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture]
Finish exchange future [startVer=AffinityTopologyVersion [topVer=162,
minorTopVer=0], resVer=AffinityTopologyVersion [topVer=162, minorTopVer=0],
err=null]
{code}
on one we have critical error(1)
{code}
2018-12-06 15:55:23.727
[ERROR][utility-#432%DPL_GRID%DplGridNodeName%][o.a.i.i.p.cache.GridCacheIoManager]
Failed processing message [senderId=1e17c56a-5213-4a1b-b94b-4575a95a2c81,
msg=GridDhtTxPrepareRequest [nearNodeId=44d27930-80e5-4eb7-b377-8b07c02c2033,
futId=1d225238761-05eea259-5c25-4a4b-8469-9dd8980e218c, miniId=105,
topVer=AffinityTopologyVersion [topVer=162, minorTopVer=0],
invalidateNearEntries={}, nearWrites=null, owned=null,
nearXidVer=GridCacheVersion [topVer=155571374, order=1545423626166, nodeOrd
er=1], subjId=44d27930-80e5-4eb7-b377-8b07c02c2033, taskNameHash=0,
preloadKeys=null, skipCompletedVers=false,
super=GridDistributedTxPrepareRequest [threadId=1281, concurrency=PESSIMISTIC,
isolation=REPEATABLE_READ, writeVer=GridCacheVersion [topVer=15557137
4, order=1545423626614, nodeOrder=96], timeout=0, reads=null, writes=ArrayList
[IgniteTxEntry [key=KeyCacheObjectImpl [part=65, val=GridServiceAssignmentsKey
[name=DPLThreadManager_service], hasValBytes=true], cacheId=-2100569601,
txKey=IgniteTxKey [key=KeyCa
cheObjectImpl [part=65, val=GridServiceAssignmentsKey
[name=DPLThreadManager_service], hasValBytes=true], cacheId=-2100569601],
val=CacheObjectImpl [val=GridServiceAssignments
[nodeId=426a4a51-1af3-4019-9769-4a58d8ece426, topVer=162,
cfg=LazyServiceConfigurat
ion [srvcClsName=com.sbt.dpl.gridgain.thread.DPLThreadManager, svcCls=,
nodeFilterCls=IgniteAllNodesPredicate], assigns=HashMap
{74979bc7-e4c3-424a-8347-2f3a589aca3e=1,
32afc0a1-156d-4998-9b64-2336b86fb1c2=1, 6b34e59e-924c-404b-9451-ddf4b8b935b5=1,
c2af8947-1
a8d-48d5-93ca-165f97399519=1, 12f327e6-0f5f-44e2-be2c-f7bf99d22eec=1,
5638cb8c-4abb-49e3-8eef-edb3d5ccad77=1, 20349aca-ecf9-4a5f-bce9-2640e54fbbb2=1,
e83f5de6-deaf-4f60-af5c-5a13d4f251a7=1, 2b4261be-feb5-4d59-be2a-6c9fcbe2fa4a=1,
c3f0bff0-08fc-4601-b9f9-33192
9065c0a=1, bac0156c-a56b-4ab8-aa7c-7d9878151e9f=1,
ccc2d442-8df4-402e-8589-d8ec3c6ec243=1, e34256a2-1bb9-4a17-86d1-21532833dded=1,
50bae4c5-16a5-48bf-a3f6-aa1b123074af=1, c5ecce59-cf6f-4be3-9861-e5c2622480a5=1,
d95ad91e-abd6-4c59-bc02-6298278f84c5=1, 035787c0
-4497-4682-9488-9be55e875175=1, e17bd18f-4a71-47f5-bdd6-64199a9bfb3a=1,
e3565de2-04a3-4107-95d9-01cdd790838b=1, dd372a51-8239-4f8f-8eef-5d6f206e971e=1,
c7ff660d-a003-493b-9aac-a6f73ad46561=1, 426a4a51-1af3-4019-9769-4a58d8ece426=1,
107810a2-c04d-452c-b4b5-d61
abf16272c=1, 99df5f9f-5bc9-4f6b-a538-25c097124f38=1,
93baebe8-c8fc-4e25-8b6a-17f925e67dce=1, 44f063cf-9ad5-4095-96a3-54e554ab9ca3=1,
d4f7a539-cc66-4d76-afd6-2d41f533c44e=1, 0be09f47-0a70-4589-a78c-6c9fb7393d43=1,
6094f19a-6754-4ce9-8892-a540a52cf775=1, 17ad38
b2-1ec5-4531-9fca-397acbfa4a98=1, 23f220b1-cddf-4ac8-8987-05ec33569855=1,
e7b59e5d-3102-4dda-9d84-95783d80940d=1, cd9fc4bb-b488-40f5-8a72-24ecf633e3b0=1,
76fd7993-7c53-4a59-b199-ece9ce6f1b32=1, 8aef872d-83cd-42f1-9641-612206f1d026=1,
158087d4-2ca2-4ee2-83be-7
a61e52c9aac=1, 59b4bfeb-7690-4844-b3b9-d7939d72c098=1,
16bd8778-33a1-4a06-a904-8792f9991921=1, 5198403c-ffb1-4b64-b523-202cc76aee59=1,
fd636af8-4dea-4df1-b6b1-099bd14ec8aa=1, aa2471f8-f5cd-445f-8b79-fa58e08783a7=1,
22b3ebec-8a3e-4320-8ccc-a0c968362222=1, b72f
a1b9-7547-452c-99d2-5b61b6f8ebec=1, c56babe1-1071-4d8e-9499-3fc211d375a0=1,
43ff2a2f-db9c-4f5e-b721-ef350451ca0a=1, 20a6ff76-53f6-4508-8e4c-c87359e625a8=1,
07377a77-24da-4de3-8993-595b6f77e199=1, 0923e9e8-b412-40ae-bbb6-c75e1923cbd8=1,
416e7265-7507-4d05-b444
-0118999146a1=1, cd12c8c4-d209-4e5b-b34e-9cf25523ee7d=1,
32552ec6-5a88-4ae6-a069-0d86594b7031=1, 89a4c1a0-33a1-40b2-a443-f15533bd13d7=1,
b6545e2e-6ee5-4c0f-8fd8-3448bc2ab546=1, ab1dedb8-919b-4373-af45-250f12c7a8af=1,
944d0054-4b92-4eae-80dc-b6a45fe415c8=1, b7
147304-51c3-4c2d-9fb7-45853b27b79f=1, 6e334bfa-05d7-4909-94fb-e812d0fd4c76=1,
a91104a0-e902-46e4-8725-47438b48b102=1, 5e5fb902-1c22-488a-bb25-87d10c8ccc0e=1,
297d27cc-4cd1-486f-a922-0a27808d8304=1, 5e069562-4ac7-48c6-8ce2-e6feb6be3d44=1,
7db65bfd-5478-4b3d-8d
2f-7f35696fe6fe=1, 66dbd7b0-c7b2-421e-bc11-edc3919e4a0c=1,
c83a8121-38e8-4409-8cdb-d4929bf4d0cf=1, f9b0d868-0a8e-41a5-9c45-73ae688ffc1e=1,
be79ee19-8cc4-467c-bb15-2b066c27d667=1, 9e54bf88-b439-4e2f-9f14-4dee1bb66a0d=1,
9eee3d1d-702e-4e02-916f-f21b4b5dc27a=1,
e347eecb-3489-4e0d-a7dc-420854b8b3e9=1, 9cdf5210-6624-4f2f-9314-3e6bf3b23587=1,
1e17c56a-5213-4a1b-b94b-4575a95a2c81=1, 1988168d-839c-471b-9a83-3c48f0a7447f=1,
835ed2f5-6fd4-43e4-84bd-504f5df0e301=1, 8852b77b-17ee-47a7-af2f-4d63babe970e=1,
d7b33e74-5683-40c4-
983d-6e2661183022=1, 16997ebc-723f-4b26-a14e-c98c40191646=1,
00dd20b3-12b4-46fe-800c-3ec405d11d98=1, b08008de-2ad0-4777-8365-2c7db6b470f3=1,
2bba7299-41f5-4635-8896-c6995425796e=1, 9d9234fe-3a77-4623-b83f-4297193ddf04=1,
fbd450b0-b545-467c-acd6-2d4acf53f239=1
, c36dc0f8-7e04-43c0-b1af-8883c42128f2=1,
8da84ab8-1d64-41c1-b904-03348fec36c6=1, fce7c69c-60c7-4006-8d2e-7363309c05d8=1,
d7412fac-3f80-4c92-806d-cf10ae545a63=1, 7bafe6ad-d5fa-45d0-8c86-cef3d3d1bd3a=1,
ce614913-d061-4e6b-ab6d-58df42fdab80=1, d1b9e739-b957-43e
9-80d2-c2a984c48639=1, af32cdf5-01f7-4196-aea6-e07ed36de5aa=1,
6e64c0d8-2bcb-428c-b64b-bd3391763d4e=1, 9b980efd-edee-462b-adad-c667e4b4ee65=1,
205dddb8-defe-4717-93a3-05953ceb406d=1, dabfb14f-5b83-469b-9798-ab89b440f379=1,
b05053e8-dc96-42b8-99bd-a4918c950aed
=1, 2645af19-64fe-4b9d-bbba-a3d9202105be=1,
3bf0cb89-bdb4-47ff-81e7-ce67377d750b=1, 667bd106-cb20-429c-adec-07293f794db9=1,
4e7d90fb-b536-4ac0-83ca-036e151bf707=1, a0d5cf3b-7c1e-4ae6-a39c-b1c98a04ea8f=1,
625eccc2-d955-4bf9-a293-f53b945b0f09=1... and 28 more}]
, hasValBytes=true][op=UPDATE, val=], prevVal=[op=NOOP, val=null],
oldVal=CacheObjectImpl [val=null, hasValBytes=true][op=UPDATE, val=],
entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null,
explicitVer=null, dhtVer=null, filters=CacheEntr
yPredicate[] [], filtersPassed=false, filtersSet=false, entry=GridCacheMapEntry
[key=KeyCacheObjectImpl [part=65, val=GridServiceAssignmentsKey
[name=DPLThreadManager_service], hasValBytes=true], val=CacheObjectImpl
[val=null, hasValBytes=true], startVer=1545
426660336, ver=GridCacheVersion [topVer=155571373, order=1544175423890,
nodeOrder=96], hash=1900127065, extras=GridCacheMvccEntryExtras
[mvcc=GridCacheMvcc [locs=null, rmts=LinkedList [GridCacheMvccCandidate
[nodeId=1e17c56a-5213-4a1b-b94b-4575a95a2c81, ver=G
ridCacheVersion [topVer=155571374, order=1545423626612, nodeOrder=96],
threadId=1942, id=219868, topVer=AffinityTopologyVersion [topVer=-1,
minorTopVer=0], reentry=null, otherNodeId=44d27930-80e5-4eb7-b377-8b07c02c2033,
otherVer=null, mappedDhtNodes=null, map
pedNearNodes=null, ownerVer=null, serOrder=null, key=KeyCacheObjectImpl
[part=65, val=GridServiceAssignmentsKey [name=DPLThreadManager_service],
hasValBytes=true],
masks=local=0|owner=0|ready=0|reentry=0|used=0|tx=1|single_implicit=0|dht_local=0|near_local=0|
removed=0|read=0, prevVer=null, nextVer=null]]]],
flags=2]GridDistributedCacheEntry [super=]GridDhtCacheEntry [rdrs=ReaderId[]
[], part=65, super=], prepared=1, locked=false, nodeId=null, locMapped=false,
expiryPlc=null, transferExpiryPlc=false, flags=0, part
UpdateCntr=0, serReadVer=null, xidVer=null]], dhtVers=null, txSize=0, plc=5,
txState=null, flags=last|sys, super=GridDistributedBaseMessage
[ver=GridCacheVersion [topVer=155571374, order=1545423626612, nodeOrder=96],
committedVers=null, rolledbackVers=null, c
nt=0, super=GridCacheIdMessage [cacheId=0]]]]]
org.apache.ignite.IgniteException: Failed to resolve nodes topology
[cacheGrp=N/A, topVer=AffinityTopologyVersion [topVer=162, minorTopVer=0],
history=[AffinityTopologyVersion [topVer=161, minorTopVer=0]], snap=Snapshot
[topVer=AffinityTopologyVersion [topVer
=161, minorTopVer=0]], locNode=ZookeeperClusterNode
[id=51dc74ab-c989-4268-b850-ed69a24cca30, addrs=[10.116.206.28], order=161,
loc=true, client=false]]
at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.resolveDiscoCache(GridDiscoveryManager.java:2111)
at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.consistentId(GridDiscoveryManager.java:1950)
at
org.apache.ignite.internal.managers.discovery.ConsistentIdMapper.mapToCompactIds(ConsistentIdMapper.java:104)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxAdapter.state(IgniteTxAdapter.java:1142)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxAdapter.state(IgniteTxAdapter.java:993)
at
org.apache.ignite.internal.processors.cache.distributed.GridDistributedTxRemoteAdapter.prepareRemoteTx(GridDistributedTxRemoteAdapter.java:407)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.startRemoteTx(IgniteTxHandler.java:1759)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processDhtTxPrepareRequest(IgniteTxHandler.java:1121)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.access$400(IgniteTxHandler.java:101)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$5.apply(IgniteTxHandler.java:205)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$5.apply(IgniteTxHandler.java:203)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1061)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:586)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:385)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:311)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:300)
at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1556)
at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1184)
at
org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:125)
at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1091)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{code}
> Multiple server node failure after a client node stopping
> ---------------------------------------------------------
>
> Key: IGNITE-10589
> URL: https://issues.apache.org/jira/browse/IGNITE-10589
> Project: Ignite
> Issue Type: Bug
> Reporter: Sergey Kosarev
> Priority: Critical
> Attachments: 16_02.tar
>
>
> after stopping a client
> we see topology change and pme finish on the coordinator,
> and at soon on another nodes we still don't see new topology, but have
> Critical error resulting nodes failure
> crd log
> {code}
> 2018-12-06 15:55:23.660 [WARN
> ][disco-event-worker-#159%DPL_GRID%DplGridNodeName%][o.a.i.i.m.d.GridDiscoveryManager]
> Node FAILED: ZookeeperClusterNode [id=979f03db-f858-44f6-8646-12034dfd5c93,
> addrs=[10.116.206.1], order=129, loc=false, client=true]
> 2018-12-06 15:55:23.660 [INFO
> ][disco-event-worker-#159%DPL_GRID%DplGridNodeName%][o.a.i.i.m.d.GridDiscoveryManager]
> Topology snapshot [ver=162, servers=128, clients=0, CPUs=7168,
> offheap=140000.0GB, heap=4000.0GB]
> 2018-12-06 15:55:23.660 [INFO
> ][disco-event-worker-#159%DPL_GRID%DplGridNodeName%][o.a.i.i.m.d.GridDiscoveryManager]
> ^-- Node [id=44D27930-80E5-4EB7-B377-8B07C02C2033, clusterState=ACTIVE]
> 2018-12-06 15:55:23.660 [INFO
> ][zk-DPL_GRID%DplGridNodeName-EventThread][o.a.i.s.d.z.i.ZookeeperDiscoveryImpl]
> Process alive nodes change [alives=128]
> 2018-12-06 15:55:23.661 [INFO
> ][disco-event-worker-#159%DPL_GRID%DplGridNodeName%][o.a.i.i.m.d.GridDiscoveryManager]
> ^-- Baseline [id=0, size=128, online=128, offline=0]
> 2018-12-06 15:55:23.661 [INFO
> ][disco-event-worker-#159%DPL_GRID%DplGridNodeName%][o.a.i.i.m.d.GridDiscoveryManager]
> Data Regions Configured:
> 2018-12-06 15:55:23.661 [INFO
> ][disco-event-worker-#159%DPL_GRID%DplGridNodeName%][o.a.i.i.m.d.GridDiscoveryManager]
> ^-- dpl_mem_plc [initSize=256.0 MiB, maxSize=556.6 GiB,
> persistenceEnabled=true]
> 2018-12-06 15:55:23.661 [INFO
> ][disco-event-worker-#159%DPL_GRID%DplGridNodeName%][o.a.i.i.m.d.GridDiscoveryManager]
> ^-- not-persisted [initSize=256.0 MiB, maxSize=556.6 GiB,
> persistenceEnabled=false]
> 2018-12-06 15:55:23.670
> [DEBUG][sys-#564%DPL_GRID%DplGridNodeName%][o.a.i.i.p.c.d.d.p.l.ExchangeLatchManager]
> Process node left 979f03db-f858-44f6-8646-12034dfd5c93
> 2018-12-06 15:55:23.670 [INFO
> ][exchange-worker-#160%DPL_GRID%DplGridNodeName%][o.a.ignite.internal.exchange.time]
> Started exchange init [topVer=AffinityTopologyVersion [topVer=162,
> minorTopVer=0], crd=true, evt=NODE_FAILED,
> evtNode=979f03db-f858-44f6-8646-12034dfd5c93, customEvt=null, allowMerge=true]
> 2018-12-06 15:55:23.712 [INFO
> ][exchange-worker-#160%DPL_GRID%DplGridNodeName%][o.a.ignite.internal.exchange.time]
> Finished exchange init [topVer=AffinityTopologyVersion [topVer=162,
> minorTopVer=0], crd=true]
> 2018-12-06 15:55:23.699 [INFO
> ][exchange-worker-#160%DPL_GRID%DplGridNodeName%][o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture]
> Finish exchange future [startVer=AffinityTopologyVersion [topVer=162,
> minorTopVer=0], resVer=AffinityTopologyVersion [topVer=162, minorTopVer=0],
> err=null]
> {code}
> on one we have critical error(1)
> {code}
> 2018-12-06 15:55:23.727
> [ERROR][utility-#432%DPL_GRID%DplGridNodeName%][o.a.i.i.p.cache.GridCacheIoManager]
> Failed processing message [senderId=1e17c56a-5213-4a1b-b94b-4575a95a2c81,
> msg=GridDhtTxPrepareRequest [nearNodeId=44d27930-80e5-4eb7-b377-8b07c02c2033,
> futId=1d225238761-05eea259-5c25-4a4b-8469-9dd8980e218c, miniId=105,
> topVer=AffinityTopologyVersion [topVer=162, minorTopVer=0],
> invalidateNearEntries={}, nearWrites=null, owned=null,
> nearXidVer=GridCacheVersion [topVer=155571374, order=1545423626166, nodeOrd
> er=1], subjId=44d27930-80e5-4eb7-b377-8b07c02c2033, taskNameHash=0,
> preloadKeys=null, skipCompletedVers=false,
> super=GridDistributedTxPrepareRequest [threadId=1281,
> concurrency=PESSIMISTIC, isolation=REPEATABLE_READ, writeVer=GridCacheVersion
> [topVer=15557137
> 4, order=1545423626614, nodeOrder=96], timeout=0, reads=null,
> writes=ArrayList [IgniteTxEntry [key=KeyCacheObjectImpl [part=65,
> val=GridServiceAssignmentsKey [name=DPLThreadManager_service],
> hasValBytes=true], cacheId=-2100569601, txKey=IgniteTxKey [key=KeyCa
> cheObjectImpl [part=65, val=GridServiceAssignmentsKey
> [name=DPLThreadManager_service], hasValBytes=true], cacheId=-2100569601],
> val=CacheObjectImpl [val=GridServiceAssignments
> [nodeId=426a4a51-1af3-4019-9769-4a58d8ece426, topVer=162,
> cfg=LazyServiceConfigurat
> ion [srvcClsName=com.sbt.dpl.gridgain.thread.DPLThreadManager, svcCls=,
> nodeFilterCls=IgniteAllNodesPredicate], assigns=HashMap
> {74979bc7-e4c3-424a-8347-2f3a589aca3e=1,
> 32afc0a1-156d-4998-9b64-2336b86fb1c2=1,
> 6b34e59e-924c-404b-9451-ddf4b8b935b5=1, c2af8947-1
> a8d-48d5-93ca-165f97399519=1, 12f327e6-0f5f-44e2-be2c-f7bf99d22eec=1,
> 5638cb8c-4abb-49e3-8eef-edb3d5ccad77=1,
> 20349aca-ecf9-4a5f-bce9-2640e54fbbb2=1,
> e83f5de6-deaf-4f60-af5c-5a13d4f251a7=1,
> 2b4261be-feb5-4d59-be2a-6c9fcbe2fa4a=1, c3f0bff0-08fc-4601-b9f9-33192
> 9065c0a=1, bac0156c-a56b-4ab8-aa7c-7d9878151e9f=1,
> ccc2d442-8df4-402e-8589-d8ec3c6ec243=1,
> e34256a2-1bb9-4a17-86d1-21532833dded=1,
> 50bae4c5-16a5-48bf-a3f6-aa1b123074af=1,
> c5ecce59-cf6f-4be3-9861-e5c2622480a5=1,
> d95ad91e-abd6-4c59-bc02-6298278f84c5=1, 035787c0
> -4497-4682-9488-9be55e875175=1, e17bd18f-4a71-47f5-bdd6-64199a9bfb3a=1,
> e3565de2-04a3-4107-95d9-01cdd790838b=1,
> dd372a51-8239-4f8f-8eef-5d6f206e971e=1,
> c7ff660d-a003-493b-9aac-a6f73ad46561=1,
> 426a4a51-1af3-4019-9769-4a58d8ece426=1, 107810a2-c04d-452c-b4b5-d61
> abf16272c=1, 99df5f9f-5bc9-4f6b-a538-25c097124f38=1,
> 93baebe8-c8fc-4e25-8b6a-17f925e67dce=1,
> 44f063cf-9ad5-4095-96a3-54e554ab9ca3=1,
> d4f7a539-cc66-4d76-afd6-2d41f533c44e=1,
> 0be09f47-0a70-4589-a78c-6c9fb7393d43=1,
> 6094f19a-6754-4ce9-8892-a540a52cf775=1, 17ad38
> b2-1ec5-4531-9fca-397acbfa4a98=1, 23f220b1-cddf-4ac8-8987-05ec33569855=1,
> e7b59e5d-3102-4dda-9d84-95783d80940d=1,
> cd9fc4bb-b488-40f5-8a72-24ecf633e3b0=1,
> 76fd7993-7c53-4a59-b199-ece9ce6f1b32=1,
> 8aef872d-83cd-42f1-9641-612206f1d026=1, 158087d4-2ca2-4ee2-83be-7
> a61e52c9aac=1, 59b4bfeb-7690-4844-b3b9-d7939d72c098=1,
> 16bd8778-33a1-4a06-a904-8792f9991921=1,
> 5198403c-ffb1-4b64-b523-202cc76aee59=1,
> fd636af8-4dea-4df1-b6b1-099bd14ec8aa=1,
> aa2471f8-f5cd-445f-8b79-fa58e08783a7=1,
> 22b3ebec-8a3e-4320-8ccc-a0c968362222=1, b72f
> a1b9-7547-452c-99d2-5b61b6f8ebec=1, c56babe1-1071-4d8e-9499-3fc211d375a0=1,
> 43ff2a2f-db9c-4f5e-b721-ef350451ca0a=1,
> 20a6ff76-53f6-4508-8e4c-c87359e625a8=1,
> 07377a77-24da-4de3-8993-595b6f77e199=1,
> 0923e9e8-b412-40ae-bbb6-c75e1923cbd8=1, 416e7265-7507-4d05-b444
> -0118999146a1=1, cd12c8c4-d209-4e5b-b34e-9cf25523ee7d=1,
> 32552ec6-5a88-4ae6-a069-0d86594b7031=1,
> 89a4c1a0-33a1-40b2-a443-f15533bd13d7=1,
> b6545e2e-6ee5-4c0f-8fd8-3448bc2ab546=1,
> ab1dedb8-919b-4373-af45-250f12c7a8af=1,
> 944d0054-4b92-4eae-80dc-b6a45fe415c8=1, b7
> 147304-51c3-4c2d-9fb7-45853b27b79f=1, 6e334bfa-05d7-4909-94fb-e812d0fd4c76=1,
> a91104a0-e902-46e4-8725-47438b48b102=1,
> 5e5fb902-1c22-488a-bb25-87d10c8ccc0e=1,
> 297d27cc-4cd1-486f-a922-0a27808d8304=1,
> 5e069562-4ac7-48c6-8ce2-e6feb6be3d44=1, 7db65bfd-5478-4b3d-8d
> 2f-7f35696fe6fe=1, 66dbd7b0-c7b2-421e-bc11-edc3919e4a0c=1,
> c83a8121-38e8-4409-8cdb-d4929bf4d0cf=1,
> f9b0d868-0a8e-41a5-9c45-73ae688ffc1e=1,
> be79ee19-8cc4-467c-bb15-2b066c27d667=1,
> 9e54bf88-b439-4e2f-9f14-4dee1bb66a0d=1,
> 9eee3d1d-702e-4e02-916f-f21b4b5dc27a=1,
> e347eecb-3489-4e0d-a7dc-420854b8b3e9=1,
> 9cdf5210-6624-4f2f-9314-3e6bf3b23587=1,
> 1e17c56a-5213-4a1b-b94b-4575a95a2c81=1,
> 1988168d-839c-471b-9a83-3c48f0a7447f=1,
> 835ed2f5-6fd4-43e4-84bd-504f5df0e301=1,
> 8852b77b-17ee-47a7-af2f-4d63babe970e=1, d7b33e74-5683-40c4-
> 983d-6e2661183022=1, 16997ebc-723f-4b26-a14e-c98c40191646=1,
> 00dd20b3-12b4-46fe-800c-3ec405d11d98=1,
> b08008de-2ad0-4777-8365-2c7db6b470f3=1,
> 2bba7299-41f5-4635-8896-c6995425796e=1,
> 9d9234fe-3a77-4623-b83f-4297193ddf04=1, fbd450b0-b545-467c-acd6-2d4acf53f239=1
> , c36dc0f8-7e04-43c0-b1af-8883c42128f2=1,
> 8da84ab8-1d64-41c1-b904-03348fec36c6=1,
> fce7c69c-60c7-4006-8d2e-7363309c05d8=1,
> d7412fac-3f80-4c92-806d-cf10ae545a63=1,
> 7bafe6ad-d5fa-45d0-8c86-cef3d3d1bd3a=1,
> ce614913-d061-4e6b-ab6d-58df42fdab80=1, d1b9e739-b957-43e
> 9-80d2-c2a984c48639=1, af32cdf5-01f7-4196-aea6-e07ed36de5aa=1,
> 6e64c0d8-2bcb-428c-b64b-bd3391763d4e=1,
> 9b980efd-edee-462b-adad-c667e4b4ee65=1,
> 205dddb8-defe-4717-93a3-05953ceb406d=1,
> dabfb14f-5b83-469b-9798-ab89b440f379=1, b05053e8-dc96-42b8-99bd-a4918c950aed
> =1, 2645af19-64fe-4b9d-bbba-a3d9202105be=1,
> 3bf0cb89-bdb4-47ff-81e7-ce67377d750b=1,
> 667bd106-cb20-429c-adec-07293f794db9=1,
> 4e7d90fb-b536-4ac0-83ca-036e151bf707=1,
> a0d5cf3b-7c1e-4ae6-a39c-b1c98a04ea8f=1,
> 625eccc2-d955-4bf9-a293-f53b945b0f09=1... and 28 more}]
> , hasValBytes=true][op=UPDATE, val=], prevVal=[op=NOOP, val=null],
> oldVal=CacheObjectImpl [val=null, hasValBytes=true][op=UPDATE, val=],
> entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null,
> explicitVer=null, dhtVer=null, filters=CacheEntr
> yPredicate[] [], filtersPassed=false, filtersSet=false,
> entry=GridCacheMapEntry [key=KeyCacheObjectImpl [part=65,
> val=GridServiceAssignmentsKey [name=DPLThreadManager_service],
> hasValBytes=true], val=CacheObjectImpl [val=null, hasValBytes=true],
> startVer=1545
> 426660336, ver=GridCacheVersion [topVer=155571373, order=1544175423890,
> nodeOrder=96], hash=1900127065, extras=GridCacheMvccEntryExtras
> [mvcc=GridCacheMvcc [locs=null, rmts=LinkedList [GridCacheMvccCandidate
> [nodeId=1e17c56a-5213-4a1b-b94b-4575a95a2c81, ver=G
> ridCacheVersion [topVer=155571374, order=1545423626612, nodeOrder=96],
> threadId=1942, id=219868, topVer=AffinityTopologyVersion [topVer=-1,
> minorTopVer=0], reentry=null,
> otherNodeId=44d27930-80e5-4eb7-b377-8b07c02c2033, otherVer=null,
> mappedDhtNodes=null, map
> pedNearNodes=null, ownerVer=null, serOrder=null, key=KeyCacheObjectImpl
> [part=65, val=GridServiceAssignmentsKey [name=DPLThreadManager_service],
> hasValBytes=true],
> masks=local=0|owner=0|ready=0|reentry=0|used=0|tx=1|single_implicit=0|dht_local=0|near_local=0|
> removed=0|read=0, prevVer=null, nextVer=null]]]],
> flags=2]GridDistributedCacheEntry [super=]GridDhtCacheEntry [rdrs=ReaderId[]
> [], part=65, super=], prepared=1, locked=false, nodeId=null, locMapped=false,
> expiryPlc=null, transferExpiryPlc=false, flags=0, part
> UpdateCntr=0, serReadVer=null, xidVer=null]], dhtVers=null, txSize=0, plc=5,
> txState=null, flags=last|sys, super=GridDistributedBaseMessage
> [ver=GridCacheVersion [topVer=155571374, order=1545423626612, nodeOrder=96],
> committedVers=null, rolledbackVers=null, c
> nt=0, super=GridCacheIdMessage [cacheId=0]]]]]
> org.apache.ignite.IgniteException: Failed to resolve nodes topology
> [cacheGrp=N/A, topVer=AffinityTopologyVersion [topVer=162, minorTopVer=0],
> history=[AffinityTopologyVersion [topVer=161, minorTopVer=0]], snap=Snapshot
> [topVer=AffinityTopologyVersion [topVer
> =161, minorTopVer=0]], locNode=ZookeeperClusterNode
> [id=51dc74ab-c989-4268-b850-ed69a24cca30, addrs=[10.116.206.28], order=161,
> loc=true, client=false]]
> at
> org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.resolveDiscoCache(GridDiscoveryManager.java:2111)
> at
> org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.consistentId(GridDiscoveryManager.java:1950)
> at
> org.apache.ignite.internal.managers.discovery.ConsistentIdMapper.mapToCompactIds(ConsistentIdMapper.java:104)
> at
> org.apache.ignite.internal.processors.cache.transactions.IgniteTxAdapter.state(IgniteTxAdapter.java:1142)
> at
> org.apache.ignite.internal.processors.cache.transactions.IgniteTxAdapter.state(IgniteTxAdapter.java:993)
> at
> org.apache.ignite.internal.processors.cache.distributed.GridDistributedTxRemoteAdapter.prepareRemoteTx(GridDistributedTxRemoteAdapter.java:407)
> at
> org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.startRemoteTx(IgniteTxHandler.java:1759)
> at
> org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processDhtTxPrepareRequest(IgniteTxHandler.java:1121)
> at
> org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.access$400(IgniteTxHandler.java:101)
> at
> org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$5.apply(IgniteTxHandler.java:205)
> at
> org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$5.apply(IgniteTxHandler.java:203)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1061)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:586)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:385)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:311)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:300)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1556)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1184)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:125)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1091)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> on many other nodes we have Critical error (2) with
> *nodeId=51dc74ab-c989-4268-b850-ed69a24cca30* of the node above and
> nearNodeid is CRD:
> {code}
> 2018-12-06 15:55:23.730
> [ERROR][utility-#386%DPL_GRID%DplGridNodeName%][o.a.i.i.p.cache.GridCacheIoManager]
> Failed processing message [senderId=1e17c56a-5213-4a1b-b94b-4575a95a2c81,
> msg=GridDhtTxPrepareRequest [nearNodeId=44d27930-80e5-4eb7-b377-8b07c02c2033,
> futId=1d225238761-05eea259-5c25-4a4b-8469-9dd8980e218c, miniId=79,
> topVer=AffinityTopologyVersion [topVer=162, minorTopVer=0],
> invalidateNearEntries={}, nearWrites=null, owned=null,
> nearXidVer=GridCacheVersion [topVer=155571374, order=1545423626166, nodeOrde
> r=1], subjId=44d27930-80e5-4eb7-b377-8b07c02c2033, taskNameHash=0,
> preloadKeys=null, skipCompletedVers=false,
> super=GridDistributedTxPrepareRequest [threadId=1281,
> concurrency=PESSIMISTIC, isolation=REPEATABLE_READ, writeVer=GridCacheVersion
> [topVer=155571374
> , order=1545423626614, nodeOrder=96], timeout=0, reads=null, writes=ArrayList
> [IgniteTxEntry [key=KeyCacheObjectImpl [part=65,
> val=GridServiceAssignmentsKey [name=DPLThreadManager_service],
> hasValBytes=true], cacheId=-2100569601, txKey=IgniteTxKey [key=KeyCac
> heObjectImpl [part=65, val=GridServiceAssignmentsKey
> [name=DPLThreadManager_service], hasValBytes=true], cacheId=-2100569601],
> val=CacheObjectImpl [val=GridServiceAssignments
> [nodeId=426a4a51-1af3-4019-9769-4a58d8ece426, topVer=162,
> cfg=LazyServiceConfigurati
> on [srvcClsName=com.sbt.dpl.gridgain.thread.DPLThreadManager, svcCls=,
> nodeFilterCls=IgniteAllNodesPredicate], assigns=HashMap
> {74979bc7-e4c3-424a-8347-2f3a589aca3e=1,
> 32afc0a1-156d-4998-9b64-2336b86fb1c2=1,
> 6b34e59e-924c-404b-9451-ddf4b8b935b5=1, c2af8947-1a
> 8d-48d5-93ca-165f97399519=1, 12f327e6-0f5f-44e2-be2c-f7bf99d22eec=1,
> 5638cb8c-4abb-49e3-8eef-edb3d5ccad77=1,
> 20349aca-ecf9-4a5f-bce9-2640e54fbbb2=1,
> e83f5de6-deaf-4f60-af5c-5a13d4f251a7=1,
> 2b4261be-feb5-4d59-be2a-6c9fcbe2fa4a=1, c3f0bff0-08fc-4601-b9f9-331929
> 065c0a=1, bac0156c-a56b-4ab8-aa7c-7d9878151e9f=1,
> ccc2d442-8df4-402e-8589-d8ec3c6ec243=1,
> e34256a2-1bb9-4a17-86d1-21532833dded=1,
> 50bae4c5-16a5-48bf-a3f6-aa1b123074af=1,
> c5ecce59-cf6f-4be3-9861-e5c2622480a5=1,
> d95ad91e-abd6-4c59-bc02-6298278f84c5=1, 035787c0-
> 4497-4682-9488-9be55e875175=1, e17bd18f-4a71-47f5-bdd6-64199a9bfb3a=1,
> e3565de2-04a3-4107-95d9-01cdd790838b=1,
> dd372a51-8239-4f8f-8eef-5d6f206e971e=1,
> c7ff660d-a003-493b-9aac-a6f73ad46561=1,
> 426a4a51-1af3-4019-9769-4a58d8ece426=1, 107810a2-c04d-452c-b4b5-d61a
> bf16272c=1, 99df5f9f-5bc9-4f6b-a538-25c097124f38=1,
> 93baebe8-c8fc-4e25-8b6a-17f925e67dce=1,
> 44f063cf-9ad5-4095-96a3-54e554ab9ca3=1,
> d4f7a539-cc66-4d76-afd6-2d41f533c44e=1,
> 0be09f47-0a70-4589-a78c-6c9fb7393d43=1,
> 6094f19a-6754-4ce9-8892-a540a52cf775=1, 17ad38b
> 2-1ec5-4531-9fca-397acbfa4a98=1, 23f220b1-cddf-4ac8-8987-05ec33569855=1,
> e7b59e5d-3102-4dda-9d84-95783d80940d=1,
> cd9fc4bb-b488-40f5-8a72-24ecf633e3b0=1,
> 76fd7993-7c53-4a59-b199-ece9ce6f1b32=1,
> 8aef872d-83cd-42f1-9641-612206f1d026=1, 158087d4-2ca2-4ee2-83be-7a
> 61e52c9aac=1, 59b4bfeb-7690-4844-b3b9-d7939d72c098=1,
> 16bd8778-33a1-4a06-a904-8792f9991921=1,
> 5198403c-ffb1-4b64-b523-202cc76aee59=1,
> fd636af8-4dea-4df1-b6b1-099bd14ec8aa=1,
> aa2471f8-f5cd-445f-8b79-fa58e08783a7=1,
> 22b3ebec-8a3e-4320-8ccc-a0c968362222=1, b72fa
> 1b9-7547-452c-99d2-5b61b6f8ebec=1, c56babe1-1071-4d8e-9499-3fc211d375a0=1,
> 43ff2a2f-db9c-4f5e-b721-ef350451ca0a=1,
> 20a6ff76-53f6-4508-8e4c-c87359e625a8=1,
> 07377a77-24da-4de3-8993-595b6f77e199=1,
> 0923e9e8-b412-40ae-bbb6-c75e1923cbd8=1, 416e7265-7507-4d05-b444-
> 0118999146a1=1, cd12c8c4-d209-4e5b-b34e-9cf25523ee7d=1,
> 32552ec6-5a88-4ae6-a069-0d86594b7031=1,
> 89a4c1a0-33a1-40b2-a443-f15533bd13d7=1,
> b6545e2e-6ee5-4c0f-8fd8-3448bc2ab546=1,
> ab1dedb8-919b-4373-af45-250f12c7a8af=1,
> 944d0054-4b92-4eae-80dc-b6a45fe415c8=1, b71
> 47304-51c3-4c2d-9fb7-45853b27b79f=1, 6e334bfa-05d7-4909-94fb-e812d0fd4c76=1,
> a91104a0-e902-46e4-8725-47438b48b102=1,
> 5e5fb902-1c22-488a-bb25-87d10c8ccc0e=1,
> 297d27cc-4cd1-486f-a922-0a27808d8304=1,
> 5e069562-4ac7-48c6-8ce2-e6feb6be3d44=1, 7db65bfd-5478-4b3d-8d2
> f-7f35696fe6fe=1, 66dbd7b0-c7b2-421e-bc11-edc3919e4a0c=1,
> c83a8121-38e8-4409-8cdb-d4929bf4d0cf=1,
> f9b0d868-0a8e-41a5-9c45-73ae688ffc1e=1,
> be79ee19-8cc4-467c-bb15-2b066c27d667=1,
> 9e54bf88-b439-4e2f-9f14-4dee1bb66a0d=1,
> 9eee3d1d-702e-4e02-916f-f21b4b5dc27a=1, e
> 347eecb-3489-4e0d-a7dc-420854b8b3e9=1,
> 9cdf5210-6624-4f2f-9314-3e6bf3b23587=1,
> 1e17c56a-5213-4a1b-b94b-4575a95a2c81=1,
> 1988168d-839c-471b-9a83-3c48f0a7447f=1,
> 835ed2f5-6fd4-43e4-84bd-504f5df0e301=1,
> 8852b77b-17ee-47a7-af2f-4d63babe970e=1, d7b33e74-5683-40c4-9
> 83d-6e2661183022=1, 16997ebc-723f-4b26-a14e-c98c40191646=1,
> 00dd20b3-12b4-46fe-800c-3ec405d11d98=1,
> b08008de-2ad0-4777-8365-2c7db6b470f3=1,
> 2bba7299-41f5-4635-8896-c6995425796e=1,
> 9d9234fe-3a77-4623-b83f-4297193ddf04=1,
> fbd450b0-b545-467c-acd6-2d4acf53f239=1,
> c36dc0f8-7e04-43c0-b1af-8883c42128f2=1,
> 8da84ab8-1d64-41c1-b904-03348fec36c6=1,
> fce7c69c-60c7-4006-8d2e-7363309c05d8=1,
> d7412fac-3f80-4c92-806d-cf10ae545a63=1,
> 7bafe6ad-d5fa-45d0-8c86-cef3d3d1bd3a=1,
> ce614913-d061-4e6b-ab6d-58df42fdab80=1, d1b9e739-b957-43e9
> -80d2-c2a984c48639=1, af32cdf5-01f7-4196-aea6-e07ed36de5aa=1,
> 6e64c0d8-2bcb-428c-b64b-bd3391763d4e=1,
> 9b980efd-edee-462b-adad-c667e4b4ee65=1,
> 205dddb8-defe-4717-93a3-05953ceb406d=1,
> dabfb14f-5b83-469b-9798-ab89b440f379=1, b05053e8-dc96-42b8-99bd-a4918c950aed=
> 1, 2645af19-64fe-4b9d-bbba-a3d9202105be=1,
> 3bf0cb89-bdb4-47ff-81e7-ce67377d750b=1,
> 667bd106-cb20-429c-adec-07293f794db9=1,
> 4e7d90fb-b536-4ac0-83ca-036e151bf707=1,
> a0d5cf3b-7c1e-4ae6-a39c-b1c98a04ea8f=1,
> 625eccc2-d955-4bf9-a293-f53b945b0f09=1... and 28 more}],
> hasValBytes=true][op=UPDATE, val=], prevVal=[op=NOOP, val=null],
> oldVal=CacheObjectImpl [val=null, hasValBytes=true][op=UPDATE, val=],
> entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null,
> explicitVer=null, dhtVer=null, filters=CacheEntry
> Predicate[] [], filtersPassed=false, filtersSet=false,
> entry=GridCacheMapEntry [key=KeyCacheObjectImpl [part=65,
> val=GridServiceAssignmentsKey [name=DPLThreadManager_service],
> hasValBytes=true], val=CacheObjectImpl [val=null, hasValBytes=true],
> startVer=15454
> 44833545, ver=GridCacheVersion [topVer=155571373, order=1544175423890,
> nodeOrder=96], hash=1900127065, extras=GridCacheMvccEntryExtras
> [mvcc=GridCacheMvcc [locs=null, rmts=LinkedList [GridCacheMvccCandidate
> [nodeId=1e17c56a-5213-4a1b-b94b-4575a95a2c81, ver=Gr
> idCacheVersion [topVer=155571374, order=1545423626612, nodeOrder=96],
> threadId=1937, id=210410, topVer=AffinityTopologyVersion [topVer=-1,
> minorTopVer=0], reentry=null,
> otherNodeId=44d27930-80e5-4eb7-b377-8b07c02c2033, otherVer=null,
> mappedDhtNodes=null, mapp
> edNearNodes=null, ownerVer=null, serOrder=null, key=KeyCacheObjectImpl
> [part=65, val=GridServiceAssignmentsKey [name=DPLThreadManager_service],
> hasValBytes=true],
> masks=local=0|owner=0|ready=0|reentry=0|used=0|tx=1|single_implicit=0|dht_local=0|near_local=0|r
> emoved=0|read=0, prevVer=null, nextVer=null]]]],
> flags=2]GridDistributedCacheEntry [super=]GridDhtCacheEntry [rdrs=ReaderId[]
> [], part=65, super=], prepared=1, locked=false, nodeId=null, locMapped=false,
> expiryPlc=null, transferExpiryPlc=false, flags=0, partU
> pdateCntr=0, serReadVer=null, xidVer=null]], dhtVers=null, txSize=0, plc=5,
> txState=null, flags=last|sys, super=GridDistributedBaseMessage
> [ver=GridCacheVersion [topVer=155571374, order=1545423626612, nodeOrder=96],
> committedVers=null, rolledbackVers=null, cn
> t=0, super=GridCacheIdMessage [cacheId=0]]]]]
> java.lang.IllegalStateException: Unable to find consistentId by UUID
> [nodeId=51dc74ab-c989-4268-b850-ed69a24cca30, topVer=AffinityTopologyVersion
> [topVer=162, minorTopVer=0]]
> at
> org.apache.ignite.internal.managers.discovery.ConsistentIdMapper.mapToCompactId(ConsistentIdMapper.java:62)
> at
> org.apache.ignite.internal.managers.discovery.ConsistentIdMapper.mapToCompactIds(ConsistentIdMapper.java:123)
> at
> org.apache.ignite.internal.processors.cache.transactions.IgniteTxAdapter.state(IgniteTxAdapter.java:1142)
> at
> org.apache.ignite.internal.processors.cache.transactions.IgniteTxAdapter.state(IgniteTxAdapter.java:993)
> at
> org.apache.ignite.internal.processors.cache.distributed.GridDistributedTxRemoteAdapter.prepareRemoteTx(GridDistributedTxRemoteAdapter.java:407)
> at
> org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.startRemoteTx(IgniteTxHandler.java:1759)
> at
> org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processDhtTxPrepareRequest(IgniteTxHandler.java:1121)
> at
> org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.access$400(IgniteTxHandler.java:101)
> at
> org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$5.apply(IgniteTxHandler.java:205)
> at
> org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$5.apply(IgniteTxHandler.java:203)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1061)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:586)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:385)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:311)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:300)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1556)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1184)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:125)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1091)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
