Hello! This looks like a transactional deadlock. Is this operation a part of transaction? Are you locking keys in the same order?
Regards, -- Ilya Kasnacheev вс, 18 окт. 2020 г. в 09:35, Kamlesh Joshi <kamlesh.jo...@ril.com>: > Hi Igniters, > > > > Our PROD cluster was frozen due to deadlock as per below logs. Couldn't > find further info on this, whether is it due to putall/removeall ? for > which cache and keys or is it due to some other issue ? > > We are using putall and removeall but not doing any ordering while using > it. > > > > Could you please check logs, tell us if you find any reason for this > deadlock. If it is due to putall/removeall, how to get info for which cache > and keys. Please find cluster configuration below, > > > > *######IgniteConfiguration###################* > > *<bean class="org.apache.ignite.configuration.IgniteConfiguration">* > > * <property name="igniteInstanceName" > value="${IGNITE_INSTANCE_NAME}" />* > > * <property name="peerClassLoadingEnabled" value="true"/>* > > * <property name="metricsLogFrequency" value="0" />* > > * <property name="failureDetectionTimeout" value="60000" />* > > * <property name="clientFailureDetectionTimeout" value="240000" />* > > * <property name="networkTimeout" value="60000" />* > > > > > > *############### Cache Configuration ################* > > *<property name="name" value="DeviceDetailsCache" />* > > * <property name="cacheMode" value="PARTITIONED" />* > > * <property name="atomicityMode" value="ATOMIC" />* > > * <property name="backups" value="1" />* > > * <property name="statisticsEnabled" value="true"/>* > > > > *Below are logs,* > > > > *[2020-10-17T16:12:17,158][INFO > ][db-checkpoint-thread-#385%EDIFCustomerCC%][GridCacheDatabaseSharedManager] > Checkpoint started [checkpointId=53afef47-ff6e-494d-a448-e35cc45b4eff, > startPtr=FileWALPointer [idx=1928027, fileOff=3575743, len=63453], > checkpointLockWait=0ms, checkpointLockHoldTime=33ms, > walCpRecordFsyncDuration=5ms, pages=76968, reason='timeout']* > > *[2020-10-17T16:12:17,224][INFO > ][wal-file-archiver%EDIFCustomerCC-#360%EDIFCustomerCC%][FileWriteAheadLogManager] > Copied file > [src=/datastore1/wal/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000000000006.wal, > dst=/datastore1/archive/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000001928026.wal]* > > *[2020-10-17T16:12:18,318][INFO > ][db-checkpoint-thread-#385%EDIFCustomerCC%][GridCacheDatabaseSharedManager] > Checkpoint finished [cpId=53afef47-ff6e-494d-a448-e35cc45b4eff, > pages=76968, markPos=FileWALPointer [idx=1928027, fileOff=3575743, > len=63453], walSegmentsCleared=5, walSegmentsCovered=[1928022 - 1928026], > markDuration=67ms, pagesWrite=706ms, fsync=454ms, total=1227ms]* > > *[2020-10-17T16:12:48,251][INFO > ][wal-file-archiver%EDIFCustomerCC-#360%EDIFCustomerCC%][FileWriteAheadLogManager] > Starting to copy WAL segment [absIdx=1928027, segIdx=7, > origFile=/datastore1/wal/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000000000007.wal, > dstFile=/datastore1/archive/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000001928027.wal]* > > *[2020-10-17T16:12:48,355][INFO > ][wal-file-archiver%EDIFCustomerCC-#360%EDIFCustomerCC%][FileWriteAheadLogManager] > Copied file > [src=/datastore1/wal/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000000000007.wal, > dst=/datastore1/archive/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000001928027.wal]* > > *[2020-10-17T16:13:27,146][INFO > ][wal-file-archiver%EDIFCustomerCC-#360%EDIFCustomerCC%][FileWriteAheadLogManager] > Starting to copy WAL segment [absIdx=1928028, segIdx=8, > origFile=/datastore1/wal/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000000000008.wal, > dstFile=/datastore1/archive/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000001928028.wal]* > > *[2020-10-17T16:13:27,265][INFO > ][wal-file-archiver%EDIFCustomerCC-#360%EDIFCustomerCC%][FileWriteAheadLogManager] > Copied file > [src=/datastore1/wal/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000000000008.wal, > dst=/datastore1/archive/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000001928028.wal]* > > *[2020-10-17T16:14:06,135][INFO > ][wal-file-archiver%EDIFCustomerCC-#360%EDIFCustomerCC%][FileWriteAheadLogManager] > Starting to copy WAL segment [absIdx=1928029, segIdx=9, > origFile=/datastore1/wal/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000000000009.wal, > dstFile=/datastore1/archive/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000001928029.wal]* > > *[2020-10-17T16:14:06,250][INFO > ][wal-file-archiver%EDIFCustomerCC-#360%EDIFCustomerCC%][FileWriteAheadLogManager] > Copied file > [src=/datastore1/wal/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000000000009.wal, > dst=/datastore1/archive/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000001928029.wal]* > > *[2020-10-17T16:15:07,128][INFO > ][wal-file-archiver%EDIFCustomerCC-#360%EDIFCustomerCC%][FileWriteAheadLogManager] > Starting to copy WAL segment [absIdx=1928030, segIdx=0, > origFile=/datastore1/wal/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000000000000.wal, > dstFile=/datastore1/archive/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000001928030.wal]* > > *[2020-10-17T16:15:07,228][INFO > ][wal-file-archiver%EDIFCustomerCC-#360%EDIFCustomerCC%][FileWriteAheadLogManager] > Copied file > [src=/datastore1/wal/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000000000000.wal, > dst=/datastore1/archive/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000001928030.wal]* > > *[2020-10-17T16:15:24,264][WARN > ][grid-timeout-worker-#263%EDIFCustomerCC%][G] >>> Possible starvation in > striped pool.* > > * Thread name: sys-stripe-40-#41%EDIFCustomerCC%* > > * Queue: [Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, > topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, > msg=GridNearSingleGetRequest [futId=1602252009083, key=KeyCacheObjectImpl > [part=40, val=null, hasValBytes=true], flags=1, > topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], > subjId=236ba74c-3e77-43db-b06d-ae757784e7d6, taskNameHash=0, createTtl=-1, > accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage > [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, > skipOnTimeout=false, msg=GridNearAtomicFullUpdateRequest > [keys=[BinaryObjectImpl [arr= true, ctx=false, start=0]], > conflictTtls=null, conflictExpireTimes=null, expiryPlc=null, initSize=0, > filter=null, parent=GridNearAtomicAbstractUpdateRequest [res=null, > flags=keepBinary]]]], Message closure [msg=GridIoMessage [plc=2, > topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, > skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602250628085, > key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, > topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], > subjId=3495693d-2021-4b80-9322-97227ab02d7a, taskNameHash=0, createTtl=-1, > accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage > [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, > skipOnTimeout=false, msg=GridNearGetRequest > [futId=608dea33571-1d1d3064-dfa1-4587-9131-439530617f71, > miniId=808dea33571-1d1d3064-dfa1-4587-9131-439530617f71, ver=null, > keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956, > minorTopVer=0], subjId=a745cdf1-ff3f-4e0f-9c97-d46319342227, > taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message > closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, > ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest > [futId=1602251655252, key=BinaryObjectImpl [arr= true, ctx=false, start=0], > flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], > subjId=d31dfbf3-95a4-42c8-be02-99cc24d174bd, taskNameHash=0, createTtl=-1, > accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage > [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, > skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602901227431, > key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, > topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], > subjId=6b2fb754-9165-4d61-891f-5970e263c396, taskNameHash=0, createTtl=-1, > accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage > [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, > skipOnTimeout=false, msg=GridNearAtomicFullUpdateRequest > [keys=[BinaryObjectImpl [arr= true, ctx=false, start=0]], > conflictTtls=null, conflictExpireTimes=null, expiryPlc=null, initSize=0, > filter=null, parent=GridNearAtomicAbstractUpdateRequest [res=null, > flags=keepBinary]]]], Message closure [msg=GridIoMessage [plc=2, > topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, > skipOnTimeout=false, msg=GridNearAtomicFullUpdateRequest > [keys=[KeyCacheObjectImpl [part=40, val=null, hasValBytes=true]], > conflictTtls=null, conflictExpireTimes=null, expiryPlc=null, initSize=0, > filter=null, parent=GridNearAtomicAbstractUpdateRequest [res=null, > flags=keepBinary]]]], Message closure [msg=GridIoMessage [plc=2, > topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, > skipOnTimeout=false, msg=GridNearGetRequest > [futId=cb782ce0571-870075cf-b885-40be-a586-b862991e3798, > miniId=db782ce0571-870075cf-b885-40be-a586-b862991e3798, ver=null, > keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956, > minorTopVer=0], subjId=fb38f867-4fb3-448b-99db-1f20ca492598, > taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message > closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, > ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest > [futId=1602252831927, key=BinaryObjectImpl [arr= true, ctx=false, start=0], > flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], > subjId=e5667fd1-0d1d-466a-aafd-3d4daafccb6b, taskNameHash=0, createTtl=-1, > accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage > [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, > skipOnTimeout=false, msg=GridNearAtomicFullUpdateRequest > [keys=[BinaryObjectImpl [arr= true, ctx=false, start=0], BinaryObjectImpl > [arr= true, ctx=false, start=0]], conflictTtls=null, > conflictExpireTimes=null, expiryPlc=null, initSize=0, filter=null, > parent=GridNearAtomicAbstractUpdateRequest [res=null, flags=keepBinary]]]], > Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, > ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest > [futId=1602251821044, key=KeyCacheObjectImpl [part=40, val=null, > hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, > minorTopVer=0], subjId=376c2540-8cd8-4cfc-8a6d-6e71f22fd3c9, > taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message > closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, > ordered=false, timeout=0, skipOnTimeout=false, > msg=GridNearAtomicFullUpdateRequest [keys=[BinaryObjectImpl [arr= true, > ctx=false, start=0]], conflictTtls=null, conflictExpireTimes=null, > expiryPlc=null, initSize=0, filter=null, > parent=GridNearAtomicAbstractUpdateRequest [res=null, flags=keepBinary]]]], > Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, > ordered=false, timeout=0, skipOnTimeout=false, > msg=GridNearAtomicFullUpdateRequest [keys=[BinaryObjectImpl [arr= true, > ctx=false, start=0]], conflictTtls=null, conflictExpireTimes=null, > expiryPlc=null, initSize=0, filter=null, > parent=GridNearAtomicAbstractUpdateRequest [res=null, flags=keepBinary]]]], > Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, > ordered=false, timeout=0, skipOnTimeout=false, > msg=GridDhtAtomicDeferredUpdateResponse [futIds=GridLongList [idx=2, > arr=[142131082,142131083]]]]], Message closure [msg=GridIoMessage [plc=2, > topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, > skipOnTimeout=false, msg=GridNearGetRequest > [futId=398dea33571-1d1d3064-dfa1-4587-9131-439530617f71, > miniId=598dea33571-1d1d3064-dfa1-4587-9131-439530617f71, ver=null, > keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956, > minorTopVer=0], subjId=a745cdf1-ff3f-4e0f-9c97-d46319342227, > taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message > closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, > ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest > [futId=1602250628128, key=BinaryObjectImpl [arr= true, ctx=false, start=0], > flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], > subjId=3495693d-2021-4b80-9322-97227ab02d7a, taskNameHash=0, createTtl=-1, > accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage > [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, > skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602252442697, > key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, > topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], > subjId=cca2a7c6-0e27-40d3-8782-3b964b41629e, taskNameHash=0, createTtl=-1, > accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage > [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, > skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251824023, > key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, > topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], > subjId=6a3462e2-b41f-47e4-b34c-db11ce061138, taskNameHash=0, createTtl=-1, > accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage > [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, > skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602901227447, > key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, > topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], > subjId=6b2fb754-9165-4d61-891f-5970e263c396, taskNameHash=0, createTtl=-1, > accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage > [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, > skipOnTimeout=false, msg=GridNearGetRequest > [futId=d0c22e33571-e189a141-f952-4b63-a8ed-395f16b0a574, > miniId=01c22e33571-e189a141-f952-4b63-a8ed-395f16b0a574, ver=null, > keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956, > minorTopVer=0], subjId=1ac4a008-e359-487e-bd60-8bc4d101085c, > taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message > closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, > ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest > [futId=1602250628143, key=KeyCacheObjectImpl [part=40, val=null, > hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, > minorTopVer=0], subjId=3495693d-2021-4b80-9322-97227ab02d7a, > taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message > closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, > ordered=false, timeout=0, skipOnTimeout=false, > msg=GridNearAtomicFullUpdateRequest [keys=[BinaryObjectImpl [arr= true, > ctx=false, start=0]], conflictTtls=null, conflictExpireTimes=null, > expiryPlc=null, initSize=0, filter=null, > parent=GridNearAtomicAbstractUpdateRequest [res=null, flags=keepBinary]]]], > Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, > ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearGetRequest > [futId=019dea33571-1d1d3064-dfa1-4587-9131-439530617f71, > miniId=119dea33571-1d1d3064-dfa1-4587-9131-439530617f71, ver=null, > keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956, > minorTopVer=0], subjId=a745cdf1-ff3f-4e0f-9c97-d46319342227, > taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message > closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, > ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest > [futId=1602250927127, key=KeyCacheObjectImpl [part=40, val=null, > hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, > minorTopVer=0], subjId=0498547b-c945-4c65-908a-6ac0ddf4dff8, > taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message > closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, > ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearGetRequest > [futId=d04e1e33571-1f7c74a2-aff6-434b-a347-d88f70581bb8, > miniId=e04e1e33571-1f7c74a2-aff6-434b-a347-d88f70581bb8, ver=null, > keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956, > minorTopVer=0], subjId=68fac742-6ea7-4bb3-9d27-6c626cd4a009, > taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message > closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, > ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest > [futId=1602889898269, key=KeyCacheObjectImpl [part=40, val=null, > hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, > minorTopVer=0], subjId=a745cdf1-ff3f-4e0f-9c97-d46319342227, > taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message > closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, > ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest > [futId=1602125222995, key=BinaryObjectImpl [arr= true, ctx=false, start=0], > flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], > subjId=9afeae4c-ae1f-47a5-b6e6-032536db894e, taskNameHash=0, createTtl=-1, > accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage > [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, > skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602250632521, > key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, > topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], > subjId=41a03e2c-88fa-4ab1-9555-4774118b82a5, taskNameHash=0, createTtl=-1, > accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage > [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, > skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602901232835, > key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, > topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], > subjId=e01970ae-87af-4602-b303-7a75c2e64aa7, taskNameHash=0, createTtl=-1, > accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage > [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, > skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602124902656, > key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, > topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], > subjId=8fff54b3-f00a-4e4e-8421-b81c23b7919e, taskNameHash=0, createTtl=-1, > accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage > [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, > skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602137511901, > key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, > topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], > subjId=b00afce2-12e7-41b1-9147-6874d7bdca3e, taskNameHash=0, createTtl=-1, > accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage > [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, > skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602252085756, > key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, > topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], > subjId=2a3e1fb1-37cc-4c16-ace5-34f9d75e8034, taskNameHash=0, createTtl=-1, > accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage > [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, > skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602252442733, > key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, > topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], > subjId=cca2a7c6-0e27-40d3-8782-3b964b41629e, taskNameHash=0, createTtl=-1, > accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage > [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, > skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602252434062, > key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, > topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], > subjId=a9252c27-1c4d-4f64-90cb-5b59a7f03eb9, taskNameHash=0, createTtl=-1, > accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage > [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, > skipOnTimeout=false, msg=GridNearGetRequest > [futId=14061b33571-81fe5115-bee4-4ff8-8dc0-3b38d25cca03, > miniId=24061b33571-81fe5115-bee4-4ff8-8dc0-3b38d25cca03, ver=null, > keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956, > minorTopVer=0], subjId=908a1300-c06b-4349-a94a-e2e4ca836cb0, > taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message > closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, > ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest > [futId=1602251967622, key=KeyCacheObjectImpl [part=40, val=null, > hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, > minorTopVer=0], subjId=921cd171-9236-4257-9bf0-81ffd57d8555, > taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message > closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, > ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearGetRequest > [futId=333c0b33571-bfe09cc5-6435-41be-a319-d4e6f5999644, > miniId=433c0b33571-bfe09cc5-6435-41be-a319-d4e6f5999644, ver=null, > keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956, > minorTopVer=0], subjId=1e50ef76-cb94-4ffd-864d-29103600d567, > taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message > closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, > ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest > [futId=1602251239976, key=KeyCacheObjectImpl [part=40, val=null, > hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, > minorTopVer=0], subjId=ac6bc567-29d3-40c2-8f62-b63f3e8edba6, > taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message > closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, > ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest > [futId=1602901308734, key=BinaryObjectImpl [arr= true, ctx=false, start=0], > flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], > subjId=6bbfc872-5022-48dc-b4a6-f8dabec35cf0, taskNameHash=0, createTtl=-1, > accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage > [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, > skipOnTimeout=false, msg=GridNearAtomicFullUpdateRequest > [keys=[BinaryObjectImpl [arr= true, ctx=false, start=0]], > conflictTtls=null, conflictExpireTimes=null, expiryPlc=null, initSize=0, > filter=null, parent=GridNearAtomicAbstractUpdateRequest [res=null, > flags=keepBinary]]]], Message closure [msg=GridIoMessage [plc=2, > topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, > skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602250879685, > key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, > topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], > subjId=7da2bd3b-fb71-4d3f-a634-8fa9c6d60d14, taskNameHash=0, createTtl=-1, > accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage > [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, > skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602252442873, > key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, > topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], > subjId=cca2a7c6-0e27-40d3-8782-3b964b41629e, taskNameHash=0, createTtl=-1, > accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage > [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, > skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602250628186, > key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, > topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], > subjId=3495693d-2021-4b80-9322-97227ab02d7a, taskNameHash=0, createTtl=-1, > accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage > [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, > skipOnTimeout=false, msg=GridNearAtomicFullUpdateRequest > [keys=[BinaryObjectImpl [arr= true, ctx=false, start=0]], > conflictTtls=null, conflictExpireTimes=null, expiryPlc=null, initSize=0, > filter=null, parent=GridNearAtomicAbstractUpdateRequest [res=null, > flags=keepBinary]]]], Message closure [msg=GridIoMessage [plc=2, > topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, > skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251824253, > key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, > topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], > subjId=6a3462e2-b41f-47e4-b34c-db11ce061138, taskNameHash=0, createTtl=-1, > accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage > [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, > skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602901233022, > key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, > topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], > subjId=e01970ae-87af-4602-b303-7a75c2e64aa7, taskNameHash=0, createTtl=-1, > accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage > [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, > skipOnTimeout=false, msg=GridNearGetRequest > [futId=34c22e33571-e189a141-f952-4b63-a8ed-395f16b0a574, > miniId=64c22e33571-e189a141-f952-4b63-a8ed-395f16b0a574, ver=null, > keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956, > minorTopVer=0], subjId=1ac4a008-e359-487e-bd60-8bc4d101085c, > taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message > closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, > ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest > [futId=1602250414898, key=KeyCacheObjectImpl [part=40, val=null, > hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, > minorTopVer=0], subjId=2a66a450-c90c-44e5-826b-ab0be7706e88, > taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message > closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, > ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearGetRequest > [futId=cab59a33571-7fc24c46-bc64-4e80-a4fc-6d6f02d8d8cd, > miniId=dab59a33571-7fc24c46-bc64-4e80-a4fc-6d6f02d8d8cd, ver=null, > keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956, > minorTopVer=0], subjId=bff99e14-d0a6-488b-9e5b-7ab3b2190382, > taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message > closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, > ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest > [futId=1602251821280, key=KeyCacheObjectImpl [part=40, val=null, > hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, > minorTopVer=0], subjId=376c2540-8cd8-4cfc-8a6d-6e71f22fd3c9, > taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message > closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, > ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest > [futId=1602251065338, key=BinaryObjectImpl [arr= true, ctx=false, start=0], > flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], > subjId=11395929-053b-4a13-b03f-c8fc670eb8fa, taskNameHash=0, createTtl=-1, > accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage > [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, > skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602123820250, > key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, > topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], > subjId=df729a44-6fd8-4dd1-b9bf-98d0aef3de75, taskNameHash=0, createTtl=-1, > accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage > [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, > skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602901376631, > key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, > topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], > subjId=a1377485-85fd-43ca-96b1-7b323e124662, taskNameHash=0, createTtl=-1, > accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage > [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, > skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251643645, > key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, > topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], > subjId=9cf24c88-db30-43c2-a8ed-2cf14c36f558, taskNameHash=0, createTtl=-1, > accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage > [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, > skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602123751037, > key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, > topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], > subjId=d2d8e1bf-43a9-47b0-bc91-97a9fb0fa498, taskNameHash=0, createTtl=-1, > accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage > [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, > skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602123619208, > key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, > topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], > subjId=beddfcbd-19e9-4a74-b87b-153ec35af598, taskNameHash=0, createTtl=-1, > accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage > [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, > skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602123751050, > key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, > topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], > subjId=d2d8e1bf-43a9-47b0-bc91-97a9fb0fa498, taskNameHash=0, createTtl=-1, > accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage > [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, > skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251821399, > key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, > topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], > subjId=376c2540-8cd8-4cfc-8a6d-6e71f22fd3c9, taskNameHash=0, createTtl=-1, > accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage > [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, > skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602901287062, > key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, > topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], > subjId=b5e6888c-3f7b-4cec-95e0-5b1acf2da830, taskNameHash=0, createTtl=-1, > accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage > [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, > skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251821413, > key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, > topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], > subjId=376c2540-8cd8-4cfc-8a6d-6e71f22fd3c9, taskNameHash=0, createTtl=-1, > accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage > [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, > skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251821422, > key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, > topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], > subjId=376c2540-8cd8-4cfc-8a6d-6e71f22fd3c9, taskNameHash=0, createTtl=-1, > accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage > [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, > skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602901315682, > key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, > topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], > subjId=0845ba78-0c30-4f97-952d-4c5b57ce8913, taskNameHash=0, createTtl=-1, > accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage > [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, > skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251655371, > key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, > topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], > subjId=d31dfbf3-95a4-42c8-be02-99cc24d174bd, taskNameHash=0, createTtl=-1, > accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage > [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, > skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602901303189, > key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, > topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], > subjId=7251566e-d022-4c5e-af08-60feaa3fb9ee, taskNameHash=0, createTtl=-1, > accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage > [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, > skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602252832012, > key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, > topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], > subjId=e5667fd1-0d1d-466a-aafd-3d4daafccb6b, taskNameHash=0, createTtl=-1, > accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage > [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, > skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251010602, > key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, > topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], > subjId=f94e3dca-bf5c-44df-bb18-87fc3a9a8ca0, taskNameHash=0, createTtl=-1, > accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage > [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, > skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251967763, > key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, > topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], > subjId=921cd171-9236-4257-9bf0-81ffd57d8555, taskNameHash=0, createTtl=-1, > accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage > [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, > skipOnTimeout=false, msg=GridNearGetRequest > [futId=276f3e33571-3bd539ba-2f6c-41d9-ba62-80ec8d57c2f8, > miniId=676f3e33571-3bd539ba-2f6c-41d9-ba62-80ec8d57c2f8, ver=null, > keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956, > minorTopVer=0], subjId=425f44e2-2600-427b-a47a-1823af73692d, > taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message > closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, > ordered=false, timeout=0, skipOnTimeout=false, > msg=GridDhtAtomicDeferredUpdateResponse [futIds=GridLongList [idx=2, > arr=[142346521,142346523]]]]], Message closure [msg=GridIoMessage [plc=2, > topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, > skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251821584, > key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, > topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], > subjId=376c2540-8cd8-4cfc-8a6d-6e71f22fd3c9, taskNameHash=0, createTtl=-1, > accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage > [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, > skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602252443040, > key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, > topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], > subjId=cca2a7c6-0e27-40d3-8782-3b964b41629e, taskNameHash=0, createTtl=-1, > accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage > [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, > skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602900084066, > key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, > topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], > subjId=2e8b3c45-2790-471a-9aa4-2afcdf1f78b5, taskNameHash=0, createTtl=-1, > accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage > [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, > skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602136587142, > key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, > topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], > subjId=c2c02269-cec2-4f96-acdd-2a131a4b27fd, taskNameHash=0, createTtl=-1, > accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage > [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, > skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251177802, > key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, > topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], > subjId=db5b0c2d-48b5-444f-a59b-7850a2b6073c, taskNameHash=0, createTtl=-1, > accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage > [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, > skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602899966390, > key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, > topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], > subjId=6c04a7b6-8b10-4329-b751-92912eaea781, taskNameHash=0, createTtl=-1, > accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage > [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, > skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251968094, > key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, > topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], > subjId=921cd171-9236-4257-9bf0-81ffd57d8555, taskNameHash=0, createTtl=-1, > accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage > [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, > skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602123464026, > key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, > topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], > subjId=95dc0be1-d639-4920-a635-3b28075ce7e7, taskNameHash=0, createTtl=-1, > accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage > [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, > skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602900044341, > key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, > topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], > subjId=8787e0aa-ae20-4c7d-bb5b-c8be99c88a3d, taskNameHash=0, createTtl=-1, > accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage > [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, > skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602250880196, > key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, > topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], > subjId=7da2bd3b-fb71-4d3f-a634-8fa9c6d60d14, taskNameHash=0, createTtl=-1, > accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage > [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, > skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602901233568, > key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, > topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], > subjId=e01970ae-87af-4602-b303-7a75c2e64aa7, taskNameHash=0, createTtl=-1, > accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage > [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, > skipOnTimeout=false, msg=GridNearGetRequest > [futId=b53d3e33571-701b81e6-a49b-44df-81c4-a063a7d1e7a6, > miniId=c53d3e33571-701b81e6-a49b-44df-81c4-a063a7d1e7a6, ver=null, > keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956, > minorTopVer=0], subjId=22c72b15-5e2a-47eb-9949-1d5c7565ddfc, > taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message > closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, > ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearGetRequest > [futId=7d4a6243571-6fff9fa2-9496-41bd-bf6e-db2d52c4a2e4, > miniId=8d4a6243571-6fff9fa2-9496-41bd-bf6e-db2d52c4a2e4, ver=null, > keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956, > minorTopVer=0], subjId=e2dff823-0090-40bf-8aab-0a3b5bcf3611, > taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message > closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, > ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest > [futId=1602900044373, key=BinaryObjectImpl [arr= true, ctx=false, start=0], > flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], > subjId=8787e0aa-ae20-4c7d-bb5b-c8be99c88a3d, taskNameHash=0, createTtl=-1, > accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage > [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, > skipOnTimeout=false, msg=GridNearGetRequest > [futId=27337243571-cdc9b3fd-927b-4c38-b7ac-52840c0d42cc, > miniId=57337243571-cdc9b3fd-927b-4c38-b7ac-52840c0d42cc, ver=null, > keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956, > minorTopVer=0], subjId=69e1b4e3-5082-4317-8ce0-19c956be89c0, > taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message > closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, > ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest > [futId=1602252009770, key=BinaryObjectImpl [arr= true, ctx=false, start=0], > flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], > subjId=236ba74c-3e77-43db-b06d-ae757784e7d6, taskNameHash=0, createTtl=-1, > accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage > [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, > skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602250415308, > key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, > topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], > subjId=2a66a450-c90c-44e5-826b-ab0be7706e88, taskNameHash=0, createTtl=-1, > accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage > [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, > skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602901287470, > key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, > topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], > subjId=b5e6888c-3f7b-4cec-95e0-5b1acf2da830, taskNameHash=0, createTtl=-1, > accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage > [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, > skipOnTimeout=false, msg=GridNearGetRequest > [futId=3cdf3e33571-8b9a9b5e-83cf-469d-b424-4feba04aa2ed, > miniId=5cdf3e33571-8b9a9b5e-83cf-469d-b424-4feba04aa2ed, ver=null, > keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956, > minorTopVer=0], subjId=95c546c4-3c34-433e-97a5-a24911be098a, > taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message > closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, > ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearGetRequest > [futId=4f7f3e33571-3bd539ba-2f6c-41d9-ba62-80ec8d57c2f8, > miniId=5f7f3e33571-3bd539ba-2f6c-41d9-ba62-80ec8d57c2f8, ver=null, > keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956, > minorTopVer=0], subjId=425f44e2-2600-427b-a47a-1823af73692d, > taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message > closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, > ordered=false, timeout=0, skipOnTimeout=false, > msg=GridNearAtomicFullUpdateRequest [keys=[BinaryObjectImpl [arr= true, > ctx=false, start=0], BinaryObjectImpl [arr= true, ctx=false, start=0]], > conflictTtls=null, conflictExpireTimes=null, expiryPlc=null, initSize=0, > filter=null, parent=GridNearAtomicAbstractUpdateRequest [res=null, > flags=keepBinary]]]], Message closure [msg=GridIoMessage [plc=2, > topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, > skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602899937408, > key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, > topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], > subjId=f3c978a8-b2fd-443e-b1c0-9197c50521e3, taskNameHash=0, createTtl=-1, > accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage > [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, > skipOnTimeout=false, msg=GridDhtAtomicDeferredUpdateResponse > [futIds=GridLongList [idx=1, arr=[142346525]]]]], Message closure > [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, > timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest > [futId=1602901287655, key=KeyCacheObjectImpl [part=40, val=null, > hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, > minorTopVer=0], subjId=b5e6888c-3f7b-4cec-95e0-5b1acf2da830, > taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message > closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, > ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearGetRequest > [futId=9e2f3e33571-055b3332-1a13-40cf-8857-662219beebf6, > miniId=be2f3e33571-055b3332-1a13-40cf-8857-662219beebf6, ver=null, > keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956, > minorTopVer=0], subjId=4e7366a0-6ba8-4fad-afae-b412601c99b1, > taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message > closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, > ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest > [futId=1602901309460, key=KeyCacheObjectImpl [part=40, val=null, > hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, > minorTopVer=0], subjId=6bbfc872-5022-48dc-b4a6-f8dabec35cf0, > taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message > closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, > ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearGetRequest > [futId=0d2f1e33571-b536da48-556d-4c6b-be8f-817b8280fb82, > miniId=3d2f1e33571-b536da48-556d-4c6b-be8f-817b8280fb82, ver=null, > keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956, > minorTopVer=0], subjId=ce6f4bab-f303-4d87-80a6-c03343400c8e, > taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message > closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, > ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest > [futId=1602124011789, key=BinaryObjectImpl [arr= true, ctx=false, start=0], > flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], > subjId=0b85712f-70b0-4d18-ae92-cf7955ef4464, taskNameHash=0, createTtl=-1, > accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage > [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, > skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602252005183, > key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, > topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], > subjId=5a368e87-d5b1-4aa5-8fdd-cf2184f0a768, taskNameHash=0, createTtl=-1, > accessTtl=-1, mvccSnapshot=null]]] Message closure [msg=GridIoMessage > [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, > skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602250950411, > key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, > topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], > subjId=0498547b-c945-4c65-908a-6ac0ddf4dff8, taskNameHash=0, createTtl=-1, > accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage > [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, > skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251034391, > key=KeyCacheObjectImpl [part=120, val=null, hasValBytes=true], flags=1, > topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], > subjId=f94e3dca-bf5c-44df-bb18-87fc3a9a8ca0, taskNameHash=0, createTtl=-1, > accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage > [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, > skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602252175793, > key=KeyCacheObjectImpl [part=120, val=null, hasValBytes=true], flags=1, > topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], > subjId=3b156c84-9eba-4de0-9861-e4974e17fe1b, taskNameHash=0, createTtl=-1, > accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage > [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, > skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602124988181, > key=KeyCacheObjectImpl [part=120, val=null, hasValBytes=true], flags=1, > topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], > subjId=b33c781b-e83a-495c-9e72-5f9d4507a717, taskNameHash=0, createTtl=-1, > accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage > [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, > skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251529166, > key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, > topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], > subjId=9a3e3502-fa33-494b-a445-b6ec254e5d59, taskNameHash=0, createTtl=-1, > accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage > [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, > skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602124901688, > key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, > topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], > subjId=31f69255-0587-49b8-bf74-8d9941967eb4, taskNameHash=0, createTtl=-1, > accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage > [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, > skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251847453, > key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, > topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], > subjId=6a3462e2-b41f-47e4-b34c-db11ce061138, taskNameHash=0, createTtl=-1, > accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage > [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, > skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602889551081, > key=KeyCacheObjectImpl [part=120, val=null, hasValBytes=true], flags=1, > topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], > subjId=bff99e14-d0a6-488b-9e5b-7ab3b2190382, taskNameHash=0, createTtl=-1, > accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage > [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, > skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251845149, > key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, > topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], > subjId=376c2540-8cd8-4cfc-8a6d-6e71f22fd3c9, taskNameHash=0, createTtl=-1, > accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage > [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, > skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602123481727, > key=KeyCacheObjectImpl [part=120, val=null, hasValBytes=true], flags=1, > topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], > subjId=95dc0be1-d639-4920-a635-3b28075ce7e7, taskNameHash=0, createTtl=-1, > accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage > [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, > skipOnTimeout=false, msg=GridNearGetRequest > [futId=1085ba33571-e48b4a3d-7144-4e73-a0c6-f2c452b56cd7, > miniId=3085ba33571-e48b4a3d-7144-4e73-a0c6-f2c452b56cd7, ver=null, > keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956, > minorTopVer=0], subjId=1769deb3-fd7b-4b36-9707-1808245f6fe9, > taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message > closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, > ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest > [futId=1602251448830, key=KeyCacheObjectImpl [part=120, val=null, > hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, > minorTopVer=0], subjId=44b5b67c-da49-468b-96af-e302b07b6a16, > taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message > closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, > ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest > [futId=1602251201103, key=KeyCacheObjectImpl [part=120, val=null, > hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, > minorTopVer=0], subjId=db5b0c2d-48b5-444f-a59b-7850a2b6073c, > taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message > closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, > ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest > [futId=1602250596135, key=KeyCacheObjectImpl [part=120, val=null, > hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, > minorTopVer=0], subjId=653fbfe7-dd96-4896-9b63-9b5fd3aec77f, > taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message > closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, > ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest > [futId=1602125326244, key=BinaryObjectImpl [arr= true, ctx=false, start=0], > flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], > subjId=c55ebb6d-4539-44e2-b14f-24a9bb10f077, taskNameHash=0, createTtl=-1, > accessTtl=-1, mvccSnapshot=null]]]]* > > * Deadlock: true* > > * Completed: 58842256* > > *Thread [name="sys-stripe-120-#121%EDIFCustomerCC%", id=153, > state=WAITING, blockCnt=7120, waitCnt=58416728]* > > * Lock > [object=java.util.concurrent.locks.ReentrantLock$NonfairSync@793496f8, > ownerName=sys-stripe-40-#41%EDIFCustomerCC%, ownerId=73]* > > * at sun.misc.Unsafe.park(Native Method)* > > * at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)* > > * at > java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)* > > * at > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)* > > * at > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)* > > * at > java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)* > > * at > java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)* > > * at > o.a.i.i.processors.cache.GridCacheMapEntry.lockEntry(GridCacheMapEntry.java:4967)* > > * at > o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.lockEntries(GridDhtAtomicCache.java:2930)* > > * at > o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:1746)* > > * at > o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal(GridDhtAtomicCache.java:1668)* > > * at > o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processNearAtomicUpdateRequest(GridDhtAtomicCache.java:3138)* > > * at > o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$400(GridDhtAtomicCache.java:135)* > > * at > o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:271)* > > * at > o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:266)* > > * at > o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)* > > * at > o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)* > > * at > o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)* > > * at > o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)* > > * at > o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)* > > * at > o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)* > > * at > o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)* > > * at > o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)* > > * at > o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)* > > * at > o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)* > > * at > o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:505)* > > * at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)* > > * at java.lang.Thread.run(Thread.java:748)* > > > > *[2020-10-17T16:15:51,276][ERROR][disco-event-worker-#342%EDIFCustomerCC%][G] > Blocked system-critical thread has been detected. This can lead to > cluster-wide undefined behaviour [threadName=sys-stripe-120, > blockedFor=77s]* > > *[2020-10-17T16:15:51,276][WARN > ][disco-event-worker-#342%EDIFCustomerCC%][G] Thread > [name="sys-stripe-120-#121%EDIFCustomerCC%", id=153, state=WAITING, > blockCnt=7120, waitCnt=58416728]* > > * Lock > [object=java.util.concurrent.locks.ReentrantLock$NonfairSync@793496f8, > ownerName=sys-stripe-40-#41%EDIFCustomerCC%, ownerId=73]* > > > > *[2020-10-17T16:15:51,276][ERROR][disco-event-worker-#342%EDIFCustomerCC%][] > Critical system error detected. Will be handled accordingly to configured > handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, > super=AbstractFailureHandler [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED, > SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext > [type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker > [name=sys-stripe-120, igniteInstanceName=EDIFCustomerCC, finished=false, > heartbeatTs=1602931473886]]]* > > *org.apache.ignite.IgniteException: GridWorker [name=sys-stripe-120, > igniteInstanceName=EDIFCustomerCC, finished=false, > heartbeatTs=1602931473886]* > > * at > org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1831) > [ignite-core-2.7.6.jar:2.7.6]* > > * at > org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1826) > [ignite-core-2.7.6.jar:2.7.6]* > > * at > org.apache.ignite.internal.worker.WorkersRegistry.onIdle(WorkersRegistry.java:233) > [ignite-core-2.7.6.jar:2.7.6]* > > * at > org.apache.ignite.internal.util.worker.GridWorker.onIdle(GridWorker.java:297) > [ignite-core-2.7.6.jar:2.7.6]* > > * at > org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body(GridDiscoveryManager.java:2827) > [ignite-core-2.7.6.jar:2.7.6]* > > * at > org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) > [ignite-core-2.7.6.jar:2.7.6]* > > * at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]* > > *[2020-10-17T16:15:51,278][WARN > ][disco-event-worker-#342%EDIFCustomerCC%][FailureProcessor] Deadlocked > threads detected (see thread dump below) [deadlockedThreadsCnt=2]* > > *[2020-10-17T16:15:55,716][WARN > ][jvm-pause-detector-worker][IgniteKernal%EDIFCustomerCC] Possible too long > JVM pause: 4429 milliseconds.* > > *[2020-10-17T16:15:55,760][WARN > ][disco-event-worker-#342%EDIFCustomerCC%][FailureProcessor] Thread dump at > 2020/10/17 16:15:55 IST* > > *Thread [name="sys-#194498%EDIFCustomerCC%", id=382762, > state=TIMED_WAITING, blockCnt=0, waitCnt=1]* > > * Lock > [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@435ed2a3, > ownerName=null, ownerId=-1]* > > * at sun.misc.Unsafe.park(Native Method)* > > * at > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)* > > * at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)* > > * at > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)* > > * at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)* > > * at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)* > > * at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)* > > * at java.lang.Thread.run(Thread.java:748)* > > > > *Thread [name="sys-#194497%EDIFCustomerCC%", id=382761, > state=TIMED_WAITING, blockCnt=0, waitCnt=1]* > > * Lock > [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@435ed2a3, > ownerName=null, ownerId=-1]* > > * at sun.misc.Unsafe.park(Native Method)* > > * at > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)* > > * at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)* > > * at > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)* > > * at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)* > > * at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)* > > * at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)* > > * at java.lang.Thread.run(Thread.java:748)* > > > > > > *Thanks and Regards,* > > *Kamlesh Joshi* > > > > > "*Confidentiality Warning*: This message and any attachments are intended > only for the use of the intended recipient(s), are confidential and may be > privileged. If you are not the intended recipient, you are hereby notified > that any review, re-transmission, conversion to hard copy, copying, > circulation or other use of this message and any attachments is strictly > prohibited. If you are not the intended recipient, please notify the sender > immediately by return email and delete this message and any attachments > from your system. > > *Virus Warning:* Although the company has taken reasonable precautions to > ensure no viruses are present in this email. The company cannot accept > responsibility for any loss or damage arising from the use of this email or > attachment." >