[ 
https://issues.apache.org/jira/browse/IGNITE-1905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Denis Magda updated IGNITE-1905:
--------------------------------
    Attachment: ServerTest.java
                ClientTest.java

> High contention for CacheLockImpl causes AssertionErrors
> --------------------------------------------------------
>
>                 Key: IGNITE-1905
>                 URL: https://issues.apache.org/jira/browse/IGNITE-1905
>             Project: Ignite
>          Issue Type: Bug
>          Components: cache
>    Affects Versions: ignite-1.4
>         Environment: Windows 7
>            Reporter: Denis Magda
>            Priority: Critical
>             Fix For: 1.5
>
>         Attachments: ClientTest.java, ServerTest.java
>
>
> When multiple threads, running on the same client node, compete for 
> CacheLockImpl this leads to AssertionErrors.
> Pseudo-code snippet, that is called from multiple threads and causes the 
> assertions, looks like this:
> {noformat}
> boolean locked = lock.tryLock(100, TimeUnit.MILLISECONDS);
> if (locked)
>     lock.unlock();
> {noformat}
> Initially the issue was detected on ignite-1.4.
> In server's node logs the following assertion appears
> {noformat}
> java.lang.AssertionError
>       at 
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtLockFuture.addOwned(GridDhtLockFuture.java:958)
>       at 
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtLockFuture.map(GridDhtLockFuture.java:918)
>       at 
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtLockFuture.onOwnerChanged(GridDhtLockFuture.java:663)
>       at 
> org.apache.ignite.internal.processors.cache.GridCacheMvccManager$2.onOwnerChanged(GridCacheMvccManager.java:155)
>       at 
> org.apache.ignite.internal.processors.cache.distributed.GridDistributedCacheEntry.checkOwnerChanged(GridDistributedCacheEntry.java:810)
>       at 
> org.apache.ignite.internal.processors.cache.distributed.GridDistributedCacheEntry.readyLock(GridDistributedCacheEntry.java:516)
>       at 
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtLockFuture.readyLocks(GridDhtLockFuture.java:576)
>       at 
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtLockFuture.map(GridDhtLockFuture.java:764)
>       at 
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter.lockAllAsync(GridDhtTransactionalCacheAdapter.java:973)
>       at 
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter.processNearLockRequest(GridDhtTransactionalCacheAdapter.java:557)
>       at 
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter.access$000(GridDhtTransactionalCacheAdapter.java:88)
>       at 
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter$2.apply(GridDhtTransactionalCacheAdapter.java:132)
>       at 
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter$2.apply(GridDhtTransactionalCacheAdapter.java:130)
>       at 
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:580)
>       at 
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:280)
>       at 
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:198)
>       at 
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$000(GridCacheIoManager.java:77)
>       at 
> org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:160)
>       at 
> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:811)
>       at 
> org.apache.ignite.internal.managers.communication.GridIoManager.access$1500(GridIoManager.java:106)
>       at 
> org.apache.ignite.internal.managers.communication.GridIoManager$5.run(GridIoManager.java:774)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>       at java.lang.Thread.run(Thread.java:745)
> {noformat}
>                       
> In addition from time to time a client node also outputs a different kind of 
> assertion:
> {noformat}
> Exception in thread "ignite-#4%sys-null%" java.lang.AssertionError
>       at 
> org.apache.ignite.internal.processors.cache.GridCacheExplicitLockSpan.markOwned(GridCacheExplicitLockSpan.java:196)
>       at 
> org.apache.ignite.internal.processors.cache.GridCacheMvccManager.markExplicitOwner(GridCacheMvccManager.java:862)
>       at 
> org.apache.ignite.internal.processors.cache.distributed.dht.colocated.GridDhtColocatedLockFuture$MiniFuture.onResult(GridDhtColocatedLockFuture.java:1412)
>       at 
> org.apache.ignite.internal.processors.cache.distributed.dht.colocated.GridDhtColocatedLockFuture.onResult(GridDhtColocatedLockFuture.java:437)
>       at 
> org.apache.ignite.internal.processors.cache.distributed.dht.colocated.GridDhtColocatedCache.processLockResponse(GridDhtColocatedCache.java:888)
>       at 
> org.apache.ignite.internal.processors.cache.distributed.dht.colocated.GridDhtColocatedCache.access$100(GridDhtColocatedCache.java:80)
>       at 
> org.apache.ignite.internal.processors.cache.distributed.dht.colocated.GridDhtColocatedCache$3.apply(GridDhtColocatedCache.java:145)
>       at 
> org.apache.ignite.internal.processors.cache.distributed.dht.colocated.GridDhtColocatedCache$3.apply(GridDhtColocatedCache.java:143)
>       at 
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:580)
>       at 
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:280)
>       at 
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:198)
>       at 
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$000(GridCacheIoManager.java:77)
>       at 
> org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:160)
>       at 
> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:811)
>       at 
> org.apache.ignite.internal.managers.communication.GridIoManager.access$1500(GridIoManager.java:106)
>       at 
> org.apache.ignite.internal.managers.communication.GridIoManager$5.run(GridIoManager.java:774)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>       at java.lang.Thread.run(Thread.java:745)
> {noformat}
> Ran the test against ignite-1.5 and got a different kind of assertions on the 
> server node:
> {noformat}
> Exception in thread "ignite-#246%sys-null%" java.lang.AssertionError: Entry 
> does not own lock for tx [locNodeId=8f8c5147-2d8f-44e6-8da8-ef8a5bae1fd3, 
> entry=GridDhtColocatedCacheEntry [super=GridDhtCacheEntry [rdrs=[], 
> locPart=GridDhtLocalPartition [id=49, mapPubSize=5, 
> rmvQueue=GridCircularBuffer [sizeMask=511, idxGen=0], state=OWNING, 
> reservations=0, empty=false, createTime=11/13/2015 15:08:42, mapPubSize=5], 
> super=GridDistributedCacheEntry [super=GridCacheMapEntry 
> [key=KeyCacheObjectImpl [val=1, hasValBytes=true], val=CacheObjectImpl 
> [val=null, hasValBytes=true], startVer=1447412910273, ver=GridCacheVersion 
> [topVer=58892922, nodeOrderDrId=1, globalTime=0, order=0], hash=-320443512, 
> extras=GridCacheMvccEntryExtras [mvcc=GridCacheMvcc 
> [locs=[GridCacheMvccCandidate [nodeId=8f8c5147-2d8f-44e6-8da8-ef8a5bae1fd3, 
> ver=GridCacheVersion [topVer=58892924, nodeOrderDrId=3, 
> globalTime=1447412968535, order=1447412964154], timeout=100, 
> ts=1447412968537, threadId=101, id=2245, topVer=AffinityTopologyVersion 
> [topVer=3, minorTopVer=0], reentry=null, 
> otherNodeId=f35fe3b2-93e7-4da5-8c59-94325ccd971b, otherVer=GridCacheVersion 
> [topVer=58892924, nodeOrderDrId=3, globalTime=1447412968535, 
> order=1447412964154], mappedDhtNodes=[TcpDiscoveryNode 
> [id=ce646644-b983-4d81-ba40-c39a057b4cc5, addrs=[0:0:0:0:0:0:0:1, 10.0.0.18, 
> 127.0.0.1], sockAddrs=[/10.0.0.18:47501, /0:0:0:0:0:0:0:1:47501, 
> /10.0.0.18:47501, /127.0.0.1:47501], discPort=47501, order=2, intOrder=2, 
> lastExchangeTime=1447412934615, loc=false, ver=1.5.0#19700101-sha1:00000000, 
> isClient=false]], mappedNearNodes=null, ownerVer=GridCacheVersion 
> [topVer=58892924, nodeOrderDrId=3, globalTime=1447412968452, 
> order=1447412964076], serOrder=null, key=KeyCacheObjectImpl [val=1, 
> hasValBytes=true], 
> masks=local=1|owner=1|ready=1|reentry=0|used=0|tx=0|single_implicit=0|dht_local=1|near_local=0|removed=0,
>  prevVer=null, nextVer=null], GridCacheMvccCandidate 
> [nodeId=8f8c5147-2d8f-44e6-8da8-ef8a5bae1fd3, ver=GridCacheVersion 
> [topVer=58892924, nodeOrderDrId=3, globalTime=1447412968537, 
> order=1447412964156], timeout=100, ts=1447412968547, threadId=99, id=2246, 
> topVer=AffinityTopologyVersion [topVer=3, minorTopVer=0], reentry=null, 
> otherNodeId=f35fe3b2-93e7-4da5-8c59-94325ccd971b, otherVer=GridCacheVersion 
> [topVer=58892924, nodeOrderDrId=3, globalTime=1447412968537, 
> order=1447412964156], mappedDhtNodes=null, mappedNearNodes=null, 
> ownerVer=GridCacheVersion [topVer=58892924, nodeOrderDrId=3, 
> globalTime=1447412968454, order=1447412964078], serOrder=null, 
> key=KeyCacheObjectImpl [val=1, hasValBytes=true], 
> masks=local=1|owner=0|ready=1|reentry=0|used=0|tx=0|single_implicit=0|dht_local=1|near_local=0|removed=0,
>  prevVer=null, nextVer=null], GridCacheMvccCandidate 
> [nodeId=8f8c5147-2d8f-44e6-8da8-ef8a5bae1fd3, ver=GridCacheVersion 
> [topVer=58892924, nodeOrderDrId=3, globalTime=1447412968539, 
> order=1447412964158], timeout=100, ts=1447412968547, threadId=75, id=2247, 
> topVer=AffinityTopologyVersion [topVer=3, minorTopVer=0], reentry=null, 
> otherNodeId=f35fe3b2-93e7-4da5-8c59-94325ccd971b, otherVer=GridCacheVersion 
> [topVer=58892924, nodeOrderDrId=3, globalTime=1447412968539, 
> order=1447412964158], mappedDhtNodes=null, mappedNearNodes=null, 
> ownerVer=GridCacheVersion [topVer=58892924, nodeOrderDrId=3, 
> globalTime=1447412968456, order=1447412964080], serOrder=null, 
> key=KeyCacheObjectImpl [val=1, hasValBytes=true], 
> masks=local=1|owner=0|ready=1|reentry=0|used=0|tx=0|single_implicit=0|dht_local=1|near_local=0|removed=0,
>  prevVer=null, nextVer=null], GridCacheMvccCandidate 
> [nodeId=8f8c5147-2d8f-44e6-8da8-ef8a5bae1fd3, ver=GridCacheVersion 
> [topVer=58892924, nodeOrderDrId=3, globalTime=1447412968541, 
> order=1447412964160], timeout=100, ts=1447412968547, threadId=83, id=2248, 
> topVer=AffinityTopologyVersion [topVer=3, minorTopVer=0], reentry=null, 
> otherNodeId=f35fe3b2-93e7-4da5-8c59-94325ccd971b, otherVer=GridCacheVersion 
> [topVer=58892924, nodeOrderDrId=3, globalTime=1447412968541, 
> order=1447412964160], mappedDhtNodes=null, mappedNearNodes=null, 
> ownerVer=GridCacheVersion [topVer=58892924, nodeOrderDrId=3, 
> globalTime=1447412968458, order=1447412964082], serOrder=null, 
> key=KeyCacheObjectImpl [val=1, hasValBytes=true], 
> masks=local=1|owner=0|ready=1|reentry=0|used=0|tx=0|single_implicit=0|dht_local=1|near_local=0|removed=0,
>  prevVer=null, nextVer=null], GridCacheMvccCandidate 
> [nodeId=8f8c5147-2d8f-44e6-8da8-ef8a5bae1fd3, ver=GridCacheVersion 
> [topVer=58892924, nodeOrderDrId=3, globalTime=1447412968543, 
> order=1447412964162], timeout=100, ts=1447412968547, threadId=86, id=2249, 
> topVer=AffinityTopologyVersion [topVer=3, minorTopVer=0], reentry=null, 
> otherNodeId=f35fe3b2-93e7-4da5-8c59-94325ccd971b, otherVer=GridCacheVersion 
> [topVer=58892924, nodeOrderDrId=3, globalTime=1447412968543, 
> order=1447412964162], mappedDhtNodes=null, mappedNearNodes=null, 
> ownerVer=GridCacheVersion [topVer=58892924, nodeOrderDrId=3, 
> globalTime=1447412968461, order=1447412964084], serOrder=null, 
> key=KeyCacheObjectImpl [val=1, hasValBytes=true], 
> masks=local=1|owner=0|ready=1|reentry=0|used=0|tx=0|single_implicit=0|dht_local=1|near_local=0|removed=0,
>  prevVer=null, nextVer=null], GridCacheMvccCandidate 
> [nodeId=8f8c5147-2d8f-44e6-8da8-ef8a5bae1fd3, ver=GridCacheVersion 
> [topVer=58892924, nodeOrderDrId=3, globalTime=1447412968545, 
> order=1447412964164], timeout=100, ts=1447412968557, threadId=69, id=2250, 
> topVer=AffinityTopologyVersion [topVer=3, minorTopVer=0], reentry=null, 
> otherNodeId=f35fe3b2-93e7-4da5-8c59-94325ccd971b, otherVer=GridCacheVersion 
> [topVer=58892924, nodeOrderDrId=3, globalTime=1447412968545, 
> order=1447412964164], mappedDhtNodes=null, mappedNearNodes=null, 
> ownerVer=GridCacheVersion [topVer=58892924, nodeOrderDrId=3, 
> globalTime=1447412968463, order=1447412964086], serOrder=null, 
> key=KeyCacheObjectImpl [val=1, hasValBytes=true], 
> masks=local=1|owner=0|ready=1|reentry=0|used=0|tx=0|single_implicit=0|dht_local=1|near_local=0|removed=0,
>  prevVer=null, nextVer=null], GridCacheMvccCandidate 
> [nodeId=8f8c5147-2d8f-44e6-8da8-ef8a5bae1fd3, ver=GridCacheVersion 
> [topVer=58892924, nodeOrderDrId=3, globalTime=1447412968547, 
> order=1447412964166], timeout=100, ts=1447412968557, threadId=95, id=2251, 
> topVer=AffinityTopologyVersion [topVer=3, minorTopVer=0], reentry=null, 
> otherNodeId=f35fe3b2-93e7-4da5-8c59-94325ccd971b, otherVer=GridCacheVersion 
> [topVer=58892924, nodeOrderDrId=3, globalTime=1447412968547, 
> order=1447412964166], mappedDhtNodes=null, mappedNearNodes=null, 
> ownerVer=GridCacheVersion [topVer=58892924, nodeOrderDrId=3, 
> globalTime=1447412968465, order=1447412964088], serOrder=null, 
> key=KeyCacheObjectImpl [val=1, hasValBytes=true], 
> masks=local=1|owner=0|ready=1|reentry=0|used=0|tx=0|single_implicit=0|dht_local=1|near_local=0|removed=0,
>  prevVer=null, nextVer=null], GridCacheMvccCandidate 
> [nodeId=8f8c5147-2d8f-44e6-8da8-ef8a5bae1fd3, ver=GridCacheVersion 
> [topVer=58892924, nodeOrderDrId=3, globalTime=1447412968550, 
> order=1447412964168], timeout=100, ts=1447412968557, threadId=102, id=2252, 
> topVer=AffinityTopologyVersion [topVer=3, minorTopVer=0], reentry=null, 
> otherNodeId=f35fe3b2-93e7-4da5-8c59-94325ccd971b, otherVer=GridCacheVersion 
> [topVer=58892924, nodeOrderDrId=3, globalTime=1447412968550, 
> order=1447412964168], mappedDhtNodes=null, mappedNearNodes=null, 
> ownerVer=GridCacheVersion [topVer=58892924, nodeOrderDrId=3, 
> globalTime=1447412968467, order=1447412964090], serOrder=null, 
> key=KeyCacheObjectImpl [val=1, hasValBytes=true], 
> masks=local=1|owner=0|ready=1|reentry=0|used=0|tx=0|single_implicit=0|dht_local=1|near_local=0|removed=0,
>  prevVer=null, nextVer=null], GridCacheMvccCandidate 
> [nodeId=8f8c5147-2d8f-44e6-8da8-ef8a5bae1fd3, ver=GridCacheVersion 
> [topVer=58892924, nodeOrderDrId=3, globalTime=1447412968552, 
> order=1447412964170], timeout=100, ts=1447412968557, threadId=67, id=2253, 
> topVer=AffinityTopologyVersion [topVer=3, minorTopVer=0], reentry=null, 
> otherNodeId=f35fe3b2-93e7-4da5-8c59-94325ccd971b, otherVer=GridCacheVersion 
> [topVer=58892924, nodeOrderDrId=3, globalTime=1447412968552, 
> order=1447412964170], mappedDhtNodes=null, mappedNearNodes=null, 
> ownerVer=GridCacheVersion [topVer=58892924, nodeOrderDrId=3, 
> globalTime=1447412968470, order=1447412964092], serOrder=null, 
> key=KeyCacheObjectImpl [val=1, hasValBytes=true], 
> masks=local=1|owner=0|ready=1|reentry=0|used=0|tx=0|single_implicit=0|dht_local=1|near_local=0|removed=0,
>  prevVer=null, nextVer=null], GridCacheMvccCandidate 
> [nodeId=8f8c5147-2d8f-44e6-8da8-ef8a5bae1fd3, ver=GridCacheVersion 
> [topVer=58892924, nodeOrderDrId=3, globalTime=1447412968554, 
> order=1447412964172], timeout=100, ts=1447412968557, threadId=78, id=2254, 
> topVer=AffinityTopologyVersion [topVer=3, minorTopVer=0], reentry=null, 
> otherNodeId=f35fe3b2-93e7-4da5-8c59-94325ccd971b, otherVer=GridCacheVersion 
> [topVer=58892924, nodeOrderDrId=3, globalTime=1447412968554, 
> order=1447412964172], mappedDhtNodes=null, mappedNearNodes=null, 
> ownerVer=GridCacheVersion [topVer=58892924, nodeOrderDrId=3, 
> globalTime=1447412968472, order=1447412964094], serOrder=null, 
> key=KeyCacheObjectImpl [val=1, hasValBytes=true], 
> masks=local=1|owner=0|ready=1|reentry=0|used=0|tx=0|single_implicit=0|dht_local=1|near_local=0|removed=0,
>  prevVer=null, nextVer=null], GridCacheMvccCandidate 
> [nodeId=8f8c5147-2d8f-44e6-8da8-ef8a5bae1fd3, ver=GridCacheVersion 
> [topVer=58892924, nodeOrderDrId=3, globalTime=1447412968556, 
> order=1447412964174], timeout=100, ts=1447412968567, threadId=70, id=2255, 
> topVer=AffinityTopologyVersion [topVer=3, minorTopVer=0], reentry=null, 
> otherNodeId=f35fe3b2-93e7-4da5-8c59-94325ccd971b, otherVer=GridCacheVersion 
> [topVer=58892924, nodeOrderDrId=3, globalTime=1447412968556, 
> order=1447412964174], mappedDhtNodes=null, mappedNearNodes=null, 
> ownerVer=GridCacheVersion [topVer=58892924, nodeOrderDrId=3, 
> globalTime=1447412968474, order=1447412964096], serOrder=null, 
> key=KeyCacheObjectImpl [val=1, hasValBytes=true], 
> masks=local=1|owner=0|ready=1|reentry=0|used=0|tx=0|single_implicit=0|dht_local=1|near_local=0|removed=0,
>  prevVer=null, nextVer=null], GridCacheMvccCandidate 
> [nodeId=8f8c5147-2d8f-44e6-8da8-ef8a5bae1fd3, ver=GridCacheVersion 
> [topVer=58892924, nodeOrderDrId=3, globalTime=1447412968559, 
> order=1447412964176], timeout=100, ts=1447412968567, threadId=79, id=2256, 
> topVer=AffinityTopologyVersion [topVer=3, minorTopVer=0], reentry=null, 
> otherNodeId=f35fe3b2-93e7-4da5-8c59-94325ccd971b, otherVer=GridCacheVersion 
> [topVer=58892924, nodeOrderDrId=3, globalTime=1447412968559, 
> order=1447412964176], mappedDhtNodes=null, mappedNearNodes=null, 
> ownerVer=GridCacheVersion [topVer=58892924, nodeOrderDrId=3, 
> globalTime=1447412968476, order=1447412964098], serOrder=null, 
> key=KeyCacheObjectImpl [val=1, hasValBytes=true], 
> masks=local=1|owner=0|ready=1|reentry=0|used=0|tx=0|single_implicit=0|dht_local=1|near_local=0|removed=0,
>  prevVer=null, nextVer=null], GridCacheMvccCandidate 
> [nodeId=8f8c5147-2d8f-44e6-8da8-ef8a5bae1fd3, ver=GridCacheVersion 
> [topVer=58892924, nodeOrderDrId=3, globalTime=1447412968561, 
> order=1447412964178], timeout=100, ts=1447412968567, threadId=103, id=2257, 
> topVer=AffinityTopologyVersion [topVer=3, minorTopVer=0], reentry=null, 
> otherNodeId=f35fe3b2-93e7-4da5-8c59-94325ccd971b, otherVer=GridCacheVersion 
> [topVer=58892924, nodeOrderDrId=3, globalTime=1447412968561, 
> order=1447412964178], mappedDhtNodes=null, mappedNearNodes=null, 
> ownerVer=GridCacheVersion [topVer=58892924, nodeOrderDrId=3, 
> globalTime=1447412968479, order=1447412964100], serOrder=null, 
> key=KeyCacheObjectImpl [val=1, hasValBytes=true], 
> masks=local=1|owner=0|ready=1|reentry=0|used=0|tx=0|single_implicit=0|dht_local=1|near_local=0|removed=0,
>  prevVer=null, nextVer=null], GridCacheMvccCandidate 
> [nodeId=8f8c5147-2d8f-44e6-8da8-ef8a5bae1fd3, ver=GridCacheVersion 
> [topVer=58892924, nodeOrderDrId=3, globalTime=1447412968563, 
> order=1447412964180], timeout=100, ts=1447412968567, threadId=90, id=2258, 
> topVer=AffinityTopologyVersion [topVer=3, minorTopVer=0], reentry=null, 
> otherNodeId=f35fe3b2-93e7-4da5-8c59-94325ccd971b, otherVer=GridCacheVersion 
> [topVer=58892924, nodeOrderDrId=3, globalTime=1447412968563, 
> order=1447412964180], mappedDhtNodes=null, mappedNearNodes=null, 
> ownerVer=GridCacheVersion [topVer=58892924, nodeOrderDrId=3, 
> globalTime=1447412968481, order=1447412964102], serOrder=null, 
> key=KeyCacheObjectImpl [val=1, hasValBytes=true], 
> masks=local=1|owner=0|ready=1|reentry=0|used=0|tx=0|single_implicit=0|dht_local=1|near_local=0|removed=0,
>  prevVer=null, nextVer=null], GridCacheMvccCandidate 
> [nodeId=8f8c5147-2d8f-44e6-8da8-ef8a5bae1fd3, ver=GridCacheVersion 
> [topVer=58892924, nodeOrderDrId=3, globalTime=1447412968566, 
> order=1447412964182], timeout=100, ts=1447412968577, threadId=66, id=2259, 
> topVer=AffinityTopologyVersion [topVer=3, minorTopVer=0], reentry=null, 
> otherNodeId=f35fe3b2-93e7-4da5-8c59-94325ccd971b, otherVer=GridCacheVersion 
> [topVer=58892924, nodeOrderDrId=3, globalTime=1447412968566, 
> order=1447412964182], mappedDhtNodes=null, mappedNearNodes=null, 
> ownerVer=GridCacheVersion [topVer=58892924, nodeOrderDrId=3, 
> globalTime=1447412968483, order=1447412964104], serOrder=null, 
> key=KeyCacheObjectImpl [val=1, hasValBytes=true], 
> masks=local=1|owner=0|ready=1|reentry=0|used=0|tx=0|single_implicit=0|dht_local=1|near_local=0|removed=0,
>  prevVer=null, nextVer=null], GridCacheMvccCandidate 
> [nodeId=8f8c5147-2d8f-44e6-8da8-ef8a5bae1fd3, ver=GridCacheVersion 
> [topVer=58892924, nodeOrderDrId=3, globalTime=1447412968568, 
> order=1447412964184], timeout=100, ts=1447412968577, threadId=82, id=2260, 
> topVer=AffinityTopologyVersion [topVer=3, minorTopVer=0], reentry=null, 
> otherNodeId=f35fe3b2-93e7-4da5-8c59-94325ccd971b, otherVer=GridCacheVersion 
> [topVer=58892924, nodeOrderDrId=3, globalTime=1447412968568, 
> order=1447412964184], mappedDhtNodes=null, mappedNearNodes=null, 
> ownerVer=GridCacheVersion [topVer=58892924, nodeOrderDrId=3, 
> globalTime=1447412968485, order=1447412964106], serOrder=null, 
> key=KeyCacheObjectImpl [val=1, hasValBytes=true], 
> masks=local=1|owner=0|ready=1|reentry=0|used=0|tx=0|single_implicit=0|dht_local=1|near_local=0|removed=0,
>  prevVer=null, nextVer=null], GridCacheMvccCandidate 
> [nodeId=8f8c5147-2d8f-44e6-8da8-ef8a5bae1fd3, ver=GridCacheVersion 
> [topVer=58892924, nodeOrderDrId=3, globalTime=1447412968570, 
> order=1447412964186], timeout=100, ts=1447412968577, threadId=91, id=2261, 
> topVer=AffinityTopologyVersion [topVer=3, minorTopVer=0], reentry=null, 
> otherNodeId=f35fe3b2-93e7-4da5-8c59-94325ccd971b, otherVer=GridCacheVersion 
> [topVer=58892924, nodeOrderDrId=3, globalTime=1447412968570, 
> order=1447412964186], mappedDhtNodes=null, mappedNearNodes=null, 
> ownerVer=GridCacheVersion [topVer=58892924, nodeOrderDrId=3, 
> globalTime=1447412968487, order=1447412964108], serOrder=null, 
> key=KeyCacheObjectImpl [val=1, hasValBytes=true], 
> masks=local=1|owner=0|ready=1|reentry=0|used=0|tx=0|single_implicit=0|dht_local=1|near_local=0|removed=0,
>  prevVer=null, nextVer=null], GridCacheMvccCandidate 
> [nodeId=8f8c5147-2d8f-44e6-8da8-ef8a5bae1fd3, ver=GridCacheVersion 
> [topVer=58892924, nodeOrderDrId=3, globalTime=1447412968584, 
> order=1447412964188], timeout=100, ts=1447412968597, threadId=94, id=2262, 
> topVer=AffinityTopologyVersion [topVer=3, minorTopVer=0], reentry=null, 
> otherNodeId=f35fe3b2-93e7-4da5-8c59-94325ccd971b, otherVer=GridCacheVersion 
> [topVer=58892924, nodeOrderDrId=3, globalTime=1447412968584, 
> order=1447412964188], mappedDhtNodes=null, mappedNearNodes=null, 
> ownerVer=GridCacheVersion [topVer=58892924, nodeOrderDrId=3, 
> globalTime=1447412968494, order=1447412964114], serOrder=null, 
> key=KeyCacheObjectImpl [val=1, hasValBytes=true], 
> masks=local=1|owner=0|ready=1|reentry=0|used=0|tx=0|single_implicit=0|dht_local=1|near_local=0|removed=0,
>  prevVer=null, nextVer=null], GridCacheMvccCandidate 
> [nodeId=8f8c5147-2d8f-44e6-8da8-ef8a5bae1fd3, ver=GridCacheVersion 
> [topVer=58892924, nodeOrderDrId=3, globalTime=1447412968585, 
> order=1447412964189], timeout=100, ts=1447412968597, threadId=68, id=2263, 
> topVer=AffinityTopologyVersion [topVer=3, minorTopVer=0], reentry=null, 
> otherNodeId=f35fe3b2-93e7-4da5-8c59-94325ccd971b, otherVer=GridCacheVersion 
> [topVer=58892924, nodeOrderDrId=3, globalTime=1447412968585, 
> order=1447412964189], mappedDhtNodes=null, mappedNearNodes=null, 
> ownerVer=GridCacheVersion [topVer=58892924, nodeOrderDrId=3, 
> globalTime=1447412968494, order=1447412964114], serOrder=null, 
> key=KeyCacheObjectImpl [val=1, hasValBytes=true], 
> masks=local=1|owner=0|ready=1|reentry=0|used=0|tx=0|single_implicit=0|dht_local=1|near_local=0|removed=0,
>  prevVer=null, nextVer=null], GridCacheMvccCandidate 
> [nodeId=8f8c5147-2d8f-44e6-8da8-ef8a5bae1fd3, ver=GridCacheVersion 
> [topVer=58892924, nodeOrderDrId=3, globalTime=1447412968586, 
> order=1447412964192], timeout=100, ts=1447412968597, threadId=84, id=2264, 
> topVer=AffinityTopologyVersion [topVer=3, minorTopVer=0], reentry=null, 
> otherNodeId=f35fe3b2-93e7-4da5-8c59-94325ccd971b, otherVer=GridCacheVersion 
> [topVer=58892924, nodeOrderDrId=3, globalTime=1447412968586, 
> order=1447412964192], mappedDhtNodes=null, mappedNearNodes=null, 
> ownerVer=GridCacheVersion [topVer=58892924, nodeOrderDrId=3, 
> globalTime=1447412968494, order=1447412964114], serOrder=null, 
> key=KeyCacheObjectImpl [val=1, hasValBytes=true], 
> masks=local=1|owner=0|ready=1|reentry=0|used=0|tx=0|single_implicit=0|dht_local=1|near_local=0|removed=0,
>  prevVer=null, nextVer=null], GridCacheMvccCandidate 
> [nodeId=8f8c5147-2d8f-44e6-8da8-ef8a5bae1fd3, ver=GridCacheVersion 
> [topVer=58892924, nodeOrderDrId=3, globalTime=1447412968591, 
> order=1447412964194], timeout=100, ts=1447412968597, threadId=85, id=2265, 
> topVer=AffinityTopologyVersion [topVer=3, minorTopVer=0], reentry=null, 
> otherNodeId=f35fe3b2-93e7-4da5-8c59-94325ccd971b, otherVer=GridCacheVersion 
> [topVer=58892924, nodeOrderDrId=3, globalTime=1447412968591, 
> order=1447412964194], mappedDhtNodes=null, mappedNearNodes=null, 
> ownerVer=GridCacheVersion [topVer=58892924, nodeOrderDrId=3, 
> globalTime=1447412968497, order=1447412964116], serOrder=null, 
> key=KeyCacheObjectImpl [val=1, hasValBytes=true], 
> masks=local=1|owner=0|ready=1|reentry=0|used=0|tx=0|single_implicit=0|dht_local=1|near_local=0|removed=0,
>  prevVer=null, nextVer=null], GridCacheMvccCandidate 
> [nodeId=8f8c5147-2d8f-44e6-8da8-ef8a5bae1fd3, ver=GridCacheVersion 
> [topVer=58892924, nodeOrderDrId=3, globalTime=1447412968593, 
> order=1447412964196], timeout=100, ts=1447412968597, threadId=104, id=2266, 
> topVer=AffinityTopologyVersion [topVer=3, minorTopVer=0], reentry=null, 
> otherNodeId=f35fe3b2-93e7-4da5-8c59-94325ccd971b, otherVer=GridCacheVersion 
> [topVer=58892924, nodeOrderDrId=3, globalTime=1447412968593, 
> order=1447412964196], mappedDhtNodes=null, mappedNearNodes=null, 
> ownerVer=GridCacheVersion [topVer=58892924, nodeOrderDrId=3, 
> globalTime=1447412968499, order=1447412964118], serOrder=null, 
> key=KeyCacheObjectImpl [val=1, hasValBytes=true], 
> masks=local=1|owner=0|ready=1|reentry=0|used=0|tx=0|single_implicit=0|dht_local=1|near_local=0|removed=0,
>  prevVer=null, nextVer=null], GridCacheMvccCandidate 
> [nodeId=8f8c5147-2d8f-44e6-8da8-ef8a5bae1fd3, ver=GridCacheVersion 
> [topVer=58892924, nodeOrderDrId=3, globalTime=1447412968595, 
> order=1447412964198], timeout=100, ts=1447412968597, threadId=92, id=2267, 
> topVer=AffinityTopologyVersion [topVer=3, minorTopVer=0], reentry=null, 
> otherNodeId=f35fe3b2-93e7-4da5-8c59-94325ccd971b, otherVer=GridCacheVersion 
> [topVer=58892924, nodeOrderDrId=3, globalTime=1447412968595, 
> order=1447412964198], mappedDhtNodes=null, mappedNearNodes=null, 
> ownerVer=GridCacheVersion [topVer=58892924, nodeOrderDrId=3, 
> globalTime=1447412968502, order=1447412964120], serOrder=null, 
> key=KeyCacheObjectImpl [val=1, hasValBytes=true], 
> masks=local=1|owner=0|ready=1|reentry=0|used=0|tx=0|single_implicit=0|dht_local=1|near_local=0|removed=0,
>  prevVer=null, nextVer=null], GridCacheMvccCandidate 
> [nodeId=8f8c5147-2d8f-44e6-8da8-ef8a5bae1fd3, ver=GridCacheVersion 
> [topVer=58892924, nodeOrderDrId=3, globalTime=1447412968596, 
> order=1447412964200], timeout=100, ts=1447412968607, threadId=88, id=2268, 
> topVer=AffinityTopologyVersion [topVer=3, minorTopVer=0], reentry=null, 
> otherNodeId=f35fe3b2-93e7-4da5-8c59-94325ccd971b, otherVer=GridCacheVersion 
> [topVer=58892924, nodeOrderDrId=3, globalTime=1447412968596, 
> order=1447412964200], mappedDhtNodes=null, mappedNearNodes=null, 
> ownerVer=GridCacheVersion [topVer=58892924, nodeOrderDrId=3, 
> globalTime=1447412968504, order=1447412964122], serOrder=null, 
> key=KeyCacheObjectImpl [val=1, hasValBytes=true], 
> masks=local=1|owner=0|ready=1|reentry=0|used=0|tx=0|single_implicit=0|dht_local=1|near_local=0|removed=0,
>  prevVer=null, nextVer=null], GridCacheMvccCandidate 
> [nodeId=8f8c5147-2d8f-44e6-8da8-ef8a5bae1fd3, ver=GridCacheVersion 
> [topVer=58892924, nodeOrderDrId=3, globalTime=1447412968601, 
> order=1447412964202], timeout=100, ts=1447412968607, threadId=98, id=2269, 
> topVer=AffinityTopologyVersion [topVer=3, minorTopVer=0], reentry=null, 
> otherNodeId=f35fe3b2-93e7-4da5-8c59-94325ccd971b, otherVer=GridCacheVersion 
> [topVer=58892924, nodeOrderDrId=3, globalTime=1447412968601, 
> order=1447412964202], mappedDhtNodes=null, mappedNearNodes=null, 
> ownerVer=GridCacheVersion [topVer=58892924, nodeOrderDrId=3, 
> globalTime=1447412968506, order=1447412964124], serOrder=null, 
> key=KeyCacheObjectImpl [val=1, hasValBytes=true], 
> masks=local=1|owner=0|ready=1|reentry=0|used=0|tx=0|single_implicit=0|dht_local=1|near_local=0|removed=0,
>  prevVer=null, nextVer=null], GridCacheMvccCandidate 
> [nodeId=8f8c5147-2d8f-44e6-8da8-ef8a5bae1fd3, ver=GridCacheVersion 
> [topVer=58892924, nodeOrderDrId=3, globalTime=1447412968603, 
> order=1447412964204], timeout=100, ts=1447412968607, threadId=72, id=2270, 
> topVer=AffinityTopologyVersion [topVer=3, minorTopVer=0], reentry=null, 
> otherNodeId=f35fe3b2-93e7-4da5-8c59-94325ccd971b, otherVer=GridCacheVersion 
> [topVer=58892924, nodeOrderDrId=3, globalTime=1447412968603, 
> order=1447412964204], mappedDhtNodes=null, mappedNearNodes=null, 
> ownerVer=GridCacheVersion [topVer=58892924, nodeOrderDrId=3, 
> globalTime=1447412968508, order=1447412964126], serOrder=null, 
> key=KeyCacheObjectImpl [val=1, hasValBytes=true], 
> masks=local=1|owner=0|ready=1|reentry=0|used=0|tx=0|single_implicit=0|dht_local=1|near_local=0|removed=0,
>  prevVer=null, nextVer=null], GridCacheMvccCandidate 
> [nodeId=8f8c5147-2d8f-44e6-8da8-ef8a5bae1fd3, ver=GridCacheVersion 
> [topVer=58892924, nodeOrderDrId=3, globalTime=1447412968605, 
> order=1447412964206], timeout=100, ts=1447412968617, threadId=96, id=2271, 
> topVer=AffinityTopologyVersion [topVer=3, minorTopVer=0], reentry=null, 
> otherNodeId=f35fe3b2-93e7-4da5-8c59-94325ccd971b, otherVer=GridCacheVersion 
> [topVer=58892924, nodeOrderDrId=3, globalTime=1447412968605, 
> order=1447412964206], mappedDhtNodes=null, mappedNearNodes=null, 
> ownerVer=GridCacheVersion [topVer=58892924, nodeOrderDrId=3, 
> globalTime=1447412968510, order=1447412964128], serOrder=null, 
> key=KeyCacheObjectImpl [val=1, hasValBytes=true], 
> masks=local=1|owner=0|ready=1|reentry=0|used=0|tx=0|single_implicit=0|dht_local=1|near_local=0|removed=0,
>  prevVer=null, nextVer=null], GridCacheMvccCandidate 
> [nodeId=8f8c5147-2d8f-44e6-8da8-ef8a5bae1fd3, ver=GridCacheVersion 
> [topVer=58892924, nodeOrderDrId=3, globalTime=1447412968606, 
> order=1447412964208], timeout=100, ts=1447412968617, threadId=80, id=2272, 
> topVer=AffinityTopologyVersion [topVer=3, minorTopVer=0], reentry=null, 
> otherNodeId=f35fe3b2-93e7-4da5-8c59-94325ccd971b, otherVer=GridCacheVersion 
> [topVer=58892924, nodeOrderDrId=3, globalTime=1447412968606, 
> order=1447412964208], mappedDhtNodes=null, mappedNearNodes=null, 
> ownerVer=GridCacheVersion [topVer=58892924, nodeOrderDrId=3, 
> globalTime=1447412968514, order=1447412964132], serOrder=null, 
> key=KeyCacheObjectImpl [val=1, hasValBytes=true], 
> masks=local=1|owner=0|ready=1|reentry=0|used=0|tx=0|single_implicit=0|dht_local=1|near_local=0|removed=0,
>  prevVer=null, nextVer=null], GridCacheMvccCandidate 
> [nodeId=8f8c5147-2d8f-44e6-8da8-ef8a5bae1fd3, ver=GridCacheVersion 
> [topVer=58892924, nodeOrderDrId=3, globalTime=1447412968606, 
> order=1447412964210], timeout=100, ts=1447412968617, threadId=81, id=2273, 
> topVer=AffinityTopologyVersion [topVer=3, minorTopVer=0], reentry=null, 
> otherNodeId=f35fe3b2-93e7-4da5-8c59-94325ccd971b, otherVer=GridCacheVersion 
> [topVer=58892924, nodeOrderDrId=3, globalTime=1447412968606, 
> order=1447412964210], mappedDhtNodes=null, mappedNearNodes=null, 
> ownerVer=GridCacheVersion [topVer=58892924, nodeOrderDrId=3, 
> globalTime=1447412968514, order=1447412964132], serOrder=null, 
> key=KeyCacheObjectImpl [val=1, hasValBytes=true], 
> masks=local=1|owner=0|ready=1|reentry=0|used=0|tx=0|single_implicit=0|dht_local=1|near_local=0|removed=0,
>  prevVer=null, nextVer=null], GridCacheMvccCandidate 
> [nodeId=8f8c5147-2d8f-44e6-8da8-ef8a5bae1fd3, ver=GridCacheVersion 
> [topVer=58892924, nodeOrderDrId=3, globalTime=1447412968610, 
> order=1447412964212], timeout=100, ts=1447412968617, threadId=97, id=2274, 
> topVer=AffinityTopologyVersion [topVer=3, minorTopVer=0], reentry=null, 
> otherNodeId=f35fe3b2-93e7-4da5-8c59-94325ccd971b, otherVer=GridCacheVersion 
> [topVer=58892924, nodeOrderDrId=3, globalTime=1447412968610, 
> order=1447412964212], mappedDhtNodes=null, mappedNearNodes=null, 
> ownerVer=GridCacheVersion [topVer=58892924, nodeOrderDrId=3, 
> globalTime=1447412968516, order=1447412964134], serOrder=null, 
> key=KeyCacheObjectImpl [val=1, hasValBytes=true], 
> masks=local=1|owner=0|ready=1|reentry=0|used=0|tx=0|single_implicit=0|dht_local=1|near_local=0|removed=0,
>  prevVer=null, nextVer=null], GridCacheMvccCandidate 
> [nodeId=8f8c5147-2d8f-44e6-8da8-ef8a5bae1fd3, ver=GridCacheVersion 
> [topVer=58892924, nodeOrderDrId=3, globalTime=1447412968612, 
> order=1447412964214], timeout=100, ts=1447412968617, threadId=65, id=2275, 
> topVer=AffinityTopologyVersion [topVer=3, minorTopVer=0], reentry=null, 
> otherNodeId=f35fe3b2-93e7-4da5-8c59-94325ccd971b, otherVer=GridCacheVersion 
> [topVer=58892924, nodeOrderDrId=3, globalTime=1447412968612, 
> order=1447412964214], mappedDhtNodes=null, mappedNearNodes=null, 
> ownerVer=GridCacheVersion [topVer=58892924, nodeOrderDrId=3, 
> globalTime=1447412968518, order=1447412964136], serOrder=null, 
> key=KeyCacheObjectImpl [val=1, hasValBytes=true], 
> masks=local=1|owner=0|ready=1|reentry=0|used=0|tx=0|single_implicit=0|dht_local=1|near_local=0|removed=0,
>  prevVer=null, nextVer=null], GridCacheMvccCandidate 
> [nodeId=8f8c5147-2d8f-44e6-8da8-ef8a5bae1fd3, ver=GridCacheVersion 
> [topVer=58892924, nodeOrderDrId=3, globalTime=1447412968614, 
> order=1447412964216], timeout=100, ts=1447412968617, threadId=89, id=2276, 
> topVer=AffinityTopologyVersion [topVer=3, minorTopVer=0], reentry=null, 
> otherNodeId=f35fe3b2-93e7-4da5-8c59-94325ccd971b, otherVer=GridCacheVersion 
> [topVer=58892924, nodeOrderDrId=3, globalTime=1447412968614, 
> order=1447412964216], mappedDhtNodes=null, mappedNearNodes=null, 
> ownerVer=GridCacheVersion [topVer=58892924, nodeOrderDrId=3, 
> globalTime=1447412968520, order=1447412964138], serOrder=null, 
> key=KeyCacheObjectImpl [val=1, hasValBytes=true], 
> masks=local=1|owner=0|ready=1|reentry=0|used=0|tx=0|single_implicit=0|dht_local=1|near_local=0|removed=0,
>  prevVer=null, nextVer=null], GridCacheMvccCandidate 
> [nodeId=8f8c5147-2d8f-44e6-8da8-ef8a5bae1fd3, ver=GridCacheVersion 
> [topVer=58892924, nodeOrderDrId=3, globalTime=1447412968616, 
> order=1447412964218], timeout=100, ts=1447412968627, threadId=73, id=2277, 
> topVer=AffinityTopologyVersion [topVer=3, minorTopVer=0], reentry=null, 
> otherNodeId=f35fe3b2-93e7-4da5-8c59-94325ccd971b, otherVer=GridCacheVersion 
> [topVer=58892924, nodeOrderDrId=3, globalTime=1447412968616, 
> order=1447412964218], mappedDhtNodes=null, mappedNearNodes=null, 
> ownerVer=GridCacheVersion [topVer=58892924, nodeOrderDrId=3, 
> globalTime=1447412968525, order=1447412964144], serOrder=null, 
> key=KeyCacheObjectImpl [val=1, hasValBytes=true], 
> masks=local=1|owner=0|ready=1|reentry=0|used=0|tx=0|single_implicit=0|dht_local=1|near_local=0|removed=0,
>  prevVer=null, nextVer=null], GridCacheMvccCandidate 
> [nodeId=8f8c5147-2d8f-44e6-8da8-ef8a5bae1fd3, ver=GridCacheVersion 
> [topVer=58892924, nodeOrderDrId=3, globalTime=1447412968616, 
> order=1447412964220], timeout=100, ts=1447412968627, threadId=74, id=2278, 
> topVer=AffinityTopologyVersion [topVer=3, minorTopVer=0], reentry=null, 
> otherNodeId=f35fe3b2-93e7-4da5-8c59-94325ccd971b, otherVer=GridCacheVersion 
> [topVer=58892924, nodeOrderDrId=3, globalTime=1447412968616, 
> order=1447412964220], mappedDhtNodes=null, mappedNearNodes=null, 
> ownerVer=GridCacheVersion [topVer=58892924, nodeOrderDrId=3, 
> globalTime=1447412968525, order=1447412964144], serOrder=null, 
> key=KeyCacheObjectImpl [val=1, hasValBytes=true], 
> masks=local=1|owner=0|ready=1|reentry=0|used=0|tx=0|single_implicit=0|dht_local=1|near_local=0|removed=0,
>  prevVer=null, nextVer=null], GridCacheMvccCandidate 
> [nodeId=8f8c5147-2d8f-44e6-8da8-ef8a5bae1fd3, ver=GridCacheVersion 
> [topVer=58892924, nodeOrderDrId=3, globalTime=1447412968616, 
> order=1447412964222], timeout=100, ts=1447412968627, threadId=71, id=2279, 
> topVer=AffinityTopologyVersion [topVer=3, minorTopVer=0], reentry=null, 
> otherNodeId=f35fe3b2-93e7-4da5-8c59-94325ccd971b, otherVer=GridCacheVersion 
> [topVer=58892924, nodeOrderDrId=3, globalTime=1447412968616, 
> order=1447412964222], mappedDhtNodes=null, mappedNearNodes=null, 
> ownerVer=GridCacheVersion [topVer=58892924, nodeOrderDrId=3, 
> globalTime=1447412968525, order=1447412964144], serOrder=null, 
> key=KeyCacheObjectImpl [val=1, hasValBytes=true], 
> masks=local=1|owner=0|ready=1|reentry=0|used=0|tx=0|single_implicit=0|dht_local=1|near_local=0|removed=0,
>  prevVer=null, nextVer=null], GridCacheMvccCandidate 
> [nodeId=8f8c5147-2d8f-44e6-8da8-ef8a5bae1fd3, ver=GridCacheVersion 
> [topVer=58892924, nodeOrderDrId=3, globalTime=1447412968621, 
> order=1447412964224], timeout=100, ts=1447412968627, threadId=87, id=2280, 
> topVer=AffinityTopologyVersion [topVer=3, minorTopVer=0], reentry=null, 
> otherNodeId=f35fe3b2-93e7-4da5-8c59-94325ccd971b, otherVer=GridCacheVersion 
> [topVer=58892924, nodeOrderDrId=3, globalTime=1447412968621, 
> order=1447412964224], mappedDhtNodes=null, mappedNearNodes=null, 
> ownerVer=GridCacheVersion [topVer=58892924, nodeOrderDrId=3, 
> globalTime=1447412968527, order=1447412964146], serOrder=null, 
> key=KeyCacheObjectImpl [val=1, hasValBytes=true], 
> masks=local=1|owner=0|ready=1|reentry=0|used=0|tx=0|single_implicit=0|dht_local=1|near_local=0|removed=0,
>  prevVer=null, nextVer=null], GridCacheMvccCandidate 
> [nodeId=8f8c5147-2d8f-44e6-8da8-ef8a5bae1fd3, ver=GridCacheVersion 
> [topVer=58892924, nodeOrderDrId=3, globalTime=1447412968624, 
> order=1447412964226], timeout=100, ts=1447412968627, threadId=76, id=2281, 
> topVer=AffinityTopologyVersion [topVer=3, minorTopVer=0], reentry=null, 
> otherNodeId=f35fe3b2-93e7-4da5-8c59-94325ccd971b, otherVer=GridCacheVersion 
> [topVer=58892924, nodeOrderDrId=3, globalTime=1447412968624, 
> order=1447412964226], mappedDhtNodes=null, mappedNearNodes=null, 
> ownerVer=GridCacheVersion [topVer=58892924, nodeOrderDrId=3, 
> globalTime=1447412968529, order=1447412964148], serOrder=null, 
> key=KeyCacheObjectImpl [val=1, hasValBytes=true], 
> masks=local=1|owner=0|ready=1|reentry=0|used=0|tx=0|single_implicit=0|dht_local=1|near_local=0|removed=0,
>  prevVer=null, nextVer=null]], rmts=null]], flags=0]]]], 
> mappedVer=GridCacheVersion [topVer=58892924, nodeOrderDrId=3, 
> globalTime=1447412968533, order=1447412964152], ver=GridCacheVersion 
> [topVer=58892922, nodeOrderDrId=1, globalTime=0, order=0], tx=null, 
> req=GridNearLockRequest [topVer=AffinityTopologyVersion [topVer=3, 
> minorTopVer=0], miniId=6b229800151-652e9fe6-66fa-4415-aeb1-1ec44adaf8d2, 
> implicitTx=false, implicitSingleTx=false, onePhaseCommit=false, 
> dhtVers=[null], subjId=null, taskNameHash=0, hasTransforms=false, 
> syncCommit=false, accessTtl=-1, retVal=true, firstClientReq=true, 
> filter=null, super=GridDistributedLockRequest 
> [nodeId=f35fe3b2-93e7-4da5-8c59-94325ccd971b, nearXidVer=GridCacheVersion 
> [topVer=58892924, nodeOrderDrId=3, globalTime=1447412968533, 
> order=1447412964152], threadId=77, 
> futId=4b229800151-652e9fe6-66fa-4415-aeb1-1ec44adaf8d2, timeout=100, 
> isInTx=false, isInvalidate=false, isRead=false, isolation=null, 
> retVals=[true], txSize=1, flags=0, keysCnt=1, 
> super=GridDistributedBaseMessage [ver=GridCacheVersion [topVer=58892924, 
> nodeOrderDrId=3, globalTime=1447412968533, order=1447412964152], 
> candsByIdx=null, committedVers=null, rolledbackVers=null, cnt=0, 
> super=GridCacheMessage [msgId=4463, depInfo=null, err=null, 
> skipPrepare=false]]]], err=null]
>       at 
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter.createLockReply(GridDhtTransactionalCacheAdapter.java:1132)
>       at 
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter.access$400(GridDhtTransactionalCacheAdapter.java:88)
>       at 
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter$10.apply(GridDhtTransactionalCacheAdapter.java:986)
>       at 
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter$10.apply(GridDhtTransactionalCacheAdapter.java:979)
>       at 
> org.apache.ignite.internal.util.future.GridEmbeddedFuture$1.applyx(GridEmbeddedFuture.java:59)
>       at 
> org.apache.ignite.internal.util.future.GridEmbeddedFuture$AsyncListener1.apply(GridEmbeddedFuture.java:297)
>       at 
> org.apache.ignite.internal.util.future.GridEmbeddedFuture$AsyncListener1.apply(GridEmbeddedFuture.java:290)
>       at 
> org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:262)
>       at 
> org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListeners(GridFutureAdapter.java:250)
>       at 
> org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:380)
>       at 
> org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:346)
>       at 
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtLockFuture.onComplete(GridDhtLockFuture.java:733)
>       at 
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtLockFuture.onDone(GridDhtLockFuture.java:711)
>       at 
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtLockFuture.onDone(GridDhtLockFuture.java:80)
>       at 
> org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:323)
>       at 
> org.apache.ignite.internal.util.future.GridCompoundFuture.checkComplete(GridCompoundFuture.java:285)
>       at 
> org.apache.ignite.internal.util.future.GridCompoundFuture.markInitialized(GridCompoundFuture.java:276)
>       at 
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtLockFuture.map(GridDhtLockFuture.java:943)
>       at 
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtLockFuture.onOwnerChanged(GridDhtLockFuture.java:666)
>       at 
> org.apache.ignite.internal.processors.cache.GridCacheMvccManager$2.onOwnerChanged(GridCacheMvccManager.java:156)
>       at 
> org.apache.ignite.internal.processors.cache.distributed.GridDistributedCacheEntry.checkOwnerChanged(GridDistributedCacheEntry.java:814)
>       at 
> org.apache.ignite.internal.processors.cache.distributed.GridDistributedCacheEntry.removeLock(GridDistributedCacheEntry.java:448)
>       at 
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheEntry.removeLock(GridDhtCacheEntry.java:319)
>       at 
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter.removeLocks(GridDhtTransactionalCacheAdapter.java:1486)
>       at 
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter.processNearUnlockRequest(GridDhtTransactionalCacheAdapter.java:1336)
>       at 
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter.access$200(GridDhtTransactionalCacheAdapter.java:88)
>       at 
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter$5.apply(GridDhtTransactionalCacheAdapter.java:150)
>       at 
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter$5.apply(GridDhtTransactionalCacheAdapter.java:148)
>       at 
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:540)
>       at 
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:273)
>       at 
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:197)
>       at 
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$000(GridCacheIoManager.java:76)
>       at 
> org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:159)
>       at 
> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:811)
>       at 
> org.apache.ignite.internal.managers.communication.GridIoManager.access$1500(GridIoManager.java:106)
>       at 
> org.apache.ignite.internal.managers.communication.GridIoManager$5.run(GridIoManager.java:774)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>       at java.lang.Thread.run(Thread.java:745)
> {noformat}
> To reproduce run attached sources in the following order:
> - Start two server nodes (ServerTest.java);
> - Start a client node (ClientTest.java).
> If the issue is not reproduced on your machine try to adjust {{lock.tryLock}} 
> timeout value.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to