[
https://issues.apache.org/jira/browse/IGNITE-7570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16773459#comment-16773459
]
Scott Feldstein commented on IGNITE-7570:
-----------------------------------------
I just encountered this on Ignite 2.7.0 with persistence turned off during a
scale test where an ignite node was intentionally killed. When it came back I
saw this error on the clients. It seems to have recovered pretty quickly
{code:java}
2019-02-20T22:04:18,995Z localhost svc: [priority='ERROR'
thread='http-nio-8080-exec-74'
class='org.apache.catalina.core.ContainerBase.[Tomcat].[localhost].[/].[di
spatcherServlet]@'] Servlet.service() for servlet [dispatcherServlet] in
context with path [] threw exception [Request processing failed; nested
exception is org.hibernate.cache.CacheException: class org.apac
he.ignite.IgniteCheckedException: Failed to process invalid partitions response
(remote node reported invalid partitions but remote topology version does not
differ from local) [topVer=AffinityTopologyVersion
[topVer=190, minorTopVer=0], rmtTopVer=AffinityTopologyVersion [topVer=190,
minorTopVer=0], part=492, nodeId=b23b132e-469e-41b0-87ff-63b5a2010411]] with
root cause
org.apache.ignite.IgniteCheckedException: Failed to process invalid partitions
response (remote node reported invalid partitions but remote topology version
does not differ from local) [topVer=AffinityTopologyVersion [topVer=190,
minorTopVer=0], rmtTopVer=AffinityTopologyVersion [topVer=190, minorTopVer=0],
part=492, nodeId=b23b132e-469e-41b0-87ff-63b5a2010411]
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridPartitionedSingleGetFuture.checkError(GridPartitionedSingleGetFuture.java:611)
~[ignite-core-2.7.0.jar:2.7.0]
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridPartitionedSingleGetFuture.onResult(GridPartitionedSingleGetFuture.java:520)
~[ignite-core-2.7.0.jar:2.7.0]
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheAdapter.processNearSingleGetResponse(GridDhtCacheAdapter.java:368)
~[ignite-core-2.7.0.jar:2.7.0]
at
org.apache.ignite.internal.processors.cache.distributed.dht.colocated.GridDhtColocatedCache.access$100(GridDhtColocatedCache.java:87)
~[ignite-core-2.7.0.jar:2.7.0]
at
org.apache.ignite.internal.processors.cache.distributed.dht.colocated.GridDhtColocatedCache$2.apply(GridDhtColocatedCache.java:132)
~[ignite-core-2.7.0.jar:2.7.0]
at
org.apache.ignite.internal.processors.cache.distributed.dht.colocated.GridDhtColocatedCache$2.apply(GridDhtColocatedCache.java:130)
~[ignite-core-2.7.0.jar:2.7.0]
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
~[ignite-core-2.7.0.jar:2.7.0]
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
~[ignite-core-2.7.0.jar:2.7.0]
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
~[ignite-core-2.7.0.jar:2.7.0]
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
~[ignite-core-2.7.0.jar:2.7.0]
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
~[ignite-core-2.7.0.jar:2.7.0]
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
~[ignite-core-2.7.0.jar:2.7.0]
at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
~[ignite-core-2.7.0.jar:2.7.0]
at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
~[ignite-core-2.7.0.jar:2.7.0]
at
org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
~[ignite-core-2.7.0.jar:2.7.0]
at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
~[ignite-core-2.7.0.jar:2.7.0]
at
org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:505)
~[ignite-core-2.7.0.jar:2.7.0]
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
~[ignite-core-2.7.0.jar:2.7.0]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]{code}
> Client nodes failed with "Failed to process invalid partitions response"
> during failover
> ----------------------------------------------------------------------------------------
>
> Key: IGNITE-7570
> URL: https://issues.apache.org/jira/browse/IGNITE-7570
> Project: Ignite
> Issue Type: Bug
> Affects Versions: 2.4
> Reporter: Ksenia Rybakova
> Priority: Major
> Attachments: ignite-base-load-config.xml, run-load.properties,
> run-load.xml
>
>
> Some client nodes fail with "Failed to process invalid partitions response"
> during failover test:
> {noformat}
> [2018-01-30 16:27:58,610][INFO ][sys-#190][GridDhtPartitionsExchangeFuture]
> Received full message, will finish exchange
> [node=80ebd2ac-1432-4bfc-bab7-d9dbf56cdeb4, resVer=AffinityTopologyVersion
> [topVer=37, minorTopVer=0]]
> [2018-01-30 16:27:58,688][INFO ][sys-#190][GridDhtPartitionsExchangeFuture]
> Finish exchange future [startVer=AffinityTopologyVersion [topVer=37,
> minorTopVer=0], resVer=AffinityTopologyVersion [topVer=37, minorTopVer=0],
> err=null]
> <16:27:58><benchmark-worker-32><yardstick> The benchmark of random operation
> failed.
> javax.cache.CacheException: class org.apache.ignite.IgniteCheckedException:
> Failed to process invalid partitions response (remote node reported invalid
> partitions but remote topology version does not differ from local)
> [topVer=AffinityTopologyVersion [topVer=37, minorTopVer=0],
> rmtTopVer=AffinityTopologyVersion [topVer=37, minorTopVer=0], part=204,
> nodeId=80ebd2ac-1432-4bfc-bab7-d9dbf56cdeb4]
> at
> org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1294)
> at
> org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.cacheException(IgniteCacheProxyImpl.java:1673)
> at
> org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.get(IgniteCacheProxyImpl.java:852)
> at
> org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.get(GatewayProtectedCacheProxy.java:676)
> at
> org.apache.ignite.yardstick.cache.load.IgniteCacheRandomOperationBenchmark.doGet(IgniteCacheRandomOperationBenchmark.java:776)
> at
> org.apache.ignite.yardstick.cache.load.IgniteCacheRandomOperationBenchmark.executeRandomOperation(IgniteCacheRandomOperationBenchmark.java:624)
> at
> org.apache.ignite.yardstick.cache.load.IgniteCacheRandomOperationBenchmark.executeOutOfTx(IgniteCacheRandomOperationBenchmark.java:602)
> at
> org.apache.ignite.yardstick.cache.load.IgniteCacheRandomOperationBenchmark.test(IgniteCacheRandomOperationBenchmark.java:207)
> at
> org.yardstickframework.impl.BenchmarkRunner$2.run(BenchmarkRunner.java:178)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: class org.apache.ignite.IgniteCheckedException: Failed to process
> invalid partitions response (remote node reported invalid partitions but
> remote topology version does not differ from local)
> [topVer=AffinityTopologyVersion [topVer=37, minorTopVer=0],
> rmtTopVer=AffinityTopologyVersion [topVer=37, minorTopVer=0], part=204,
> nodeId=80ebd2ac-1432-4bfc-bab7-d9dbf56cdeb4]
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.GridPartitionedSingleGetFuture.checkError(GridPartitionedSingleGetFuture.java:596)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.GridPartitionedSingleGetFuture.onResult(GridPartitionedSingleGetFuture.java:505)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheAdapter.processNearSingleGetResponse(GridDhtCacheAdapter.java:349)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$1400(GridDhtAtomicCache.java:130)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$15.apply(GridDhtAtomicCache.java:422)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$15.apply(GridDhtAtomicCache.java:417)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1060)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:579)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:378)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:304)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:99)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:293)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1555)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1183)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1090)
> at
> org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
> ... 1 more
> Finishing main test [ts=1517318878863, date=Tue Jan 30 16:27:58 MSK 2018]
> ERROR: Shutting down benchmark driver to unexpected exception.
> {noformat}
> Test config:
> CacheRandomOperationBenchmark
> - 20 server nodes, 10 client nodes at 10 hosts
> - 34 caches with different configs with and without PDS, 3 backups
> - preload amount 250
> - key range 500K
> - operations: PUT PUT_ALL GET GET_ALL INVOKE INVOKE_ALL REMOVE REMOVE_ALL
> PUT_IF_ABSENT REPLACE
> - 2 of 20 servers are being restarted every 15 minutes
> Complete yardstick configs are attached.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
