[ https://issues.apache.org/jira/browse/IGNITE-3211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Semen Boikov updated IGNITE-3211: --------------------------------- Priority: Critical (was: Major) > "Failed to reinitialize local partitions", "Failed to wait for completion of > partition map exchange" errors during failover test > -------------------------------------------------------------------------------------------------------------------------------- > > Key: IGNITE-3211 > URL: https://issues.apache.org/jira/browse/IGNITE-3211 > Project: Ignite > Issue Type: Bug > Affects Versions: 1.6 > Reporter: Ksenia Rybakova > Assignee: Semen Boikov > Priority: Critical > Fix For: 1.8 > > > "Failed to reinitialize local partitions (preloading will be stopped)" and > "Failed to wait for completion of partition map exchange (preloading will not > start)" errors occured during failover load test. Complete stack trace see > below. > Load config: > - 1 client, 20 servers (5 servers per 1 host) > - warmup 60 > - duration 66h > - preload 5M > - key range 10M > - operations: PUT PUT_ALL GET GET_ALL INVOKE INVOKE_ALL REMOVE REMOVE_ALL > PUT_IF_ABSENT REPLACE > - backups count 3 > - 3 servers restart every 15 min with 30 sec step, pause between stop and > start 5min > {noformat} > [08:32:21,002][ERROR][exchange-worker-#83%null%][GridDhtPartitionsExchangeFuture] > Failed to reinitialize local partitions (preloading will be stopped): > GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=39, > minorTopVer=1], nodeId=20ddc8b7, evt=DISCOVERY_CUSTOM_EVT] > class org.apache.ignite.IgniteException: null > at > org.apache.ignite.internal.util.tostring.GridToStringBuilder.toStringImpl(GridToStringBuilder.java:506) > at > org.apache.ignite.internal.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:297) > at > org.apache.ignite.internal.processors.cache.transactions.IgniteTxLocalAdapter.toString(IgniteTxLocalAdapter.java:3743) > at > org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocalAdapter.toString(GridDhtTxLocalAdapter.java:868) > at > org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocal.toString(GridDhtTxLocal.java:703) > at java.lang.String.valueOf(String.java:2849) > at java.lang.StringBuilder.append(StringBuilder.java:128) > at > org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.dumpPendingObjects(GridCachePartitionExchangeManager.java:1172) > at > org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.dumpDebugInfo(GridCachePartitionExchangeManager.java:1150) > at > org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.dumpPendingObjects(GridDhtPartitionsExchangeFuture.java:894) > at > org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.waitPartitionRelease(GridDhtPartitionsExchangeFuture.java:769) > at > org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.distributedExchange(GridDhtPartitionsExchangeFuture.java:715) > at > org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:472) > at > org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:1333) > at > org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) > at java.lang.Thread.run(Thread.java:745) > Caused by: class org.apache.ignite.IgniteException: null > at > org.apache.ignite.internal.util.tostring.GridToStringBuilder.toStringImpl(GridToStringBuilder.java:506) > at > org.apache.ignite.internal.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:364) > at > org.apache.ignite.internal.processors.cache.transactions.IgniteTxStateImpl.toString(IgniteTxStateImpl.java:443) > at java.lang.String.valueOf(String.java:2849) > at > org.apache.ignite.internal.util.GridStringBuilder.a(GridStringBuilder.java:101) > at > org.apache.ignite.internal.util.tostring.GridToStringBuilder.toStringImpl(GridToStringBuilder.java:474) > ... 15 more > Caused by: java.util.ConcurrentModificationException > at > java.util.LinkedHashMap$LinkedHashIterator.nextEntry(LinkedHashMap.java:394) > at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:413) > at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:412) > at java.util.AbstractMap.toString(AbstractMap.java:518) > at java.lang.String.valueOf(String.java:2849) > at > org.apache.ignite.internal.util.GridStringBuilder.a(GridStringBuilder.java:101) > at > org.apache.ignite.internal.util.tostring.GridToStringBuilder.toStringImpl(GridToStringBuilder.java:474) > ... 20 more > [08:32:21,072][ERROR][exchange-worker-#83%null%][GridCachePartitionExchangeManager] > Failed to wait for completion of partition map exchange (preloading will not > start): GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false, > reassign=false, discoEvt=DiscoveryCustomEvent [customMsg=null, > affTopVer=AffinityTopologyVersion [topVer=39, minorTopVer=1], > super=DiscoveryEvent [evtNode=TcpDiscoveryNode > [id=20ddc8b7-fc62-4d8c-be98-2e5edf60a419, addrs=[10.20.0.216, 127.0.0.1], > sockAddrs=[fosters-216/10.20.0.216:47500, /10.20.0.216:47500, > /127.0.0.1:47500], discPort=47500, order=1, intOrder=1, > lastExchangeTime=1464363141003, loc=true, ver=1.6.0#20160525-sha1:48321a40, > isClient=false], topVer=39, nodeId8=20ddc8b7, msg=null, > type=DISCOVERY_CUSTOM_EVT, tstamp=1464363130370]], crd=TcpDiscoveryNode > [id=20ddc8b7-fc62-4d8c-be98-2e5edf60a419, addrs=[10.20.0.216, 127.0.0.1], > sockAddrs=[fosters-216/10.20.0.216:47500, /10.20.0.216:47500, > /127.0.0.1:47500], discPort=47500, order=1, intOrder=1, > lastExchangeTime=1464363141003, loc=true, ver=1.6.0#20160525-sha1:48321a40, > isClient=false], exchId=GridDhtPartitionExchangeId > [topVer=AffinityTopologyVersion [topVer=39, minorTopVer=1], nodeId=20ddc8b7, > evt=DISCOVERY_CUSTOM_EVT], added=true, initFut=GridFutureAdapter [resFlag=2, > res=false, startTime=1464363130370, endTime=1464363141057, > ignoreInterrupts=false, state=DONE], init=false, topSnapshot=null, > lastVer=GridCacheVersion [topVer=75840698, time=1464363140466, > order=1464462577173, nodeOrder=38], partReleaseFut=GridCompoundFuture > [rdc=null, initFlag=1, lsnrCalls=2, done=false, cancelled=false, err=null, > futs=[true, false, true]], affChangeMsg=CacheAffinityChangeMessage > [id=9560ad2f451-621c2001-9d60-40f4-ab21-20834a2b6b33, > topVer=AffinityTopologyVersion [topVer=39, minorTopVer=0], exchId=null, > partsMsg=null, exchangeNeeded=true], skipPreload=false, > clientOnlyExchange=false, initTs=1464363130370, centralizedAff=false, > evtLatch=0, remaining=[e7f032f5-4ec6-48c8-9cd4-ac78b0f8ccc4, > a550929c-1f4b-4de3-8d57-47b4a8232010, c7ff42e7-acd5-4cb4-9cd5-179aa18b88b0, > fc79bff1-480f-44b3-9915-2548f937726c, c013837c-b46e-4667-9e47-e3c41412b5a1, > ba89970d-8908-4276-9d3c-b679a335c2d2, a7f0feb2-b932-4843-ac93-560fc547700e, > 3c462bba-1d12-43ab-ad21-761f9ef94aa1, 9c184df6-e5e0-49b3-9836-65932d58be6a, > 317b61fc-186b-4bf7-9622-6c4094409814, 425b461c-5117-403d-af2b-e323b7c1aa39, > a4d0911c-d54a-4a36-b911-b74eddc3f0cc, d0e84363-590d-45bf-bad3-b88cde5d03fe, > b51cc458-c884-41ce-9e5b-b83ea111218b, 6970f13f-fe82-4583-a128-af93f86e99e7, > a9b8fe06-cda9-4907-871d-0684bcc2c3ca, c978e55f-4de6-4ffd-98b1-459215db2232, > cab5d0e0-7365-4774-8f99-d9f131c5d896, 0859feda-b70d-4fab-870a-4db18bd5984b], > srvNodes=[TcpDiscoveryNode [id=20ddc8b7-fc62-4d8c-be98-2e5edf60a419, > addrs=[10.20.0.216, 127.0.0.1], sockAddrs=[fosters-216/10.20.0.216:47500, > /10.20.0.216:47500, /127.0.0.1:47500], discPort=47500, order=1, intOrder=1, > lastExchangeTime=1464363141003, loc=true, ver=1.6.0#20160525-sha1:48321a40, > isClient=false], TcpDiscoveryNode [id=cab5d0e0-7365-4774-8f99-d9f131c5d896, > addrs=[10.20.0.221, 127.0.0.1], sockAddrs=[fosters-221/10.20.0.221:47500, > /10.20.0.221:47500, /127.0.0.1:47500], discPort=47500, order=2, intOrder=2, > lastExchangeTime=1464360659615, loc=false, ver=1.6.0#20160525-sha1:48321a40, > isClient=false], TcpDiscoveryNode [id=c7ff42e7-acd5-4cb4-9cd5-179aa18b88b0, > addrs=[10.20.0.222, 127.0.0.1], sockAddrs=[fosters-222/10.20.0.222:47500, > /10.20.0.222:47500, /127.0.0.1:47500], discPort=47500, order=3, intOrder=3, > lastExchangeTime=1464360660162, loc=false, ver=1.6.0#20160525-sha1:48321a40, > isClient=false], TcpDiscoveryNode [id=0859feda-b70d-4fab-870a-4db18bd5984b, > addrs=[10.20.0.223, 127.0.0.1], sockAddrs=[fosters-223/10.20.0.223:47500, > /10.20.0.223:47500, /127.0.0.1:47500], discPort=47500, order=4, intOrder=4, > lastExchangeTime=1464360660689, loc=false, ver=1.6.0#20160525-sha1:48321a40, > isClient=false], TcpDiscoveryNode [id=ba89970d-8908-4276-9d3c-b679a335c2d2, > addrs=[10.20.0.216, 127.0.0.1], sockAddrs=[fosters-216/10.20.0.216:47501, > /10.20.0.216:47501, /127.0.0.1:47501], discPort=47501, order=5, intOrder=5, > lastExchangeTime=1464360661276, loc=false, ver=1.6.0#20160525-sha1:48321a40, > isClient=false], TcpDiscoveryNode [id=317b61fc-186b-4bf7-9622-6c4094409814, > addrs=[10.20.0.221, 127.0.0.1], sockAddrs=[fosters-221/10.20.0.221:47501, > /10.20.0.221:47501, /127.0.0.1:47501], discPort=47501, order=6, intOrder=6, > lastExchangeTime=1464360662086, loc=false, ver=1.6.0#20160525-sha1:48321a40, > isClient=false], TcpDiscoveryNode [id=425b461c-5117-403d-af2b-e323b7c1aa39, > addrs=[10.20.0.222, 127.0.0.1], sockAddrs=[fosters-222/10.20.0.222:47501, > /10.20.0.222:47501, /127.0.0.1:47501], discPort=47501, order=7, intOrder=7, > lastExchangeTime=1464360662471, loc=false, ver=1.6.0#20160525-sha1:48321a40, > isClient=false], TcpDiscoveryNode [id=a550929c-1f4b-4de3-8d57-47b4a8232010, > addrs=[10.20.0.223, 127.0.0.1], sockAddrs=[fosters-223/10.20.0.223:47501, > /10.20.0.223:47501, /127.0.0.1:47501], discPort=47501, order=8, intOrder=8, > lastExchangeTime=1464360662886, loc=false, ver=1.6.0#20160525-sha1:48321a40, > isClient=false], TcpDiscoveryNode [id=e7f032f5-4ec6-48c8-9cd4-ac78b0f8ccc4, > addrs=[10.20.0.216, 127.0.0.1], sockAddrs=[fosters-216/10.20.0.216:47502, > /10.20.0.216:47502, /127.0.0.1:47502], discPort=47502, order=9, intOrder=9, > lastExchangeTime=1464360663373, loc=false, ver=1.6.0#20160525-sha1:48321a40, > isClient=false], TcpDiscoveryNode [id=b51cc458-c884-41ce-9e5b-b83ea111218b, > addrs=[10.20.0.221, 127.0.0.1], sockAddrs=[fosters-221/10.20.0.221:47502, > /10.20.0.221:47502, /127.0.0.1:47502], discPort=47502, order=10, intOrder=10, > lastExchangeTime=1464360664058, loc=false, ver=1.6.0#20160525-sha1:48321a40, > isClient=false], TcpDiscoveryNode [id=fc79bff1-480f-44b3-9915-2548f937726c, > addrs=[10.20.0.222, 127.0.0.1], sockAddrs=[fosters-222/10.20.0.222:47502, > /10.20.0.222:47502, /127.0.0.1:47502], discPort=47502, order=11, intOrder=11, > lastExchangeTime=1464360664685, loc=false, ver=1.6.0#20160525-sha1:48321a40, > isClient=false], TcpDiscoveryNode [id=a7f0feb2-b932-4843-ac93-560fc547700e, > addrs=[10.20.0.223, 127.0.0.1], sockAddrs=[fosters-223/10.20.0.223:47502, > /10.20.0.223:47502, /127.0.0.1:47502], discPort=47502, order=12, intOrder=12, > lastExchangeTime=1464360665216, loc=false, ver=1.6.0#20160525-sha1:48321a40, > isClient=false], TcpDiscoveryNode [id=9c184df6-e5e0-49b3-9836-65932d58be6a, > addrs=[10.20.0.216, 127.0.0.1], sockAddrs=[fosters-216/10.20.0.216:47503, > /10.20.0.216:47503, /127.0.0.1:47503], discPort=47503, order=13, intOrder=13, > lastExchangeTime=1464360665661, loc=false, ver=1.6.0#20160525-sha1:48321a40, > isClient=false], TcpDiscoveryNode [id=6970f13f-fe82-4583-a128-af93f86e99e7, > addrs=[10.20.0.216, 127.0.0.1], sockAddrs=[fosters-216/10.20.0.216:47504, > /10.20.0.216:47504, /127.0.0.1:47504], discPort=47504, order=17, intOrder=17, > lastExchangeTime=1464360668158, loc=false, ver=1.6.0#20160525-sha1:48321a40, > isClient=false], TcpDiscoveryNode [id=d0e84363-590d-45bf-bad3-b88cde5d03fe, > addrs=[10.20.0.221, 127.0.0.1], sockAddrs=[fosters-221/10.20.0.221:47504, > /10.20.0.221:47504, /127.0.0.1:47504], discPort=47504, order=18, intOrder=18, > lastExchangeTime=1464360668768, loc=false, ver=1.6.0#20160525-sha1:48321a40, > isClient=false], TcpDiscoveryNode [id=c013837c-b46e-4667-9e47-e3c41412b5a1, > addrs=[10.20.0.222, 127.0.0.1], sockAddrs=[fosters-222/10.20.0.222:47504, > /10.20.0.222:47504, /127.0.0.1:47504], discPort=47504, order=19, intOrder=19, > lastExchangeTime=1464360669745, loc=false, ver=1.6.0#20160525-sha1:48321a40, > isClient=false], TcpDiscoveryNode [id=3c462bba-1d12-43ab-ad21-761f9ef94aa1, > addrs=[10.20.0.223, 127.0.0.1], sockAddrs=[fosters-223/10.20.0.223:47504, > /10.20.0.223:47504, /127.0.0.1:47504], discPort=47504, order=20, intOrder=20, > lastExchangeTime=1464360669836, loc=false, ver=1.6.0#20160525-sha1:48321a40, > isClient=false], TcpDiscoveryNode [id=c978e55f-4de6-4ffd-98b1-459215db2232, > addrs=[10.20.0.221, 127.0.0.1], sockAddrs=[fosters-221/10.20.0.221:47503, > /10.20.0.221:47503, /127.0.0.1:47503], discPort=47503, order=37, intOrder=28, > lastExchangeTime=1464362949633, loc=false, ver=1.6.0#20160525-sha1:48321a40, > isClient=false], TcpDiscoveryNode [id=a9b8fe06-cda9-4907-871d-0684bcc2c3ca, > addrs=[10.20.0.222, 127.0.0.1], sockAddrs=[fosters-222/10.20.0.222:47503, > /10.20.0.222:47503, /127.0.0.1:47503], discPort=47503, order=38, intOrder=29, > lastExchangeTime=1464362981162, loc=false, ver=1.6.0#20160525-sha1:48321a40, > isClient=false], TcpDiscoveryNode [id=a4d0911c-d54a-4a36-b911-b74eddc3f0cc, > addrs=[10.20.0.223, 127.0.0.1], sockAddrs=[fosters-223/10.20.0.223:47503, > /10.20.0.223:47503, /127.0.0.1:47503], discPort=47503, order=39, intOrder=30, > lastExchangeTime=1464363011016, loc=false, ver=1.6.0#20160525-sha1:48321a40, > isClient=false]], super=GridFutureAdapter [resFlag=1, res=class > o.a.i.IgniteException: null, startTime=1464363130370, endTime=1464363141003, > ignoreInterrupts=false, state=DONE]] > class org.apache.ignite.IgniteCheckedException: null > at > org.apache.ignite.internal.util.IgniteUtils.cast(IgniteUtils.java:7067) > at > org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:168) > at > org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:117) > at > org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:1335) > at > org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.util.ConcurrentModificationException > at > java.util.LinkedHashMap$LinkedHashIterator.nextEntry(LinkedHashMap.java:394) > at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:413) > at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:412) > at java.util.AbstractMap.toString(AbstractMap.java:518) > at java.lang.String.valueOf(String.java:2849) > at > org.apache.ignite.internal.util.GridStringBuilder.a(GridStringBuilder.java:101) > at > org.apache.ignite.internal.util.tostring.GridToStringBuilder.toStringImpl(GridToStringBuilder.java:474) > at > org.apache.ignite.internal.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:364) > at > org.apache.ignite.internal.processors.cache.transactions.IgniteTxStateImpl.toString(IgniteTxStateImpl.java:443) > at java.lang.String.valueOf(String.java:2849) > at > org.apache.ignite.internal.util.GridStringBuilder.a(GridStringBuilder.java:101) > at > org.apache.ignite.internal.util.tostring.GridToStringBuilder.toStringImpl(GridToStringBuilder.java:474) > at > org.apache.ignite.internal.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:297) > at > org.apache.ignite.internal.processors.cache.transactions.IgniteTxLocalAdapter.toString(IgniteTxLocalAdapter.java:3743) > at > org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocalAdapter.toString(GridDhtTxLocalAdapter.java:868) > at > org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocal.toString(GridDhtTxLocal.java:703) > at java.lang.String.valueOf(String.java:2849) > at java.lang.StringBuilder.append(StringBuilder.java:128) > at > org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.dumpPendingObjects(GridCachePartitionExchangeManager.java:1172) > at > org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.dumpDebugInfo(GridCachePartitionExchangeManager.java:1150) > at > org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.dumpPendingObjects(GridDhtPartitionsExchangeFuture.java:894) > at > org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.waitPartitionRelease(GridDhtPartitionsExchangeFuture.java:769) > at > org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.distributedExchange(GridDhtPartitionsExchangeFuture.java:715) > at > org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:472) > at > org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:1333) > ... 2 more > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)