Hi, I have see that your grid suffer from long running operations:
[19:20:19,285][WARNING][grid-timeout-worker-#63%null%][GridCachePartitionExchangeManager] Found long running cache future [startTime=19:18:55.302, curTime=19:20:19.280, fut=GridDhtAtomicSingleUpdateFuture [key=MyProject.TokenSecondMap [idHash=1526386777, hash=2035509954, processingSecond=1174038983, token=38888], nearReaderEntry=null]] [19:20:19,286][WARNING][grid-timeout-worker-#63%null%][GridCachePartitionExchangeManager] Found long running cache future [startTime=19:18:55.302, curTime=19:20:19.280, ... by that reason it can not changing topology fast. This is topology before killing nodes: [19:17:42,341][INFO][disco-event-worker-#100%null%][GridDiscoveryManager] Topology snapshot [ver=30, servers=30, clients=0, CPUs=168, heap=120.0GB] and the same minor topology version after remaping failed out: [19:20:41,302][WARNING][exchange-worker-#103%null%][GridCachePartitionExchangeManager] Ready affinity version: AffinityTopologyVersion [topVer=30, minorTopVer=10] But actual exchange future have required 33 version already: [topVer=AffinityTopologyVersion [topVer=33, minorTopVer=0], nodeId=7484419b, evt=NODE_FAILED] Remap was failed, because grid had not been in time to change topology. You should avoid message about long running operations (reduce the load or change logic), otherwise exchange will going slow and you got the exception. Also, long exchange can hang whole grid, not only lead to the streamer exception. -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/what-does-remaps-mean-in-DataStreamerImpl-java-tp12033p12440.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
