Ksenia Rybakova created IGNITE-3402: ---------------------------------------
Summary: "Remote node ID is not as expected" errors at driver during load test Key: IGNITE-3402 URL: https://issues.apache.org/jira/browse/IGNITE-3402 Project: Ignite Issue Type: Bug Components: general Reporter: Ksenia Rybakova Fix For: 1.7 A lot of errors at some drivers after several minutes after grid is up during load test: {noformat} [14:33:42] (err) Failed to execute compound future reducer: Compound future listener []class org.apache.ignite.IgniteCheckedException: Failed to send message (node may have left the grid or TCP connection cannot be established due to fir at org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:1129) at org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:1193) at org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl$Buffer.submit(DataStreamerImpl.java:1378) at org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl$Buffer.update(DataStreamerImpl.java:1173) at org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl.load0(DataStreamerImpl.java:756) at org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl.addDataInternal(DataStreamerImpl.java:568) at org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl.addData(DataStreamerImpl.java:604) at org.apache.ignite.yardstick.cache.load.IgniteCacheRandomOperationBenchmark$1.run(IgniteCacheRandomOperationBenchmark.java:383) Caused by: class org.apache.ignite.spi.IgniteSpiException: Failed to send message to remote node: TcpDiscoveryNode [id=11da20f5-87c5-4321-8163-ffef85f2d54c, addrs=[10.20.0.215, 127.0.0.1], sockAddrs=[/127.0.0.1:47503, fosters-215/10.20.0 at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:1986) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:1926) at org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:1124) ... 7 more Caused by: class org.apache.ignite.IgniteCheckedException: Failed to connect to node (is node still alive?). Make sure that each GridComputeTask and GridCacheTransaction has a timeout set in order to prevent parties from waiting forever at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:2489) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioClient(TcpCommunicationSpi.java:2130) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:2024) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:1960) ... 9 more Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to connect to address: fosters-215/10.20.0.215:47103 at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:2494) ... 12 more Caused by: class org.apache.ignite.IgniteCheckedException: Failed to read remote node recovery handshake (connection closed). at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.safeHandshake(TcpCommunicationSpi.java:2699) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:2361) ... 12 more Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to connect to address: /127.0.0.1:47103 at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:2494) ... 12 more Caused by: class org.apache.ignite.IgniteCheckedException: Remote node ID is not as expected [expected=11da20f5-87c5-4321-8163-ffef85f2d54c, rcvd=03bc7d54-4fec-481d-af24-6f5408b32fe2] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.safeHandshake(TcpCommunicationSpi.java:2604) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:2361) ... 12 more {noformat} Tested version: 1.5.27 Load config: - 7 hosts with 4 servers and 3 drivers each (28 servers and 21 drivers in total) - benchmark: IgniteCacheRandomOperationBenchmark - preload: 2M - key range: 4M - warmup: 30s - duration: 10h - operations: PUT PUT_ALL GET GET_ALL INVOKE INVOKE_ALL REMOVE REMOVE_ALL PUT_IF_ABSENT REPLACE -backup count: 1 - GC: G1 - heap size: 8Gb for servers, 4Gb for drivers - caches: {noformat} <bean name="atomic" class="org.apache.ignite.configuration.CacheConfiguration"> <property name="name" value="atomic"/> <property name="cacheMode" value="PARTITIONED"/> <property name="atomicityMode" value="ATOMIC"/> <property name="swapEnabled" value="false"/> </bean> <bean name="atomic-evict" class="org.apache.ignite.configuration.CacheConfiguration"> <property name="name" value="atomic-evict"/> <property name="cacheMode" value="PARTITIONED"/> <property name="atomicityMode" value="ATOMIC"/> <property name="swapEnabled" value="false"/> <property name="offHeapMaxMemory" value="#{1024L*1024L*1024L*6L}"/> <property name="evictionPolicy"> <bean class="org.apache.ignite.cache.eviction.fifo.FifoEvictionPolicy"> <property name="maxSize" value="750000"/> </bean> </property> </bean> <bean name="atomic-offheap" class="org.apache.ignite.configuration.CacheConfiguration"> <property name="name" value="atomic-offheap"/> <property name="cacheMode" value="PARTITIONED"/> <property name="atomicityMode" value="ATOMIC"/> <property name="swapEnabled" value="false"/> <property name="memoryMode" value="OFFHEAP_TIERED"/> </bean> <bean name="atomic-offheap-values" class="org.apache.ignite.configuration.CacheConfiguration"> <property name="name" value="atomic-offheap-values"/> <property name="cacheMode" value="PARTITIONED"/> <property name="atomicityMode" value="ATOMIC"/> <property name="swapEnabled" value="false"/> <property name="memoryMode" value="OFFHEAP_VALUES"/> </bean> <bean name="tx" class="org.apache.ignite.configuration.CacheConfiguration"> <property name="name" value="tx"/> <property name="cacheMode" value="PARTITIONED"/> <property name="atomicityMode" value="TRANSACTIONAL"/> <property name="swapEnabled" value="false"/> </bean> <bean name="tx-evict" class="org.apache.ignite.configuration.CacheConfiguration"> <property name="name" value="tx-evict"/> <property name="cacheMode" value="PARTITIONED"/> <property name="atomicityMode" value="TRANSACTIONAL"/> <property name="swapEnabled" value="false"/> <property name="offHeapMaxMemory" value="#{1024L*1024L*1024L*6L}"/> <property name="evictionPolicy"> <bean class="org.apache.ignite.cache.eviction.lru.LruEvictionPolicy"> <property name="maxSize" value="750000"/> </bean> </property> </bean> <bean name="compute" class="org.apache.ignite.configuration.CacheConfiguration"> <property name="name" value="compute"/> <property name="cacheMode" value="PARTITIONED"/> <property name="atomicityMode" value="TRANSACTIONAL"/> <property name="swapEnabled" value="false"/> <property name="offHeapMaxMemory" value="#{1024L*1024L*1024L*6L}"/> <property name="evictionPolicy"> <bean class="org.apache.ignite.cache.eviction.fifo.FifoEvictionPolicy"> <property name="maxSize" value="750000"/> </bean> </property> </bean> <bean name="tx-offheap" class="org.apache.ignite.configuration.CacheConfiguration"> <property name="name" value="tx-offheap"/> <property name="cacheMode" value="PARTITIONED"/> <property name="atomicityMode" value="TRANSACTIONAL"/> <property name="swapEnabled" value="false"/> <property name="memoryMode" value="OFFHEAP_TIERED"/> </bean> <bean name="tx-offheap-values" class="org.apache.ignite.configuration.CacheConfiguration"> <property name="name" value="tx-offheap-values"/> <property name="cacheMode" value="PARTITIONED"/> <property name="atomicityMode" value="TRANSACTIONAL"/> <property name="swapEnabled" value="false"/> <property name="memoryMode" value="OFFHEAP_VALUES"/> </bean> <bean name="atomic-index" class="org.apache.ignite.configuration.CacheConfiguration"> <property name="name" value="atomic-index"/> <property name="cacheMode" value="PARTITIONED"/> <property name="atomicityMode" value="ATOMIC"/> <property name="swapEnabled" value="false"/> <property name="indexedTypes"> <list> <value>java.lang.Integer</value> <value>org.apache.ignite.yardstick.cache.model.Person1</value> </list> </property> </bean> <bean name="query" class="org.apache.ignite.configuration.CacheConfiguration"> <property name="name" value="query"/> <property name="cacheMode" value="PARTITIONED"/> <property name="atomicityMode" value="ATOMIC"/> <property name="swapEnabled" value="false"/> <!-- Configure query entities --> <property name="queryEntities"> <list> <bean class="org.apache.ignite.cache.QueryEntity"> <property name="keyType" value="java.lang.Integer"/> <property name="valueType" value="org.apache.ignite.yardstick.cache.model.Person"/> <property name="fields"> <map> <entry key="id" value="java.lang.Integer"/> <entry key="orgId" value="java.lang.Integer"/> <entry key="firstName" value="java.lang.String"/> <entry key="lastName" value="java.lang.String"/> <entry key="salary" value="java.lang.Double"/> </map> </property> <property name="indexes"> <list> <bean class="org.apache.ignite.cache.QueryIndex"> <constructor-arg value="id"/> </bean> <!-- Group index. --> <bean class="org.apache.ignite.cache.QueryIndex"> <constructor-arg> <list> <value>id</value> <value>salary</value> </list> </constructor-arg> <constructor-arg value="SORTED"/> </bean> </list> </property> </bean> </list> </property> </bean> <bean name="orgCache" class="org.apache.ignite.configuration.CacheConfiguration"> <property name="name" value="orgCache"/> <property name="cacheMode" value="PARTITIONED"/> <property name="atomicityMode" value="ATOMIC"/> <property name="swapEnabled" value="false"/> <!-- Configure query entities --> <property name="queryEntities"> <list> <bean class="org.apache.ignite.cache.QueryEntity"> <property name="keyType" value="java.lang.Integer"/> <property name="valueType" value="org.apache.ignite.yardstick.cache.model.Organization"/> <property name="fields"> <map> <entry key="id" value="java.lang.Integer"/> <entry key="name" value="java.lang.String"/> </map> </property> <property name="indexes"> <list> <bean class="org.apache.ignite.cache.QueryIndex"> <constructor-arg value="id"/> </bean> <bean class="org.apache.ignite.cache.QueryIndex"> <constructor-arg value="name"/> </bean> </list> </property> </bean> </list> </property> </bean> {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)