[ https://issues.apache.org/jira/browse/IGNITE-5457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alexei Scherbakov updated IGNITE-5457: -------------------------------------- Description: I observe buggy behavior in case of simulated split brain. Nodes in DataCenter1 (where coordinator is located) are slowly leave grid, while nodes in DataCenter2 stay in grid forever. In logs I see multiple attemps to kick coordinator by communcation by connect timeout, but number of nodes does not change. Note what my failureDetectionTimeout is significantly higher than communication connect timeout. Looks like coordinator cannot be kicked from topology by TcpCommuncationSpi. {noformat} 19:14:03.289 [WARN ] [o.a.i.s.c.tcp.TcpCommunicationSpi] [T:] - Connect timed out (consider increasing 'connTimeout' configuration property) [addr=grid457.ca.sbrf.ru/10.116.206.193:47100, connTimeout=120000] 19:14:03.289 [ERROR] [o.a.i.s.c.tcp.TcpCommunicationSpi] [T:] - TcpCommunicationSpi failed to establish connection to node, node will be dropped from cluster [rmtNode=TcpDiscoveryNode [id=a8ac1b24-8377-4064-a3d9-02bad9c6f2bb, addrs=[10.116.206.193], sockAddrs=[grid457.ca.sbrf.ru/10.116.206.193:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1496936257121, loc=false, ver=1.10.3#20170604-sha1:30521a17, isClient=false]] org.apache.ignite.IgniteCheckedException: Failed to connect to node (is node still alive?). Make sure that each ComputeTask and cache Transaction has a timeout set in order to prevent parties from waiting forever in case of network issues [nodeId=a8ac1b24-8377-4064-a3d9-02bad9c6f2bb, addrs=[grid457.ca.sbrf.ru/10.116.206.193:47100]] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3022) [ignite-core-1.10.3.ea10.jar:1.10.3.ea10] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioClient(TcpCommunicationSpi.java:2636) [ignite-core-1.10.3.ea10.jar:1.10.3.ea10] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:2528) [ignite-core-1.10.3.ea10.jar:1.10.3.ea10] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.access$5800(TcpCommunicationSpi.java:245) [ignite-core-1.10.3.ea10.jar:1.10.3.ea10] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$CommunicationWorker.processDisconnect(TcpCommunicationSpi.java:3830) [ignite-core-1.10.3.ea10.jar:1.10.3.ea10] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$CommunicationWorker.body(TcpCommunicationSpi.java:3656) [ignite-core-1.10.3.ea10.jar:1.10.3.ea10] at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62) [ignite-core-1.10.3.ea10.jar:1.10.3.ea10] Suppressed: org.apache.ignite.IgniteCheckedException: Failed to connect to address [addr=grid457.ca.sbrf.ru/10.116.206.193:47100, err=null] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3027) [ignite-core-1.10.3.ea10.jar:1.10.3.ea10] ... 6 common frames omitted Caused by: java.net.SocketTimeoutException: null at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:118) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:2884) ... 6 common frames omitted 19:14:37.967 [INFO ] [o.a.i.i.IgniteKernal%DPL_GRID%grid880] [T:] - Metrics for local node (to disable set 'metricsLogFrequency' to 0) ^-- Node [id=21e01ea7, name=DPL_GRID%grid880, uptime=00:37:00:200] ^-- H/N/C [hosts=144, nodes=160, CPUs=8064] ^-- CPU [cur=0.2%, avg=2.37%, GC=0%] ^-- PageMemory [pages=604144] ^-- Heap [used=33396MB, free=49.04%, comm=65536MB] ^-- Non heap [used=171MB, free=-1%, comm=173MB] ^-- Public thread pool [active=0, idle=0, qSize=0] ^-- System thread pool [active=0, idle=0, qSize=0] ^-- Outbound messages queue [size=0] {noformat} Number of nodes is same as before kick attempt. was: I observe buggy behavior in case of simulated split brain. Nodes in DataCenter1 (where coordinator is located) are slowly leave grid, while nodes in DataCenter2 stay in grid forever. In logs I see multiple attemps to kick coordinator by communcation by connect timeout, but number of nodes does not change. Note what my failureDetectionTimeout is significantly higher than communication connect timeout. Looks like coordinator cannot be kicked from topology by TcpCommuncationSpi. {noformat} 19:14:03.289 [WARN ] [o.a.i.s.c.tcp.TcpCommunicationSpi] [T:] - Connect timed out (consider increasing 'connTimeout' configuration property) [addr=grid457.ca.sbrf.ru/10.116.206.193:47100, connTimeout=120000] 19:14:03.289 [ERROR] [o.a.i.s.c.tcp.TcpCommunicationSpi] [T:] - TcpCommunicationSpi failed to establish connection to node, node will be dropped from cluster [rmtNode=TcpDiscoveryNode [id=a8ac1b24-8377-4064-a3d9-02bad9c6f2bb, addrs=[10.116.206.193], sockAddrs=[grid457.ca.sbrf.ru/10.116.206.193:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1496936257121, loc=false, ver=1.10.3#20170604-sha1:30521a17, isClient=false]] org.apache.ignite.IgniteCheckedException: Failed to connect to node (is node still alive?). Make sure that each ComputeTask and cache Transaction has a timeout set in order to prevent parties from waiting forever in case of network issues [nodeId=a8ac1b24-8377-4064-a3d9-02bad9c6f2bb, addrs=[grid457.ca.sbrf.ru/10.116.206.193:47100]] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3022) [ignite-core-1.10.3.ea10.jar:1.10.3.ea10] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioClient(TcpCommunicationSpi.java:2636) [ignite-core-1.10.3.ea10.jar:1.10.3.ea10] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:2528) [ignite-core-1.10.3.ea10.jar:1.10.3.ea10] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.access$5800(TcpCommunicationSpi.java:245) [ignite-core-1.10.3.ea10.jar:1.10.3.ea10] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$CommunicationWorker.processDisconnect(TcpCommunicationSpi.java:3830) [ignite-core-1.10.3.ea10.jar:1.10.3.ea10] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$CommunicationWorker.body(TcpCommunicationSpi.java:3656) [ignite-core-1.10.3.ea10.jar:1.10.3.ea10] at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62) [ignite-core-1.10.3.ea10.jar:1.10.3.ea10] Suppressed: org.apache.ignite.IgniteCheckedException: Failed to connect to address [addr=grid457.ca.sbrf.ru/10.116.206.193:47100, err=null] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3027) [ignite-core-1.10.3.ea10.jar:1.10.3.ea10] ... 6 common frames omitted Caused by: java.net.SocketTimeoutException: null at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:118) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:2884) ... 6 common frames omitted 19:14:37.967 [INFO ] [o.a.i.i.IgniteKernal%DPL_GRID%grid880] [T:] - Metrics for local node (to disable set 'metricsLogFrequency' to 0) ^-- Node [id=21e01ea7, name=DPL_GRID%grid880, uptime=00:37:00:200] ^-- H/N/C [hosts=144, nodes=160, CPUs=8064] ^-- CPU [cur=0.2%, avg=2.37%, GC=0%] ^-- PageMemory [pages=604144] ^-- Heap [used=33396MB, free=49.04%, comm=65536MB] ^-- Non heap [used=171MB, free=-1%, comm=173MB] ^-- Public thread pool [active=0, idle=0, qSize=0] ^-- System thread pool [active=0, idle=0, qSize=0] ^-- Outbound messages queue [size=0] {noformat} > Weird discovery behavior on split brain. > ---------------------------------------- > > Key: IGNITE-5457 > URL: https://issues.apache.org/jira/browse/IGNITE-5457 > Project: Ignite > Issue Type: Bug > Components: general > Affects Versions: 2.0 > Reporter: Alexei Scherbakov > Priority: Critical > Fix For: 2.2 > > > I observe buggy behavior in case of simulated split brain. > Nodes in DataCenter1 (where coordinator is located) are slowly leave grid, > while nodes in DataCenter2 stay in grid forever. > In logs I see multiple attemps to kick coordinator by communcation by connect > timeout, but number of nodes does not change. > Note what my failureDetectionTimeout is significantly higher than > communication connect timeout. > Looks like coordinator cannot be kicked from topology by TcpCommuncationSpi. > {noformat} > 19:14:03.289 [WARN ] [o.a.i.s.c.tcp.TcpCommunicationSpi] [T:] - Connect timed > out (consider increasing 'connTimeout' configuration property) > [addr=grid457.ca.sbrf.ru/10.116.206.193:47100, connTimeout=120000] > 19:14:03.289 [ERROR] [o.a.i.s.c.tcp.TcpCommunicationSpi] [T:] - > TcpCommunicationSpi failed to establish connection to node, node will be > dropped from cluster [rmtNode=TcpDiscoveryNode > [id=a8ac1b24-8377-4064-a3d9-02bad9c6f2bb, addrs=[10.116.206.193], > sockAddrs=[grid457.ca.sbrf.ru/10.116.206.193:47500], discPort=47500, order=1, > intOrder=1, lastExchangeTime=1496936257121, loc=false, > ver=1.10.3#20170604-sha1:30521a17, isClient=false]] > org.apache.ignite.IgniteCheckedException: Failed to connect to node (is node > still alive?). Make sure that each ComputeTask and cache Transaction has a > timeout set in order to prevent parties from waiting forever in case of > network issues [nodeId=a8ac1b24-8377-4064-a3d9-02bad9c6f2bb, > addrs=[grid457.ca.sbrf.ru/10.116.206.193:47100]] > at > org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3022) > [ignite-core-1.10.3.ea10.jar:1.10.3.ea10] > at > org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioClient(TcpCommunicationSpi.java:2636) > [ignite-core-1.10.3.ea10.jar:1.10.3.ea10] > at > org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:2528) > [ignite-core-1.10.3.ea10.jar:1.10.3.ea10] > at > org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.access$5800(TcpCommunicationSpi.java:245) > [ignite-core-1.10.3.ea10.jar:1.10.3.ea10] > at > org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$CommunicationWorker.processDisconnect(TcpCommunicationSpi.java:3830) > [ignite-core-1.10.3.ea10.jar:1.10.3.ea10] > at > org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$CommunicationWorker.body(TcpCommunicationSpi.java:3656) > [ignite-core-1.10.3.ea10.jar:1.10.3.ea10] > at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62) > [ignite-core-1.10.3.ea10.jar:1.10.3.ea10] > Suppressed: org.apache.ignite.IgniteCheckedException: Failed to connect > to address [addr=grid457.ca.sbrf.ru/10.116.206.193:47100, err=null] > at > org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3027) > [ignite-core-1.10.3.ea10.jar:1.10.3.ea10] > ... 6 common frames omitted > Caused by: java.net.SocketTimeoutException: null > at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:118) > at > org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:2884) > ... 6 common frames omitted > 19:14:37.967 [INFO ] [o.a.i.i.IgniteKernal%DPL_GRID%grid880] [T:] - > Metrics for local node (to disable set 'metricsLogFrequency' to 0) > ^-- Node [id=21e01ea7, name=DPL_GRID%grid880, uptime=00:37:00:200] > ^-- H/N/C [hosts=144, nodes=160, CPUs=8064] > ^-- CPU [cur=0.2%, avg=2.37%, GC=0%] > ^-- PageMemory [pages=604144] > ^-- Heap [used=33396MB, free=49.04%, comm=65536MB] > ^-- Non heap [used=171MB, free=-1%, comm=173MB] > ^-- Public thread pool [active=0, idle=0, qSize=0] > ^-- System thread pool [active=0, idle=0, qSize=0] > ^-- Outbound messages queue [size=0] > {noformat} > Number of nodes is same as before kick attempt. -- This message was sent by Atlassian JIRA (v6.3.15#6346)