[ https://issues.apache.org/jira/browse/IGNITE-5115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16709900#comment-16709900 ]
Amelchev Nikita commented on IGNITE-5115: ----------------------------------------- [~agoncharuk], In my fix, the behavior is the same, except for *disconnect discovery, not stopping*. I call the method _segmentLocalNodeOnSendFail()_. It notifies discovery with the _EVT_NODE_SEGMENTED_ event and then _DiscoveryWorker_ process it and call the _onSegmentation()_ method where discovery disconnecting from topology and applying segmentation policy. > Investigation of failing tests of coordinator node failure > ----------------------------------------------------------- > > Key: IGNITE-5115 > URL: https://issues.apache.org/jira/browse/IGNITE-5115 > Project: Ignite > Issue Type: Bug > Components: messaging > Reporter: Sergey Chugunov > Assignee: Amelchev Nikita > Priority: Major > Labels: MakeTeamcityGreenAgain > Fix For: 2.8 > > > Tests *customEventCoordinatorFailure1/2* from *TcpDiscoverySelfTest* are > flaky on TC and sometimes hang with the following assertion in logs: > {code} > Exception in thread "tcp-disco-msg-worker-#5245%tcp.TcpDiscoverySelfTest0%" > java.lang.AssertionError > at > org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNodesRing.removeNode(TcpDiscoveryNodesRing.java:353) > at > org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.processNodeFailedMessage(ServerImpl.java:4670) > at > org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.processMessage(ServerImpl.java:2567) > at > org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.processMessage(ServerImpl.java:2366) > at > org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerAdapter.body(ServerImpl.java:6485) > at > org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2456) > at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62) > {code} > It seems that this happens because tests' implementation drops connections of > *TcpCommunicatonSpi* on coordinator node with *simulateNodeFailure* method. > At the same time tests leave *TcpDiscoverySpi* operational; it receives > subsequent NodeFailed message and throws the assertion error shown above. > The whole situation looks legitimate as it is possible to imagine a situation > when CommSPI connections on coordinator fail for some reason while DiscoSPI > connections are healthy. > It is needed to investigate the situation deeper, figure out whether the root > cause is using of *simulateNodeFailure* or not and propose a solution if the > error may happen in the real life. -- This message was sent by Atlassian JIRA (v7.6.3#76005)