[ https://issues.apache.org/jira/browse/IGNITE-13111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17138300#comment-17138300 ]
Vladimir Steshin edited comment on IGNITE-13111 at 6/17/20, 10:06 AM: ---------------------------------------------------------------------- I find IGNITE-13016 as better solution. We cannot rely on ping interval because two nodes are involved in backward connection checking. They work with same but shifted ping intervals. If node N asks N+2 to check N+1, N+2 waits for the rest of its failureDetectionTimeout. But ping and failureDetectionTimeout on N are shifted in comparision with N+2. N can fail before N+2 has waited for ping from N+1. was (Author: vladsz83): I find IGNITE-13016 or IGNITE-13014 better solution. We cannot rely on ping interval because two nodes are involved in backward connection checking. They work with same but shifted ping intervals. If node N asks N+2 to check N+1, N+2 waits for the rest of its failureDetectionTimeout. But ping and failureDetectionTimeout on N are shifted in comparision with N+2. N can fail before N+2 has waited for ping from N+1. > Simplify backward checking of node connection. > ---------------------------------------------- > > Key: IGNITE-13111 > URL: https://issues.apache.org/jira/browse/IGNITE-13111 > Project: Ignite > Issue Type: Improvement > Reporter: Vladimir Steshin > Assignee: Vladimir Steshin > Priority: Major > Labels: iep-45 > Attachments: FailureDetectionResearch.patch, > FailureDetectionResearch.txt, FailureDetectionResearch_fixed.txt, > WostCaseStepByStep.txt > > > We should fix several drawbacks in the backward checking of failed node. They > prolong node failure detection upto: > ServerImpl.CON_CHECK_INTERVAL + 2 * > IgniteConfiguretion.failureDetectionTimeout + 300ms. > See: > * ‘_NodeFailureResearch.patch_'. It creates test 'FailureDetectionResearch' > which emulates long answears on a failed node and measures failure detection > delays. > * '_FailureDetectionResearch.txt_' - results of the test. > * '_FailureDetectionResearch_fixed.txt_' - results of the test after this fix. > * '_WostCaseStepByStep.txt_' - description how the worst case happens. > *Suggestion:* > 1) We can simplify backward connection checking as we implement IGNITE-13012. > Once we get robust, predictable connection ping, we don't need to check > previous node because we can see whether it sent ping to current node within > failure detection timeout. If not, previous node can be considered lost. > Instead of: > {code:java} > // Node cannot connect to it's next (for local node it's previous). > // Need to check connectivity to it. > long rcvdTime = lastRingMsgReceivedTime; > long now = U.currentTimeMillis(); > // We got message from previous in less than double > connection check interval. > boolean ok = rcvdTime + effectiveExchangeTimeout() >= > now; > TcpDiscoveryNode previous = null; > if (ok) { > // Check case when previous node suddenly died. > This will speed up > // node failing. > Checking connection to previous node > } > {code} > we could wait for ping from previous node. Scenario: > * n1 (Node1) failed to connect to n2. > * n1 asks n3 to establish connection instead of n2. > * n3 waits for ping form n2 for the rest of failure detection timeout. > * If n3 received ping from n2, it connects with n1. Or answers n1 that n2 is > considered alive. > 2) Then, seems we can remove: > {code:java} > ServerImpl.SocketReader.isConnectionRefused(SocketAddress addr); > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)