[ https://issues.apache.org/jira/browse/GEODE-9764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17447553#comment-17447553 ]
Anthony Baker commented on GEODE-9764: -------------------------------------- I would argue that for certain messages like replication of values, timeouts alone are insufficient. To maintain consistency, we have to replicate the change or revert it. I think that implies the need for timeouts as well failure detection improvements. > Request-Response Messaging Should Time Out > ------------------------------------------ > > Key: GEODE-9764 > URL: https://issues.apache.org/jira/browse/GEODE-9764 > Project: Geode > Issue Type: Improvement > Components: messaging > Reporter: Bill Burcham > Assignee: Bill Burcham > Priority: Major > > There is a weakness in the P2P/DirectChannel messaging architecture, in that > it never gives up on a request (in a request-response scenario). As a result > a bug (software fault) anywhere from the point where the requesting thread > hands off the {{DistributionMessage}} e.g. to > {{ClusterDistributionManager.putOutgoing(DistributionMessage)}}, to the point > where that request is ultimately fulfilled on a (one) receiver, can result in > a hang (of some task on the send side, which is waiting for a response). > Well it's a little worse than that because any code in the return (response) > path can also cause disruption of the (response) flow, thereby leaving the > requesting task hanging. > If the code in the request path (primarily in P2P messaging) and the code in > the response path (P2P messaging and TBD higher-level code) were perfect this > might not be a problem. But there is a fair amount of code there and we have > some evidence that it is currently not perfect, nor do we expect it to become > perfect and stay that way. That being the case it seems prudent to institute > response timeouts so that bugs of this sort (which disrupt request-response > message flow) don't result in hangs. > It's TBD if we want to go a step further and institute retries. The latter > would entail introducing duplicate-suppression (conflation) in P2P messaging. > We might also add exponential backoff (open-loop) or back-pressure > (closed-loop) to prevent a flood of retries when the system is at or near the > point of thrashing. > But even without retries, a configurable timeout might have good ROI as a > first step. This would entail: > * adding a configuration parameter to specify the timeout value > * changing ReplyProcessor21 and others TBD to "give up" after the timeout has > elapsed > * changing higher-level code dependent on request-reply messaging so it > properly handles the situations where we might have to "give up" > This issue affects all versions of Geode. -- This message was sent by Atlassian Jira (v8.20.1#820001)