Hi Maninda, Since we are using RDBMS to poll the node status, the cluster will not end up in situation 1,2 or 3. With this approach we consider a node unreachable when it cannot access the database. Therefore an unreachable node can never be the leader.
As you have mentioned, we are currently using the RDBMS as an atomic global variable to create the coordinator entry. On Tue, Aug 2, 2016 at 5:22 PM, Maninda Edirisooriya <mani...@wso2.com> wrote: > Hi Asanka, > > As I understand the accuracy of electing the leader correctly is dependent > on the election mechanism with RDBMS because there can be edge cases like, > > 1. Unreachable leader activates during the election process: Then who > becomes the leader? > 2. The elected leader becomes unreachable before the election is > completed: Then will there be a situation where there is no leader? > 3. A leader and a set of nodes are disconnected from the other part of the > cluster and while the leader is trying to remove unreachable members other > part is calling an election to make a leader: Who will win? > > RDBMS based election algorithm should handle such cases without bringing > the cluster to an inconsistent state or dead lock in all concurrent cases. > If all these kind of cases cannot be handled isn't it better to keep the > current hazelcast clustering and use the RDBMS only to handle the split > brain scenario? In other words when a new hazelcast leader is elected it > should be updated in the RDBMS. If another split party has already elected > a leader, the node who is going to write it to RDBMS should avoid updating > it. Simply, the RDBMS can be used as an atomic global variable to keep the > leader name by modifying the hazelcast clustering. WDYT? > > Thanks. > > > *Maninda Edirisooriya* > Senior Software Engineer > > *WSO2, Inc.*lean.enterprise.middleware. > > *Blog* : http://maninda.blogspot.com/ > *E-mail* : mani...@wso2.com > *Skype* : @manindae > *Twitter* : @maninda > > On Thu, Jul 28, 2016 at 4:38 PM, Asanka Abeyweera <asank...@wso2.com> > wrote: > >> Hi Akila, >> >> Let me explain the issue in a different way. Let's assume the MB nodes >> are using two different network interfaces for Hazelcast communication and >> database communication. With such a configuration, there can be failures >> only in the network interface used for Hazelcast communication in some >> nodes. When this happens, there will be two or more Hazelcast clusters due >> to the network segmentation, and as a result there will be multiple >> coordinators. Since every node still have access to the database, multiple >> coordinators can affect the correctness of the data stored in the DB. But >> if we used a RDBMS based approach we won't have multiple coordinators due >> to a network partition in Hazelcast. This is one advantage we get from this >> approach. >> >> Even when we use Zookeeper or RAFT the same issue will be there since we >> are using different interfaces for Hazelcast communication and DB >> communication. >> >> >> On Thu, Jul 28, 2016 at 2:56 PM, Akila Ravihansa Perera < >> raviha...@wso2.com> wrote: >> >>> Hi, >>> >>> What's the advantage of using RDBMS (even as an alternative) to >>> implement a leader/coordinator election? If the network connection to DB >>> fails then this will be a single point of failure. I don't think we can >>> scale RDBMS instances and expect the election algorithm to work. That would >>> be reducing this problem to another problem (electing coordinator RDBMS >>> instance). >>> >>> IMHO it would be better to look at Zookeeper Atomic Broadcast (ZAB) [1] >>> or RAFT leader election [2] algorithms which have already proven results. >>> >>> [1] https://cwiki.apache.org/confluence/display/ZOOKEEPER/Zab1.0 >>> [2] http://libraft.io/ >>> >>> Thanks. >>> >>> On Thu, Jul 28, 2016 at 1:42 PM, Nandika Jayawardana <nand...@wso2.com> >>> wrote: >>> >>>> +1 to make it a common component . We have the clustering >>>> implementation for BPEL component based on hazelcast. If the coordination >>>> is available at RDBMS level, we can remove hazelcast dependancy. >>>> >>>> Regards >>>> Nandika >>>> >>>> On Thu, Jul 28, 2016 at 1:28 PM, Hasitha Aravinda <hasi...@wso2.com> >>>> wrote: >>>> >>>>> Can we make it a common component, which is not hard coupled with MB. >>>>> BPS has the same requirement. >>>>> >>>>> Thanks, >>>>> Hasitha. >>>>> >>>>> On Thu, Jul 28, 2016 at 9:47 AM, Asanka Abeyweera <asank...@wso2.com> >>>>> wrote: >>>>> >>>>>> Hi All, >>>>>> >>>>>> In MB, we have used a coordinator based approach to manage >>>>>> distributed messaging algorithm in the cluster. Currently Hazelcast is >>>>>> used >>>>>> to elect the coordinator. But one issue we faced with Hazelcast is, >>>>>> during >>>>>> a network segmentation (split brain), Hazelcast can elect two or more >>>>>> coordinators in the cluster. This affects the correctness of the >>>>>> distributed messaging algorithm since there are some tables in the >>>>>> database >>>>>> that should only be edited by a single node (i.e. coordinator). >>>>>> >>>>>> As a solution to this problem we have implemented minimum node count >>>>>> based approach [1] to deactivate set of partitioned nodes to stop >>>>>> multiple >>>>>> nodes becoming coordinators until the network segmentation issue is >>>>>> fixed. >>>>>> >>>>>> As an alternative solution, we are thinking of implementing an RDBMS >>>>>> based approach to elect the coordinator node in the cluster. By doing >>>>>> this >>>>>> we can make sure that even during a network segmentation only one node >>>>>> will >>>>>> be elected as the coordinator node since the election is happening >>>>>> through >>>>>> the database. >>>>>> >>>>>> The algorithm will use a polling mechanism to check the validity of >>>>>> the nodes. To make the election algorithm scalable, only the coordinator >>>>>> node will be checking status of all the nodes in the cluster and it will >>>>>> inform other nodes through database when a member is added/left. The >>>>>> nodes >>>>>> will be only checking for the status of the coordinator node. When a node >>>>>> detect that coordinator is invalid it will go for a election to elect a >>>>>> new >>>>>> coordinator. >>>>>> >>>>>> We are currently working on a POC to test how this works with MB's >>>>>> slot based messaging algorithm. >>>>>> >>>>>> thoughts? >>>>>> >>>>>> [1] https://wso2.org/jira/browse/MB-1664 >>>>>> >>>>>> -- >>>>>> Asanka Abeyweera >>>>>> Senior Software Engineer >>>>>> WSO2 Inc. >>>>>> >>>>>> Phone: +94 712228648 >>>>>> Blog: a5anka.github.io >>>>>> >>>>>> <https://wso2.com/signature> >>>>>> >>>>>> _______________________________________________ >>>>>> Architecture mailing list >>>>>> Architecture@wso2.org >>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> -- >>>>> Hasitha Aravinda, >>>>> Associate Technical Lead, >>>>> WSO2 Inc. >>>>> Email: hasi...@wso2.com >>>>> Mobile : +94 718 210 200 >>>>> >>>>> _______________________________________________ >>>>> Architecture mailing list >>>>> Architecture@wso2.org >>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>> >>>>> >>>> >>>> >>>> -- >>>> Nandika Jayawardana >>>> WSO2 Inc ; http://wso2.com >>>> lean.enterprise.middleware >>>> >>>> _______________________________________________ >>>> Architecture mailing list >>>> Architecture@wso2.org >>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>> >>>> >>> >>> >>> -- >>> Akila Ravihansa Perera >>> WSO2 Inc.; http://wso2.com/ >>> >>> Blog: http://ravihansa3000.blogspot.com >>> >>> _______________________________________________ >>> Architecture mailing list >>> Architecture@wso2.org >>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>> >>> >> >> >> -- >> Asanka Abeyweera >> Senior Software Engineer >> WSO2 Inc. >> >> Phone: +94 712228648 >> Blog: a5anka.github.io >> >> <https://wso2.com/signature> >> >> _______________________________________________ >> Architecture mailing list >> Architecture@wso2.org >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> >> > -- Asanka Abeyweera Senior Software Engineer WSO2 Inc. Phone: +94 712228648 Blog: a5anka.github.io <https://wso2.com/signature>
_______________________________________________ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
