+1 On Mon, Nov 7, 2016 at 12:40 PM, Anjana Fernando <anj...@wso2.com> wrote:
> Hi Ramith, > > Sure. Actually, I was talking with SameeraR to take over this and create a > common component which has the required coordination functionality. The > idea is to create a component, where the providers can be plugged in, such > as the RDBMS based one, ZK, or any other container specific provider that > maybe out there. > > Cheers, > Anjana. > > On Mon, Nov 7, 2016 at 12:38 PM, Ramith Jayasinghe <ram...@wso2.com> > wrote: > >> this might require some work.. shall we have a chat? >> >> On Thu, Nov 3, 2016 at 3:52 PM, Anjana Fernando <anj...@wso2.com> wrote: >> >>> Ping! .. >>> >>> On Wed, Nov 2, 2016 at 5:03 PM, Anjana Fernando <anj...@wso2.com> wrote: >>> >>>> Hi, >>>> >>>> On Wed, Nov 2, 2016 at 3:14 PM, Asanka Abeyweera <asank...@wso2.com> >>>> wrote: >>>> >>>>> Hi Anjana, >>>>> >>>>> Currently, the implementation is part of the MB code (not a common >>>>> component). >>>>> >>>> >>>> Okay, can we please get it as a common component. >>>> >>>> Cheers, >>>> Anjana. >>>> >>>> >>>>> >>>>> On Wed, Nov 2, 2016 at 3:00 PM, Anjana Fernando <anj...@wso2.com> >>>>> wrote: >>>>> >>>>>> Hi Asanka/Ramith, >>>>>> >>>>>> So for C5 based Streaming Analytics solution, we need coordination >>>>>> functionality there as well. Is the functionality mentioned here created >>>>>> as >>>>>> a common component or baked in to the MB code? .. if so, can we please >>>>>> get >>>>>> it implemented it as a generic component, so other products can also use >>>>>> it. >>>>>> >>>>>> Cheers, >>>>>> Anjana. >>>>>> >>>>>> On Tue, Aug 9, 2016 at 3:10 PM, Anjana Fernando <anj...@wso2.com> >>>>>> wrote: >>>>>> >>>>>>> Great! .. >>>>>>> >>>>>>> Cheers, >>>>>>> Anjana. >>>>>>> >>>>>>> On Tue, Aug 9, 2016 at 1:49 PM, Asanka Abeyweera <asank...@wso2.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Anjana, >>>>>>>> >>>>>>>> Thank you for the suggestion. We have already done a similar thing. >>>>>>>> We have added a backoff time after creating the leader entry and check >>>>>>>> if >>>>>>>> the leader entry is the entry created by self before informing the >>>>>>>> leader >>>>>>>> change. >>>>>>>> >>>>>>>> On Tue, Aug 9, 2016 at 12:27 PM, Anjana Fernando <anj...@wso2.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> I see, thanks for the clarification, looks good! .. I think small >>>>>>>>> thing to consider is, to avoid the situation where, the current >>>>>>>>> leader goes >>>>>>>>> away, and two other competes to become the leader, and the first one >>>>>>>>> and >>>>>>>>> the second one checks (reads) the table to check the last heartbeat >>>>>>>>> and >>>>>>>>> figures out that the leader is outdated at the same time, and then >>>>>>>>> first >>>>>>>>> one delete the entry and puts his one, and after that, second one >>>>>>>>> will also >>>>>>>>> delete the existing one and put his one, so both will think they >>>>>>>>> became the >>>>>>>>> leader, due to the condition that both succeeded in adding the entry >>>>>>>>> without an error. So this can probably be fixed by checking back >>>>>>>>> after a >>>>>>>>> bit of time if the current node is actually me, which >>>>>>>>> probabilistically >>>>>>>>> will work well, if that time period is sufficient big enough than a >>>>>>>>> typical >>>>>>>>> database transaction required by a node to do the earlier operations. >>>>>>>>> Or >>>>>>>>> else, we should make sure the database transaction level used in this >>>>>>>>> scenario is at least REPEATABLE_READ, where when we read the record, >>>>>>>>> it >>>>>>>>> will lock it throughout the transaction. So some DBMSs does not >>>>>>>>> support >>>>>>>>> REPEATABLE_READ, where in that case, we should be able to use >>>>>>>>> SERIALIZABLE, >>>>>>>>> which most of them support. >>>>>>>>> >>>>>>>>> Cheers, >>>>>>>>> Anjana. >>>>>>>>> >>>>>>>>> On Tue, Aug 9, 2016 at 11:11 AM, Maninda Edirisooriya < >>>>>>>>> mani...@wso2.com> wrote: >>>>>>>>> >>>>>>>>>> Hi Anjana, >>>>>>>>>> >>>>>>>>>> After having an offline chat with Asanka what I understood was >>>>>>>>>> that the leader election was done completely via the database but >>>>>>>>>> with no >>>>>>>>>> network communication. The leader is mentioned in the database >>>>>>>>>> first. Then >>>>>>>>>> the leader updates the node data periodically in the database. If >>>>>>>>>> some node >>>>>>>>>> realizes the data in the DB are outdated that means the leader was >>>>>>>>>> disconnected. Then that node will look at the created timestamp of >>>>>>>>>> the >>>>>>>>>> leader entry. If that is not very recent that means there was no new >>>>>>>>>> leader >>>>>>>>>> elected recently. So he will try to update the leader entry with his >>>>>>>>>> ID. As >>>>>>>>>> I understand there the leader entry is using the leader ID and the >>>>>>>>>> timestamp as the primary key. Even several nodes try to do it >>>>>>>>>> simultaneously only one node will successfully be able to update the >>>>>>>>>> entry >>>>>>>>>> with the help of atomicity provided by the DB. Others members will >>>>>>>>>> note the >>>>>>>>>> timestamp of the leader was updated so will accept the first one who >>>>>>>>>> updates as the leader. Even after the leader is elected, the leader >>>>>>>>>> will >>>>>>>>>> only notify node data via updating DB instead of network calls. >>>>>>>>>> Other nodes >>>>>>>>>> will just observe it and check the latest timestmps of the entry. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> *Maninda Edirisooriya* >>>>>>>>>> Senior Software Engineer >>>>>>>>>> >>>>>>>>>> *WSO2, Inc.*lean.enterprise.middleware. >>>>>>>>>> >>>>>>>>>> *Blog* : http://maninda.blogspot.com/ >>>>>>>>>> *E-mail* : mani...@wso2.com >>>>>>>>>> *Skype* : @manindae >>>>>>>>>> *Twitter* : @maninda >>>>>>>>>> >>>>>>>>>> On Tue, Aug 9, 2016 at 10:13 AM, Anjana Fernando <anj...@wso2.com >>>>>>>>>> > wrote: >>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> I just noticed this thread. I've some concerns on this >>>>>>>>>>> implementations. First of all, I don't think the statement >>>>>>>>>>> mentioned here >>>>>>>>>>> saying an external service such as ZooKeeper doesn't work, is >>>>>>>>>>> correct. >>>>>>>>>>> Because, if you have a ZK cluster (it is suppose to be used as a >>>>>>>>>>> cluster), >>>>>>>>>>> you will not have any issues. All the nodes have a list of >>>>>>>>>>> endpoints to all >>>>>>>>>>> the ZK nodes and they connect to those, and ZK has a quorum based >>>>>>>>>>> mechanism >>>>>>>>>>> in keeping its state. So this makes sure, all the users have a >>>>>>>>>>> single >>>>>>>>>>> version of the ZK data. >>>>>>>>>>> >>>>>>>>>>> Also, I guess the fundamental problem here in the split brain >>>>>>>>>>> situation is, we need one external entity taking the decision (e.g. >>>>>>>>>>> ZK >>>>>>>>>>> cluster), because it should have oversight to the whole >>>>>>>>>>> environment. I >>>>>>>>>>> don't see how this RDBMS mechanism would solve that. Because, what >>>>>>>>>>> it gives >>>>>>>>>>> is a central location of state persistence. But the decisions of >>>>>>>>>>> making who >>>>>>>>>>> is the leader is taken by the users, which can be problematic. >>>>>>>>>>> Where when >>>>>>>>>>> we have a network partition scenario in that occasion, two groups >>>>>>>>>>> of users >>>>>>>>>>> will be overriding each other in the centralized RDBMS data >>>>>>>>>>> repeatedly and >>>>>>>>>>> it will go on forever, where in the ZK situation, there will be >>>>>>>>>>> only one >>>>>>>>>>> leader, and the guys in the other partition simply won't be able to >>>>>>>>>>> reach >>>>>>>>>>> the leader, until its network issues are sorted. >>>>>>>>>>> >>>>>>>>>>> So I also think, as Imesh mentioned, creating a coordination >>>>>>>>>>> algorithm from scratch may not be a wise decision, and we should >>>>>>>>>>> use proven >>>>>>>>>>> technology/libraries to do that. And on a side note, the main >>>>>>>>>>> reason for >>>>>>>>>>> not using ZK for this earlier was because of the hassle of bringing >>>>>>>>>>> up >>>>>>>>>>> another set of servers when our products are clustered, and we knew >>>>>>>>>>> that >>>>>>>>>>> the split brain scenario will occur in HZ, but maybe now we should >>>>>>>>>>> give an >>>>>>>>>>> extension point probably to plug into an external service if for >>>>>>>>>>> some >>>>>>>>>>> applications the split brain scenario is a show stopper. >>>>>>>>>>> >>>>>>>>>>> Cheers, >>>>>>>>>>> Anjana. >>>>>>>>>>> >>>>>>>>>>> On Tue, Aug 9, 2016 at 4:45 AM, Kasun Indrasiri <ka...@wso2.com> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi Ramith/Asanka, >>>>>>>>>>>> >>>>>>>>>>>> ESB/DSS natask impl is also based on HZ. I guess if this model >>>>>>>>>>>> works for the MB, we should make it generic for all such >>>>>>>>>>>> coordination >>>>>>>>>>>> requirements. (Thinking about using this in ESB 5.1)? >>>>>>>>>>>> >>>>>>>>>>>> On Fri, Aug 5, 2016 at 3:58 AM, Sajini De Silva < >>>>>>>>>>>> saj...@wso2.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi Maninda, >>>>>>>>>>>>> >>>>>>>>>>>>> Locking the database will be supported by some databases but >>>>>>>>>>>>> there will be huge performance impact. So we cannot use that >>>>>>>>>>>>> approach. If >>>>>>>>>>>>> this approach cannot be adapted the only thing we can do is >>>>>>>>>>>>> queue wise >>>>>>>>>>>>> load balancing through slot coordinator. But in this case we >>>>>>>>>>>>> cannot >>>>>>>>>>>>> guarantee that load balance will be equally distributed since >>>>>>>>>>>>> some queues >>>>>>>>>>>>> can be loaded while some will be idle. Also we cannot have >>>>>>>>>>>>> multiple slot >>>>>>>>>>>>> coordinators having same queue as it may cause several >>>>>>>>>>>>> complications such >>>>>>>>>>>>> as, same slot is assigned to two nodes by different subscribers, >>>>>>>>>>>>> message >>>>>>>>>>>>> duplication etc. Actually this slot architecture was discussed in >>>>>>>>>>>>> a >>>>>>>>>>>>> separate mail thread before it is implemented. >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks >>>>>>>>>>>>> >>>>>>>>>>>>> On Fri, Aug 5, 2016 at 3:12 PM, Maninda Edirisooriya < >>>>>>>>>>>>> mani...@wso2.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hi Sajini, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Yes that is what I meant. As the number of slots are >>>>>>>>>>>>>> proportional to the number of messages passing through the >>>>>>>>>>>>>> cluster, slot >>>>>>>>>>>>>> delivery should not be handled by the coordinator when there is >>>>>>>>>>>>>> only one >>>>>>>>>>>>>> coordinator in the cluster which is a bottleneck for scaling >>>>>>>>>>>>>> messages >>>>>>>>>>>>>> passing through the cluster. If there is only a single >>>>>>>>>>>>>> coordinator, it >>>>>>>>>>>>>> should handle operations that are not proportional to messages >>>>>>>>>>>>>> throughput >>>>>>>>>>>>>> of the cluster. Then only the tasks like subscriber adding / >>>>>>>>>>>>>> removing >>>>>>>>>>>>>> should be handled by the coordinator. As this is not the current >>>>>>>>>>>>>> implementation, we can consider multiple coordinator approach. >>>>>>>>>>>>>> Then the >>>>>>>>>>>>>> number of coordinators should be scalable with the message >>>>>>>>>>>>>> throughout. I am >>>>>>>>>>>>>> not sure whether locking the database per transaction would >>>>>>>>>>>>>> achieve this >>>>>>>>>>>>>> coordinator scalability in the multiple coordinator >>>>>>>>>>>>>> implementation. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> *Maninda Edirisooriya* >>>>>>>>>>>>>> Senior Software Engineer >>>>>>>>>>>>>> >>>>>>>>>>>>>> *WSO2, Inc.*lean.enterprise.middleware. >>>>>>>>>>>>>> >>>>>>>>>>>>>> *Blog* : http://maninda.blogspot.com/ >>>>>>>>>>>>>> *E-mail* : mani...@wso2.com >>>>>>>>>>>>>> *Skype* : @manindae >>>>>>>>>>>>>> *Twitter* : @maninda >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Fri, Aug 5, 2016 at 2:42 PM, Sajini De Silva < >>>>>>>>>>>>>> saj...@wso2.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Maninda, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Fri, Aug 5, 2016 at 2:28 PM, Maninda Edirisooriya < >>>>>>>>>>>>>>> mani...@wso2.com> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> @Sajini, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> But the number of slots are proportional to the number of >>>>>>>>>>>>>>>> messages pass through the MB which needs to be handled by the >>>>>>>>>>>>>>>> coordinator. >>>>>>>>>>>>>>>> That is what I meant by "information related to meta data of >>>>>>>>>>>>>>>> messages pass >>>>>>>>>>>>>>>> through a single coordinator". Ideally after the senders and >>>>>>>>>>>>>>>> receivers are >>>>>>>>>>>>>>>> subscribed to the cluster, coordinator should have nothing to >>>>>>>>>>>>>>>> do until they >>>>>>>>>>>>>>>> are removed or changed. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Even though it is possible to have multiple coordinators >>>>>>>>>>>>>>> after having en effort (Lock the database for a whole >>>>>>>>>>>>>>> transaction or the >>>>>>>>>>>>>>> work load distribution as described by Ramith) , coordinator >>>>>>>>>>>>>>> may have >>>>>>>>>>>>>>> different work to do other than subscriber adding and removing. >>>>>>>>>>>>>>> As I said >>>>>>>>>>>>>>> earlier our MB message distribution system is based on slot >>>>>>>>>>>>>>> architecture >>>>>>>>>>>>>>> and slots will managed by the coordinator. You can read [1] to >>>>>>>>>>>>>>> understand >>>>>>>>>>>>>>> more about slot architecture in MB. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [1] http://sajinid.blogspot.com/2015/03/wso2-message-broker- >>>>>>>>>>>>>>> 300-slot-based.html >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> @Ramith, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> +1 for multiple coordinators by partitioning the cluster >>>>>>>>>>>>>>>> which maintains the simplicity and correctness of the >>>>>>>>>>>>>>>> algorithm than >>>>>>>>>>>>>>>> compromising simplicity with a less important factor like >>>>>>>>>>>>>>>> "delivering a >>>>>>>>>>>>>>>> good mix of messages". >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> *Maninda Edirisooriya* >>>>>>>>>>>>>>>> Senior Software Engineer >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> *WSO2, Inc.*lean.enterprise.middleware. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> *Blog* : http://maninda.blogspot.com/ >>>>>>>>>>>>>>>> *E-mail* : mani...@wso2.com >>>>>>>>>>>>>>>> *Skype* : @manindae >>>>>>>>>>>>>>>> *Twitter* : @maninda >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Fri, Aug 5, 2016 at 2:05 PM, Ramith Jayasinghe < >>>>>>>>>>>>>>>> ram...@wso2.com> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> @Imesh, >>>>>>>>>>>>>>>>> We can prove that doing leader election using >>>>>>>>>>>>>>>>> a lib (where we maintain cluster state in another place, >>>>>>>>>>>>>>>>> a.k.a DB) will not >>>>>>>>>>>>>>>>> solve our original problem (this also relates to our past >>>>>>>>>>>>>>>>> experience with >>>>>>>>>>>>>>>>> both the zookeeper and hazelcast). >>>>>>>>>>>>>>>>> We can make this implementation a common component if >>>>>>>>>>>>>>>>> other products have a use of it. BPS might be able to use it >>>>>>>>>>>>>>>>> since their >>>>>>>>>>>>>>>>> data is also in the database. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> @Malaka: >>>>>>>>>>>>>>>>> VFS scenario can't be solved by relying on this >>>>>>>>>>>>>>>>> implementation. why? you can have the access to DB but not VFS >>>>>>>>>>>>>>>>> resources/file (and vice versa). this is the same point we >>>>>>>>>>>>>>>>> explained before. >>>>>>>>>>>>>>>>> in Ntask implementation, if tasks are stored in the >>>>>>>>>>>>>>>>> database then using this implementation makes sense. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> @Akila, >>>>>>>>>>>>>>>>> implementing (distributed) a queue algorithm is >>>>>>>>>>>>>>>>> non-trivial. Having one coordinator (single source of truth) >>>>>>>>>>>>>>>>> keeps things >>>>>>>>>>>>>>>>> simple hence it's a conscious design decision we agreed >>>>>>>>>>>>>>>>> during the initial >>>>>>>>>>>>>>>>> stages. However, possible extension to this scheme is to have >>>>>>>>>>>>>>>>> multiple >>>>>>>>>>>>>>>>> coordinators ( each responsible for coordinating a subset of >>>>>>>>>>>>>>>>> queues in the >>>>>>>>>>>>>>>>> cluster), that will be some what similar to kafka. >>>>>>>>>>>>>>>>> Even if its preferable to have no coordinator at-all, (to >>>>>>>>>>>>>>>>> decide how messages are disseminated in the cluster) that >>>>>>>>>>>>>>>>> will make us >>>>>>>>>>>>>>>>> give up desired behaviour such as delivering a good mix of >>>>>>>>>>>>>>>>> messages (from >>>>>>>>>>>>>>>>> different publishers) to consumers in a cluster. having said >>>>>>>>>>>>>>>>> this, we have >>>>>>>>>>>>>>>>> an ongoing research on how to improve the algorithm and we >>>>>>>>>>>>>>>>> like to try out >>>>>>>>>>>>>>>>> both these approaches. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Fri, Aug 5, 2016 at 1:31 PM, Malaka Silva < >>>>>>>>>>>>>>>>> mal...@wso2.com> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> The same issue with Hazelcast can be experienced with ESB >>>>>>>>>>>>>>>>>> inbounds (running on top of NTASK) and VFS distribution >>>>>>>>>>>>>>>>>> locks. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> The idea of only single worker works at a given time >>>>>>>>>>>>>>>>>> breaks if there is a Hazelcast heart beat fails. This will >>>>>>>>>>>>>>>>>> make two workers >>>>>>>>>>>>>>>>>> to work in parallel. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Also with distributed locking there is no guarantee that >>>>>>>>>>>>>>>>>> file is only process only by one worker. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> So in the case of network fail >>>>>>>>>>>>>>>>>> with DB >>>>>>>>>>>>>>>>>> make sense to stop processing until it's recovered. >>>>>>>>>>>>>>>>>> Also making this component generic ESB can reuse. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Fri, Aug 5, 2016 at 9:21 AM, Asitha Nanayakkara < >>>>>>>>>>>>>>>>>> asi...@wso2.com> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Hi Imesh, >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> On Fri, Aug 5, 2016 at 7:33 AM, Imesh Gunaratne < >>>>>>>>>>>>>>>>>>> im...@wso2.com> wrote: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> On Fri, Aug 5, 2016 at 7:31 AM, Imesh Gunaratne < >>>>>>>>>>>>>>>>>>>> im...@wso2.com> wrote: >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> You can see here [3] how K8S has implemented leader >>>>>>>>>>>>>>>>>>>>> election feature for the products deployed on top of that >>>>>>>>>>>>>>>>>>>>> to utilize. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Correction: Please refer [4]. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 4, 2016 at 7:27 PM, Asanka Abeyweera < >>>>>>>>>>>>>>>>>>>>>> asank...@wso2.com> wrote: >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Hi Imesh, >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> We are not implementing this to overcome a >>>>>>>>>>>>>>>>>>>>>>> limitation in the coordination algorithm available in >>>>>>>>>>>>>>>>>>>>>>> the Hazlecast. We are >>>>>>>>>>>>>>>>>>>>>>> implementing this since we need an RDBMS based >>>>>>>>>>>>>>>>>>>>>>> coordination algorithm (not >>>>>>>>>>>>>>>>>>>>>>> a network based algorithm). >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Are you saying that database connections do not use >>>>>>>>>>>>>>>>>>>>> the same network used by Hazelcast? >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> The reason is, a network based election algorithm >>>>>>>>>>>>>>>>>>>>>>> will always elect multiple leaders when the network is >>>>>>>>>>>>>>>>>>>>>>> partitioned. But if >>>>>>>>>>>>>>>>>>>>>>> we use a RDBMS based algorithm this will not happen. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> I do not think your argument is correct. If there is >>>>>>>>>>>>>>>>>>>>> a problem with the network, it may apply to both >>>>>>>>>>>>>>>>>>>>> Hazelcast based solution >>>>>>>>>>>>>>>>>>>>> and database based solution. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Yes, if the same network interface is used network >>>>>>>>>>>>>>>>>>> partion will cause all types of connections to be >>>>>>>>>>>>>>>>>>> partitioned. But user can >>>>>>>>>>>>>>>>>>> use multiple network interfaces for database, Hazelcast and >>>>>>>>>>>>>>>>>>> thrift. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Following is the scenario we are trying to solve in MB. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> In MB all the details related to messages, >>>>>>>>>>>>>>>>>>> subscriptions, queues, topics etc are stored in database. >>>>>>>>>>>>>>>>>>> And we operate >>>>>>>>>>>>>>>>>>> depending on that information. If the MB node can't connect >>>>>>>>>>>>>>>>>>> to the database >>>>>>>>>>>>>>>>>>> that means the node is ineffective in the cluster until it >>>>>>>>>>>>>>>>>>> can make a >>>>>>>>>>>>>>>>>>> database connection. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> We have seen instances where Hazelcast cluster get >>>>>>>>>>>>>>>>>>> partitioned for some time period in networks, Reasons were, >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> 1. Due to heavy load Hazelcast couldn't process or >>>>>>>>>>>>>>>>>>> send (some times both) hearbeats, hence a network >>>>>>>>>>>>>>>>>>> partition for Hazelcast >>>>>>>>>>>>>>>>>>> cluster >>>>>>>>>>>>>>>>>>> 2. An actual network partition of Hazelcast cluster >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> In both scenarios the database connection was working. >>>>>>>>>>>>>>>>>>> In that case we get two coordinators elected through >>>>>>>>>>>>>>>>>>> Hazelcast and working >>>>>>>>>>>>>>>>>>> on the same database to deliver the messages. this leads to >>>>>>>>>>>>>>>>>>> inconsistencies >>>>>>>>>>>>>>>>>>> in the cluster behavior (for instances duplicate message >>>>>>>>>>>>>>>>>>> delivery, >>>>>>>>>>>>>>>>>>> corrupred subscription states etc) . >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Since the point of interest for MB is the database, we >>>>>>>>>>>>>>>>>>> decided to do the coordinator election through database as >>>>>>>>>>>>>>>>>>> well. If the >>>>>>>>>>>>>>>>>>> node can't connect to the database, then the MB won't >>>>>>>>>>>>>>>>>>> operate anyway. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>>>>>> Asitha >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> [4] http://blog.kubernetes.io/ >>>>>>>>>>>>>>>>>>>>> 2016/01/simple-leader-election-with-Kubernetes.html >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 4, 2016 at 7:16 PM, Imesh Gunaratne < >>>>>>>>>>>>>>>>>>>>>>> im...@wso2.com> wrote: >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Hi Asanka, >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Do we really need to implement a leader election >>>>>>>>>>>>>>>>>>>>>>>> algorithm on our own? AFAIU this is a complex problem >>>>>>>>>>>>>>>>>>>>>>>> which has been >>>>>>>>>>>>>>>>>>>>>>>> already solved by several algorithms [1]. IMO it would >>>>>>>>>>>>>>>>>>>>>>>> be better to go >>>>>>>>>>>>>>>>>>>>>>>> ahead with an existing well established implementation >>>>>>>>>>>>>>>>>>>>>>>> on etcd [1] or >>>>>>>>>>>>>>>>>>>>>>>> Consul [2]. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Those provide HTTP APIs for clients to make leader >>>>>>>>>>>>>>>>>>>>>>>> election calls. [3] is a client library written in >>>>>>>>>>>>>>>>>>>>>>>> Node.js for etcd based >>>>>>>>>>>>>>>>>>>>>>>> leader election. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> [1] https://www.projectcalico. >>>>>>>>>>>>>>>>>>>>>>>> org/using-etcd-for-elections >>>>>>>>>>>>>>>>>>>>>>>> [2] https://www.consul.io/docs >>>>>>>>>>>>>>>>>>>>>>>> /guides/leader-election.html >>>>>>>>>>>>>>>>>>>>>>>> [3] https://www.npmjs.com/package/etcd-leader >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> On Wed, Aug 3, 2016 at 5:12 PM, Asanka Abeyweera < >>>>>>>>>>>>>>>>>>>>>>>> asank...@wso2.com> wrote: >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> Hi Maninda, >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> Since we are using RDBMS to poll the node status, >>>>>>>>>>>>>>>>>>>>>>>>> the cluster will not end up in situation 1,2 or 3. >>>>>>>>>>>>>>>>>>>>>>>>> With this approach we >>>>>>>>>>>>>>>>>>>>>>>>> consider a node unreachable when it cannot access the >>>>>>>>>>>>>>>>>>>>>>>>> database. Therefore >>>>>>>>>>>>>>>>>>>>>>>>> an unreachable node can never be the leader. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> As you have mentioned, we are currently using the >>>>>>>>>>>>>>>>>>>>>>>>> RDBMS as an atomic global variable to create the >>>>>>>>>>>>>>>>>>>>>>>>> coordinator entry. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Aug 2, 2016 at 5:22 PM, Maninda >>>>>>>>>>>>>>>>>>>>>>>>> Edirisooriya <mani...@wso2.com> wrote: >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> Hi Asanka, >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> As I understand the accuracy of electing the >>>>>>>>>>>>>>>>>>>>>>>>>> leader correctly is dependent on the election >>>>>>>>>>>>>>>>>>>>>>>>>> mechanism with RDBMS because >>>>>>>>>>>>>>>>>>>>>>>>>> there can be edge cases like, >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> 1. Unreachable leader activates during the >>>>>>>>>>>>>>>>>>>>>>>>>> election process: Then who becomes the leader? >>>>>>>>>>>>>>>>>>>>>>>>>> 2. The elected leader becomes unreachable before >>>>>>>>>>>>>>>>>>>>>>>>>> the election is completed: Then will there be a >>>>>>>>>>>>>>>>>>>>>>>>>> situation where there is no >>>>>>>>>>>>>>>>>>>>>>>>>> leader? >>>>>>>>>>>>>>>>>>>>>>>>>> 3. A leader and a set of nodes are disconnected >>>>>>>>>>>>>>>>>>>>>>>>>> from the other part of the cluster and while the >>>>>>>>>>>>>>>>>>>>>>>>>> leader is trying to remove >>>>>>>>>>>>>>>>>>>>>>>>>> unreachable members other part is calling an >>>>>>>>>>>>>>>>>>>>>>>>>> election to make a leader: Who >>>>>>>>>>>>>>>>>>>>>>>>>> will win? >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> RDBMS based election algorithm should handle such >>>>>>>>>>>>>>>>>>>>>>>>>> cases without bringing the cluster to an >>>>>>>>>>>>>>>>>>>>>>>>>> inconsistent state or dead lock in >>>>>>>>>>>>>>>>>>>>>>>>>> all concurrent cases. If all these kind of cases >>>>>>>>>>>>>>>>>>>>>>>>>> cannot be handled isn't it >>>>>>>>>>>>>>>>>>>>>>>>>> better to keep the current hazelcast clustering and >>>>>>>>>>>>>>>>>>>>>>>>>> use the RDBMS only to >>>>>>>>>>>>>>>>>>>>>>>>>> handle the split brain scenario? In other words when >>>>>>>>>>>>>>>>>>>>>>>>>> a new hazelcast leader >>>>>>>>>>>>>>>>>>>>>>>>>> is elected it should be updated in the RDBMS. If >>>>>>>>>>>>>>>>>>>>>>>>>> another split party has >>>>>>>>>>>>>>>>>>>>>>>>>> already elected a leader, the node who is going to >>>>>>>>>>>>>>>>>>>>>>>>>> write it to RDBMS should >>>>>>>>>>>>>>>>>>>>>>>>>> avoid updating it. Simply, the RDBMS can be used as >>>>>>>>>>>>>>>>>>>>>>>>>> an atomic global >>>>>>>>>>>>>>>>>>>>>>>>>> variable to keep the leader name by modifying the >>>>>>>>>>>>>>>>>>>>>>>>>> hazelcast clustering. >>>>>>>>>>>>>>>>>>>>>>>>>> WDYT? >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks. >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> *Maninda Edirisooriya* >>>>>>>>>>>>>>>>>>>>>>>>>> Senior Software Engineer >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> *WSO2, Inc.*lean.enterprise.middleware. >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> *Blog* : http://maninda.blogspot.com/ >>>>>>>>>>>>>>>>>>>>>>>>>> *E-mail* : mani...@wso2.com >>>>>>>>>>>>>>>>>>>>>>>>>> *Skype* : @manindae >>>>>>>>>>>>>>>>>>>>>>>>>> *Twitter* : @maninda >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Jul 28, 2016 at 4:38 PM, Asanka Abeyweera >>>>>>>>>>>>>>>>>>>>>>>>>> <asank...@wso2.com> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Akila, >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Let me explain the issue in a different way. >>>>>>>>>>>>>>>>>>>>>>>>>>> Let's assume the MB nodes are using two different >>>>>>>>>>>>>>>>>>>>>>>>>>> network interfaces for >>>>>>>>>>>>>>>>>>>>>>>>>>> Hazelcast communication and database communication. >>>>>>>>>>>>>>>>>>>>>>>>>>> With such a >>>>>>>>>>>>>>>>>>>>>>>>>>> configuration, there can be failures only in the >>>>>>>>>>>>>>>>>>>>>>>>>>> network interface used for >>>>>>>>>>>>>>>>>>>>>>>>>>> Hazelcast communication in some nodes. When this >>>>>>>>>>>>>>>>>>>>>>>>>>> happens, there will be two >>>>>>>>>>>>>>>>>>>>>>>>>>> or more Hazelcast clusters due to the network >>>>>>>>>>>>>>>>>>>>>>>>>>> segmentation, and as a result >>>>>>>>>>>>>>>>>>>>>>>>>>> there will be multiple coordinators. Since every >>>>>>>>>>>>>>>>>>>>>>>>>>> node still have access to >>>>>>>>>>>>>>>>>>>>>>>>>>> the database, multiple coordinators can affect the >>>>>>>>>>>>>>>>>>>>>>>>>>> correctness of the data >>>>>>>>>>>>>>>>>>>>>>>>>>> stored in the DB. But if we used a RDBMS based >>>>>>>>>>>>>>>>>>>>>>>>>>> approach we won't have >>>>>>>>>>>>>>>>>>>>>>>>>>> multiple coordinators due to a network partition in >>>>>>>>>>>>>>>>>>>>>>>>>>> Hazelcast. This is one >>>>>>>>>>>>>>>>>>>>>>>>>>> advantage we get from this approach. >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Even when we use Zookeeper or RAFT the same >>>>>>>>>>>>>>>>>>>>>>>>>>> issue will be there since we are using different >>>>>>>>>>>>>>>>>>>>>>>>>>> interfaces for Hazelcast >>>>>>>>>>>>>>>>>>>>>>>>>>> communication and DB communication. >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Jul 28, 2016 at 2:56 PM, Akila Ravihansa >>>>>>>>>>>>>>>>>>>>>>>>>>> Perera <raviha...@wso2.com> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> What's the advantage of using RDBMS (even as an >>>>>>>>>>>>>>>>>>>>>>>>>>>> alternative) to implement a leader/coordinator >>>>>>>>>>>>>>>>>>>>>>>>>>>> election? If the network >>>>>>>>>>>>>>>>>>>>>>>>>>>> connection to DB fails then this will be a single >>>>>>>>>>>>>>>>>>>>>>>>>>>> point of failure. I don't >>>>>>>>>>>>>>>>>>>>>>>>>>>> think we can scale RDBMS instances and expect the >>>>>>>>>>>>>>>>>>>>>>>>>>>> election algorithm to >>>>>>>>>>>>>>>>>>>>>>>>>>>> work. That would be reducing this problem to >>>>>>>>>>>>>>>>>>>>>>>>>>>> another problem (electing >>>>>>>>>>>>>>>>>>>>>>>>>>>> coordinator RDBMS instance). >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> IMHO it would be better to look at Zookeeper >>>>>>>>>>>>>>>>>>>>>>>>>>>> Atomic Broadcast (ZAB) [1] or RAFT leader election >>>>>>>>>>>>>>>>>>>>>>>>>>>> [2] algorithms which >>>>>>>>>>>>>>>>>>>>>>>>>>>> have already proven results. >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> [1] https://cwiki.apache.org/c >>>>>>>>>>>>>>>>>>>>>>>>>>>> onfluence/display/ZOOKEEPER/Zab1.0 >>>>>>>>>>>>>>>>>>>>>>>>>>>> [2] http://libraft.io/ >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks. >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Jul 28, 2016 at 1:42 PM, Nandika >>>>>>>>>>>>>>>>>>>>>>>>>>>> Jayawardana <nand...@wso2.com> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> +1 to make it a common component . We have the >>>>>>>>>>>>>>>>>>>>>>>>>>>>> clustering implementation for BPEL component >>>>>>>>>>>>>>>>>>>>>>>>>>>>> based on hazelcast. If the >>>>>>>>>>>>>>>>>>>>>>>>>>>>> coordination is available at RDBMS level, we can >>>>>>>>>>>>>>>>>>>>>>>>>>>>> remove hazelcast >>>>>>>>>>>>>>>>>>>>>>>>>>>>> dependancy. >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Regards >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Nandika >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Jul 28, 2016 at 1:28 PM, Hasitha >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Aravinda <hasi...@wso2.com> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Can we make it a common component, which is >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> not hard coupled with MB. BPS has the same >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> requirement. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hasitha. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Jul 28, 2016 at 9:47 AM, Asanka >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Abeyweera <asank...@wso2.com> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi All, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> In MB, we have used a coordinator based >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> approach to manage distributed messaging >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> algorithm in the cluster. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Currently Hazelcast is used to elect the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> coordinator. But one issue we >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> faced with Hazelcast is, during a network >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> segmentation (split brain), >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hazelcast can elect two or more coordinators in >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the cluster. This affects >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the correctness of the distributed messaging >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> algorithm since there are some >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> tables in the database that should only be >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> edited by a single node (i.e. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> coordinator). >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> As a solution to this problem we have >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> implemented minimum node count based approach >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1] to deactivate set of >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partitioned nodes to stop multiple nodes >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> becoming coordinators until the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> network segmentation issue is fixed. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> As an alternative solution, we are thinking >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of implementing an RDBMS based approach to >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> elect the coordinator node in >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the cluster. By doing this we can make sure >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that even during a network >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> segmentation only one node will be elected as >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the coordinator node since >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the election is happening through the database. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The algorithm will use a polling mechanism >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to check the validity of the nodes. To make the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> election algorithm >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> scalable, only the coordinator node will be >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> checking status of all the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nodes in the cluster and it will inform other >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nodes through database when a >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> member is added/left. The nodes will be only >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> checking for the status of the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> coordinator node. When a node detect that >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> coordinator is invalid it will go >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for a election to elect a new coordinator. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> We are currently working on a POC to test >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> how this works with MB's slot based messaging >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> algorithm. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> thoughts? >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1] https://wso2.org/jira/browse/MB-1664 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Asanka Abeyweera >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Senior Software Engineer >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> WSO2 Inc. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Phone: +94 712228648 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Blog: a5anka.github.io >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <https://wso2.com/signature> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ______________________________ >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _________________ >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Architecture@wso2.org >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/ >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mailman/listinfo/architecture >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hasitha Aravinda, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Associate Technical Lead, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> WSO2 Inc. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Email: hasi...@wso2.com >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Mobile : +94 718 210 200 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ______________________________ >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _________________ >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Architecture@wso2.org >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/ >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mailman/listinfo/architecture >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Nandika Jayawardana >>>>>>>>>>>>>>>>>>>>>>>>>>>>> WSO2 Inc ; http://wso2.com >>>>>>>>>>>>>>>>>>>>>>>>>>>>> lean.enterprise.middleware >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> ______________________________ >>>>>>>>>>>>>>>>>>>>>>>>>>>>> _________________ >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Architecture@wso2.org >>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/ >>>>>>>>>>>>>>>>>>>>>>>>>>>>> mailman/listinfo/architecture >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>>>>>>> Akila Ravihansa Perera >>>>>>>>>>>>>>>>>>>>>>>>>>>> WSO2 Inc.; http://wso2.com/ >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Blog: http://ravihansa3000.blogspot.com >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>>>>>>>>>>>>>>>>> Architecture@wso2.org >>>>>>>>>>>>>>>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/ >>>>>>>>>>>>>>>>>>>>>>>>>>>> mailman/listinfo/architecture >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>>>>>> Asanka Abeyweera >>>>>>>>>>>>>>>>>>>>>>>>>>> Senior Software Engineer >>>>>>>>>>>>>>>>>>>>>>>>>>> WSO2 Inc. >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Phone: +94 712228648 >>>>>>>>>>>>>>>>>>>>>>>>>>> Blog: a5anka.github.io >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> <https://wso2.com/signature> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>>>>>>>>>>>>>>>> Architecture@wso2.org >>>>>>>>>>>>>>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/ >>>>>>>>>>>>>>>>>>>>>>>>>>> mailman/listinfo/architecture >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>>>> Asanka Abeyweera >>>>>>>>>>>>>>>>>>>>>>>>> Senior Software Engineer >>>>>>>>>>>>>>>>>>>>>>>>> WSO2 Inc. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> Phone: +94 712228648 >>>>>>>>>>>>>>>>>>>>>>>>> Blog: a5anka.github.io >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> <https://wso2.com/signature> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>>>>>>>>>>>>>> Architecture@wso2.org >>>>>>>>>>>>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/ >>>>>>>>>>>>>>>>>>>>>>>>> mailman/listinfo/architecture >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>>> *Imesh Gunaratne* >>>>>>>>>>>>>>>>>>>>>>>> Software Architect >>>>>>>>>>>>>>>>>>>>>>>> WSO2 Inc: http://wso2.com >>>>>>>>>>>>>>>>>>>>>>>> T: +94 11 214 5345 M: +94 77 374 2057 >>>>>>>>>>>>>>>>>>>>>>>> W: https://medium.com/@imesh TW: @imesh >>>>>>>>>>>>>>>>>>>>>>>> lean. enterprise. middleware >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>>>>>>>>>>>>> Architecture@wso2.org >>>>>>>>>>>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/ >>>>>>>>>>>>>>>>>>>>>>>> mailman/listinfo/architecture >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>> Asanka Abeyweera >>>>>>>>>>>>>>>>>>>>>>> Senior Software Engineer >>>>>>>>>>>>>>>>>>>>>>> WSO2 Inc. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Phone: +94 712228648 >>>>>>>>>>>>>>>>>>>>>>> Blog: a5anka.github.io >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> <https://wso2.com/signature> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>>>>>>>>>>>> Architecture@wso2.org >>>>>>>>>>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/ >>>>>>>>>>>>>>>>>>>>>>> mailman/listinfo/architecture >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>> Ramith Jayasinghe >>>>>>>>>>>>>>>>>>>>>> Technical Lead >>>>>>>>>>>>>>>>>>>>>> WSO2 Inc., http://wso2.com >>>>>>>>>>>>>>>>>>>>>> lean.enterprise.middleware >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> E: ram...@wso2.com >>>>>>>>>>>>>>>>>>>>>> P: +94 772534930 >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>>>>>>>>>>> Architecture@wso2.org >>>>>>>>>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/ >>>>>>>>>>>>>>>>>>>>>> mailman/listinfo/architecture >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>> *Imesh Gunaratne* >>>>>>>>>>>>>>>>>>>>> Software Architect >>>>>>>>>>>>>>>>>>>>> WSO2 Inc: http://wso2.com >>>>>>>>>>>>>>>>>>>>> T: +94 11 214 5345 M: +94 77 374 2057 >>>>>>>>>>>>>>>>>>>>> W: https://medium.com/@imesh TW: @imesh >>>>>>>>>>>>>>>>>>>>> lean. enterprise. middleware >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>> *Imesh Gunaratne* >>>>>>>>>>>>>>>>>>>> Software Architect >>>>>>>>>>>>>>>>>>>> WSO2 Inc: http://wso2.com >>>>>>>>>>>>>>>>>>>> T: +94 11 214 5345 M: +94 77 374 2057 >>>>>>>>>>>>>>>>>>>> W: https://medium.com/@imesh TW: @imesh >>>>>>>>>>>>>>>>>>>> lean. enterprise. middleware >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>>>>>>>>> Architecture@wso2.org >>>>>>>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/ >>>>>>>>>>>>>>>>>>>> mailman/listinfo/architecture >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>> *Asitha Nanayakkara* <http://asitha.github.io/> >>>>>>>>>>>>>>>>>>> Senior Software Engineer >>>>>>>>>>>>>>>>>>> WSO2, Inc. <http://wso2.com/> >>>>>>>>>>>>>>>>>>> Mob: +94 77 853 0682 >>>>>>>>>>>>>>>>>>> [image: https://wso2.com/signature] >>>>>>>>>>>>>>>>>>> <https://wso2.com/signature> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>>>>>>>> Architecture@wso2.org >>>>>>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/ >>>>>>>>>>>>>>>>>>> mailman/listinfo/architecture >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Best Regards, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Malaka Silva >>>>>>>>>>>>>>>>>> Senior Technical Lead >>>>>>>>>>>>>>>>>> M: +94 777 219 791 >>>>>>>>>>>>>>>>>> Tel : 94 11 214 5345 >>>>>>>>>>>>>>>>>> Fax :94 11 2145300 >>>>>>>>>>>>>>>>>> Skype : malaka.sampath.silva >>>>>>>>>>>>>>>>>> LinkedIn : http://www.linkedin.com/pub/ >>>>>>>>>>>>>>>>>> malaka-silva/6/33/77 >>>>>>>>>>>>>>>>>> Blog : http://mrmalakasilva.blogspot.com/ >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> WSO2, Inc. >>>>>>>>>>>>>>>>>> lean . enterprise . middleware >>>>>>>>>>>>>>>>>> https://wso2.com/signature >>>>>>>>>>>>>>>>>> http://www.wso2.com/about/team/malaka-silva/ >>>>>>>>>>>>>>>>>> <http://wso2.com/about/team/malaka-silva/> >>>>>>>>>>>>>>>>>> https://store.wso2.com/store/ >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Don't make Trees rare, we should keep them with care >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>>>>>>> Architecture@wso2.org >>>>>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/ >>>>>>>>>>>>>>>>>> mailman/listinfo/architecture >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>> Ramith Jayasinghe >>>>>>>>>>>>>>>>> Technical Lead >>>>>>>>>>>>>>>>> WSO2 Inc., http://wso2.com >>>>>>>>>>>>>>>>> lean.enterprise.middleware >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> E: ram...@wso2.com >>>>>>>>>>>>>>>>> P: +94 772534930 >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>>>>>> Architecture@wso2.org >>>>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/ >>>>>>>>>>>>>>>>> mailman/listinfo/architecture >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> Sajini De SIlva >>>>>>>>>>>>>>> Senior Software Engineer; WSO2 Inc.; http://wso2.com , >>>>>>>>>>>>>>> Email: saj...@wso2.com >>>>>>>>>>>>>>> Blog: http://sajinid.blogspot.com/ >>>>>>>>>>>>>>> Git hub profile: https://github.com/sajinidesilva >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Phone: +94 712797729 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> Sajini De SIlva >>>>>>>>>>>>> Senior Software Engineer; WSO2 Inc.; http://wso2.com , >>>>>>>>>>>>> Email: saj...@wso2.com >>>>>>>>>>>>> Blog: http://sajinid.blogspot.com/ >>>>>>>>>>>>> Git hub profile: https://github.com/sajinidesilva >>>>>>>>>>>>> >>>>>>>>>>>>> Phone: +94 712797729 >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>> Architecture@wso2.org >>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> Kasun Indrasiri >>>>>>>>>>>> Director, Integration Technologies >>>>>>>>>>>> WSO2, Inc.; http://wso2.com >>>>>>>>>>>> lean.enterprise.middleware >>>>>>>>>>>> >>>>>>>>>>>> cell: +1 650 450 2293 >>>>>>>>>>>> Blog : http://kasunpanorama.blogspot.com/ >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> *Anjana Fernando* >>>>>>>>>>> Associate Director / Architect >>>>>>>>>>> WSO2 Inc. | http://wso2.com >>>>>>>>>>> lean . enterprise . middleware >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> Architecture mailing list >>>>>>>>>>> Architecture@wso2.org >>>>>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> *Anjana Fernando* >>>>>>>>> Associate Director / Architect >>>>>>>>> WSO2 Inc. | http://wso2.com >>>>>>>>> lean . enterprise . middleware >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Asanka Abeyweera >>>>>>>> Senior Software Engineer >>>>>>>> WSO2 Inc. >>>>>>>> >>>>>>>> Phone: +94 712228648 >>>>>>>> Blog: a5anka.github.io >>>>>>>> >>>>>>>> <https://wso2.com/signature> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> *Anjana Fernando* >>>>>>> Associate Director / Architect >>>>>>> WSO2 Inc. | http://wso2.com >>>>>>> lean . enterprise . middleware >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> *Anjana Fernando* >>>>>> Associate Director / Architect >>>>>> WSO2 Inc. | http://wso2.com >>>>>> lean . enterprise . middleware >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Asanka Abeyweera >>>>> Senior Software Engineer >>>>> WSO2 Inc. >>>>> >>>>> Phone: +94 712228648 >>>>> Blog: a5anka.github.io >>>>> >>>>> <https://wso2.com/signature> >>>>> >>>> >>>> >>>> >>>> -- >>>> *Anjana Fernando* >>>> Associate Director / Architect >>>> WSO2 Inc. | http://wso2.com >>>> lean . enterprise . middleware >>>> >>> >>> >>> >>> -- >>> *Anjana Fernando* >>> Associate Director / Architect >>> WSO2 Inc. | http://wso2.com >>> lean . enterprise . middleware >>> >> >> >> >> -- >> Ramith Jayasinghe >> Technical Lead >> WSO2 Inc., http://wso2.com >> lean.enterprise.middleware >> >> E: ram...@wso2.com >> P: +94 772534930 >> > > > > -- > *Anjana Fernando* > Associate Director / Architect > WSO2 Inc. | http://wso2.com > lean . enterprise . middleware > -- Ramith Jayasinghe Technical Lead WSO2 Inc., http://wso2.com lean.enterprise.middleware E: ram...@wso2.com P: +94 772534930
_______________________________________________ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
