Hi Maninda, We are not using one coordinator to send and receive messages. All the nodes in the cluster can receive and send messages to MB and messages will be written to database by multiple nodes. Also messages will be read from the database by multiple nodes. In MB we have a concept called slots. Basically when messages are read from database one node read a chunk of messages which is called a slot. Coordinator is only responsible of creating these slots, distributes slots and status updating related to slots.
Thanks On Fri, Aug 5, 2016 at 1:45 PM, Maninda Edirisooriya <mani...@wso2.com> wrote: > > On Fri, Aug 5, 2016 at 12:00 PM, Hasitha Hiranya <hasit...@wso2.com> > wrote: > >> Hi, >> >> >> On Fri, Aug 5, 2016 at 11:31 AM, Akila Ravihansa Perera < >> raviha...@wso2.com> wrote: >> >>> Hi, >>> >>> I think the original problem here is that MB needs to absolutely >>> guarantee the integrity of the data written to the database. And if I >>> understood correctly, only the coordinator can write specific entries to >>> the database which is a unique scenario for MB. Any network based leader >>> election algorithm (even database based one) can cause split brain due to >>> network/latency issues, but using a database based leader election >>> algorithm is safe here because if it cannot connect to the database then no >>> harm can be done to the data integrity. Am I correct to say that? >>> >>> However, on a separate node, it would be better to look into a new >>> design where MB doesn't depend on a coordinator to write data into the db. >>> This is a major bottleneck to scale. >>> >> >> True. But MB is naturally stateful. There are tasks only one node should >> perform on behalf of the cluster for integrity of data. I think CAP theorem >> is in action here. We can only achieve only two out of three factors >> (Consistency/Availability and Partition tolerance). [1] >> >> [1]. https://dzone.com/articles/better-explaining-cap-theorem >> > > It does not seem problematic implementing the RDBMS based leader election > due to unavoidable split brain issue. As RDBMS network connection is a must > to function of the MB, it is fine to dependent on RDBMS connection. But the > scalability aspect can be improved as Akila has stated above. Although > metadata updates like adding / removing subscribers need a single master to > coordinate, message receiving and sending could be scalable. In other words > information related to metadata of messages should not pass through a > single coordinator at all which is an absolute bottleneck. May be that > should need to be discussed in a separate thread which is out of topic. > > Thanks. > >> >>> Thanks. >>> >>> On Fri, Aug 5, 2016 at 11:21 AM, Imesh Gunaratne <im...@wso2.com> wrote: >>> >>>> Hi Asitha/Asanka, >>>> >>>> I think it is clear that the issue we have here is mostly related to >>>> Hazelcast. >>>> >>>> Now to solve that problem I think it would be better to go ahead with a >>>> generic leader election system for the entire platform rather than writing >>>> one specific to MB. This requirement is there in several other products and >>>> for some a database driven approach might not work. >>>> >>>> Therefore it would be better if we can decouple this from the product >>>> and use an interface to talk to a leader election module. This module can >>>> either be implemented as a separate component or utilize an existing system >>>> such as etcd. >>>> >>>> To start with I think it would be better to evaluate what etcd and >>>> consul has to offer and check whether they fit to our requirement. >>>> >>>> Thanks >>>> >>>> On Fri, Aug 5, 2016 at 10:12 AM, Asanka Abeyweera <asank...@wso2.com> >>>> wrote: >>>> >>>>> Hi Imesh, >>>>> >>>>> On Fri, Aug 5, 2016 at 7:33 AM, Imesh Gunaratne <im...@wso2.com> >>>>> wrote: >>>>> >>>>>> >>>>>> >>>>>> On Fri, Aug 5, 2016 at 7:31 AM, Imesh Gunaratne <im...@wso2.com> >>>>>> wrote: >>>>>>> >>>>>>> >>>>>>> You can see here [3] how K8S has implemented leader election feature >>>>>>> for the products deployed on top of that to utilize. >>>>>>> >>>>>> >>>>>> Correction: Please refer [4]. >>>>>> >>>>>> >>>>>>> >>>>>>> >>>>>>>> On Thu, Aug 4, 2016 at 7:27 PM, Asanka Abeyweera <asank...@wso2.com >>>>>>>> > wrote: >>>>>>>> >>>>>>>>> Hi Imesh, >>>>>>>>> >>>>>>>>> We are not implementing this to overcome a limitation in the >>>>>>>>> coordination algorithm available in the Hazlecast. We are >>>>>>>>> implementing this >>>>>>>>> since we need an RDBMS based coordination algorithm (not a network >>>>>>>>> based >>>>>>>>> algorithm). >>>>>>>>> >>>>>>>> >>>>>>> Are you saying that database connections do not use the same >>>>>>> network used by Hazelcast? >>>>>>> >>>>>> >>>>> Yes, This is most problematic when two interfaces are used for >>>>> Hazelcast communication and RDBMS communication. Additionally there is an >>>>> edge case even when a single interface is used for both Hazelcast and >>>>> RDBMS >>>>> communication. When a cluster merge after a network segmentation, there >>>>> can >>>>> be a delay in Hazelcast detecting the cluster merge. If a database is >>>>> accessed by multiple coordinators during this time, there can be message >>>>> delivery issues like message duplication. Therefore we cannot ignore this >>>>> issue even when the same network is used for Hazelcast and database >>>>> connections. >>>>> >>>>> >>>>> >>>>>> The reason is, a network based election algorithm will always elect >>>>>>>>> multiple leaders when the network is partitioned. But if we use a >>>>>>>>> RDBMS >>>>>>>>> based algorithm this will not happen. >>>>>>>>> >>>>>>>> >>>>>>> I do not think your argument is correct. If there is a problem with >>>>>>> the network, it may apply to both Hazelcast based solution and database >>>>>>> based solution. >>>>>>> >>>>>>> [4] http://blog.kubernetes.io/2016/01/simple-leader-election >>>>>>> -with-Kubernetes.html >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> On Thu, Aug 4, 2016 at 7:16 PM, Imesh Gunaratne <im...@wso2.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hi Asanka, >>>>>>>>>> >>>>>>>>>> Do we really need to implement a leader election algorithm on our >>>>>>>>>> own? AFAIU this is a complex problem which has been already solved by >>>>>>>>>> several algorithms [1]. IMO it would be better to go ahead with an >>>>>>>>>> existing >>>>>>>>>> well established implementation on etcd [1] or Consul [2]. >>>>>>>>>> >>>>>>>>>> Those provide HTTP APIs for clients to make leader election >>>>>>>>>> calls. [3] is a client library written in Node.js for etcd based >>>>>>>>>> leader >>>>>>>>>> election. >>>>>>>>>> >>>>>>>>>> [1] https://www.projectcalico.org/using-etcd-for-elections >>>>>>>>>> [2] https://www.consul.io/docs/guides/leader-election.html >>>>>>>>>> [3] https://www.npmjs.com/package/etcd-leader >>>>>>>>>> >>>>>>>>>> Thanks >>>>>>>>>> >>>>>>>>>> On Wed, Aug 3, 2016 at 5:12 PM, Asanka Abeyweera < >>>>>>>>>> asank...@wso2.com> wrote: >>>>>>>>>> >>>>>>>>>>> Hi Maninda, >>>>>>>>>>> >>>>>>>>>>> Since we are using RDBMS to poll the node status, the cluster >>>>>>>>>>> will not end up in situation 1,2 or 3. With this approach we >>>>>>>>>>> consider a >>>>>>>>>>> node unreachable when it cannot access the database. Therefore an >>>>>>>>>>> unreachable node can never be the leader. >>>>>>>>>>> >>>>>>>>>>> As you have mentioned, we are currently using the RDBMS as an >>>>>>>>>>> atomic global variable to create the coordinator entry. >>>>>>>>>>> >>>>>>>>>>> On Tue, Aug 2, 2016 at 5:22 PM, Maninda Edirisooriya < >>>>>>>>>>> mani...@wso2.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi Asanka, >>>>>>>>>>>> >>>>>>>>>>>> As I understand the accuracy of electing the leader correctly >>>>>>>>>>>> is dependent on the election mechanism with RDBMS because there >>>>>>>>>>>> can be edge >>>>>>>>>>>> cases like, >>>>>>>>>>>> >>>>>>>>>>>> 1. Unreachable leader activates during the election process: >>>>>>>>>>>> Then who becomes the leader? >>>>>>>>>>>> 2. The elected leader becomes unreachable before the election >>>>>>>>>>>> is completed: Then will there be a situation where there is no >>>>>>>>>>>> leader? >>>>>>>>>>>> 3. A leader and a set of nodes are disconnected from the other >>>>>>>>>>>> part of the cluster and while the leader is trying to remove >>>>>>>>>>>> unreachable >>>>>>>>>>>> members other part is calling an election to make a leader: Who >>>>>>>>>>>> will win? >>>>>>>>>>>> >>>>>>>>>>>> RDBMS based election algorithm should handle such cases without >>>>>>>>>>>> bringing the cluster to an inconsistent state or dead lock in all >>>>>>>>>>>> concurrent cases. If all these kind of cases cannot be handled >>>>>>>>>>>> isn't it >>>>>>>>>>>> better to keep the current hazelcast clustering and use the RDBMS >>>>>>>>>>>> only to >>>>>>>>>>>> handle the split brain scenario? In other words when a new >>>>>>>>>>>> hazelcast leader >>>>>>>>>>>> is elected it should be updated in the RDBMS. If another split >>>>>>>>>>>> party has >>>>>>>>>>>> already elected a leader, the node who is going to write it to >>>>>>>>>>>> RDBMS should >>>>>>>>>>>> avoid updating it. Simply, the RDBMS can be used as an atomic >>>>>>>>>>>> global >>>>>>>>>>>> variable to keep the leader name by modifying the hazelcast >>>>>>>>>>>> clustering. >>>>>>>>>>>> WDYT? >>>>>>>>>>>> >>>>>>>>>>>> Thanks. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> *Maninda Edirisooriya* >>>>>>>>>>>> Senior Software Engineer >>>>>>>>>>>> >>>>>>>>>>>> *WSO2, Inc.*lean.enterprise.middleware. >>>>>>>>>>>> >>>>>>>>>>>> *Blog* : http://maninda.blogspot.com/ >>>>>>>>>>>> *E-mail* : mani...@wso2.com >>>>>>>>>>>> *Skype* : @manindae >>>>>>>>>>>> *Twitter* : @maninda >>>>>>>>>>>> >>>>>>>>>>>> On Thu, Jul 28, 2016 at 4:38 PM, Asanka Abeyweera < >>>>>>>>>>>> asank...@wso2.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi Akila, >>>>>>>>>>>>> >>>>>>>>>>>>> Let me explain the issue in a different way. Let's assume the >>>>>>>>>>>>> MB nodes are using two different network interfaces for Hazelcast >>>>>>>>>>>>> communication and database communication. With such a >>>>>>>>>>>>> configuration, there >>>>>>>>>>>>> can be failures only in the network interface used for Hazelcast >>>>>>>>>>>>> communication in some nodes. When this happens, there will be two >>>>>>>>>>>>> or more >>>>>>>>>>>>> Hazelcast clusters due to the network segmentation, and as a >>>>>>>>>>>>> result there >>>>>>>>>>>>> will be multiple coordinators. Since every node still have access >>>>>>>>>>>>> to the >>>>>>>>>>>>> database, multiple coordinators can affect the correctness of the >>>>>>>>>>>>> data >>>>>>>>>>>>> stored in the DB. But if we used a RDBMS based approach we won't >>>>>>>>>>>>> have >>>>>>>>>>>>> multiple coordinators due to a network partition in Hazelcast. >>>>>>>>>>>>> This is one >>>>>>>>>>>>> advantage we get from this approach. >>>>>>>>>>>>> >>>>>>>>>>>>> Even when we use Zookeeper or RAFT the same issue will be >>>>>>>>>>>>> there since we are using different interfaces for Hazelcast >>>>>>>>>>>>> communication >>>>>>>>>>>>> and DB communication. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Thu, Jul 28, 2016 at 2:56 PM, Akila Ravihansa Perera < >>>>>>>>>>>>> raviha...@wso2.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>> >>>>>>>>>>>>>> What's the advantage of using RDBMS (even as an alternative) >>>>>>>>>>>>>> to implement a leader/coordinator election? If the network >>>>>>>>>>>>>> connection to DB >>>>>>>>>>>>>> fails then this will be a single point of failure. I don't think >>>>>>>>>>>>>> we can >>>>>>>>>>>>>> scale RDBMS instances and expect the election algorithm to work. >>>>>>>>>>>>>> That would >>>>>>>>>>>>>> be reducing this problem to another problem (electing >>>>>>>>>>>>>> coordinator RDBMS >>>>>>>>>>>>>> instance). >>>>>>>>>>>>>> >>>>>>>>>>>>>> IMHO it would be better to look at Zookeeper Atomic Broadcast >>>>>>>>>>>>>> (ZAB) [1] or RAFT leader election [2] algorithms which have >>>>>>>>>>>>>> already proven >>>>>>>>>>>>>> results. >>>>>>>>>>>>>> >>>>>>>>>>>>>> [1] https://cwiki.apache.org/confluence/display/ZOOKEEPER/Za >>>>>>>>>>>>>> b1.0 >>>>>>>>>>>>>> [2] http://libraft.io/ >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks. >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Thu, Jul 28, 2016 at 1:42 PM, Nandika Jayawardana < >>>>>>>>>>>>>> nand...@wso2.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> +1 to make it a common component . We have the clustering >>>>>>>>>>>>>>> implementation for BPEL component based on hazelcast. If the >>>>>>>>>>>>>>> coordination >>>>>>>>>>>>>>> is available at RDBMS level, we can remove hazelcast dependancy. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Regards >>>>>>>>>>>>>>> Nandika >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Thu, Jul 28, 2016 at 1:28 PM, Hasitha Aravinda < >>>>>>>>>>>>>>> hasi...@wso2.com> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Can we make it a common component, which is not hard >>>>>>>>>>>>>>>> coupled with MB. BPS has the same requirement. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>> Hasitha. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Thu, Jul 28, 2016 at 9:47 AM, Asanka Abeyweera < >>>>>>>>>>>>>>>> asank...@wso2.com> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hi All, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> In MB, we have used a coordinator based approach to manage >>>>>>>>>>>>>>>>> distributed messaging algorithm in the cluster. Currently >>>>>>>>>>>>>>>>> Hazelcast is used >>>>>>>>>>>>>>>>> to elect the coordinator. But one issue we faced with >>>>>>>>>>>>>>>>> Hazelcast is, during >>>>>>>>>>>>>>>>> a network segmentation (split brain), Hazelcast can elect two >>>>>>>>>>>>>>>>> or more >>>>>>>>>>>>>>>>> coordinators in the cluster. This affects the correctness of >>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>> distributed messaging algorithm since there are some tables >>>>>>>>>>>>>>>>> in the database >>>>>>>>>>>>>>>>> that should only be edited by a single node (i.e. >>>>>>>>>>>>>>>>> coordinator). >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> As a solution to this problem we have implemented minimum >>>>>>>>>>>>>>>>> node count based approach [1] to deactivate set of >>>>>>>>>>>>>>>>> partitioned nodes to >>>>>>>>>>>>>>>>> stop multiple nodes becoming coordinators until the network >>>>>>>>>>>>>>>>> segmentation >>>>>>>>>>>>>>>>> issue is fixed. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> As an alternative solution, we are thinking of >>>>>>>>>>>>>>>>> implementing an RDBMS based approach to elect the coordinator >>>>>>>>>>>>>>>>> node in the >>>>>>>>>>>>>>>>> cluster. By doing this we can make sure that even during a >>>>>>>>>>>>>>>>> network >>>>>>>>>>>>>>>>> segmentation only one node will be elected as the coordinator >>>>>>>>>>>>>>>>> node since >>>>>>>>>>>>>>>>> the election is happening through the database. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> The algorithm will use a polling mechanism to check the >>>>>>>>>>>>>>>>> validity of the nodes. To make the election algorithm >>>>>>>>>>>>>>>>> scalable, only the >>>>>>>>>>>>>>>>> coordinator node will be checking status of all the nodes in >>>>>>>>>>>>>>>>> the cluster >>>>>>>>>>>>>>>>> and it will inform other nodes through database when a member >>>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>> added/left. The nodes will be only checking for the status of >>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>> coordinator node. When a node detect that coordinator is >>>>>>>>>>>>>>>>> invalid it will go >>>>>>>>>>>>>>>>> for a election to elect a new coordinator. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> We are currently working on a POC to test how this works >>>>>>>>>>>>>>>>> with MB's slot based messaging algorithm. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> thoughts? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [1] https://wso2.org/jira/browse/MB-1664 >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>> Asanka Abeyweera >>>>>>>>>>>>>>>>> Senior Software Engineer >>>>>>>>>>>>>>>>> WSO2 Inc. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Phone: +94 712228648 >>>>>>>>>>>>>>>>> Blog: a5anka.github.io >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> <https://wso2.com/signature> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>>>>>> Architecture@wso2.org >>>>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/ >>>>>>>>>>>>>>>>> mailman/listinfo/architecture >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>> Hasitha Aravinda, >>>>>>>>>>>>>>>> Associate Technical Lead, >>>>>>>>>>>>>>>> WSO2 Inc. >>>>>>>>>>>>>>>> Email: hasi...@wso2.com >>>>>>>>>>>>>>>> Mobile : +94 718 210 200 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>>>>> Architecture@wso2.org >>>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> Nandika Jayawardana >>>>>>>>>>>>>>> WSO2 Inc ; http://wso2.com >>>>>>>>>>>>>>> lean.enterprise.middleware >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>>>> Architecture@wso2.org >>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> Akila Ravihansa Perera >>>>>>>>>>>>>> WSO2 Inc.; http://wso2.com/ >>>>>>>>>>>>>> >>>>>>>>>>>>>> Blog: http://ravihansa3000.blogspot.com >>>>>>>>>>>>>> >>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>>> Architecture@wso2.org >>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> Asanka Abeyweera >>>>>>>>>>>>> Senior Software Engineer >>>>>>>>>>>>> WSO2 Inc. >>>>>>>>>>>>> >>>>>>>>>>>>> Phone: +94 712228648 >>>>>>>>>>>>> Blog: a5anka.github.io >>>>>>>>>>>>> >>>>>>>>>>>>> <https://wso2.com/signature> >>>>>>>>>>>>> >>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>> Architecture@wso2.org >>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Asanka Abeyweera >>>>>>>>>>> Senior Software Engineer >>>>>>>>>>> WSO2 Inc. >>>>>>>>>>> >>>>>>>>>>> Phone: +94 712228648 >>>>>>>>>>> Blog: a5anka.github.io >>>>>>>>>>> >>>>>>>>>>> <https://wso2.com/signature> >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> Architecture mailing list >>>>>>>>>>> Architecture@wso2.org >>>>>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> *Imesh Gunaratne* >>>>>>>>>> Software Architect >>>>>>>>>> WSO2 Inc: http://wso2.com >>>>>>>>>> T: +94 11 214 5345 M: +94 77 374 2057 >>>>>>>>>> W: https://medium.com/@imesh TW: @imesh >>>>>>>>>> lean. enterprise. middleware >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> Architecture mailing list >>>>>>>>>> Architecture@wso2.org >>>>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Asanka Abeyweera >>>>>>>>> Senior Software Engineer >>>>>>>>> WSO2 Inc. >>>>>>>>> >>>>>>>>> Phone: +94 712228648 >>>>>>>>> Blog: a5anka.github.io >>>>>>>>> >>>>>>>>> <https://wso2.com/signature> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> Architecture mailing list >>>>>>>>> Architecture@wso2.org >>>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Ramith Jayasinghe >>>>>>>> Technical Lead >>>>>>>> WSO2 Inc., http://wso2.com >>>>>>>> lean.enterprise.middleware >>>>>>>> >>>>>>>> E: ram...@wso2.com >>>>>>>> P: +94 772534930 >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Architecture mailing list >>>>>>>> Architecture@wso2.org >>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> *Imesh Gunaratne* >>>>>>> Software Architect >>>>>>> WSO2 Inc: http://wso2.com >>>>>>> T: +94 11 214 5345 M: +94 77 374 2057 >>>>>>> W: https://medium.com/@imesh TW: @imesh >>>>>>> lean. enterprise. middleware >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> *Imesh Gunaratne* >>>>>> Software Architect >>>>>> WSO2 Inc: http://wso2.com >>>>>> T: +94 11 214 5345 M: +94 77 374 2057 >>>>>> W: https://medium.com/@imesh TW: @imesh >>>>>> lean. enterprise. middleware >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Architecture mailing list >>>>>> Architecture@wso2.org >>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Asanka Abeyweera >>>>> Senior Software Engineer >>>>> WSO2 Inc. >>>>> >>>>> Phone: +94 712228648 >>>>> Blog: a5anka.github.io >>>>> >>>>> <https://wso2.com/signature> >>>>> >>>>> _______________________________________________ >>>>> Architecture mailing list >>>>> Architecture@wso2.org >>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>> >>>>> >>>> >>>> >>>> -- >>>> *Imesh Gunaratne* >>>> Software Architect >>>> WSO2 Inc: http://wso2.com >>>> T: +94 11 214 5345 M: +94 77 374 2057 >>>> W: https://medium.com/@imesh TW: @imesh >>>> lean. enterprise. middleware >>>> >>>> >>>> _______________________________________________ >>>> Architecture mailing list >>>> Architecture@wso2.org >>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>> >>>> >>> >>> >>> -- >>> Akila Ravihansa Perera >>> WSO2 Inc.; http://wso2.com/ >>> >>> Blog: http://ravihansa3000.blogspot.com >>> >>> _______________________________________________ >>> Architecture mailing list >>> Architecture@wso2.org >>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>> >>> >> >> >> -- >> *Hasitha Abeykoon* >> Senior Software Engineer; WSO2, Inc.; http://wso2.com >> *cell:* *+94 719363063* >> *blog: **abeykoon.blogspot.com* <http://abeykoon.blogspot.com> >> >> >> _______________________________________________ >> Architecture mailing list >> Architecture@wso2.org >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> >> > > _______________________________________________ > Architecture mailing list > Architecture@wso2.org > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > > -- Sajini De SIlva Senior Software Engineer; WSO2 Inc.; http://wso2.com , Email: saj...@wso2.com Blog: http://sajinid.blogspot.com/ Git hub profile: https://github.com/sajinidesilva Phone: +94 712797729
_______________________________________________ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
