this might require some work.. shall we have a chat? On Thu, Nov 3, 2016 at 3:52 PM, Anjana Fernando <anj...@wso2.com> wrote:
> Ping! .. > > On Wed, Nov 2, 2016 at 5:03 PM, Anjana Fernando <anj...@wso2.com> wrote: > >> Hi, >> >> On Wed, Nov 2, 2016 at 3:14 PM, Asanka Abeyweera <asank...@wso2.com> >> wrote: >> >>> Hi Anjana, >>> >>> Currently, the implementation is part of the MB code (not a common >>> component). >>> >> >> Okay, can we please get it as a common component. >> >> Cheers, >> Anjana. >> >> >>> >>> On Wed, Nov 2, 2016 at 3:00 PM, Anjana Fernando <anj...@wso2.com> wrote: >>> >>>> Hi Asanka/Ramith, >>>> >>>> So for C5 based Streaming Analytics solution, we need coordination >>>> functionality there as well. Is the functionality mentioned here created as >>>> a common component or baked in to the MB code? .. if so, can we please get >>>> it implemented it as a generic component, so other products can also use >>>> it. >>>> >>>> Cheers, >>>> Anjana. >>>> >>>> On Tue, Aug 9, 2016 at 3:10 PM, Anjana Fernando <anj...@wso2.com> >>>> wrote: >>>> >>>>> Great! .. >>>>> >>>>> Cheers, >>>>> Anjana. >>>>> >>>>> On Tue, Aug 9, 2016 at 1:49 PM, Asanka Abeyweera <asank...@wso2.com> >>>>> wrote: >>>>> >>>>>> Hi Anjana, >>>>>> >>>>>> Thank you for the suggestion. We have already done a similar thing. >>>>>> We have added a backoff time after creating the leader entry and check if >>>>>> the leader entry is the entry created by self before informing the leader >>>>>> change. >>>>>> >>>>>> On Tue, Aug 9, 2016 at 12:27 PM, Anjana Fernando <anj...@wso2.com> >>>>>> wrote: >>>>>> >>>>>>> I see, thanks for the clarification, looks good! .. I think small >>>>>>> thing to consider is, to avoid the situation where, the current leader >>>>>>> goes >>>>>>> away, and two other competes to become the leader, and the first one and >>>>>>> the second one checks (reads) the table to check the last heartbeat and >>>>>>> figures out that the leader is outdated at the same time, and then first >>>>>>> one delete the entry and puts his one, and after that, second one will >>>>>>> also >>>>>>> delete the existing one and put his one, so both will think they became >>>>>>> the >>>>>>> leader, due to the condition that both succeeded in adding the entry >>>>>>> without an error. So this can probably be fixed by checking back after a >>>>>>> bit of time if the current node is actually me, which probabilistically >>>>>>> will work well, if that time period is sufficient big enough than a >>>>>>> typical >>>>>>> database transaction required by a node to do the earlier operations. Or >>>>>>> else, we should make sure the database transaction level used in this >>>>>>> scenario is at least REPEATABLE_READ, where when we read the record, it >>>>>>> will lock it throughout the transaction. So some DBMSs does not support >>>>>>> REPEATABLE_READ, where in that case, we should be able to use >>>>>>> SERIALIZABLE, >>>>>>> which most of them support. >>>>>>> >>>>>>> Cheers, >>>>>>> Anjana. >>>>>>> >>>>>>> On Tue, Aug 9, 2016 at 11:11 AM, Maninda Edirisooriya < >>>>>>> mani...@wso2.com> wrote: >>>>>>> >>>>>>>> Hi Anjana, >>>>>>>> >>>>>>>> After having an offline chat with Asanka what I understood was that >>>>>>>> the leader election was done completely via the database but with no >>>>>>>> network communication. The leader is mentioned in the database first. >>>>>>>> Then >>>>>>>> the leader updates the node data periodically in the database. If some >>>>>>>> node >>>>>>>> realizes the data in the DB are outdated that means the leader was >>>>>>>> disconnected. Then that node will look at the created timestamp of the >>>>>>>> leader entry. If that is not very recent that means there was no new >>>>>>>> leader >>>>>>>> elected recently. So he will try to update the leader entry with his >>>>>>>> ID. As >>>>>>>> I understand there the leader entry is using the leader ID and the >>>>>>>> timestamp as the primary key. Even several nodes try to do it >>>>>>>> simultaneously only one node will successfully be able to update the >>>>>>>> entry >>>>>>>> with the help of atomicity provided by the DB. Others members will >>>>>>>> note the >>>>>>>> timestamp of the leader was updated so will accept the first one who >>>>>>>> updates as the leader. Even after the leader is elected, the leader >>>>>>>> will >>>>>>>> only notify node data via updating DB instead of network calls. Other >>>>>>>> nodes >>>>>>>> will just observe it and check the latest timestmps of the entry. >>>>>>>> >>>>>>>> >>>>>>>> *Maninda Edirisooriya* >>>>>>>> Senior Software Engineer >>>>>>>> >>>>>>>> *WSO2, Inc.*lean.enterprise.middleware. >>>>>>>> >>>>>>>> *Blog* : http://maninda.blogspot.com/ >>>>>>>> *E-mail* : mani...@wso2.com >>>>>>>> *Skype* : @manindae >>>>>>>> *Twitter* : @maninda >>>>>>>> >>>>>>>> On Tue, Aug 9, 2016 at 10:13 AM, Anjana Fernando <anj...@wso2.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> I just noticed this thread. I've some concerns on this >>>>>>>>> implementations. First of all, I don't think the statement mentioned >>>>>>>>> here >>>>>>>>> saying an external service such as ZooKeeper doesn't work, is correct. >>>>>>>>> Because, if you have a ZK cluster (it is suppose to be used as a >>>>>>>>> cluster), >>>>>>>>> you will not have any issues. All the nodes have a list of endpoints >>>>>>>>> to all >>>>>>>>> the ZK nodes and they connect to those, and ZK has a quorum based >>>>>>>>> mechanism >>>>>>>>> in keeping its state. So this makes sure, all the users have a single >>>>>>>>> version of the ZK data. >>>>>>>>> >>>>>>>>> Also, I guess the fundamental problem here in the split brain >>>>>>>>> situation is, we need one external entity taking the decision (e.g. ZK >>>>>>>>> cluster), because it should have oversight to the whole environment. I >>>>>>>>> don't see how this RDBMS mechanism would solve that. Because, what it >>>>>>>>> gives >>>>>>>>> is a central location of state persistence. But the decisions of >>>>>>>>> making who >>>>>>>>> is the leader is taken by the users, which can be problematic. Where >>>>>>>>> when >>>>>>>>> we have a network partition scenario in that occasion, two groups of >>>>>>>>> users >>>>>>>>> will be overriding each other in the centralized RDBMS data >>>>>>>>> repeatedly and >>>>>>>>> it will go on forever, where in the ZK situation, there will be only >>>>>>>>> one >>>>>>>>> leader, and the guys in the other partition simply won't be able to >>>>>>>>> reach >>>>>>>>> the leader, until its network issues are sorted. >>>>>>>>> >>>>>>>>> So I also think, as Imesh mentioned, creating a coordination >>>>>>>>> algorithm from scratch may not be a wise decision, and we should use >>>>>>>>> proven >>>>>>>>> technology/libraries to do that. And on a side note, the main reason >>>>>>>>> for >>>>>>>>> not using ZK for this earlier was because of the hassle of bringing up >>>>>>>>> another set of servers when our products are clustered, and we knew >>>>>>>>> that >>>>>>>>> the split brain scenario will occur in HZ, but maybe now we should >>>>>>>>> give an >>>>>>>>> extension point probably to plug into an external service if for some >>>>>>>>> applications the split brain scenario is a show stopper. >>>>>>>>> >>>>>>>>> Cheers, >>>>>>>>> Anjana. >>>>>>>>> >>>>>>>>> On Tue, Aug 9, 2016 at 4:45 AM, Kasun Indrasiri <ka...@wso2.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hi Ramith/Asanka, >>>>>>>>>> >>>>>>>>>> ESB/DSS natask impl is also based on HZ. I guess if this model >>>>>>>>>> works for the MB, we should make it generic for all such coordination >>>>>>>>>> requirements. (Thinking about using this in ESB 5.1)? >>>>>>>>>> >>>>>>>>>> On Fri, Aug 5, 2016 at 3:58 AM, Sajini De Silva <saj...@wso2.com> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Hi Maninda, >>>>>>>>>>> >>>>>>>>>>> Locking the database will be supported by some databases but >>>>>>>>>>> there will be huge performance impact. So we cannot use that >>>>>>>>>>> approach. If >>>>>>>>>>> this approach cannot be adapted the only thing we can do is queue >>>>>>>>>>> wise >>>>>>>>>>> load balancing through slot coordinator. But in this case we cannot >>>>>>>>>>> guarantee that load balance will be equally distributed since some >>>>>>>>>>> queues >>>>>>>>>>> can be loaded while some will be idle. Also we cannot have >>>>>>>>>>> multiple slot >>>>>>>>>>> coordinators having same queue as it may cause several >>>>>>>>>>> complications such >>>>>>>>>>> as, same slot is assigned to two nodes by different subscribers, >>>>>>>>>>> message >>>>>>>>>>> duplication etc. Actually this slot architecture was discussed in a >>>>>>>>>>> separate mail thread before it is implemented. >>>>>>>>>>> >>>>>>>>>>> Thanks >>>>>>>>>>> >>>>>>>>>>> On Fri, Aug 5, 2016 at 3:12 PM, Maninda Edirisooriya < >>>>>>>>>>> mani...@wso2.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi Sajini, >>>>>>>>>>>> >>>>>>>>>>>> Yes that is what I meant. As the number of slots are >>>>>>>>>>>> proportional to the number of messages passing through the >>>>>>>>>>>> cluster, slot >>>>>>>>>>>> delivery should not be handled by the coordinator when there is >>>>>>>>>>>> only one >>>>>>>>>>>> coordinator in the cluster which is a bottleneck for scaling >>>>>>>>>>>> messages >>>>>>>>>>>> passing through the cluster. If there is only a single >>>>>>>>>>>> coordinator, it >>>>>>>>>>>> should handle operations that are not proportional to messages >>>>>>>>>>>> throughput >>>>>>>>>>>> of the cluster. Then only the tasks like subscriber adding / >>>>>>>>>>>> removing >>>>>>>>>>>> should be handled by the coordinator. As this is not the current >>>>>>>>>>>> implementation, we can consider multiple coordinator approach. >>>>>>>>>>>> Then the >>>>>>>>>>>> number of coordinators should be scalable with the message >>>>>>>>>>>> throughout. I am >>>>>>>>>>>> not sure whether locking the database per transaction would >>>>>>>>>>>> achieve this >>>>>>>>>>>> coordinator scalability in the multiple coordinator implementation. >>>>>>>>>>>> >>>>>>>>>>>> Thanks. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> *Maninda Edirisooriya* >>>>>>>>>>>> Senior Software Engineer >>>>>>>>>>>> >>>>>>>>>>>> *WSO2, Inc.*lean.enterprise.middleware. >>>>>>>>>>>> >>>>>>>>>>>> *Blog* : http://maninda.blogspot.com/ >>>>>>>>>>>> *E-mail* : mani...@wso2.com >>>>>>>>>>>> *Skype* : @manindae >>>>>>>>>>>> *Twitter* : @maninda >>>>>>>>>>>> >>>>>>>>>>>> On Fri, Aug 5, 2016 at 2:42 PM, Sajini De Silva < >>>>>>>>>>>> saj...@wso2.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi Maninda, >>>>>>>>>>>>> >>>>>>>>>>>>> On Fri, Aug 5, 2016 at 2:28 PM, Maninda Edirisooriya < >>>>>>>>>>>>> mani...@wso2.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> @Sajini, >>>>>>>>>>>>>> >>>>>>>>>>>>>> But the number of slots are proportional to the number of >>>>>>>>>>>>>> messages pass through the MB which needs to be handled by the >>>>>>>>>>>>>> coordinator. >>>>>>>>>>>>>> That is what I meant by "information related to meta data of >>>>>>>>>>>>>> messages pass >>>>>>>>>>>>>> through a single coordinator". Ideally after the senders and >>>>>>>>>>>>>> receivers are >>>>>>>>>>>>>> subscribed to the cluster, coordinator should have nothing to do >>>>>>>>>>>>>> until they >>>>>>>>>>>>>> are removed or changed. >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Even though it is possible to have multiple coordinators after >>>>>>>>>>>>> having en effort (Lock the database for a whole transaction or >>>>>>>>>>>>> the work >>>>>>>>>>>>> load distribution as described by Ramith) , coordinator may have >>>>>>>>>>>>> different >>>>>>>>>>>>> work to do other than subscriber adding and removing. As I said >>>>>>>>>>>>> earlier our >>>>>>>>>>>>> MB message distribution system is based on slot architecture and >>>>>>>>>>>>> slots will >>>>>>>>>>>>> managed by the coordinator. You can read [1] to understand more >>>>>>>>>>>>> about slot >>>>>>>>>>>>> architecture in MB. >>>>>>>>>>>>> >>>>>>>>>>>>> [1] http://sajinid.blogspot.com/2015/03/wso2-message-broker- >>>>>>>>>>>>> 300-slot-based.html >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> @Ramith, >>>>>>>>>>>>>> >>>>>>>>>>>>>> +1 for multiple coordinators by partitioning the cluster >>>>>>>>>>>>>> which maintains the simplicity and correctness of the algorithm >>>>>>>>>>>>>> than >>>>>>>>>>>>>> compromising simplicity with a less important factor like >>>>>>>>>>>>>> "delivering a >>>>>>>>>>>>>> good mix of messages". >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> *Maninda Edirisooriya* >>>>>>>>>>>>>> Senior Software Engineer >>>>>>>>>>>>>> >>>>>>>>>>>>>> *WSO2, Inc.*lean.enterprise.middleware. >>>>>>>>>>>>>> >>>>>>>>>>>>>> *Blog* : http://maninda.blogspot.com/ >>>>>>>>>>>>>> *E-mail* : mani...@wso2.com >>>>>>>>>>>>>> *Skype* : @manindae >>>>>>>>>>>>>> *Twitter* : @maninda >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Fri, Aug 5, 2016 at 2:05 PM, Ramith Jayasinghe < >>>>>>>>>>>>>> ram...@wso2.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> @Imesh, >>>>>>>>>>>>>>> We can prove that doing leader election using a lib (where >>>>>>>>>>>>>>> we maintain cluster state in another place, a.k.a DB) will not >>>>>>>>>>>>>>> solve our >>>>>>>>>>>>>>> original problem (this also relates to our past experience with >>>>>>>>>>>>>>> both the >>>>>>>>>>>>>>> zookeeper and hazelcast). >>>>>>>>>>>>>>> We can make this implementation a common component if other >>>>>>>>>>>>>>> products have a use of it. BPS might be able to use it since >>>>>>>>>>>>>>> their data is >>>>>>>>>>>>>>> also in the database. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> @Malaka: >>>>>>>>>>>>>>> VFS scenario can't be solved by relying on this >>>>>>>>>>>>>>> implementation. why? you can have the access to DB but not VFS >>>>>>>>>>>>>>> resources/file (and vice versa). this is the same point we >>>>>>>>>>>>>>> explained before. >>>>>>>>>>>>>>> in Ntask implementation, if tasks are stored in the >>>>>>>>>>>>>>> database then using this implementation makes sense. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> @Akila, >>>>>>>>>>>>>>> implementing (distributed) a queue algorithm is >>>>>>>>>>>>>>> non-trivial. Having one coordinator (single source of truth) >>>>>>>>>>>>>>> keeps things >>>>>>>>>>>>>>> simple hence it's a conscious design decision we agreed during >>>>>>>>>>>>>>> the initial >>>>>>>>>>>>>>> stages. However, possible extension to this scheme is to have >>>>>>>>>>>>>>> multiple >>>>>>>>>>>>>>> coordinators ( each responsible for coordinating a subset of >>>>>>>>>>>>>>> queues in the >>>>>>>>>>>>>>> cluster), that will be some what similar to kafka. >>>>>>>>>>>>>>> Even if its preferable to have no coordinator at-all, (to >>>>>>>>>>>>>>> decide how messages are disseminated in the cluster) that will >>>>>>>>>>>>>>> make us >>>>>>>>>>>>>>> give up desired behaviour such as delivering a good mix of >>>>>>>>>>>>>>> messages (from >>>>>>>>>>>>>>> different publishers) to consumers in a cluster. having said >>>>>>>>>>>>>>> this, we have >>>>>>>>>>>>>>> an ongoing research on how to improve the algorithm and we like >>>>>>>>>>>>>>> to try out >>>>>>>>>>>>>>> both these approaches. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Fri, Aug 5, 2016 at 1:31 PM, Malaka Silva < >>>>>>>>>>>>>>> mal...@wso2.com> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> The same issue with Hazelcast can be experienced with ESB >>>>>>>>>>>>>>>> inbounds (running on top of NTASK) and VFS distribution locks. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> The idea of only single worker works at a given time breaks >>>>>>>>>>>>>>>> if there is a Hazelcast heart beat fails. This will make two >>>>>>>>>>>>>>>> workers to >>>>>>>>>>>>>>>> work in parallel. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Also with distributed locking there is no guarantee that >>>>>>>>>>>>>>>> file is only process only by one worker. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> So in the case of network fail >>>>>>>>>>>>>>>> with DB >>>>>>>>>>>>>>>> make sense to stop processing until it's recovered. >>>>>>>>>>>>>>>> Also making this component generic ESB can reuse. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Fri, Aug 5, 2016 at 9:21 AM, Asitha Nanayakkara < >>>>>>>>>>>>>>>> asi...@wso2.com> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hi Imesh, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Fri, Aug 5, 2016 at 7:33 AM, Imesh Gunaratne < >>>>>>>>>>>>>>>>> im...@wso2.com> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Fri, Aug 5, 2016 at 7:31 AM, Imesh Gunaratne < >>>>>>>>>>>>>>>>>> im...@wso2.com> wrote: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> You can see here [3] how K8S has implemented leader >>>>>>>>>>>>>>>>>>> election feature for the products deployed on top of that >>>>>>>>>>>>>>>>>>> to utilize. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Correction: Please refer [4]. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> On Thu, Aug 4, 2016 at 7:27 PM, Asanka Abeyweera < >>>>>>>>>>>>>>>>>>>> asank...@wso2.com> wrote: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Hi Imesh, >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> We are not implementing this to overcome a limitation >>>>>>>>>>>>>>>>>>>>> in the coordination algorithm available in the Hazlecast. >>>>>>>>>>>>>>>>>>>>> We are >>>>>>>>>>>>>>>>>>>>> implementing this since we need an RDBMS based >>>>>>>>>>>>>>>>>>>>> coordination algorithm (not >>>>>>>>>>>>>>>>>>>>> a network based algorithm). >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Are you saying that database connections do not use the >>>>>>>>>>>>>>>>>>> same network used by Hazelcast? >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> The reason is, a network based election algorithm will >>>>>>>>>>>>>>>>>>>>> always elect multiple leaders when the network is >>>>>>>>>>>>>>>>>>>>> partitioned. But if we >>>>>>>>>>>>>>>>>>>>> use a RDBMS based algorithm this will not happen. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> I do not think your argument is correct. If there is a >>>>>>>>>>>>>>>>>>> problem with the network, it may apply to both Hazelcast >>>>>>>>>>>>>>>>>>> based solution >>>>>>>>>>>>>>>>>>> and database based solution. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Yes, if the same network interface is used network partion >>>>>>>>>>>>>>>>> will cause all types of connections to be partitioned. But >>>>>>>>>>>>>>>>> user can use >>>>>>>>>>>>>>>>> multiple network interfaces for database, Hazelcast and >>>>>>>>>>>>>>>>> thrift. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Following is the scenario we are trying to solve in MB. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> In MB all the details related to messages, subscriptions, >>>>>>>>>>>>>>>>> queues, topics etc are stored in database. And we operate >>>>>>>>>>>>>>>>> depending on that >>>>>>>>>>>>>>>>> information. If the MB node can't connect to the database >>>>>>>>>>>>>>>>> that means the >>>>>>>>>>>>>>>>> node is ineffective in the cluster until it can make a >>>>>>>>>>>>>>>>> database connection. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> We have seen instances where Hazelcast cluster get >>>>>>>>>>>>>>>>> partitioned for some time period in networks, Reasons were, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 1. Due to heavy load Hazelcast couldn't process or >>>>>>>>>>>>>>>>> send (some times both) hearbeats, hence a network >>>>>>>>>>>>>>>>> partition for Hazelcast >>>>>>>>>>>>>>>>> cluster >>>>>>>>>>>>>>>>> 2. An actual network partition of Hazelcast cluster >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> In both scenarios the database connection was working. In >>>>>>>>>>>>>>>>> that case we get two coordinators elected through Hazelcast >>>>>>>>>>>>>>>>> and working on >>>>>>>>>>>>>>>>> the same database to deliver the messages. this leads to >>>>>>>>>>>>>>>>> inconsistencies in >>>>>>>>>>>>>>>>> the cluster behavior (for instances duplicate message >>>>>>>>>>>>>>>>> delivery, corrupred >>>>>>>>>>>>>>>>> subscription states etc) . >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Since the point of interest for MB is the database, we >>>>>>>>>>>>>>>>> decided to do the coordinator election through database as >>>>>>>>>>>>>>>>> well. If the >>>>>>>>>>>>>>>>> node can't connect to the database, then the MB won't operate >>>>>>>>>>>>>>>>> anyway. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>>>> Asitha >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> [4] http://blog.kubernetes.io/ >>>>>>>>>>>>>>>>>>> 2016/01/simple-leader-election-with-Kubernetes.html >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> On Thu, Aug 4, 2016 at 7:16 PM, Imesh Gunaratne < >>>>>>>>>>>>>>>>>>>>> im...@wso2.com> wrote: >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Hi Asanka, >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Do we really need to implement a leader election >>>>>>>>>>>>>>>>>>>>>> algorithm on our own? AFAIU this is a complex problem >>>>>>>>>>>>>>>>>>>>>> which has been >>>>>>>>>>>>>>>>>>>>>> already solved by several algorithms [1]. IMO it would >>>>>>>>>>>>>>>>>>>>>> be better to go >>>>>>>>>>>>>>>>>>>>>> ahead with an existing well established implementation >>>>>>>>>>>>>>>>>>>>>> on etcd [1] or >>>>>>>>>>>>>>>>>>>>>> Consul [2]. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Those provide HTTP APIs for clients to make leader >>>>>>>>>>>>>>>>>>>>>> election calls. [3] is a client library written in >>>>>>>>>>>>>>>>>>>>>> Node.js for etcd based >>>>>>>>>>>>>>>>>>>>>> leader election. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> [1] https://www.projectcalico. >>>>>>>>>>>>>>>>>>>>>> org/using-etcd-for-elections >>>>>>>>>>>>>>>>>>>>>> [2] https://www.consul.io/docs >>>>>>>>>>>>>>>>>>>>>> /guides/leader-election.html >>>>>>>>>>>>>>>>>>>>>> [3] https://www.npmjs.com/package/etcd-leader >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> On Wed, Aug 3, 2016 at 5:12 PM, Asanka Abeyweera < >>>>>>>>>>>>>>>>>>>>>> asank...@wso2.com> wrote: >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Hi Maninda, >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Since we are using RDBMS to poll the node status, >>>>>>>>>>>>>>>>>>>>>>> the cluster will not end up in situation 1,2 or 3. With >>>>>>>>>>>>>>>>>>>>>>> this approach we >>>>>>>>>>>>>>>>>>>>>>> consider a node unreachable when it cannot access the >>>>>>>>>>>>>>>>>>>>>>> database. Therefore >>>>>>>>>>>>>>>>>>>>>>> an unreachable node can never be the leader. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> As you have mentioned, we are currently using the >>>>>>>>>>>>>>>>>>>>>>> RDBMS as an atomic global variable to create the >>>>>>>>>>>>>>>>>>>>>>> coordinator entry. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> On Tue, Aug 2, 2016 at 5:22 PM, Maninda Edirisooriya >>>>>>>>>>>>>>>>>>>>>>> <mani...@wso2.com> wrote: >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Hi Asanka, >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> As I understand the accuracy of electing the leader >>>>>>>>>>>>>>>>>>>>>>>> correctly is dependent on the election mechanism with >>>>>>>>>>>>>>>>>>>>>>>> RDBMS because there >>>>>>>>>>>>>>>>>>>>>>>> can be edge cases like, >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> 1. Unreachable leader activates during the election >>>>>>>>>>>>>>>>>>>>>>>> process: Then who becomes the leader? >>>>>>>>>>>>>>>>>>>>>>>> 2. The elected leader becomes unreachable before >>>>>>>>>>>>>>>>>>>>>>>> the election is completed: Then will there be a >>>>>>>>>>>>>>>>>>>>>>>> situation where there is no >>>>>>>>>>>>>>>>>>>>>>>> leader? >>>>>>>>>>>>>>>>>>>>>>>> 3. A leader and a set of nodes are disconnected >>>>>>>>>>>>>>>>>>>>>>>> from the other part of the cluster and while the >>>>>>>>>>>>>>>>>>>>>>>> leader is trying to remove >>>>>>>>>>>>>>>>>>>>>>>> unreachable members other part is calling an election >>>>>>>>>>>>>>>>>>>>>>>> to make a leader: Who >>>>>>>>>>>>>>>>>>>>>>>> will win? >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> RDBMS based election algorithm should handle such >>>>>>>>>>>>>>>>>>>>>>>> cases without bringing the cluster to an inconsistent >>>>>>>>>>>>>>>>>>>>>>>> state or dead lock in >>>>>>>>>>>>>>>>>>>>>>>> all concurrent cases. If all these kind of cases >>>>>>>>>>>>>>>>>>>>>>>> cannot be handled isn't it >>>>>>>>>>>>>>>>>>>>>>>> better to keep the current hazelcast clustering and >>>>>>>>>>>>>>>>>>>>>>>> use the RDBMS only to >>>>>>>>>>>>>>>>>>>>>>>> handle the split brain scenario? In other words when a >>>>>>>>>>>>>>>>>>>>>>>> new hazelcast leader >>>>>>>>>>>>>>>>>>>>>>>> is elected it should be updated in the RDBMS. If >>>>>>>>>>>>>>>>>>>>>>>> another split party has >>>>>>>>>>>>>>>>>>>>>>>> already elected a leader, the node who is going to >>>>>>>>>>>>>>>>>>>>>>>> write it to RDBMS should >>>>>>>>>>>>>>>>>>>>>>>> avoid updating it. Simply, the RDBMS can be used as an >>>>>>>>>>>>>>>>>>>>>>>> atomic global >>>>>>>>>>>>>>>>>>>>>>>> variable to keep the leader name by modifying the >>>>>>>>>>>>>>>>>>>>>>>> hazelcast clustering. >>>>>>>>>>>>>>>>>>>>>>>> WDYT? >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Thanks. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> *Maninda Edirisooriya* >>>>>>>>>>>>>>>>>>>>>>>> Senior Software Engineer >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> *WSO2, Inc.*lean.enterprise.middleware. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> *Blog* : http://maninda.blogspot.com/ >>>>>>>>>>>>>>>>>>>>>>>> *E-mail* : mani...@wso2.com >>>>>>>>>>>>>>>>>>>>>>>> *Skype* : @manindae >>>>>>>>>>>>>>>>>>>>>>>> *Twitter* : @maninda >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> On Thu, Jul 28, 2016 at 4:38 PM, Asanka Abeyweera < >>>>>>>>>>>>>>>>>>>>>>>> asank...@wso2.com> wrote: >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> Hi Akila, >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> Let me explain the issue in a different way. Let's >>>>>>>>>>>>>>>>>>>>>>>>> assume the MB nodes are using two different network >>>>>>>>>>>>>>>>>>>>>>>>> interfaces for >>>>>>>>>>>>>>>>>>>>>>>>> Hazelcast communication and database communication. >>>>>>>>>>>>>>>>>>>>>>>>> With such a >>>>>>>>>>>>>>>>>>>>>>>>> configuration, there can be failures only in the >>>>>>>>>>>>>>>>>>>>>>>>> network interface used for >>>>>>>>>>>>>>>>>>>>>>>>> Hazelcast communication in some nodes. When this >>>>>>>>>>>>>>>>>>>>>>>>> happens, there will be two >>>>>>>>>>>>>>>>>>>>>>>>> or more Hazelcast clusters due to the network >>>>>>>>>>>>>>>>>>>>>>>>> segmentation, and as a result >>>>>>>>>>>>>>>>>>>>>>>>> there will be multiple coordinators. Since every node >>>>>>>>>>>>>>>>>>>>>>>>> still have access to >>>>>>>>>>>>>>>>>>>>>>>>> the database, multiple coordinators can affect the >>>>>>>>>>>>>>>>>>>>>>>>> correctness of the data >>>>>>>>>>>>>>>>>>>>>>>>> stored in the DB. But if we used a RDBMS based >>>>>>>>>>>>>>>>>>>>>>>>> approach we won't have >>>>>>>>>>>>>>>>>>>>>>>>> multiple coordinators due to a network partition in >>>>>>>>>>>>>>>>>>>>>>>>> Hazelcast. This is one >>>>>>>>>>>>>>>>>>>>>>>>> advantage we get from this approach. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> Even when we use Zookeeper or RAFT the same issue >>>>>>>>>>>>>>>>>>>>>>>>> will be there since we are using different interfaces >>>>>>>>>>>>>>>>>>>>>>>>> for Hazelcast >>>>>>>>>>>>>>>>>>>>>>>>> communication and DB communication. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Jul 28, 2016 at 2:56 PM, Akila Ravihansa >>>>>>>>>>>>>>>>>>>>>>>>> Perera <raviha...@wso2.com> wrote: >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> What's the advantage of using RDBMS (even as an >>>>>>>>>>>>>>>>>>>>>>>>>> alternative) to implement a leader/coordinator >>>>>>>>>>>>>>>>>>>>>>>>>> election? If the network >>>>>>>>>>>>>>>>>>>>>>>>>> connection to DB fails then this will be a single >>>>>>>>>>>>>>>>>>>>>>>>>> point of failure. I don't >>>>>>>>>>>>>>>>>>>>>>>>>> think we can scale RDBMS instances and expect the >>>>>>>>>>>>>>>>>>>>>>>>>> election algorithm to >>>>>>>>>>>>>>>>>>>>>>>>>> work. That would be reducing this problem to another >>>>>>>>>>>>>>>>>>>>>>>>>> problem (electing >>>>>>>>>>>>>>>>>>>>>>>>>> coordinator RDBMS instance). >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> IMHO it would be better to look at Zookeeper >>>>>>>>>>>>>>>>>>>>>>>>>> Atomic Broadcast (ZAB) [1] or RAFT leader election >>>>>>>>>>>>>>>>>>>>>>>>>> [2] algorithms which >>>>>>>>>>>>>>>>>>>>>>>>>> have already proven results. >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> [1] https://cwiki.apache.org/c >>>>>>>>>>>>>>>>>>>>>>>>>> onfluence/display/ZOOKEEPER/Zab1.0 >>>>>>>>>>>>>>>>>>>>>>>>>> [2] http://libraft.io/ >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks. >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Jul 28, 2016 at 1:42 PM, Nandika >>>>>>>>>>>>>>>>>>>>>>>>>> Jayawardana <nand...@wso2.com> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> +1 to make it a common component . We have the >>>>>>>>>>>>>>>>>>>>>>>>>>> clustering implementation for BPEL component based >>>>>>>>>>>>>>>>>>>>>>>>>>> on hazelcast. If the >>>>>>>>>>>>>>>>>>>>>>>>>>> coordination is available at RDBMS level, we can >>>>>>>>>>>>>>>>>>>>>>>>>>> remove hazelcast >>>>>>>>>>>>>>>>>>>>>>>>>>> dependancy. >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Regards >>>>>>>>>>>>>>>>>>>>>>>>>>> Nandika >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Jul 28, 2016 at 1:28 PM, Hasitha >>>>>>>>>>>>>>>>>>>>>>>>>>> Aravinda <hasi...@wso2.com> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Can we make it a common component, which is not >>>>>>>>>>>>>>>>>>>>>>>>>>>> hard coupled with MB. BPS has the same requirement. >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>>>>>>>>>>>> Hasitha. >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Jul 28, 2016 at 9:47 AM, Asanka >>>>>>>>>>>>>>>>>>>>>>>>>>>> Abeyweera <asank...@wso2.com> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi All, >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> In MB, we have used a coordinator based >>>>>>>>>>>>>>>>>>>>>>>>>>>>> approach to manage distributed messaging >>>>>>>>>>>>>>>>>>>>>>>>>>>>> algorithm in the cluster. >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Currently Hazelcast is used to elect the >>>>>>>>>>>>>>>>>>>>>>>>>>>>> coordinator. But one issue we >>>>>>>>>>>>>>>>>>>>>>>>>>>>> faced with Hazelcast is, during a network >>>>>>>>>>>>>>>>>>>>>>>>>>>>> segmentation (split brain), >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hazelcast can elect two or more coordinators in >>>>>>>>>>>>>>>>>>>>>>>>>>>>> the cluster. This affects >>>>>>>>>>>>>>>>>>>>>>>>>>>>> the correctness of the distributed messaging >>>>>>>>>>>>>>>>>>>>>>>>>>>>> algorithm since there are some >>>>>>>>>>>>>>>>>>>>>>>>>>>>> tables in the database that should only be edited >>>>>>>>>>>>>>>>>>>>>>>>>>>>> by a single node (i.e. >>>>>>>>>>>>>>>>>>>>>>>>>>>>> coordinator). >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> As a solution to this problem we have >>>>>>>>>>>>>>>>>>>>>>>>>>>>> implemented minimum node count based approach [1] >>>>>>>>>>>>>>>>>>>>>>>>>>>>> to deactivate set of >>>>>>>>>>>>>>>>>>>>>>>>>>>>> partitioned nodes to stop multiple nodes becoming >>>>>>>>>>>>>>>>>>>>>>>>>>>>> coordinators until the >>>>>>>>>>>>>>>>>>>>>>>>>>>>> network segmentation issue is fixed. >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> As an alternative solution, we are thinking of >>>>>>>>>>>>>>>>>>>>>>>>>>>>> implementing an RDBMS based approach to elect the >>>>>>>>>>>>>>>>>>>>>>>>>>>>> coordinator node in the >>>>>>>>>>>>>>>>>>>>>>>>>>>>> cluster. By doing this we can make sure that even >>>>>>>>>>>>>>>>>>>>>>>>>>>>> during a network >>>>>>>>>>>>>>>>>>>>>>>>>>>>> segmentation only one node will be elected as the >>>>>>>>>>>>>>>>>>>>>>>>>>>>> coordinator node since >>>>>>>>>>>>>>>>>>>>>>>>>>>>> the election is happening through the database. >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> The algorithm will use a polling mechanism to >>>>>>>>>>>>>>>>>>>>>>>>>>>>> check the validity of the nodes. To make the >>>>>>>>>>>>>>>>>>>>>>>>>>>>> election algorithm scalable, >>>>>>>>>>>>>>>>>>>>>>>>>>>>> only the coordinator node will be checking status >>>>>>>>>>>>>>>>>>>>>>>>>>>>> of all the nodes in the >>>>>>>>>>>>>>>>>>>>>>>>>>>>> cluster and it will inform other nodes through >>>>>>>>>>>>>>>>>>>>>>>>>>>>> database when a member is >>>>>>>>>>>>>>>>>>>>>>>>>>>>> added/left. The nodes will be only checking for >>>>>>>>>>>>>>>>>>>>>>>>>>>>> the status of the >>>>>>>>>>>>>>>>>>>>>>>>>>>>> coordinator node. When a node detect that >>>>>>>>>>>>>>>>>>>>>>>>>>>>> coordinator is invalid it will go >>>>>>>>>>>>>>>>>>>>>>>>>>>>> for a election to elect a new coordinator. >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> We are currently working on a POC to test how >>>>>>>>>>>>>>>>>>>>>>>>>>>>> this works with MB's slot based messaging >>>>>>>>>>>>>>>>>>>>>>>>>>>>> algorithm. >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> thoughts? >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1] https://wso2.org/jira/browse/MB-1664 >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Asanka Abeyweera >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Senior Software Engineer >>>>>>>>>>>>>>>>>>>>>>>>>>>>> WSO2 Inc. >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Phone: +94 712228648 >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Blog: a5anka.github.io >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> <https://wso2.com/signature> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> ______________________________ >>>>>>>>>>>>>>>>>>>>>>>>>>>>> _________________ >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Architecture@wso2.org >>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/ >>>>>>>>>>>>>>>>>>>>>>>>>>>>> mailman/listinfo/architecture >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>>>>>>> Hasitha Aravinda, >>>>>>>>>>>>>>>>>>>>>>>>>>>> Associate Technical Lead, >>>>>>>>>>>>>>>>>>>>>>>>>>>> WSO2 Inc. >>>>>>>>>>>>>>>>>>>>>>>>>>>> Email: hasi...@wso2.com >>>>>>>>>>>>>>>>>>>>>>>>>>>> Mobile : +94 718 210 200 >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>>>>>>>>>>>>>>>>> Architecture@wso2.org >>>>>>>>>>>>>>>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/ >>>>>>>>>>>>>>>>>>>>>>>>>>>> mailman/listinfo/architecture >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>>>>>> Nandika Jayawardana >>>>>>>>>>>>>>>>>>>>>>>>>>> WSO2 Inc ; http://wso2.com >>>>>>>>>>>>>>>>>>>>>>>>>>> lean.enterprise.middleware >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>>>>>>>>>>>>>>>> Architecture@wso2.org >>>>>>>>>>>>>>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/ >>>>>>>>>>>>>>>>>>>>>>>>>>> mailman/listinfo/architecture >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>>>>> Akila Ravihansa Perera >>>>>>>>>>>>>>>>>>>>>>>>>> WSO2 Inc.; http://wso2.com/ >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> Blog: http://ravihansa3000.blogspot.com >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>>>>>>>>>>>>>>> Architecture@wso2.org >>>>>>>>>>>>>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/ >>>>>>>>>>>>>>>>>>>>>>>>>> mailman/listinfo/architecture >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>>>> Asanka Abeyweera >>>>>>>>>>>>>>>>>>>>>>>>> Senior Software Engineer >>>>>>>>>>>>>>>>>>>>>>>>> WSO2 Inc. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> Phone: +94 712228648 >>>>>>>>>>>>>>>>>>>>>>>>> Blog: a5anka.github.io >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> <https://wso2.com/signature> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>>>>>>>>>>>>>> Architecture@wso2.org >>>>>>>>>>>>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/ >>>>>>>>>>>>>>>>>>>>>>>>> mailman/listinfo/architecture >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>> Asanka Abeyweera >>>>>>>>>>>>>>>>>>>>>>> Senior Software Engineer >>>>>>>>>>>>>>>>>>>>>>> WSO2 Inc. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Phone: +94 712228648 >>>>>>>>>>>>>>>>>>>>>>> Blog: a5anka.github.io >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> <https://wso2.com/signature> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>>>>>>>>>>>> Architecture@wso2.org >>>>>>>>>>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/ >>>>>>>>>>>>>>>>>>>>>>> mailman/listinfo/architecture >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>> *Imesh Gunaratne* >>>>>>>>>>>>>>>>>>>>>> Software Architect >>>>>>>>>>>>>>>>>>>>>> WSO2 Inc: http://wso2.com >>>>>>>>>>>>>>>>>>>>>> T: +94 11 214 5345 M: +94 77 374 2057 >>>>>>>>>>>>>>>>>>>>>> W: https://medium.com/@imesh TW: @imesh >>>>>>>>>>>>>>>>>>>>>> lean. enterprise. middleware >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>>>>>>>>>>> Architecture@wso2.org >>>>>>>>>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/ >>>>>>>>>>>>>>>>>>>>>> mailman/listinfo/architecture >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>> Asanka Abeyweera >>>>>>>>>>>>>>>>>>>>> Senior Software Engineer >>>>>>>>>>>>>>>>>>>>> WSO2 Inc. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Phone: +94 712228648 >>>>>>>>>>>>>>>>>>>>> Blog: a5anka.github.io >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> <https://wso2.com/signature> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>>>>>>>>>> Architecture@wso2.org >>>>>>>>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/ >>>>>>>>>>>>>>>>>>>>> mailman/listinfo/architecture >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>> Ramith Jayasinghe >>>>>>>>>>>>>>>>>>>> Technical Lead >>>>>>>>>>>>>>>>>>>> WSO2 Inc., http://wso2.com >>>>>>>>>>>>>>>>>>>> lean.enterprise.middleware >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> E: ram...@wso2.com >>>>>>>>>>>>>>>>>>>> P: +94 772534930 >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>>>>>>>>> Architecture@wso2.org >>>>>>>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/ >>>>>>>>>>>>>>>>>>>> mailman/listinfo/architecture >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>> *Imesh Gunaratne* >>>>>>>>>>>>>>>>>>> Software Architect >>>>>>>>>>>>>>>>>>> WSO2 Inc: http://wso2.com >>>>>>>>>>>>>>>>>>> T: +94 11 214 5345 M: +94 77 374 2057 >>>>>>>>>>>>>>>>>>> W: https://medium.com/@imesh TW: @imesh >>>>>>>>>>>>>>>>>>> lean. enterprise. middleware >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>> *Imesh Gunaratne* >>>>>>>>>>>>>>>>>> Software Architect >>>>>>>>>>>>>>>>>> WSO2 Inc: http://wso2.com >>>>>>>>>>>>>>>>>> T: +94 11 214 5345 M: +94 77 374 2057 >>>>>>>>>>>>>>>>>> W: https://medium.com/@imesh TW: @imesh >>>>>>>>>>>>>>>>>> lean. enterprise. middleware >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>>>>>>> Architecture@wso2.org >>>>>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/ >>>>>>>>>>>>>>>>>> mailman/listinfo/architecture >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>> *Asitha Nanayakkara* <http://asitha.github.io/> >>>>>>>>>>>>>>>>> Senior Software Engineer >>>>>>>>>>>>>>>>> WSO2, Inc. <http://wso2.com/> >>>>>>>>>>>>>>>>> Mob: +94 77 853 0682 >>>>>>>>>>>>>>>>> [image: https://wso2.com/signature] >>>>>>>>>>>>>>>>> <https://wso2.com/signature> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>>>>>> Architecture@wso2.org >>>>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/ >>>>>>>>>>>>>>>>> mailman/listinfo/architecture >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Best Regards, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Malaka Silva >>>>>>>>>>>>>>>> Senior Technical Lead >>>>>>>>>>>>>>>> M: +94 777 219 791 >>>>>>>>>>>>>>>> Tel : 94 11 214 5345 >>>>>>>>>>>>>>>> Fax :94 11 2145300 >>>>>>>>>>>>>>>> Skype : malaka.sampath.silva >>>>>>>>>>>>>>>> LinkedIn : http://www.linkedin.com/pub/malaka-silva/6/33/77 >>>>>>>>>>>>>>>> Blog : http://mrmalakasilva.blogspot.com/ >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> WSO2, Inc. >>>>>>>>>>>>>>>> lean . enterprise . middleware >>>>>>>>>>>>>>>> https://wso2.com/signature >>>>>>>>>>>>>>>> http://www.wso2.com/about/team/malaka-silva/ >>>>>>>>>>>>>>>> <http://wso2.com/about/team/malaka-silva/> >>>>>>>>>>>>>>>> https://store.wso2.com/store/ >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Don't make Trees rare, we should keep them with care >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>>>>> Architecture@wso2.org >>>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> Ramith Jayasinghe >>>>>>>>>>>>>>> Technical Lead >>>>>>>>>>>>>>> WSO2 Inc., http://wso2.com >>>>>>>>>>>>>>> lean.enterprise.middleware >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> E: ram...@wso2.com >>>>>>>>>>>>>>> P: +94 772534930 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>>>> Architecture@wso2.org >>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> Sajini De SIlva >>>>>>>>>>>>> Senior Software Engineer; WSO2 Inc.; http://wso2.com , >>>>>>>>>>>>> Email: saj...@wso2.com >>>>>>>>>>>>> Blog: http://sajinid.blogspot.com/ >>>>>>>>>>>>> Git hub profile: https://github.com/sajinidesilva >>>>>>>>>>>>> >>>>>>>>>>>>> Phone: +94 712797729 >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Sajini De SIlva >>>>>>>>>>> Senior Software Engineer; WSO2 Inc.; http://wso2.com , >>>>>>>>>>> Email: saj...@wso2.com >>>>>>>>>>> Blog: http://sajinid.blogspot.com/ >>>>>>>>>>> Git hub profile: https://github.com/sajinidesilva >>>>>>>>>>> >>>>>>>>>>> Phone: +94 712797729 >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> Architecture mailing list >>>>>>>>>>> Architecture@wso2.org >>>>>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Kasun Indrasiri >>>>>>>>>> Director, Integration Technologies >>>>>>>>>> WSO2, Inc.; http://wso2.com >>>>>>>>>> lean.enterprise.middleware >>>>>>>>>> >>>>>>>>>> cell: +1 650 450 2293 >>>>>>>>>> Blog : http://kasunpanorama.blogspot.com/ >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> *Anjana Fernando* >>>>>>>>> Associate Director / Architect >>>>>>>>> WSO2 Inc. | http://wso2.com >>>>>>>>> lean . enterprise . middleware >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> Architecture mailing list >>>>>>>>> Architecture@wso2.org >>>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> *Anjana Fernando* >>>>>>> Associate Director / Architect >>>>>>> WSO2 Inc. | http://wso2.com >>>>>>> lean . enterprise . middleware >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Asanka Abeyweera >>>>>> Senior Software Engineer >>>>>> WSO2 Inc. >>>>>> >>>>>> Phone: +94 712228648 >>>>>> Blog: a5anka.github.io >>>>>> >>>>>> <https://wso2.com/signature> >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> *Anjana Fernando* >>>>> Associate Director / Architect >>>>> WSO2 Inc. | http://wso2.com >>>>> lean . enterprise . middleware >>>>> >>>> >>>> >>>> >>>> -- >>>> *Anjana Fernando* >>>> Associate Director / Architect >>>> WSO2 Inc. | http://wso2.com >>>> lean . enterprise . middleware >>>> >>> >>> >>> >>> -- >>> Asanka Abeyweera >>> Senior Software Engineer >>> WSO2 Inc. >>> >>> Phone: +94 712228648 >>> Blog: a5anka.github.io >>> >>> <https://wso2.com/signature> >>> >> >> >> >> -- >> *Anjana Fernando* >> Associate Director / Architect >> WSO2 Inc. | http://wso2.com >> lean . enterprise . middleware >> > > > > -- > *Anjana Fernando* > Associate Director / Architect > WSO2 Inc. | http://wso2.com > lean . enterprise . middleware > -- Ramith Jayasinghe Technical Lead WSO2 Inc., http://wso2.com lean.enterprise.middleware E: ram...@wso2.com P: +94 772534930
_______________________________________________ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
