Hi Shaheed, As i have locally tested, member fault case working fine in the current master. Can you share us with the complete stratos logs once you upgrade your setup to the latest code in master? Also, you can enable debug logs for autosclaer and drools as below in <STRATOS-HOME>/repository/conf/log4j.properties
log4j.logger.org.apache.stratos.autoscaler=DEBUG log4j.logger.org.apache.stratos.autoscaler.rule.RuleLog=DEBUG Are you running a separate CEP? In that case, if you could share the CEP logs, then it will help us to narrow down the problem. Thanks, Reka On Mon, May 18, 2015 at 9:07 PM, Reka Thirunavukkarasu <r...@wso2.com> wrote: > Hi Shaheed, > > We had been using member fault detection to test the termination-behavior > in beta2 and after that like one week before. So, i believe that It will > work in the latest master. However we will also verify this again. > > Thanks, > Reka > > On Mon, May 18, 2015 at 7:53 PM, Imesh Gunaratne <im...@apache.org> wrote: > >> Thanks Shaheed! I will verify the second problem where Stratos is not >> detecting manually terminated members. >> >> Thanks >> >> On Mon, May 18, 2015 at 3:39 PM, Shaheedur Haque (shahhaqu) < >> shahh...@cisco.com> wrote: >> >>> Ack. We are just in the middle of doing getting sync’d up again to >>> master, and it sounds like that might fix the persistence issue. >>> >>> >>> >>> I guess that leaves the Cartridge Agent reconnect side of the problem… >>> >>> >>> >>> *From:* Lahiru Sandaruwan [mailto:lahi...@wso2.com] >>> *Sent:* 17 May 2015 03:06 >>> >>> *To:* dev >>> *Cc:* Ryan Du Plessis (rdupless); Luca Martini (lmartini) >>> *Subject:* Re: Clustered deployments of Stratos >>> >>> >>> >>> Hi Shaheed, >>> >>> >>> >>> Similarly it would be a great help, if you can verify all these issues >>> in latest code, since we have been fixing a lot of issues in recent days, >>> as a result of RC1 testing. >>> >>> >>> >>> Thanks. >>> >>> >>> >>> On Fri, May 15, 2015 at 9:42 PM, Imesh Gunaratne <im...@apache.org> >>> wrote: >>> >>> Hi Shaheed, >>> >>> >>> >>> Thanks for the quick response, after analyzing the results you have >>> provided again, it looks like only the deployment policies are missing >>> after the failover. We have fixed this issue in commit >>> revision: 0c515aa013850575ddcfa2e299da5f0ec250ebc3 >>> >>> >>> >>> >>> http://mail-archives.apache.org/mod_mbox/incubator-stratos-commits/201504.mbox/%3c22eed4e8639c401a8fda637fa6bb4...@git.apache.org%3E >>> >>> >>> >>> Would you mind verifying whether this is there in your runtime? >>> >>> >>> >>> Thanks >>> >>> >>> >>> >>> >>> On Fri, May 15, 2015 at 9:02 PM, Shaheedur Haque (shahhaqu) < >>> shahh...@cisco.com> wrote: >>> >>> The latter; we never have both Stratos instances running. >>> >>> >>> >>> *From:* Imesh Gunaratne [mailto:im...@apache.org] >>> *Sent:* 15 May 2015 16:17 >>> *To:* dev >>> *Cc:* Ryan Du Plessis (rdupless); Luca Martini (lmartini) >>> >>> >>> *Subject:* Re: Clustered deployments of Stratos >>> >>> >>> >>> Hi Shaheed, >>> >>> >>> >>> Do you have both active and passive Stratos nodes running at the same >>> time or do you start the passive node once the active node goes down? >>> >>> >>> >>> Thanks >>> >>> >>> >>> On Fri, May 15, 2015 at 6:31 PM, Shaheedur Haque (shahhaqu) < >>> shahh...@cisco.com> wrote: >>> >>> Hi Imesh, >>> >>> >>> >>> I finally got round to a proper series of tests, and here are the >>> conclusions: >>> >>> >>> >>> · In Stratos 4.0, after a Pacemaker driven failover, the newly >>> Active Stratos has lost all Cartridge Definitions. >>> >>> · In current [1] Stratos 4.1, after a Pacemaker driven failover, >>> the newly Active Stratos: >>> >>> o Has lost all Deployment Policies. >>> >>> o Has lost contact with the Cartridge Agents, and all VMs are stuck >>> with whatever state they had before the failover. >>> >>> · Note: I have not verified if Cartridge Groups are lost or not. >>> >>> >>> >>> I include the test results below at [2] and [3]. I am concerned as to >>> whether 4.1 is ready for GA on this basis, so though more testing is no >>> doubt possible (e.g. Cartridge Groups) I wanted to get this info to the >>> list ASAP. >>> >>> >>> >>> Thanks, Shaheed >>> >>> >>> >>> [1] A recent build somewhere between beta 1 and beta 2, but I don’t >>> think any relevant fixes have been made in master. >>> >>> >>> >>> [2] Persistence test output from Stratos 4.1. Note: >>> >>> >>> >>> 1. In the build I have, the CLI is broken for a couple of >>> commands; these are supplemented by direct “curl” commands further down. >>> >>> 2. I’ve used one of our commands to show the instances and their >>> state for a given application since there is not a compact JSON or >>> convenient Startos CLI for that. >>> >>> >>> >>> *PERSISTENCE TEST, BEFORE FAILOVER* >>> >>> *================================* >>> >>> >>> >>> stratos> list-tenants >>> >>> Tenants: >>> >>> >>> +-----------------------+-----------+------------------+--------+------------------------------+ >>> >>> | Domain | Tenant ID | Email | State | >>> Created Date | >>> >>> >>> +-----------------------+-----------+------------------+--------+------------------------------+ >>> >>> | cloud1.qmog.cisco.com | 1 | clo...@cisco.com | Active | Fri >>> May 15 04:46:58 MDT 2015 | >>> >>> >>> +-----------------------+-----------+------------------+--------+------------------------------+ >>> >>> >>> >>> stratos> list-network-partitions >>> >>> Network partitions found: >>> >>> +----------------------+----------------------+ >>> >>> | Network Partition ID | Number of Partitions | >>> >>> +----------------------+----------------------+ >>> >>> | RegionOne | 1 | >>> >>> +----------------------+----------------------+ >>> >>> >>> >>> stratos> list-deployment-policies >>> >>> Deployment policies found: >>> >>> +-------------------+---------------+ >>> >>> | ID | Accessibility | >>> >>> +-------------------+---------------+ >>> >>> | static-2-ha | 1 | >>> >>> +-------------------+---------------+ >>> >>> | autoscale-2-10-ha | 1 | >>> >>> +-------------------+---------------+ >>> >>> | autoscale-1-5 | 1 | >>> >>> +-------------------+---------------+ >>> >>> | static-1 | 1 | >>> >>> +-------------------+---------------+ >>> >>> >>> >>> stratos> list-application-policies >>> >>> Error in listing application policies >>> >>> No application policies found >>> >>> >>> >>> stratos> list-autoscaling-policies >>> >>> Error in listing autoscaling policies >>> >>> No autoscaling policies found >>> >>> >>> >>> stratos> list-cartridges >>> >>> Cartridges found: >>> >>> >>> +------------------+-------------+------------------+----------------------------+---------+--------------+ >>> >>> | Type | Category | Name | >>> Description | Version | Multi-Tenant | >>> >>> >>> +------------------+-------------+------------------+----------------------------+---------+--------------+ >>> >>> | cartridge-proxy | Application | cartridge-proxy | cartridge-proxy >>> Cartridge | 1 | false | >>> >>> >>> +------------------+-------------+------------------+----------------------------+---------+--------------+ >>> >>> | cisco-sample-vm | Application | cisco-sample-vm | cisco-sample-vm >>> Cartridge | 1 | false | >>> >>> >>> +------------------+-------------+------------------+----------------------------+---------+--------------+ >>> >>> | cisco-qvpc-cf-01 | Application | cisco-qvpc-cf-01 | cisco-qvpc-cf-01 >>> Cartridge | 1 | false | >>> >>> >>> +------------------+-------------+------------------+----------------------------+---------+--------------+ >>> >>> | cisco-qvpc-cf-02 | Application | cisco-qvpc-cf-02 | cisco-qvpc-cf-02 >>> Cartridge | 1 | false | >>> >>> >>> +------------------+-------------+------------------+----------------------------+---------+--------------+ >>> >>> | cisco-qvpc-si | Application | cisco-qvpc-si | cisco-qvpc-si >>> Cartridge | 1 | false | >>> >>> >>> +------------------+-------------+------------------+----------------------------+---------+--------------+ >>> >>> | cisco-qvpc-sf | Application | cisco-qvpc-sf | cisco-qvpc-sf >>> Cartridge | 1 | false | >>> >>> >>> +------------------+-------------+------------------+----------------------------+---------+--------------+ >>> >>> >>> >>> stratos> list-applications >>> >>> Applications found: >>> >>> +-----------------+-----------------+----------+ >>> >>> | Application ID | Alias | Status | >>> >>> +-----------------+-----------------+----------+ >>> >>> | cartridge-proxy | cartridge-proxy | Deployed | >>> >>> +-----------------+-----------------+----------+ >>> >>> | cisco-sample-vm | cisco-sample-vm | Deployed | >>> >>> +-----------------+-----------------+----------+ >>> >>> >>> >>> $ curl -uadmin:admin -k -H'Content-type: application/json' >>> https://localhost:9443/api/autoscalingPolicies >>> >>> >>> [{"id":"economyPolicy","instanceRoundingFactor":0,"isPublic":false,"loadThresholds":""}] >>> >>> >>> >>> $ curl -uadmin:admin -k -H'Content-type: application/json' >>> https://localhost:9443/api/applicationPolicies >>> >>> >>> [{"algorithm":"one-after-another","id":"default-iaas","networkPartitions":["RegionOne"],"properties":{"name":"networkPartitionGroups","value":"RegionOne"}}] >>> >>> >>> >>> >>> >>> *PERSISTENCE TEST, AFTER FAILOVER* >>> >>> *===============================* >>> >>> >>> >>> stratos> list-tenants >>> >>> Tenants: >>> >>> >>> +-----------------------+-----------+------------------+--------+------------------------------+ >>> >>> | Domain | Tenant ID | Email | State | >>> Created Date | >>> >>> >>> +-----------------------+-----------+------------------+--------+------------------------------+ >>> >>> | cloud1.qmog.cisco.com | 1 | clo...@cisco.com | Active | Fri >>> May 15 05:26:52 MDT 2015 | >>> >>> >>> +-----------------------+-----------+------------------+--------+------------------------------+ >>> >>> >>> >>> stratos> list-network-partitions >>> >>> Network partitions found: >>> >>> +----------------------+----------------------+ >>> >>> | Network Partition ID | Number of Partitions | >>> >>> +----------------------+----------------------+ >>> >>> | RegionOne | 1 | >>> >>> +----------------------+----------------------+ >>> >>> >>> >>> stratos> list-deployment-policies >>> >>> No deployment policies found >>> >>> >>> >>> stratos> list-application-policies >>> >>> Error in listing application policies >>> >>> No application policies found >>> >>> >>> >>> stratos> list-autoscaling-policies >>> >>> Error in listing autoscaling policies >>> >>> No autoscaling policies found >>> >>> >>> >>> stratos> list-cartridges >>> >>> Cartridges found: >>> >>> >>> +------------------+-------------+------------------+----------------------------+---------+--------------+ >>> >>> | Type | Category | Name | >>> Description | Version | Multi-Tenant | >>> >>> >>> +------------------+-------------+------------------+----------------------------+---------+--------------+ >>> >>> | cartridge-proxy | Application | cartridge-proxy | cartridge-proxy >>> Cartridge | 1 | false | >>> >>> >>> +------------------+-------------+------------------+----------------------------+---------+--------------+ >>> >>> | cisco-sample-vm | Application | cisco-sample-vm | cisco-sample-vm >>> Cartridge | 1 | false | >>> >>> >>> +------------------+-------------+------------------+----------------------------+---------+--------------+ >>> >>> | cisco-qvpc-cf-01 | Application | cisco-qvpc-cf-01 | cisco-qvpc-cf-01 >>> Cartridge | 1 | false | >>> >>> >>> +------------------+-------------+------------------+----------------------------+---------+--------------+ >>> >>> | cisco-qvpc-cf-02 | Application | cisco-qvpc-cf-02 | cisco-qvpc-cf-02 >>> Cartridge | 1 | false | >>> >>> >>> +------------------+-------------+------------------+----------------------------+---------+--------------+ >>> >>> | cisco-qvpc-si | Application | cisco-qvpc-si | cisco-qvpc-si >>> Cartridge | 1 | false | >>> >>> >>> +------------------+-------------+------------------+----------------------------+---------+--------------+ >>> >>> | cisco-qvpc-sf | Application | cisco-qvpc-sf | cisco-qvpc-sf >>> Cartridge | 1 | false | >>> >>> >>> +------------------+-------------+------------------+----------------------------+---------+--------------+ >>> >>> >>> >>> stratos> list-applications >>> >>> Applications found: >>> >>> +-----------------+-----------------+----------+ >>> >>> | Application ID | Alias | Status | >>> >>> +-----------------+-----------------+----------+ >>> >>> | cartridge-proxy | cartridge-proxy | Deployed | >>> >>> +-----------------+-----------------+----------+ >>> >>> | cisco-sample-vm | cisco-sample-vm | Deployed | >>> >>> +-----------------+-----------------+----------+ >>> >>> >>> >>> $ curl -uadmin:admin -k -H'Content-type: application/json' >>> https://localhost:9443/api/autoscalingPolicies >>> >>> >>> [{"id":"economyPolicy","instanceRoundingFactor":0,"isPublic":false,"loadThresholds":""}] >>> >>> >>> >>> $ curl -uadmin:admin -k -H'Content-type: application/json' >>> https://localhost:9443/api/applicationPolicies >>> >>> >>> [{"algorithm":"one-after-another","id":"default-iaas","networkPartitions":["RegionOne"],"properties":{"name":"networkPartitionGroups","value":"RegionOne"}}] >>> >>> >>> >>> [3] Cartridge test output from Stratos 4.1. Note: >>> >>> >>> >>> 1. We do not use a VIP for Stratos, either for 4.0 or 4.1. >>> >>> 2. We expect the Cartridge Agent to use a DNS lookup when it ends >>> up reconnecting, and this worked just fine in Stratos 4.0. >>> >>> >>> >>> *CARTRIDGE TEST, BEFORE FAILOVER* >>> >>> *==============================* >>> >>> >>> >>> $ ./bin/orchestration subscription list-instances --admin cisco-sample-vm >>> >>> cisco-sample-vm: applicationInstances 1, groupInstances 0, >>> clusterInstances 1, members 1 (Active 1) >>> >>> cisco-sample-vm: 172.16.180.30/10.0.0.101: status Active >>> >>> >>> >>> *CARTRIDGE TEST, AFTER FAILOVER* >>> >>> *=============================* >>> >>> >>> >>> $ ./bin/orchestration subscription list-instances --admin cisco-sample-vm >>> >>> cisco-sample-vm: applicationInstances 1, groupInstances 0, >>> clusterInstances 1, members 1 (Active 1) >>> >>> cisco-sample-vm: 172.16.180.30/10.0.0.101: status Active >>> >>> >>> >>> *CARTRIDGE TEST, AFTER FAILOVER WAIT 5 MINUTES, THEN KILL INSTANCE, >>> THEN WAIT 2 MINUTES* >>> >>> >>> *===================================================================================* >>> >>> >>> >>> $ ./bin/orchestration subscription list-instances --admin cisco-sample-vm >>> >>> cisco-sample-vm: applicationInstances 1, groupInstances 0, >>> clusterInstances 1, members 1 (Active 1) >>> >>> cisco-sample-vm: 172.16.180.30/10.0.0.101: status Active >>> >>> >>> >>> >>> >>> >>> >>> *From:* Imesh Gunaratne [mailto:im...@apache.org] >>> *Sent:* 14 May 2015 20:34 >>> >>> >>> *To:* dev >>> *Subject:* Re: Clustered deployments of Stratos >>> >>> >>> >>> It would be better to use the REST API to query and see whether the >>> relevant entities are persisted. Since data is stored in binary format in >>> the registry it would be difficult to query the database and verify this. >>> >>> >>> >>> On Thu, May 14, 2015 at 10:47 PM, Shaheedur Haque (shahhaqu) < >>> shahh...@cisco.com> wrote: >>> >>> I looked at REG_RESOURCEs a9s well as a few others) but I’m afraid I am >>> going to need more specifics. >>> >>> >>> >>> For example, what query would you recommend to look at say deployment >>> policies and cartridge definitions? >>> >>> >>> >>> *From:* Imesh Gunaratne [mailto:im...@apache.org] >>> *Sent:* 09 May 2015 09:08 >>> >>> >>> *To:* dev >>> *Subject:* Re: Clustered deployments of Stratos >>> >>> >>> >>> Yes you could refer the tables that have the prefix "REG_". >>> >>> >>> >>> On Sat, May 9, 2015 at 4:11 AM, Shaheedur Haque (shahhaqu) < >>> shahh...@cisco.com> wrote: >>> >>> Can you suggest what tables to look at? >>> >>> >>> >>> *From:* Imesh Gunaratne [mailto:im...@apache.org] >>> *Sent:* 07 May 2015 18:00 >>> >>> >>> *To:* dev >>> *Subject:* Re: Clustered deployments of Stratos >>> >>> >>> >>> Hi Shaheed, >>> >>> >>> >>> Thanks for the clarification! May be the problem is with the MySQL >>> active-passive configuration. >>> >>> >>> >>> I understand that you are switching the same OpenStack volume from >>> active node to the passive node (when the passive node becomes active) >>> therefore technically it should work. May be we need to investigate this >>> problem further by analysing whether data is persisted properly in the >>> active node before the passive node becomes active. >>> >>> >>> >>> Thanks >>> >>> >>> >>> On Tue, May 5, 2015 at 4:22 PM, Shaheedur Haque (shahhaqu) < >>> shahh...@cisco.com> wrote: >>> >>> The data is not synchronised between the active and passive nodes. For >>> clarity, this is the HA model we had, much as described in the blog: >>> >>> >>> >>> · 2 nodes, with Pacemaker in active-passive mode. >>> >>> · Under Pacemaker control: >>> >>> o We run MySQL in active-passive mode, using a single OpenStack >>> volume which we attach/reattach as the active role moves around nodes. >>> >>> o As the Pacemaker moves the volume, and thus MySQL around on node >>> failures, ActiveMQ and Stratos are moved around too. >>> >>> o Thus, everything operates in active-passive mode. >>> >>> >>> >>> Even in this model, as the active Stratos 4.0 is moved around (i.e. the >>> Stratos JVM on the old active node has gone with the node, and Pacemaker >>> starts up a new Stratos JVM on what used to be the passive node), we found >>> that the Cartridge Definition objects were found to be missing and, as a >>> clumsy workaround [1], we had to replay the stored copied of them into >>> Stratos using the REST API. >>> >>> >>> >>> With Stratos 4.1, using the new object names , early indications are >>> *Deployment >>> Policies* and *Application Deployment* policies are lost as the active >>> fails over to the passive. If anything, these objects are more likely to >>> hit the problems of [1], since Stratos 4.1 expects these to be tweaked on >>> the fly (min/max etc). >>> >>> >>> >>> Thanks, Shaheed >>> >>> >>> >>> [1] Clearly, this loses any changes that were not in the stored copies. >>> >>> >>> >>> *From:* Imesh Gunaratne [mailto:im...@apache.org] >>> *Sent:* 03 May 2015 06:43 >>> *To:* dev@stratos.apache.org >>> >>> >>> *Subject:* Re: Clustered deployments of Stratos >>> >>> >>> >>> Hi Shaheed, >>> >>> >>> >>> Thanks for taking time to test this! >>> >>> >>> >>> Just to clarify the exact problem, do you mean that data is not >>> synchronized between the active and passive nodes or they are not persisted >>> in the active node? >>> >>> >>> >>> Thanks >>> >>> >>> On Sunday, May 3, 2015, Shaheedur Haque (shahhaqu) <shahh...@cisco.com> >>> wrote: >>> >>> >>> I have been looking into our use of Linux HA to setup an Active-Passive >>> configuration. Testing indicates that in 4.1 (beta1), several objects seem >>> not to be persisted properly. This includes at least: >>> >>> - Cartridges >>> - Deployment policies >>> >>> Am I missing something? Is it safe to workaround this by replaying those >>> objects? >>> ------------------------------ >>> >>> *From:* Imesh Gunaratne [im...@apache.org] >>> *Sent:* 23 April 2015 10:47 >>> *To:* dev >>> *Subject:* Re: Clustered deployments of Stratos >>> >>> Hi Shaheed, >>> >>> >>> >>> Currently N-way clustering is still not possible with CC, AS & SM. We >>> completed the initial phase of this feature however it was not completed. >>> You could refer mail thread "[Discuss] Clustering Feature Implementation >>> for 4.1.0-Alpha Release" for details. >>> >>> >>> >>> However at present [1] is valid. We could use Linux HA and deploy CC, AS >>> and SM in Active-Passive mode. >>> >>> >>> >>> Thanks >>> >>> >>> >>> >>> >>> >>> >>> On Thu, Apr 23, 2015 at 2:41 PM, Shaheedur Haque (shahhaqu) < >>> shahh...@cisco.com> wrote: >>> >>> Hi, >>> >>> >>> >>> We currently try to achieve HA with Stratos using something so >>> unpleasant that I won’t even describe it here J. It has been suggested >>> that Stratos has, for a while now, supported a clustered mode of deployment >>> where, given N servers: >>> >>> >>> >>> · The SM, AS and MB operate in a N-way clustered mode >>> >>> · The CEP operates in a N-way loadsharing mode >>> >>> · The Cartridge Agents can react to a failure in one of the N >>> CEPs by failing over to one of the other N-1 remaining servers >>> >>> >>> >>> In looking for documentation on how to set this up, I came across these >>> two write-ups [1] and [2]. Questions: >>> >>> >>> >>> · Both these documents mention only using N=2. Is that still >>> correct? >>> >>> · [1] Seems recently written, and [2] is a little older but not >>> much. Are both documents still regarded as current? >>> >>> >>> >>> Also, I’d love to hear any other experiences people have of running >>> configurations like this. >>> >>> >>> >>> Thanks, Shaheed >>> >>> >>> >>> [1] >>> https://cwiki.apache.org/confluence/display/STRATOS/4.1.0+Configuring+HA+Using+Pacemaker+and+Heartbeat >>> >>> [2] http://blog.lasindu.com/2014/08/wso2-private-paas-supporting.html >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> -- >>> >>> Imesh Gunaratne >>> >>> >>> >>> Technical Lead, WSO2 >>> >>> Committer & PMC Member, Apache Stratos >>> >>> >>> >>> -- >>> >>> Imesh Gunaratne >>> >>> >>> >>> Senior Technical Lead, WSO2 >>> >>> Committer & PMC Member, Apache Stratos >>> >>> >>> >>> >>> >>> >>> >>> -- >>> >>> Imesh Gunaratne >>> >>> >>> >>> Senior Technical Lead, WSO2 >>> >>> Committer & PMC Member, Apache Stratos >>> >>> >>> >>> >>> >>> -- >>> >>> Imesh Gunaratne >>> >>> >>> >>> Senior Technical Lead, WSO2 >>> >>> Committer & PMC Member, Apache Stratos >>> >>> >>> >>> >>> >>> -- >>> >>> Imesh Gunaratne >>> >>> >>> >>> Senior Technical Lead, WSO2 >>> >>> Committer & PMC Member, Apache Stratos >>> >>> >>> >>> >>> >>> -- >>> >>> Imesh Gunaratne >>> >>> >>> >>> Senior Technical Lead, WSO2 >>> >>> Committer & PMC Member, Apache Stratos >>> >>> >>> >>> >>> >>> -- >>> >>> Imesh Gunaratne >>> >>> >>> >>> Senior Technical Lead, WSO2 >>> >>> Committer & PMC Member, Apache Stratos >>> >>> >>> >>> >>> >>> -- >>> >>> -- >>> Lahiru Sandaruwan >>> >>> Committer and PMC member, Apache Stratos, >>> Senior Software Engineer, >>> WSO2 Inc., http://wso2.com >>> >>> lean.enterprise.middleware >>> >>> phone: +94773325954 >>> email: lahi...@wso2.com blog: http://lahiruwrites.blogspot.com/ >>> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146 >>> >>> >>> >> >> >> >> -- >> Imesh Gunaratne >> >> Senior Technical Lead, WSO2 >> Committer & PMC Member, Apache Stratos >> > > > > -- > Reka Thirunavukkarasu > Senior Software Engineer, > WSO2, Inc.:http://wso2.com, > Mobile: +94776442007 > > > -- Reka Thirunavukkarasu Senior Software Engineer, WSO2, Inc.:http://wso2.com, Mobile: +94776442007
