Hi Shaheed, Thanks for the quick response, after analyzing the results you have provided again, it looks like only the deployment policies are missing after the failover. We have fixed this issue in commit revision: 0c515aa013850575ddcfa2e299da5f0ec250ebc3
http://mail-archives.apache.org/mod_mbox/incubator-stratos-commits/201504.mbox/%3c22eed4e8639c401a8fda637fa6bb4...@git.apache.org%3E Would you mind verifying whether this is there in your runtime? Thanks On Fri, May 15, 2015 at 9:02 PM, Shaheedur Haque (shahhaqu) < shahh...@cisco.com> wrote: > The latter; we never have both Stratos instances running. > > > > *From:* Imesh Gunaratne [mailto:im...@apache.org] > *Sent:* 15 May 2015 16:17 > *To:* dev > *Cc:* Ryan Du Plessis (rdupless); Luca Martini (lmartini) > > *Subject:* Re: Clustered deployments of Stratos > > > > Hi Shaheed, > > > > Do you have both active and passive Stratos nodes running at the same time > or do you start the passive node once the active node goes down? > > > > Thanks > > > > On Fri, May 15, 2015 at 6:31 PM, Shaheedur Haque (shahhaqu) < > shahh...@cisco.com> wrote: > > Hi Imesh, > > > > I finally got round to a proper series of tests, and here are the > conclusions: > > > > · In Stratos 4.0, after a Pacemaker driven failover, the newly > Active Stratos has lost all Cartridge Definitions. > > · In current [1] Stratos 4.1, after a Pacemaker driven failover, > the newly Active Stratos: > > o Has lost all Deployment Policies. > > o Has lost contact with the Cartridge Agents, and all VMs are stuck > with whatever state they had before the failover. > > · Note: I have not verified if Cartridge Groups are lost or not. > > > > I include the test results below at [2] and [3]. I am concerned as to > whether 4.1 is ready for GA on this basis, so though more testing is no > doubt possible (e.g. Cartridge Groups) I wanted to get this info to the > list ASAP. > > > > Thanks, Shaheed > > > > [1] A recent build somewhere between beta 1 and beta 2, but I don’t think > any relevant fixes have been made in master. > > > > [2] Persistence test output from Stratos 4.1. Note: > > > > 1. In the build I have, the CLI is broken for a couple of commands; > these are supplemented by direct “curl” commands further down. > > 2. I’ve used one of our commands to show the instances and their > state for a given application since there is not a compact JSON or > convenient Startos CLI for that. > > > > *PERSISTENCE TEST, BEFORE FAILOVER* > > *================================* > > > > stratos> list-tenants > > Tenants: > > > +-----------------------+-----------+------------------+--------+------------------------------+ > > | Domain | Tenant ID | Email | State | Created > Date | > > > +-----------------------+-----------+------------------+--------+------------------------------+ > > | cloud1.qmog.cisco.com | 1 | clo...@cisco.com | Active | Fri May > 15 04:46:58 MDT 2015 | > > > +-----------------------+-----------+------------------+--------+------------------------------+ > > > > stratos> list-network-partitions > > Network partitions found: > > +----------------------+----------------------+ > > | Network Partition ID | Number of Partitions | > > +----------------------+----------------------+ > > | RegionOne | 1 | > > +----------------------+----------------------+ > > > > stratos> list-deployment-policies > > Deployment policies found: > > +-------------------+---------------+ > > | ID | Accessibility | > > +-------------------+---------------+ > > | static-2-ha | 1 | > > +-------------------+---------------+ > > | autoscale-2-10-ha | 1 | > > +-------------------+---------------+ > > | autoscale-1-5 | 1 | > > +-------------------+---------------+ > > | static-1 | 1 | > > +-------------------+---------------+ > > > > stratos> list-application-policies > > Error in listing application policies > > No application policies found > > > > stratos> list-autoscaling-policies > > Error in listing autoscaling policies > > No autoscaling policies found > > > > stratos> list-cartridges > > Cartridges found: > > > +------------------+-------------+------------------+----------------------------+---------+--------------+ > > | Type | Category | Name | > Description | Version | Multi-Tenant | > > > +------------------+-------------+------------------+----------------------------+---------+--------------+ > > | cartridge-proxy | Application | cartridge-proxy | cartridge-proxy > Cartridge | 1 | false | > > > +------------------+-------------+------------------+----------------------------+---------+--------------+ > > | cisco-sample-vm | Application | cisco-sample-vm | cisco-sample-vm > Cartridge | 1 | false | > > > +------------------+-------------+------------------+----------------------------+---------+--------------+ > > | cisco-qvpc-cf-01 | Application | cisco-qvpc-cf-01 | cisco-qvpc-cf-01 > Cartridge | 1 | false | > > > +------------------+-------------+------------------+----------------------------+---------+--------------+ > > | cisco-qvpc-cf-02 | Application | cisco-qvpc-cf-02 | cisco-qvpc-cf-02 > Cartridge | 1 | false | > > > +------------------+-------------+------------------+----------------------------+---------+--------------+ > > | cisco-qvpc-si | Application | cisco-qvpc-si | cisco-qvpc-si > Cartridge | 1 | false | > > > +------------------+-------------+------------------+----------------------------+---------+--------------+ > > | cisco-qvpc-sf | Application | cisco-qvpc-sf | cisco-qvpc-sf > Cartridge | 1 | false | > > > +------------------+-------------+------------------+----------------------------+---------+--------------+ > > > > stratos> list-applications > > Applications found: > > +-----------------+-----------------+----------+ > > | Application ID | Alias | Status | > > +-----------------+-----------------+----------+ > > | cartridge-proxy | cartridge-proxy | Deployed | > > +-----------------+-----------------+----------+ > > | cisco-sample-vm | cisco-sample-vm | Deployed | > > +-----------------+-----------------+----------+ > > > > $ curl -uadmin:admin -k -H'Content-type: application/json' > https://localhost:9443/api/autoscalingPolicies > > > [{"id":"economyPolicy","instanceRoundingFactor":0,"isPublic":false,"loadThresholds":""}] > > > > $ curl -uadmin:admin -k -H'Content-type: application/json' > https://localhost:9443/api/applicationPolicies > > > [{"algorithm":"one-after-another","id":"default-iaas","networkPartitions":["RegionOne"],"properties":{"name":"networkPartitionGroups","value":"RegionOne"}}] > > > > > > *PERSISTENCE TEST, AFTER FAILOVER* > > *===============================* > > > > stratos> list-tenants > > Tenants: > > > +-----------------------+-----------+------------------+--------+------------------------------+ > > | Domain | Tenant ID | Email | State | Created > Date | > > > +-----------------------+-----------+------------------+--------+------------------------------+ > > | cloud1.qmog.cisco.com | 1 | clo...@cisco.com | Active | Fri May > 15 05:26:52 MDT 2015 | > > > +-----------------------+-----------+------------------+--------+------------------------------+ > > > > stratos> list-network-partitions > > Network partitions found: > > +----------------------+----------------------+ > > | Network Partition ID | Number of Partitions | > > +----------------------+----------------------+ > > | RegionOne | 1 | > > +----------------------+----------------------+ > > > > stratos> list-deployment-policies > > No deployment policies found > > > > stratos> list-application-policies > > Error in listing application policies > > No application policies found > > > > stratos> list-autoscaling-policies > > Error in listing autoscaling policies > > No autoscaling policies found > > > > stratos> list-cartridges > > Cartridges found: > > > +------------------+-------------+------------------+----------------------------+---------+--------------+ > > | Type | Category | Name | > Description | Version | Multi-Tenant | > > > +------------------+-------------+------------------+----------------------------+---------+--------------+ > > | cartridge-proxy | Application | cartridge-proxy | cartridge-proxy > Cartridge | 1 | false | > > > +------------------+-------------+------------------+----------------------------+---------+--------------+ > > | cisco-sample-vm | Application | cisco-sample-vm | cisco-sample-vm > Cartridge | 1 | false | > > > +------------------+-------------+------------------+----------------------------+---------+--------------+ > > | cisco-qvpc-cf-01 | Application | cisco-qvpc-cf-01 | cisco-qvpc-cf-01 > Cartridge | 1 | false | > > > +------------------+-------------+------------------+----------------------------+---------+--------------+ > > | cisco-qvpc-cf-02 | Application | cisco-qvpc-cf-02 | cisco-qvpc-cf-02 > Cartridge | 1 | false | > > > +------------------+-------------+------------------+----------------------------+---------+--------------+ > > | cisco-qvpc-si | Application | cisco-qvpc-si | cisco-qvpc-si > Cartridge | 1 | false | > > > +------------------+-------------+------------------+----------------------------+---------+--------------+ > > | cisco-qvpc-sf | Application | cisco-qvpc-sf | cisco-qvpc-sf > Cartridge | 1 | false | > > > +------------------+-------------+------------------+----------------------------+---------+--------------+ > > > > stratos> list-applications > > Applications found: > > +-----------------+-----------------+----------+ > > | Application ID | Alias | Status | > > +-----------------+-----------------+----------+ > > | cartridge-proxy | cartridge-proxy | Deployed | > > +-----------------+-----------------+----------+ > > | cisco-sample-vm | cisco-sample-vm | Deployed | > > +-----------------+-----------------+----------+ > > > > $ curl -uadmin:admin -k -H'Content-type: application/json' > https://localhost:9443/api/autoscalingPolicies > > > [{"id":"economyPolicy","instanceRoundingFactor":0,"isPublic":false,"loadThresholds":""}] > > > > $ curl -uadmin:admin -k -H'Content-type: application/json' > https://localhost:9443/api/applicationPolicies > > > [{"algorithm":"one-after-another","id":"default-iaas","networkPartitions":["RegionOne"],"properties":{"name":"networkPartitionGroups","value":"RegionOne"}}] > > > > [3] Cartridge test output from Stratos 4.1. Note: > > > > 1. We do not use a VIP for Stratos, either for 4.0 or 4.1. > > 2. We expect the Cartridge Agent to use a DNS lookup when it ends up > reconnecting, and this worked just fine in Stratos 4.0. > > > > *CARTRIDGE TEST, BEFORE FAILOVER* > > *==============================* > > > > $ ./bin/orchestration subscription list-instances --admin cisco-sample-vm > > cisco-sample-vm: applicationInstances 1, groupInstances 0, > clusterInstances 1, members 1 (Active 1) > > cisco-sample-vm: 172.16.180.30/10.0.0.101: status Active > > > > *CARTRIDGE TEST, AFTER FAILOVER* > > *=============================* > > > > $ ./bin/orchestration subscription list-instances --admin cisco-sample-vm > > cisco-sample-vm: applicationInstances 1, groupInstances 0, > clusterInstances 1, members 1 (Active 1) > > cisco-sample-vm: 172.16.180.30/10.0.0.101: status Active > > > > *CARTRIDGE TEST, AFTER FAILOVER WAIT 5 MINUTES, THEN KILL INSTANCE, THEN > WAIT 2 MINUTES* > > > *===================================================================================* > > > > $ ./bin/orchestration subscription list-instances --admin cisco-sample-vm > > cisco-sample-vm: applicationInstances 1, groupInstances 0, > clusterInstances 1, members 1 (Active 1) > > cisco-sample-vm: 172.16.180.30/10.0.0.101: status Active > > > > > > > > *From:* Imesh Gunaratne [mailto:im...@apache.org] > *Sent:* 14 May 2015 20:34 > > > *To:* dev > *Subject:* Re: Clustered deployments of Stratos > > > > It would be better to use the REST API to query and see whether the > relevant entities are persisted. Since data is stored in binary format in > the registry it would be difficult to query the database and verify this. > > > > On Thu, May 14, 2015 at 10:47 PM, Shaheedur Haque (shahhaqu) < > shahh...@cisco.com> wrote: > > I looked at REG_RESOURCEs a9s well as a few others) but I’m afraid I am > going to need more specifics. > > > > For example, what query would you recommend to look at say deployment > policies and cartridge definitions? > > > > *From:* Imesh Gunaratne [mailto:im...@apache.org] > *Sent:* 09 May 2015 09:08 > > > *To:* dev > *Subject:* Re: Clustered deployments of Stratos > > > > Yes you could refer the tables that have the prefix "REG_". > > > > On Sat, May 9, 2015 at 4:11 AM, Shaheedur Haque (shahhaqu) < > shahh...@cisco.com> wrote: > > Can you suggest what tables to look at? > > > > *From:* Imesh Gunaratne [mailto:im...@apache.org] > *Sent:* 07 May 2015 18:00 > > > *To:* dev > *Subject:* Re: Clustered deployments of Stratos > > > > Hi Shaheed, > > > > Thanks for the clarification! May be the problem is with the MySQL > active-passive configuration. > > > > I understand that you are switching the same OpenStack volume from active > node to the passive node (when the passive node becomes active) therefore > technically it should work. May be we need to investigate this problem > further by analysing whether data is persisted properly in the active node > before the passive node becomes active. > > > > Thanks > > > > On Tue, May 5, 2015 at 4:22 PM, Shaheedur Haque (shahhaqu) < > shahh...@cisco.com> wrote: > > The data is not synchronised between the active and passive nodes. For > clarity, this is the HA model we had, much as described in the blog: > > > > · 2 nodes, with Pacemaker in active-passive mode. > > · Under Pacemaker control: > > o We run MySQL in active-passive mode, using a single OpenStack volume > which we attach/reattach as the active role moves around nodes. > > o As the Pacemaker moves the volume, and thus MySQL around on node > failures, ActiveMQ and Stratos are moved around too. > > o Thus, everything operates in active-passive mode. > > > > Even in this model, as the active Stratos 4.0 is moved around (i.e. the > Stratos JVM on the old active node has gone with the node, and Pacemaker > starts up a new Stratos JVM on what used to be the passive node), we found > that the Cartridge Definition objects were found to be missing and, as a > clumsy workaround [1], we had to replay the stored copied of them into > Stratos using the REST API. > > > > With Stratos 4.1, using the new object names , early indications are > *Deployment > Policies* and *Application Deployment* policies are lost as the active > fails over to the passive. If anything, these objects are more likely to > hit the problems of [1], since Stratos 4.1 expects these to be tweaked on > the fly (min/max etc). > > > > Thanks, Shaheed > > > > [1] Clearly, this loses any changes that were not in the stored copies. > > > > *From:* Imesh Gunaratne [mailto:im...@apache.org] > *Sent:* 03 May 2015 06:43 > *To:* dev@stratos.apache.org > > > *Subject:* Re: Clustered deployments of Stratos > > > > Hi Shaheed, > > > > Thanks for taking time to test this! > > > > Just to clarify the exact problem, do you mean that data is not > synchronized between the active and passive nodes or they are not persisted > in the active node? > > > > Thanks > > > On Sunday, May 3, 2015, Shaheedur Haque (shahhaqu) <shahh...@cisco.com> > wrote: > > > I have been looking into our use of Linux HA to setup an Active-Passive > configuration. Testing indicates that in 4.1 (beta1), several objects seem > not to be persisted properly. This includes at least: > > - Cartridges > - Deployment policies > > Am I missing something? Is it safe to workaround this by replaying those > objects? > ------------------------------ > > *From:* Imesh Gunaratne [im...@apache.org] > *Sent:* 23 April 2015 10:47 > *To:* dev > *Subject:* Re: Clustered deployments of Stratos > > Hi Shaheed, > > > > Currently N-way clustering is still not possible with CC, AS & SM. We > completed the initial phase of this feature however it was not completed. > You could refer mail thread "[Discuss] Clustering Feature Implementation > for 4.1.0-Alpha Release" for details. > > > > However at present [1] is valid. We could use Linux HA and deploy CC, AS > and SM in Active-Passive mode. > > > > Thanks > > > > > > > > On Thu, Apr 23, 2015 at 2:41 PM, Shaheedur Haque (shahhaqu) < > shahh...@cisco.com> wrote: > > Hi, > > > > We currently try to achieve HA with Stratos using something so unpleasant > that I won’t even describe it here J. It has been suggested that Stratos > has, for a while now, supported a clustered mode of deployment where, given > N servers: > > > > · The SM, AS and MB operate in a N-way clustered mode > > · The CEP operates in a N-way loadsharing mode > > · The Cartridge Agents can react to a failure in one of the N CEPs > by failing over to one of the other N-1 remaining servers > > > > In looking for documentation on how to set this up, I came across these > two write-ups [1] and [2]. Questions: > > > > · Both these documents mention only using N=2. Is that still > correct? > > · [1] Seems recently written, and [2] is a little older but not > much. Are both documents still regarded as current? > > > > Also, I’d love to hear any other experiences people have of running > configurations like this. > > > > Thanks, Shaheed > > > > [1] > https://cwiki.apache.org/confluence/display/STRATOS/4.1.0+Configuring+HA+Using+Pacemaker+and+Heartbeat > > [2] http://blog.lasindu.com/2014/08/wso2-private-paas-supporting.html > > > > > > > > > > > > -- > > Imesh Gunaratne > > > > Technical Lead, WSO2 > > Committer & PMC Member, Apache Stratos > > > > -- > > Imesh Gunaratne > > > > Senior Technical Lead, WSO2 > > Committer & PMC Member, Apache Stratos > > > > > > > > -- > > Imesh Gunaratne > > > > Senior Technical Lead, WSO2 > > Committer & PMC Member, Apache Stratos > > > > > > -- > > Imesh Gunaratne > > > > Senior Technical Lead, WSO2 > > Committer & PMC Member, Apache Stratos > > > > > > -- > > Imesh Gunaratne > > > > Senior Technical Lead, WSO2 > > Committer & PMC Member, Apache Stratos > > > > > > -- > > Imesh Gunaratne > > > > Senior Technical Lead, WSO2 > > Committer & PMC Member, Apache Stratos > -- Imesh Gunaratne Senior Technical Lead, WSO2 Committer & PMC Member, Apache Stratos
