Hi Isuru, I did a fix to send the payload separately to place we call jclouds, without setting it to iaasProvider early.
Thanks. On Thu, Apr 2, 2015 at 9:37 AM, Isuru Haththotuwa <isu...@apache.org> wrote: > Hi Lahiru, > > On Thu, Apr 2, 2015 at 9:11 AM, Lahiru Sandaruwan <lahi...@wso2.com> > wrote: > >> Thanks Martin for the logs. So the reason is clear why happens this. We >> keep the payload par cartridge type in iaas provider object, in cloud >> controller context. >> >> When second member approaches the instance started method, it rewrites >> the payload in iaasProvider object. But the first instance is still not >> really created in iaas, and it runs in a different thread. Therefore the >> payload belong to second instance, is passed to second instance as well as >> the first instance. >> > Nice work finding this! So the resolution would be to change the cartridge > type -> payload mapping to alias -> payload? > >> >> I will fix it. >> >> Thanks. >> >> On Wed, Apr 1, 2015 at 11:38 PM, Martin Eppel (meppel) <mep...@cisco.com> >> wrote: >> >>> Hi Lahiru, >>> >>> >>> >>> Attached are the logs, >>> >>> >>> >>> Thanks >>> >>> >>> >>> Martin >>> >>> >>> >>> *From:* Lahiru Sandaruwan [mailto:lahi...@wso2.com] >>> *Sent:* Wednesday, April 01, 2015 4:11 AM >>> >>> *To:* dev >>> *Subject:* Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not >>> going active ... was RE: Stratos 4.1.0 : Issue when deploying application >>> with single group and 3 cartridges, no dependencies ... >>> >>> >>> >>> HI Martin, >>> >>> >>> >>> Could you please send the debug logs enabling debug on following >>> classes? I have improved few debug logs to track payload, therefore please >>> take a pull before reproducing with debug logs. >>> >>> >>> log4j.logger.org.apache.stratos.autoscaler.applications.parser.DefaultApplicationParser=DEBUG >>> >>> >>> log4j.logger.org.apache.stratos.cloud.controller.services.impl.InstanceCreator=DEBUG >>> >>> >>> log4j.logger.org.apache.stratos.cloud.controller.services.impl.CloudControllerServiceImpl=DEBUG >>> >>> Thanks. >>> >>> >>> >>> >>> >>> On Wed, Apr 1, 2015 at 10:50 AM, Lahiru Sandaruwan <lahi...@wso2.com> >>> wrote: >>> >>> I will have a look Martin. >>> >>> >>> >>> Thanks. >>> >>> >>> >>> On Wed, Apr 1, 2015 at 6:54 AM, Martin Eppel (meppel) <mep...@cisco.com> >>> wrote: >>> >>> Hi Lahiru, >>> >>> >>> >>> I retested the scenario to make sure proper unique aliases are provided >>> and they are, please double check the provided artifacts. Issue remains the >>> same, one VM gets stuck in INITIALIZED until terminated and restarted. As >>> pointed out before, the cartridge alias in the cartridge agent is the same >>> in 2 cartridges (although provided aliases in the application json is >>> unique). After restarting the faulty member cartridge agent alias has the >>> proper alias and goes into active. >>> >>> I also provided the cartridge-agent logs for the member which is faulty, >>> before restart and after restart (cartridge-agent-70~INITIALIZED, ACTIVE) >>> >>> >>> >>> Please see attached logs, artifacts, >>> >>> >>> >>> Thanks >>> >>> >>> >>> Martin >>> >>> >>> >>> *From:* Martin Eppel (meppel) >>> *Sent:* Tuesday, March 31, 2015 1:22 PM >>> >>> >>> *To:* dev@stratos.apache.org >>> *Subject:* RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not >>> going active ... was RE: Stratos 4.1.0 : Issue when deploying application >>> with single group and 3 cartridges, no dependencies ... >>> >>> >>> >>> Lahiru, >>> >>> >>> >>> The aliases are all unique, please check the the application json, and >>> as I said in my analysis it I think this is what I saw but I can’t be sure. >>> So this is not an issue of starting the application with duplicate aliases >>> but rather something else , >>> >>> >>> >>> Thanks >>> >>> >>> >>> Martin >>> >>> >>> >>> *From:* Lahiru Sandaruwan [mailto:lahi...@wso2.com <lahi...@wso2.com>] >>> *Sent:* Tuesday, March 31, 2015 12:58 PM >>> *To:* dev >>> *Subject:* Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not >>> going active ... was RE: Stratos 4.1.0 : Issue when deploying application >>> with single group and 3 cartridges, no dependencies ... >>> >>> >>> >>> Hi Martin, >>> >>> >>> >>> On Wed, Apr 1, 2015 at 12:34 AM, Martin Eppel (meppel) <mep...@cisco.com> >>> wrote: >>> >>> Hi Lahiru, >>> >>> >>> >>> I tested the fix and noticed the following issue: >>> >>> >>> >>> From the 4 cartridges 3 instances go Active, however one instance got >>> stuck in INITIALIZED state. >>> >>> >>> >>> One of the things I noticed while the member was stuck in INITIALIZED >>> state is that the system received 2 active events for the same membered: >>> >>> >>> >>> *TID: [0] [STRATOS] [2015-03-31 18:10:27,645] INFO >>> {org.apache.stratos.cloud.controller.messaging.publisher.TopologyEventPublisher} >>> - Publishing member activated event: [service-name] c2 [cluster-id] >>> subscription-G1-G2-G3.c2-1x1.c2.domain [cluster-instance-id] >>> subscription-G1-G2-G3-1 [member-id] >>> subscription-G1-G2-G3.c2-1x1.c2.domainf939e8dc-e0cc-408a-8343-ac86e479549f >>> [network-partition-id] RegionOne [partition-id] whole-region* >>> >>> *TID: [0] [STRATOS] [2015-03-31 18:10:31,059] INFO >>> {org.apache.stratos.cloud.controller.messaging.publisher.TopologyEventPublisher} >>> - Publishing member activated event: [service-name] c2 [cluster-id] >>> subscription-G1-G2-G3.c2-1x1.c2.domain [cluster-instance-id] >>> subscription-G1-G2-G3-1 [member-id] >>> subscription-G1-G2-G3.c2-1x1.c2.domainf939e8dc-e0cc-408a-8343-ac86e479549f >>> [network-partition-id] RegionOne [partition-id] whole-region* >>> >>> >>> >>> As another note, but I can’t be sure since the log got overwritten is >>> that it appeared to me that at one point (while the cartridge was stuck in >>> INITIALIZED state is that the cartridge alias of 2 cartridges was the same >>> (~71, ~70: 71: CARTRIDGE_ALIAS=c2-1x1), but I can’t say for sure since when >>> I double checked a new instance of the cartridge was created (which went >>> ACTIVE). >>> >>> >>> >>> >>> >>> Can i assume this occurred when two cartridges use same alias? May be we >>> have to put a validation on Application as we cannot use same alias in two >>> cartridges. >>> >>> >>> >>> If so, could you please try to reproduce this with different aliases? >>> >>> >>> >>> Thanks. >>> >>> >>> >>> >>> >>> >>> >>> Eventually the system terminated the “faulty” member and the cartridge >>> came up and went into active state: >>> >>> >>> >>> *“Pending state of member expired, member will be moved to obsolete >>> list. [pending member] >>> subscription-G1-G2-G3.c2-1x0.c2.domainfe9a30cf-fffc-4a6c-9b3b-f419498895b9* >>> ” >>> >>> … >>> >>> “*Member activated: [service] c2 [cluster] >>> subscription-G1-G2-G3.c2-1x0.c2.domain [member] >>> subscription-G1-G2-G3.c2-1x0.c2.domain44a2ae3c-9968-41de-9e10-7212ef3788c2* >>> ” >>> >>> >>> >>> Cartridge alias after all 4 cartridges went ACTIVE: >>> >>> >>> >>> 70: CARTRIDGE_ALIAS=c2-1x0 >>> >>> 71: CARTRIDGE_ALIAS=c2-1x1 >>> >>> >>> >>> 69: CARTRIDGE_ALIAS=c1-1x0 >>> >>> 68: CARTRIDGE_ALIAS=c1-1x1 >>> >>> >>> >>> I attached the logs and artefacts >>> >>> >>> >>> Thanks >>> >>> >>> >>> Martin >>> >>> >>> >>> *From:* Lahiru Sandaruwan [mailto:lahi...@wso2.com] >>> *Sent:* Tuesday, March 31, 2015 2:24 AM >>> >>> >>> *To:* dev >>> *Subject:* Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not >>> going active ... was RE: Stratos 4.1.0 : Issue when deploying application >>> with single group and 3 cartridges, no dependencies ... >>> >>> >>> >>> Hi Martin, >>> >>> >>> >>> I have reproduced the issue and pushed a fix. Please verify. >>> >>> >>> >>> Thanks. >>> >>> >>> >>> On Tue, Mar 31, 2015 at 8:32 AM, Lahiru Sandaruwan <lahi...@wso2.com> >>> wrote: >>> >>> Hi Martin, >>> >>> >>> >>> On Tue, Mar 31, 2015 at 5:41 AM, Martin Eppel (meppel) <mep...@cisco.com> >>> wrote: >>> >>> Hi Lahiru, >>> >>> >>> >>> I tested a new scenario, where I have 2 groups with cartridges and no >>> start up dependencies and I see again the issue with cartridge instances >>> being stuck in “started” state. I also noticed the following log message: >>> >>> *“Artifact repositories not found for application signup, artifact >>> updated event not sent: [application-id] subscription-G1-G2-G3*”. >>> >>> >>> >>> >>> >>> Will have a look. >>> >>> >>> >>> Thanks. >>> >>> Please find attached the artefacts and log messages, >>> >>> >>> >>> Thanks >>> >>> >>> >>> Martin >>> >>> >>> >>> *From:* Martin Eppel (meppel) >>> *Sent:* Monday, March 30, 2015 2:36 PM >>> >>> >>> *To:* dev@stratos.apache.org >>> *Subject:* RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not >>> going active ... was RE: Stratos 4.1.0 : Issue when deploying application >>> with single group and 3 cartridges, no dependencies ... >>> >>> >>> >>> Hi Lahiru, >>> >>> >>> >>> I tested both scenarios, >>> >>> >>> >>> 1. Single cartridge in single group and it worked, cartridge >>> instance starts up and goes active, >>> >>> 2. Multiple cartridges with startup dependencies : >>> >>> All instances are started up (in right sequence) and go active >>> >>> JIRA https://issues.apache.org/jira/browse/STRATOS-1292[image: >>> http://wikicentral.cisco.com/images/icons/linkext7.gif] >>> <https://issues.apache.org/jira/browse/STRATOS-1292> is being tested >>> and verified >>> >>> >>> >>> Removing / Un-deploying the application caused an issue as one of the >>> VMs did not get terminated (throwing exceptions in the log). I am not sure >>> if this is already a known issue related to to the ongoing discussion of >>> application removal / un-deployment. >>> >>> Just in case, I zipped up the logs and attached it to the email thread >>> (single-group-multiple-cartridges-startup.zip). Let me know if the >>> un-deployment issue is the same already discussed or if this is a new issue >>> ? >>> >>> >>> >>> Same error occurred in the scenario where multiple cartridges were >>> started up (all go active) and then application was removed, I attached the >>> log for this scenario as well >>> (wscarbon-application-single-group-multiple-cartridges-no-startup-remval-error.log.zip) >>> >>> >>> >>> >>> >>> Thanks >>> >>> >>> >>> Martin >>> >>> >>> >>> >>> >>> *From:* Martin Eppel (meppel) >>> *Sent:* Monday, March 30, 2015 7:47 AM >>> *To:* dev@stratos.apache.org >>> *Subject:* RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not >>> going active ... was RE: Stratos 4.1.0 : Issue when deploying application >>> with single group and 3 cartridges, no dependencies ... >>> >>> >>> >>> Thanks Lahiru, >>> >>> >>> >>> Will do and let you know >>> >>> >>> >>> Martin >>> >>> >>> >>> *From:* Lahiru Sandaruwan [mailto:lahi...@wso2.com <lahi...@wso2.com>] >>> *Sent:* Monday, March 30, 2015 3:32 AM >>> *To:* dev >>> *Subject:* Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not >>> going active ... was RE: Stratos 4.1.0 : Issue when deploying application >>> with single group and 3 cartridges, no dependencies ... >>> >>> >>> >>> Hi Martin, >>> >>> >>> >>> I was able to reproduce the issue and committed a fix. >>> >>> Could you please verify? >>> >>> >>> >>> Thanks. >>> >>> >>> >>> On Sun, Mar 29, 2015 at 5:43 PM, Lahiru Sandaruwan <lahi...@wso2.com> >>> wrote: >>> >>> Hi Martin, >>> >>> >>> >>> According to the logs attached, it seems the cartridge agent waits for >>> the artifact updated event, and somehow it is not sent. >>> >>> >>> >>> Looking into it. >>> >>> >>> >>> Thanks. >>> >>> >>> >>> On Sat, Mar 28, 2015 at 6:48 AM, Imesh Gunaratne <im...@apache.org> >>> wrote: >>> >>> Hi Martin, >>> >>> >>> >>> Thanks for the quick response on this! >>> >>> I will use these artifacts to re-produce the problem locally and will >>> let you know the results soon. >>> >>> >>> >>> Thanks >>> >>> >>> >>> On Sat, Mar 28, 2015 at 4:19 AM, Martin Eppel (meppel) <mep...@cisco.com> >>> wrote: >>> >>> Hi Imesh, >>> >>> >>> >>> As discussed please find attached the “real” artefacts (application, >>> cartridge group, cartridge): >>> >>> >>> >>> Attached it as well to the jira >>> >>> >>> >>> Thanks >>> >>> >>> >>> Martin >>> >>> >>> >>> >>> >>> *From:* Martin Eppel (meppel) >>> *Sent:* Friday, March 27, 2015 11:37 AM >>> *To:* dev@stratos.apache.org >>> *Subject:* RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not >>> going active ... was RE: Stratos 4.1.0 : Issue when deploying application >>> with single group and 3 cartridges, no dependencies ... >>> >>> >>> >>> Hi Imesh, >>> >>> >>> >>> Originally I tested with 3 cartridges but then realized that it also >>> occurs with a single cartridge in a group – see also the JIRA for the >>> respective notes and artefacts (single group, single cartridge) >>> >>> >>> >>> Thanks >>> >>> >>> >>> Martin >>> >>> >>> >>> *From:* Imesh Gunaratne [mailto:im...@apache.org <im...@apache.org>] >>> *Sent:* Friday, March 27, 2015 11:34 AM >>> *To:* dev >>> *Subject:* Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not >>> going active ... was RE: Stratos 4.1.0 : Issue when deploying application >>> with single group and 3 cartridges, no dependencies ... >>> >>> >>> >>> Hi Martin, >>> >>> >>> >>> Thanks for reporting this problem. You have mentioned that there are >>> three cartridges in the application but in the given application.json I >>> only can see one cartridge. Are we missing anything here? Will it be >>> possible for you to attach the complete set of Stratos artifacts that you >>> are using in a zip file? >>> >>> >>> >>> Thanks >>> >>> Imesh >>> >>> >>> >>> On Fri, Mar 27, 2015 at 11:51 PM, Martin Eppel (meppel) < >>> mep...@cisco.com> wrote: >>> >>> Forgot to mention, this is the same issue as reported in JIRA >>> https://issues.apache.org/jira/browse/STRATOS-1292 >>> >>> >>> >>> Thanks >>> >>> >>> >>> Martin >>> >>> >>> >>> *From:* Martin Eppel (meppel) >>> *Sent:* Friday, March 27, 2015 10:52 AM >>> *To:* dev@stratos.apache.org >>> *Subject:* Issue (?): Stratos 4.1.0 - Cartridges in a group are not >>> going active ... was RE: Stratos 4.1.0 : Issue when deploying application >>> with single group and 3 cartridges, no dependencies ... >>> >>> >>> >>> I retested with a similar scenario (single cartridges in group) with >>> debug logs turned on, please see logs attached. >>> >>> >>> >>> Cartridge is stuck in “Starting” state >>> >>> >>> >>> Thanks >>> >>> >>> >>> Martin >>> >>> >>> >>> *From:* Martin Eppel (meppel) >>> *Sent:* Monday, March 23, 2015 7:15 PM >>> *To:* dev@stratos.apache.org >>> *Subject:* Stratos 4.1.0 : Issue when deploying application with single >>> group and 3 cartridges, no dependencies ... >>> >>> >>> >>> Hi, >>> >>> >>> >>> I see an issue when I subscribe a single group with 3 cartridges, no >>> dependencies – they never go active instead they all are stuck in >>> “Starting”. I attached the server log and the cartridge-agent logs >>> >>> >>> >>> >>> >>> Member status: >>> >>> subscription-c1-c2-c3: applicationInstances 1, groupInstances 1, >>> clusterInstances 3, members 3 (Starting 3) >>> >>> c2: 172.16.2.18/172.28.40.69: status Starting >>> >>> c3: 172.16.2.19/172.28.40.70: status Starting >>> >>> c1: 172.16.2.17/172.28.40.68: status Starting >>> >>> >>> >>> Let me know if I should go ahead and open a JIRA >>> >>> >>> >>> >>> >>> Thanks >>> >>> >>> >>> Martin >>> >>> >>> >>> >>> >>> -- >>> >>> Imesh Gunaratne >>> >>> >>> >>> Technical Lead, WSO2 >>> >>> Committer & PMC Member, Apache Stratos >>> >>> >>> >>> >>> >>> -- >>> >>> Imesh Gunaratne >>> >>> >>> >>> Technical Lead, WSO2 >>> >>> Committer & PMC Member, Apache Stratos >>> >>> >>> >>> >>> >>> -- >>> >>> -- >>> Lahiru Sandaruwan >>> >>> Committer and PMC member, Apache Stratos, >>> Senior Software Engineer, >>> WSO2 Inc., http://wso2.com >>> >>> lean.enterprise.middleware >>> >>> phone: +94773325954 >>> email: lahi...@wso2.com blog: http://lahiruwrites.blogspot.com/ >>> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146 >>> >>> >>> >>> >>> >>> >>> >>> -- >>> >>> -- >>> Lahiru Sandaruwan >>> >>> Committer and PMC member, Apache Stratos, >>> Senior Software Engineer, >>> WSO2 Inc., http://wso2.com >>> >>> lean.enterprise.middleware >>> >>> phone: +94773325954 >>> email: lahi...@wso2.com blog: http://lahiruwrites.blogspot.com/ >>> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146 >>> >>> >>> >>> >>> >>> >>> >>> -- >>> >>> -- >>> Lahiru Sandaruwan >>> >>> Committer and PMC member, Apache Stratos, >>> Senior Software Engineer, >>> WSO2 Inc., http://wso2.com >>> >>> lean.enterprise.middleware >>> >>> phone: +94773325954 >>> email: lahi...@wso2.com blog: http://lahiruwrites.blogspot.com/ >>> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146 >>> >>> >>> >>> >>> >>> >>> >>> -- >>> >>> -- >>> Lahiru Sandaruwan >>> >>> Committer and PMC member, Apache Stratos, >>> Senior Software Engineer, >>> WSO2 Inc., http://wso2.com >>> >>> lean.enterprise.middleware >>> >>> phone: +94773325954 >>> email: lahi...@wso2.com blog: http://lahiruwrites.blogspot.com/ >>> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146 >>> >>> >>> >>> >>> >>> >>> >>> -- >>> >>> -- >>> Lahiru Sandaruwan >>> >>> Committer and PMC member, Apache Stratos, >>> Senior Software Engineer, >>> WSO2 Inc., http://wso2.com >>> >>> lean.enterprise.middleware >>> >>> phone: +94773325954 >>> email: lahi...@wso2.com blog: http://lahiruwrites.blogspot.com/ >>> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146 >>> >>> >>> >>> >>> >>> >>> >>> -- >>> >>> -- >>> Lahiru Sandaruwan >>> >>> Committer and PMC member, Apache Stratos, >>> Senior Software Engineer, >>> WSO2 Inc., http://wso2.com >>> >>> lean.enterprise.middleware >>> >>> phone: +94773325954 >>> email: lahi...@wso2.com blog: http://lahiruwrites.blogspot.com/ >>> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146 >>> >>> >>> >>> >>> >>> >>> >>> -- >>> >>> -- >>> Lahiru Sandaruwan >>> >>> Committer and PMC member, Apache Stratos, >>> Senior Software Engineer, >>> WSO2 Inc., http://wso2.com >>> >>> lean.enterprise.middleware >>> >>> phone: +94773325954 >>> email: lahi...@wso2.com blog: http://lahiruwrites.blogspot.com/ >>> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146 >>> >>> >>> >> >> >> >> -- >> -- >> Lahiru Sandaruwan >> Committer and PMC member, Apache Stratos, >> Senior Software Engineer, >> WSO2 Inc., http://wso2.com >> lean.enterprise.middleware >> >> phone: +94773325954 >> email: lahi...@wso2.com blog: http://lahiruwrites.blogspot.com/ >> linked-in: >> http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146 >> >> -- >> <http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146> >> <http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146> >> Thanks and Regards, >> >> Isuru H. >> <http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146> >> +94 716 358 048 <http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146>* >> <http://wso2.com/>* >> >> >> * <http://wso2.com/>* >> >> >> -- -- Lahiru Sandaruwan Committer and PMC member, Apache Stratos, Senior Software Engineer, WSO2 Inc., http://wso2.com lean.enterprise.middleware phone: +94773325954 email: lahi...@wso2.com blog: http://lahiruwrites.blogspot.com/ linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146