Thanks Martin for the logs. So the reason is clear why happens this. We keep the payload par cartridge type in iaas provider object, in cloud controller context.
When second member approaches the instance started method, it rewrites the payload in iaasProvider object. But the first instance is still not really created in iaas, and it runs in a different thread. Therefore the payload belong to second instance, is passed to second instance as well as the first instance. I will fix it. Thanks. On Wed, Apr 1, 2015 at 11:38 PM, Martin Eppel (meppel) <mep...@cisco.com> wrote: > Hi Lahiru, > > > > Attached are the logs, > > > > Thanks > > > > Martin > > > > *From:* Lahiru Sandaruwan [mailto:lahi...@wso2.com] > *Sent:* Wednesday, April 01, 2015 4:11 AM > > *To:* dev > *Subject:* Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not > going active ... was RE: Stratos 4.1.0 : Issue when deploying application > with single group and 3 cartridges, no dependencies ... > > > > HI Martin, > > > > Could you please send the debug logs enabling debug on following classes? > I have improved few debug logs to track payload, therefore please take a > pull before reproducing with debug logs. > > > log4j.logger.org.apache.stratos.autoscaler.applications.parser.DefaultApplicationParser=DEBUG > > > log4j.logger.org.apache.stratos.cloud.controller.services.impl.InstanceCreator=DEBUG > > > log4j.logger.org.apache.stratos.cloud.controller.services.impl.CloudControllerServiceImpl=DEBUG > > Thanks. > > > > > > On Wed, Apr 1, 2015 at 10:50 AM, Lahiru Sandaruwan <lahi...@wso2.com> > wrote: > > I will have a look Martin. > > > > Thanks. > > > > On Wed, Apr 1, 2015 at 6:54 AM, Martin Eppel (meppel) <mep...@cisco.com> > wrote: > > Hi Lahiru, > > > > I retested the scenario to make sure proper unique aliases are provided > and they are, please double check the provided artifacts. Issue remains the > same, one VM gets stuck in INITIALIZED until terminated and restarted. As > pointed out before, the cartridge alias in the cartridge agent is the same > in 2 cartridges (although provided aliases in the application json is > unique). After restarting the faulty member cartridge agent alias has the > proper alias and goes into active. > > I also provided the cartridge-agent logs for the member which is faulty, > before restart and after restart (cartridge-agent-70~INITIALIZED, ACTIVE) > > > > Please see attached logs, artifacts, > > > > Thanks > > > > Martin > > > > *From:* Martin Eppel (meppel) > *Sent:* Tuesday, March 31, 2015 1:22 PM > > > *To:* dev@stratos.apache.org > *Subject:* RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not > going active ... was RE: Stratos 4.1.0 : Issue when deploying application > with single group and 3 cartridges, no dependencies ... > > > > Lahiru, > > > > The aliases are all unique, please check the the application json, and as > I said in my analysis it I think this is what I saw but I can’t be sure. So > this is not an issue of starting the application with duplicate aliases but > rather something else , > > > > Thanks > > > > Martin > > > > *From:* Lahiru Sandaruwan [mailto:lahi...@wso2.com <lahi...@wso2.com>] > *Sent:* Tuesday, March 31, 2015 12:58 PM > *To:* dev > *Subject:* Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not > going active ... was RE: Stratos 4.1.0 : Issue when deploying application > with single group and 3 cartridges, no dependencies ... > > > > Hi Martin, > > > > On Wed, Apr 1, 2015 at 12:34 AM, Martin Eppel (meppel) <mep...@cisco.com> > wrote: > > Hi Lahiru, > > > > I tested the fix and noticed the following issue: > > > > From the 4 cartridges 3 instances go Active, however one instance got > stuck in INITIALIZED state. > > > > One of the things I noticed while the member was stuck in INITIALIZED > state is that the system received 2 active events for the same membered: > > > > *TID: [0] [STRATOS] [2015-03-31 18:10:27,645] INFO > {org.apache.stratos.cloud.controller.messaging.publisher.TopologyEventPublisher} > - Publishing member activated event: [service-name] c2 [cluster-id] > subscription-G1-G2-G3.c2-1x1.c2.domain [cluster-instance-id] > subscription-G1-G2-G3-1 [member-id] > subscription-G1-G2-G3.c2-1x1.c2.domainf939e8dc-e0cc-408a-8343-ac86e479549f > [network-partition-id] RegionOne [partition-id] whole-region* > > *TID: [0] [STRATOS] [2015-03-31 18:10:31,059] INFO > {org.apache.stratos.cloud.controller.messaging.publisher.TopologyEventPublisher} > - Publishing member activated event: [service-name] c2 [cluster-id] > subscription-G1-G2-G3.c2-1x1.c2.domain [cluster-instance-id] > subscription-G1-G2-G3-1 [member-id] > subscription-G1-G2-G3.c2-1x1.c2.domainf939e8dc-e0cc-408a-8343-ac86e479549f > [network-partition-id] RegionOne [partition-id] whole-region* > > > > As another note, but I can’t be sure since the log got overwritten is that > it appeared to me that at one point (while the cartridge was stuck in > INITIALIZED state is that the cartridge alias of 2 cartridges was the same > (~71, ~70: 71: CARTRIDGE_ALIAS=c2-1x1), but I can’t say for sure since when > I double checked a new instance of the cartridge was created (which went > ACTIVE). > > > > > > Can i assume this occurred when two cartridges use same alias? May be we > have to put a validation on Application as we cannot use same alias in two > cartridges. > > > > If so, could you please try to reproduce this with different aliases? > > > > Thanks. > > > > > > > > Eventually the system terminated the “faulty” member and the cartridge > came up and went into active state: > > > > *“Pending state of member expired, member will be moved to obsolete list. > [pending member] > subscription-G1-G2-G3.c2-1x0.c2.domainfe9a30cf-fffc-4a6c-9b3b-f419498895b9* > ” > > … > > “*Member activated: [service] c2 [cluster] > subscription-G1-G2-G3.c2-1x0.c2.domain [member] > subscription-G1-G2-G3.c2-1x0.c2.domain44a2ae3c-9968-41de-9e10-7212ef3788c2* > ” > > > > Cartridge alias after all 4 cartridges went ACTIVE: > > > > 70: CARTRIDGE_ALIAS=c2-1x0 > > 71: CARTRIDGE_ALIAS=c2-1x1 > > > > 69: CARTRIDGE_ALIAS=c1-1x0 > > 68: CARTRIDGE_ALIAS=c1-1x1 > > > > I attached the logs and artefacts > > > > Thanks > > > > Martin > > > > *From:* Lahiru Sandaruwan [mailto:lahi...@wso2.com] > *Sent:* Tuesday, March 31, 2015 2:24 AM > > > *To:* dev > *Subject:* Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not > going active ... was RE: Stratos 4.1.0 : Issue when deploying application > with single group and 3 cartridges, no dependencies ... > > > > Hi Martin, > > > > I have reproduced the issue and pushed a fix. Please verify. > > > > Thanks. > > > > On Tue, Mar 31, 2015 at 8:32 AM, Lahiru Sandaruwan <lahi...@wso2.com> > wrote: > > Hi Martin, > > > > On Tue, Mar 31, 2015 at 5:41 AM, Martin Eppel (meppel) <mep...@cisco.com> > wrote: > > Hi Lahiru, > > > > I tested a new scenario, where I have 2 groups with cartridges and no > start up dependencies and I see again the issue with cartridge instances > being stuck in “started” state. I also noticed the following log message: > > *“Artifact repositories not found for application signup, artifact updated > event not sent: [application-id] subscription-G1-G2-G3*”. > > > > > > Will have a look. > > > > Thanks. > > Please find attached the artefacts and log messages, > > > > Thanks > > > > Martin > > > > *From:* Martin Eppel (meppel) > *Sent:* Monday, March 30, 2015 2:36 PM > > > *To:* dev@stratos.apache.org > *Subject:* RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not > going active ... was RE: Stratos 4.1.0 : Issue when deploying application > with single group and 3 cartridges, no dependencies ... > > > > Hi Lahiru, > > > > I tested both scenarios, > > > > 1. Single cartridge in single group and it worked, cartridge > instance starts up and goes active, > > 2. Multiple cartridges with startup dependencies : > > All instances are started up (in right sequence) and go active > > JIRA https://issues.apache.org/jira/browse/STRATOS-1292[image: > http://wikicentral.cisco.com/images/icons/linkext7.gif] > <https://issues.apache.org/jira/browse/STRATOS-1292> is being tested and > verified > > > > Removing / Un-deploying the application caused an issue as one of the VMs > did not get terminated (throwing exceptions in the log). I am not sure if > this is already a known issue related to to the ongoing discussion of > application removal / un-deployment. > > Just in case, I zipped up the logs and attached it to the email thread > (single-group-multiple-cartridges-startup.zip). Let me know if the > un-deployment issue is the same already discussed or if this is a new issue > ? > > > > Same error occurred in the scenario where multiple cartridges were started > up (all go active) and then application was removed, I attached the log for > this scenario as well > (wscarbon-application-single-group-multiple-cartridges-no-startup-remval-error.log.zip) > > > > > > Thanks > > > > Martin > > > > > > *From:* Martin Eppel (meppel) > *Sent:* Monday, March 30, 2015 7:47 AM > *To:* dev@stratos.apache.org > *Subject:* RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not > going active ... was RE: Stratos 4.1.0 : Issue when deploying application > with single group and 3 cartridges, no dependencies ... > > > > Thanks Lahiru, > > > > Will do and let you know > > > > Martin > > > > *From:* Lahiru Sandaruwan [mailto:lahi...@wso2.com <lahi...@wso2.com>] > *Sent:* Monday, March 30, 2015 3:32 AM > *To:* dev > *Subject:* Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not > going active ... was RE: Stratos 4.1.0 : Issue when deploying application > with single group and 3 cartridges, no dependencies ... > > > > Hi Martin, > > > > I was able to reproduce the issue and committed a fix. > > Could you please verify? > > > > Thanks. > > > > On Sun, Mar 29, 2015 at 5:43 PM, Lahiru Sandaruwan <lahi...@wso2.com> > wrote: > > Hi Martin, > > > > According to the logs attached, it seems the cartridge agent waits for the > artifact updated event, and somehow it is not sent. > > > > Looking into it. > > > > Thanks. > > > > On Sat, Mar 28, 2015 at 6:48 AM, Imesh Gunaratne <im...@apache.org> wrote: > > Hi Martin, > > > > Thanks for the quick response on this! > > I will use these artifacts to re-produce the problem locally and will let > you know the results soon. > > > > Thanks > > > > On Sat, Mar 28, 2015 at 4:19 AM, Martin Eppel (meppel) <mep...@cisco.com> > wrote: > > Hi Imesh, > > > > As discussed please find attached the “real” artefacts (application, > cartridge group, cartridge): > > > > Attached it as well to the jira > > > > Thanks > > > > Martin > > > > > > *From:* Martin Eppel (meppel) > *Sent:* Friday, March 27, 2015 11:37 AM > *To:* dev@stratos.apache.org > *Subject:* RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not > going active ... was RE: Stratos 4.1.0 : Issue when deploying application > with single group and 3 cartridges, no dependencies ... > > > > Hi Imesh, > > > > Originally I tested with 3 cartridges but then realized that it also > occurs with a single cartridge in a group – see also the JIRA for the > respective notes and artefacts (single group, single cartridge) > > > > Thanks > > > > Martin > > > > *From:* Imesh Gunaratne [mailto:im...@apache.org <im...@apache.org>] > *Sent:* Friday, March 27, 2015 11:34 AM > *To:* dev > *Subject:* Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not > going active ... was RE: Stratos 4.1.0 : Issue when deploying application > with single group and 3 cartridges, no dependencies ... > > > > Hi Martin, > > > > Thanks for reporting this problem. You have mentioned that there are three > cartridges in the application but in the given application.json I only can > see one cartridge. Are we missing anything here? Will it be possible for > you to attach the complete set of Stratos artifacts that you are using in a > zip file? > > > > Thanks > > Imesh > > > > On Fri, Mar 27, 2015 at 11:51 PM, Martin Eppel (meppel) <mep...@cisco.com> > wrote: > > Forgot to mention, this is the same issue as reported in JIRA > https://issues.apache.org/jira/browse/STRATOS-1292 > > > > Thanks > > > > Martin > > > > *From:* Martin Eppel (meppel) > *Sent:* Friday, March 27, 2015 10:52 AM > *To:* dev@stratos.apache.org > *Subject:* Issue (?): Stratos 4.1.0 - Cartridges in a group are not going > active ... was RE: Stratos 4.1.0 : Issue when deploying application with > single group and 3 cartridges, no dependencies ... > > > > I retested with a similar scenario (single cartridges in group) with > debug logs turned on, please see logs attached. > > > > Cartridge is stuck in “Starting” state > > > > Thanks > > > > Martin > > > > *From:* Martin Eppel (meppel) > *Sent:* Monday, March 23, 2015 7:15 PM > *To:* dev@stratos.apache.org > *Subject:* Stratos 4.1.0 : Issue when deploying application with single > group and 3 cartridges, no dependencies ... > > > > Hi, > > > > I see an issue when I subscribe a single group with 3 cartridges, no > dependencies – they never go active instead they all are stuck in > “Starting”. I attached the server log and the cartridge-agent logs > > > > > > Member status: > > subscription-c1-c2-c3: applicationInstances 1, groupInstances 1, > clusterInstances 3, members 3 (Starting 3) > > c2: 172.16.2.18/172.28.40.69: status Starting > > c3: 172.16.2.19/172.28.40.70: status Starting > > c1: 172.16.2.17/172.28.40.68: status Starting > > > > Let me know if I should go ahead and open a JIRA > > > > > > Thanks > > > > Martin > > > > > > -- > > Imesh Gunaratne > > > > Technical Lead, WSO2 > > Committer & PMC Member, Apache Stratos > > > > > > -- > > Imesh Gunaratne > > > > Technical Lead, WSO2 > > Committer & PMC Member, Apache Stratos > > > > > > -- > > -- > Lahiru Sandaruwan > > Committer and PMC member, Apache Stratos, > Senior Software Engineer, > WSO2 Inc., http://wso2.com > > lean.enterprise.middleware > > phone: +94773325954 > email: lahi...@wso2.com blog: http://lahiruwrites.blogspot.com/ > linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146 > > > > > > > > -- > > -- > Lahiru Sandaruwan > > Committer and PMC member, Apache Stratos, > Senior Software Engineer, > WSO2 Inc., http://wso2.com > > lean.enterprise.middleware > > phone: +94773325954 > email: lahi...@wso2.com blog: http://lahiruwrites.blogspot.com/ > linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146 > > > > > > > > -- > > -- > Lahiru Sandaruwan > > Committer and PMC member, Apache Stratos, > Senior Software Engineer, > WSO2 Inc., http://wso2.com > > lean.enterprise.middleware > > phone: +94773325954 > email: lahi...@wso2.com blog: http://lahiruwrites.blogspot.com/ > linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146 > > > > > > > > -- > > -- > Lahiru Sandaruwan > > Committer and PMC member, Apache Stratos, > Senior Software Engineer, > WSO2 Inc., http://wso2.com > > lean.enterprise.middleware > > phone: +94773325954 > email: lahi...@wso2.com blog: http://lahiruwrites.blogspot.com/ > linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146 > > > > > > > > -- > > -- > Lahiru Sandaruwan > > Committer and PMC member, Apache Stratos, > Senior Software Engineer, > WSO2 Inc., http://wso2.com > > lean.enterprise.middleware > > phone: +94773325954 > email: lahi...@wso2.com blog: http://lahiruwrites.blogspot.com/ > linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146 > > > > > > > > -- > > -- > Lahiru Sandaruwan > > Committer and PMC member, Apache Stratos, > Senior Software Engineer, > WSO2 Inc., http://wso2.com > > lean.enterprise.middleware > > phone: +94773325954 > email: lahi...@wso2.com blog: http://lahiruwrites.blogspot.com/ > linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146 > > > > > > > > -- > > -- > Lahiru Sandaruwan > > Committer and PMC member, Apache Stratos, > Senior Software Engineer, > WSO2 Inc., http://wso2.com > > lean.enterprise.middleware > > phone: +94773325954 > email: lahi...@wso2.com blog: http://lahiruwrites.blogspot.com/ > linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146 > > > -- -- Lahiru Sandaruwan Committer and PMC member, Apache Stratos, Senior Software Engineer, WSO2 Inc., http://wso2.com lean.enterprise.middleware phone: +94773325954 email: lahi...@wso2.com blog: http://lahiruwrites.blogspot.com/ linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146