Hi Martin, Could you please try the fix i have committed?
Thanks, On Thu, Apr 2, 2015 at 10:35 AM, Martin Eppel (meppel) <mep...@cisco.com> wrote: > Great, > > > > Thanks > > > > Martin > > > > *From:* isu...@wso2.com [mailto:isu...@wso2.com] *On Behalf Of *Isuru > Haththotuwa > *Sent:* Wednesday, April 01, 2015 9:08 PM > > *To:* dev > *Subject:* Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not > going active ... was RE: Stratos 4.1.0 : Issue when deploying application > with single group and 3 cartridges, no dependencies ... > > > > Hi Lahiru, > > > On Thu, Apr 2, 2015 at 9:11 AM, Lahiru Sandaruwan <lahi...@wso2.com> > wrote: > > Thanks Martin for the logs. So the reason is clear why happens this. We > keep the payload par cartridge type in iaas provider object, in cloud > controller context. > > > > When second member approaches the instance started method, it rewrites the > payload in iaasProvider object. But the first instance is still not really > created in iaas, and it runs in a different thread. Therefore the payload > belong to second instance, is passed to second instance as well as the > first instance. > > Nice work finding this! So the resolution would be to change the cartridge > type -> payload mapping to alias -> payload? > > > > I will fix it. > > > > Thanks. > > > > On Wed, Apr 1, 2015 at 11:38 PM, Martin Eppel (meppel) <mep...@cisco.com> > wrote: > > Hi Lahiru, > > > > Attached are the logs, > > > > Thanks > > > > Martin > > > > *From:* Lahiru Sandaruwan [mailto:lahi...@wso2.com] > *Sent:* Wednesday, April 01, 2015 4:11 AM > > > *To:* dev > *Subject:* Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not > going active ... was RE: Stratos 4.1.0 : Issue when deploying application > with single group and 3 cartridges, no dependencies ... > > > > HI Martin, > > > > Could you please send the debug logs enabling debug on following classes? > I have improved few debug logs to track payload, therefore please take a > pull before reproducing with debug logs. > > > log4j.logger.org.apache.stratos.autoscaler.applications.parser.DefaultApplicationParser=DEBUG > > > log4j.logger.org.apache.stratos.cloud.controller.services.impl.InstanceCreator=DEBUG > > > log4j.logger.org.apache.stratos.cloud.controller.services.impl.CloudControllerServiceImpl=DEBUG > > Thanks. > > > > > > On Wed, Apr 1, 2015 at 10:50 AM, Lahiru Sandaruwan <lahi...@wso2.com> > wrote: > > I will have a look Martin. > > > > Thanks. > > > > On Wed, Apr 1, 2015 at 6:54 AM, Martin Eppel (meppel) <mep...@cisco.com> > wrote: > > Hi Lahiru, > > > > I retested the scenario to make sure proper unique aliases are provided > and they are, please double check the provided artifacts. Issue remains the > same, one VM gets stuck in INITIALIZED until terminated and restarted. As > pointed out before, the cartridge alias in the cartridge agent is the same > in 2 cartridges (although provided aliases in the application json is > unique). After restarting the faulty member cartridge agent alias has the > proper alias and goes into active. > > I also provided the cartridge-agent logs for the member which is faulty, > before restart and after restart (cartridge-agent-70~INITIALIZED, ACTIVE) > > > > Please see attached logs, artifacts, > > > > Thanks > > > > Martin > > > > *From:* Martin Eppel (meppel) > *Sent:* Tuesday, March 31, 2015 1:22 PM > > > *To:* dev@stratos.apache.org > *Subject:* RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not > going active ... was RE: Stratos 4.1.0 : Issue when deploying application > with single group and 3 cartridges, no dependencies ... > > > > Lahiru, > > > > The aliases are all unique, please check the the application json, and as > I said in my analysis it I think this is what I saw but I can’t be sure. So > this is not an issue of starting the application with duplicate aliases but > rather something else , > > > > Thanks > > > > Martin > > > > *From:* Lahiru Sandaruwan [mailto:lahi...@wso2.com <lahi...@wso2.com>] > *Sent:* Tuesday, March 31, 2015 12:58 PM > *To:* dev > *Subject:* Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not > going active ... was RE: Stratos 4.1.0 : Issue when deploying application > with single group and 3 cartridges, no dependencies ... > > > > Hi Martin, > > > > On Wed, Apr 1, 2015 at 12:34 AM, Martin Eppel (meppel) <mep...@cisco.com> > wrote: > > Hi Lahiru, > > > > I tested the fix and noticed the following issue: > > > > From the 4 cartridges 3 instances go Active, however one instance got > stuck in INITIALIZED state. > > > > One of the things I noticed while the member was stuck in INITIALIZED > state is that the system received 2 active events for the same membered: > > > > *TID: [0] [STRATOS] [2015-03-31 18:10:27,645] INFO > {org.apache.stratos.cloud.controller.messaging.publisher.TopologyEventPublisher} > - Publishing member activated event: [service-name] c2 [cluster-id] > subscription-G1-G2-G3.c2-1x1.c2.domain [cluster-instance-id] > subscription-G1-G2-G3-1 [member-id] > subscription-G1-G2-G3.c2-1x1.c2.domainf939e8dc-e0cc-408a-8343-ac86e479549f > [network-partition-id] RegionOne [partition-id] whole-region* > > *TID: [0] [STRATOS] [2015-03-31 18:10:31,059] INFO > {org.apache.stratos.cloud.controller.messaging.publisher.TopologyEventPublisher} > - Publishing member activated event: [service-name] c2 [cluster-id] > subscription-G1-G2-G3.c2-1x1.c2.domain [cluster-instance-id] > subscription-G1-G2-G3-1 [member-id] > subscription-G1-G2-G3.c2-1x1.c2.domainf939e8dc-e0cc-408a-8343-ac86e479549f > [network-partition-id] RegionOne [partition-id] whole-region* > > > > As another note, but I can’t be sure since the log got overwritten is that > it appeared to me that at one point (while the cartridge was stuck in > INITIALIZED state is that the cartridge alias of 2 cartridges was the same > (~71, ~70: 71: CARTRIDGE_ALIAS=c2-1x1), but I can’t say for sure since when > I double checked a new instance of the cartridge was created (which went > ACTIVE). > > > > > > Can i assume this occurred when two cartridges use same alias? May be we > have to put a validation on Application as we cannot use same alias in two > cartridges. > > > > If so, could you please try to reproduce this with different aliases? > > > > Thanks. > > > > > > > > Eventually the system terminated the “faulty” member and the cartridge > came up and went into active state: > > > > *“Pending state of member expired, member will be moved to obsolete list. > [pending member] > subscription-G1-G2-G3.c2-1x0.c2.domainfe9a30cf-fffc-4a6c-9b3b-f419498895b9* > ” > > … > > “*Member activated: [service] c2 [cluster] > subscription-G1-G2-G3.c2-1x0.c2.domain [member] > subscription-G1-G2-G3.c2-1x0.c2.domain44a2ae3c-9968-41de-9e10-7212ef3788c2* > ” > > > > Cartridge alias after all 4 cartridges went ACTIVE: > > > > 70: CARTRIDGE_ALIAS=c2-1x0 > > 71: CARTRIDGE_ALIAS=c2-1x1 > > > > 69: CARTRIDGE_ALIAS=c1-1x0 > > 68: CARTRIDGE_ALIAS=c1-1x1 > > > > I attached the logs and artefacts > > > > Thanks > > > > Martin > > > > *From:* Lahiru Sandaruwan [mailto:lahi...@wso2.com] > *Sent:* Tuesday, March 31, 2015 2:24 AM > > > *To:* dev > *Subject:* Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not > going active ... was RE: Stratos 4.1.0 : Issue when deploying application > with single group and 3 cartridges, no dependencies ... > > > > Hi Martin, > > > > I have reproduced the issue and pushed a fix. Please verify. > > > > Thanks. > > > > On Tue, Mar 31, 2015 at 8:32 AM, Lahiru Sandaruwan <lahi...@wso2.com> > wrote: > > Hi Martin, > > > > On Tue, Mar 31, 2015 at 5:41 AM, Martin Eppel (meppel) <mep...@cisco.com> > wrote: > > Hi Lahiru, > > > > I tested a new scenario, where I have 2 groups with cartridges and no > start up dependencies and I see again the issue with cartridge instances > being stuck in “started” state. I also noticed the following log message: > > *“Artifact repositories not found for application signup, artifact updated > event not sent: [application-id] subscription-G1-G2-G3*”. > > > > > > Will have a look. > > > > Thanks. > > Please find attached the artefacts and log messages, > > > > Thanks > > > > Martin > > > > *From:* Martin Eppel (meppel) > *Sent:* Monday, March 30, 2015 2:36 PM > > > *To:* dev@stratos.apache.org > *Subject:* RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not > going active ... was RE: Stratos 4.1.0 : Issue when deploying application > with single group and 3 cartridges, no dependencies ... > > > > Hi Lahiru, > > > > I tested both scenarios, > > > > 1. Single cartridge in single group and it worked, cartridge > instance starts up and goes active, > > 2. Multiple cartridges with startup dependencies : > > All instances are started up (in right sequence) and go active > > JIRA https://issues.apache.org/jira/browse/STRATOS-1292[image: > http://wikicentral.cisco.com/images/icons/linkext7.gif] > <https://issues.apache.org/jira/browse/STRATOS-1292> is being tested and > verified > > > > Removing / Un-deploying the application caused an issue as one of the VMs > did not get terminated (throwing exceptions in the log). I am not sure if > this is already a known issue related to to the ongoing discussion of > application removal / un-deployment. > > Just in case, I zipped up the logs and attached it to the email thread > (single-group-multiple-cartridges-startup.zip). Let me know if the > un-deployment issue is the same already discussed or if this is a new issue > ? > > > > Same error occurred in the scenario where multiple cartridges were started > up (all go active) and then application was removed, I attached the log for > this scenario as well > (wscarbon-application-single-group-multiple-cartridges-no-startup-remval-error.log.zip) > > > > > > Thanks > > > > Martin > > > > > > *From:* Martin Eppel (meppel) > *Sent:* Monday, March 30, 2015 7:47 AM > *To:* dev@stratos.apache.org > *Subject:* RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not > going active ... was RE: Stratos 4.1.0 : Issue when deploying application > with single group and 3 cartridges, no dependencies ... > > > > Thanks Lahiru, > > > > Will do and let you know > > > > Martin > > > > *From:* Lahiru Sandaruwan [mailto:lahi...@wso2.com <lahi...@wso2.com>] > *Sent:* Monday, March 30, 2015 3:32 AM > *To:* dev > *Subject:* Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not > going active ... was RE: Stratos 4.1.0 : Issue when deploying application > with single group and 3 cartridges, no dependencies ... > > > > Hi Martin, > > > > I was able to reproduce the issue and committed a fix. > > Could you please verify? > > > > Thanks. > > > > On Sun, Mar 29, 2015 at 5:43 PM, Lahiru Sandaruwan <lahi...@wso2.com> > wrote: > > Hi Martin, > > > > According to the logs attached, it seems the cartridge agent waits for the > artifact updated event, and somehow it is not sent. > > > > Looking into it. > > > > Thanks. > > > > On Sat, Mar 28, 2015 at 6:48 AM, Imesh Gunaratne <im...@apache.org> wrote: > > Hi Martin, > > > > Thanks for the quick response on this! > > I will use these artifacts to re-produce the problem locally and will let > you know the results soon. > > > > Thanks > > > > On Sat, Mar 28, 2015 at 4:19 AM, Martin Eppel (meppel) <mep...@cisco.com> > wrote: > > Hi Imesh, > > > > As discussed please find attached the “real” artefacts (application, > cartridge group, cartridge): > > > > Attached it as well to the jira > > > > Thanks > > > > Martin > > > > > > *From:* Martin Eppel (meppel) > *Sent:* Friday, March 27, 2015 11:37 AM > *To:* dev@stratos.apache.org > *Subject:* RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not > going active ... was RE: Stratos 4.1.0 : Issue when deploying application > with single group and 3 cartridges, no dependencies ... > > > > Hi Imesh, > > > > Originally I tested with 3 cartridges but then realized that it also > occurs with a single cartridge in a group – see also the JIRA for the > respective notes and artefacts (single group, single cartridge) > > > > Thanks > > > > Martin > > > > *From:* Imesh Gunaratne [mailto:im...@apache.org <im...@apache.org>] > *Sent:* Friday, March 27, 2015 11:34 AM > *To:* dev > *Subject:* Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not > going active ... was RE: Stratos 4.1.0 : Issue when deploying application > with single group and 3 cartridges, no dependencies ... > > > > Hi Martin, > > > > Thanks for reporting this problem. You have mentioned that there are three > cartridges in the application but in the given application.json I only can > see one cartridge. Are we missing anything here? Will it be possible for > you to attach the complete set of Stratos artifacts that you are using in a > zip file? > > > > Thanks > > Imesh > > > > On Fri, Mar 27, 2015 at 11:51 PM, Martin Eppel (meppel) <mep...@cisco.com> > wrote: > > Forgot to mention, this is the same issue as reported in JIRA > https://issues.apache.org/jira/browse/STRATOS-1292 > > > > Thanks > > > > Martin > > > > *From:* Martin Eppel (meppel) > *Sent:* Friday, March 27, 2015 10:52 AM > *To:* dev@stratos.apache.org > *Subject:* Issue (?): Stratos 4.1.0 - Cartridges in a group are not going > active ... was RE: Stratos 4.1.0 : Issue when deploying application with > single group and 3 cartridges, no dependencies ... > > > > I retested with a similar scenario (single cartridges in group) with > debug logs turned on, please see logs attached. > > > > Cartridge is stuck in “Starting” state > > > > Thanks > > > > Martin > > > > *From:* Martin Eppel (meppel) > *Sent:* Monday, March 23, 2015 7:15 PM > *To:* dev@stratos.apache.org > *Subject:* Stratos 4.1.0 : Issue when deploying application with single > group and 3 cartridges, no dependencies ... > > > > Hi, > > > > I see an issue when I subscribe a single group with 3 cartridges, no > dependencies – they never go active instead they all are stuck in > “Starting”. I attached the server log and the cartridge-agent logs > > > > > > Member status: > > subscription-c1-c2-c3: applicationInstances 1, groupInstances 1, > clusterInstances 3, members 3 (Starting 3) > > c2: 172.16.2.18/172.28.40.69: status Starting > > c3: 172.16.2.19/172.28.40.70: status Starting > > c1: 172.16.2.17/172.28.40.68: status Starting > > > > Let me know if I should go ahead and open a JIRA > > > > > > Thanks > > > > Martin > > > > > > -- > > Imesh Gunaratne > > > > Technical Lead, WSO2 > > Committer & PMC Member, Apache Stratos > > > > > > -- > > Imesh Gunaratne > > > > Technical Lead, WSO2 > > Committer & PMC Member, Apache Stratos > > > > > > -- > > -- > Lahiru Sandaruwan > > Committer and PMC member, Apache Stratos, > Senior Software Engineer, > WSO2 Inc., http://wso2.com > > lean.enterprise.middleware > > phone: +94773325954 > email: lahi...@wso2.com blog: http://lahiruwrites.blogspot.com/ > linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146 > > > > > > > > -- > > -- > Lahiru Sandaruwan > > Committer and PMC member, Apache Stratos, > Senior Software Engineer, > WSO2 Inc., http://wso2.com > > lean.enterprise.middleware > > phone: +94773325954 > email: lahi...@wso2.com blog: http://lahiruwrites.blogspot.com/ > linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146 > > > > > > > > -- > > -- > Lahiru Sandaruwan > > Committer and PMC member, Apache Stratos, > Senior Software Engineer, > WSO2 Inc., http://wso2.com > > lean.enterprise.middleware > > phone: +94773325954 > email: lahi...@wso2.com blog: http://lahiruwrites.blogspot.com/ > linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146 > > > > > > > > -- > > -- > Lahiru Sandaruwan > > Committer and PMC member, Apache Stratos, > Senior Software Engineer, > WSO2 Inc., http://wso2.com > > lean.enterprise.middleware > > phone: +94773325954 > email: lahi...@wso2.com blog: http://lahiruwrites.blogspot.com/ > linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146 > > > > > > > > -- > > -- > Lahiru Sandaruwan > > Committer and PMC member, Apache Stratos, > Senior Software Engineer, > WSO2 Inc., http://wso2.com > > lean.enterprise.middleware > > phone: +94773325954 > email: lahi...@wso2.com blog: http://lahiruwrites.blogspot.com/ > linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146 > > > > > > > > -- > > -- > Lahiru Sandaruwan > > Committer and PMC member, Apache Stratos, > Senior Software Engineer, > WSO2 Inc., http://wso2.com > > lean.enterprise.middleware > > phone: +94773325954 > email: lahi...@wso2.com blog: http://lahiruwrites.blogspot.com/ > linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146 > > > > > > > > -- > > -- > Lahiru Sandaruwan > > Committer and PMC member, Apache Stratos, > Senior Software Engineer, > WSO2 Inc., http://wso2.com > > lean.enterprise.middleware > > phone: +94773325954 > email: lahi...@wso2.com blog: http://lahiruwrites.blogspot.com/ > linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146 > > > > > > > > -- > > -- > Lahiru Sandaruwan > > Committer and PMC member, Apache Stratos, > Senior Software Engineer, > WSO2 Inc., http://wso2.com > > lean.enterprise.middleware > > phone: +94773325954 > email: lahi...@wso2.com blog: http://lahiruwrites.blogspot.com/ > linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146 > > -- > <http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146> > > <http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146> > > > > > > *Thanks and Regards, Isuru H. > <http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146>* > > > > +94 716 358 048 <http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146> > > > > -- -- Lahiru Sandaruwan Committer and PMC member, Apache Stratos, Senior Software Engineer, WSO2 Inc., http://wso2.com lean.enterprise.middleware phone: +94773325954 email: lahi...@wso2.com blog: http://lahiruwrites.blogspot.com/ linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
