HI Martin, Could you please send the debug logs enabling debug on following classes? I have improved few debug logs to track payload, therefore please take a pull before reproducing with debug logs.
log4j.logger.org.apache.stratos.autoscaler.applications.parser.DefaultApplicationParser=DEBUG log4j.logger.org.apache.stratos.cloud.controller.services.impl.InstanceCreator=DEBUG log4j.logger.org.apache.stratos.cloud.controller.services.impl.CloudControllerServiceImpl=DEBUG Thanks. On Wed, Apr 1, 2015 at 10:50 AM, Lahiru Sandaruwan <lahi...@wso2.com> wrote: > I will have a look Martin. > > Thanks. > > On Wed, Apr 1, 2015 at 6:54 AM, Martin Eppel (meppel) <mep...@cisco.com> > wrote: > >> Hi Lahiru, >> >> >> >> I retested the scenario to make sure proper unique aliases are provided >> and they are, please double check the provided artifacts. Issue remains the >> same, one VM gets stuck in INITIALIZED until terminated and restarted. As >> pointed out before, the cartridge alias in the cartridge agent is the same >> in 2 cartridges (although provided aliases in the application json is >> unique). After restarting the faulty member cartridge agent alias has the >> proper alias and goes into active. >> >> I also provided the cartridge-agent logs for the member which is faulty, >> before restart and after restart (cartridge-agent-70~INITIALIZED, ACTIVE) >> >> >> >> Please see attached logs, artifacts, >> >> >> >> Thanks >> >> >> >> Martin >> >> >> >> *From:* Martin Eppel (meppel) >> *Sent:* Tuesday, March 31, 2015 1:22 PM >> >> *To:* dev@stratos.apache.org >> *Subject:* RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not >> going active ... was RE: Stratos 4.1.0 : Issue when deploying application >> with single group and 3 cartridges, no dependencies ... >> >> >> >> Lahiru, >> >> >> >> The aliases are all unique, please check the the application json, and as >> I said in my analysis it I think this is what I saw but I can’t be sure. So >> this is not an issue of starting the application with duplicate aliases but >> rather something else , >> >> >> >> Thanks >> >> >> >> Martin >> >> >> >> *From:* Lahiru Sandaruwan [mailto:lahi...@wso2.com <lahi...@wso2.com>] >> *Sent:* Tuesday, March 31, 2015 12:58 PM >> *To:* dev >> *Subject:* Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not >> going active ... was RE: Stratos 4.1.0 : Issue when deploying application >> with single group and 3 cartridges, no dependencies ... >> >> >> >> Hi Martin, >> >> >> >> On Wed, Apr 1, 2015 at 12:34 AM, Martin Eppel (meppel) <mep...@cisco.com> >> wrote: >> >> Hi Lahiru, >> >> >> >> I tested the fix and noticed the following issue: >> >> >> >> From the 4 cartridges 3 instances go Active, however one instance got >> stuck in INITIALIZED state. >> >> >> >> One of the things I noticed while the member was stuck in INITIALIZED >> state is that the system received 2 active events for the same membered: >> >> >> >> *TID: [0] [STRATOS] [2015-03-31 18:10:27,645] INFO >> {org.apache.stratos.cloud.controller.messaging.publisher.TopologyEventPublisher} >> - Publishing member activated event: [service-name] c2 [cluster-id] >> subscription-G1-G2-G3.c2-1x1.c2.domain [cluster-instance-id] >> subscription-G1-G2-G3-1 [member-id] >> subscription-G1-G2-G3.c2-1x1.c2.domainf939e8dc-e0cc-408a-8343-ac86e479549f >> [network-partition-id] RegionOne [partition-id] whole-region* >> >> *TID: [0] [STRATOS] [2015-03-31 18:10:31,059] INFO >> {org.apache.stratos.cloud.controller.messaging.publisher.TopologyEventPublisher} >> - Publishing member activated event: [service-name] c2 [cluster-id] >> subscription-G1-G2-G3.c2-1x1.c2.domain [cluster-instance-id] >> subscription-G1-G2-G3-1 [member-id] >> subscription-G1-G2-G3.c2-1x1.c2.domainf939e8dc-e0cc-408a-8343-ac86e479549f >> [network-partition-id] RegionOne [partition-id] whole-region* >> >> >> >> As another note, but I can’t be sure since the log got overwritten is >> that it appeared to me that at one point (while the cartridge was stuck in >> INITIALIZED state is that the cartridge alias of 2 cartridges was the same >> (~71, ~70: 71: CARTRIDGE_ALIAS=c2-1x1), but I can’t say for sure since when >> I double checked a new instance of the cartridge was created (which went >> ACTIVE). >> >> >> >> >> >> Can i assume this occurred when two cartridges use same alias? May be we >> have to put a validation on Application as we cannot use same alias in two >> cartridges. >> >> >> >> If so, could you please try to reproduce this with different aliases? >> >> >> >> Thanks. >> >> >> >> >> >> >> >> Eventually the system terminated the “faulty” member and the cartridge >> came up and went into active state: >> >> >> >> *“Pending state of member expired, member will be moved to obsolete list. >> [pending member] >> subscription-G1-G2-G3.c2-1x0.c2.domainfe9a30cf-fffc-4a6c-9b3b-f419498895b9* >> ” >> >> … >> >> “*Member activated: [service] c2 [cluster] >> subscription-G1-G2-G3.c2-1x0.c2.domain [member] >> subscription-G1-G2-G3.c2-1x0.c2.domain44a2ae3c-9968-41de-9e10-7212ef3788c2* >> ” >> >> >> >> Cartridge alias after all 4 cartridges went ACTIVE: >> >> >> >> 70: CARTRIDGE_ALIAS=c2-1x0 >> >> 71: CARTRIDGE_ALIAS=c2-1x1 >> >> >> >> 69: CARTRIDGE_ALIAS=c1-1x0 >> >> 68: CARTRIDGE_ALIAS=c1-1x1 >> >> >> >> I attached the logs and artefacts >> >> >> >> Thanks >> >> >> >> Martin >> >> >> >> *From:* Lahiru Sandaruwan [mailto:lahi...@wso2.com] >> *Sent:* Tuesday, March 31, 2015 2:24 AM >> >> >> *To:* dev >> *Subject:* Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not >> going active ... was RE: Stratos 4.1.0 : Issue when deploying application >> with single group and 3 cartridges, no dependencies ... >> >> >> >> Hi Martin, >> >> >> >> I have reproduced the issue and pushed a fix. Please verify. >> >> >> >> Thanks. >> >> >> >> On Tue, Mar 31, 2015 at 8:32 AM, Lahiru Sandaruwan <lahi...@wso2.com> >> wrote: >> >> Hi Martin, >> >> >> >> On Tue, Mar 31, 2015 at 5:41 AM, Martin Eppel (meppel) <mep...@cisco.com> >> wrote: >> >> Hi Lahiru, >> >> >> >> I tested a new scenario, where I have 2 groups with cartridges and no >> start up dependencies and I see again the issue with cartridge instances >> being stuck in “started” state. I also noticed the following log message: >> >> *“Artifact repositories not found for application signup, artifact >> updated event not sent: [application-id] subscription-G1-G2-G3*”. >> >> >> >> >> >> Will have a look. >> >> >> >> Thanks. >> >> Please find attached the artefacts and log messages, >> >> >> >> Thanks >> >> >> >> Martin >> >> >> >> *From:* Martin Eppel (meppel) >> *Sent:* Monday, March 30, 2015 2:36 PM >> >> >> *To:* dev@stratos.apache.org >> *Subject:* RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not >> going active ... was RE: Stratos 4.1.0 : Issue when deploying application >> with single group and 3 cartridges, no dependencies ... >> >> >> >> Hi Lahiru, >> >> >> >> I tested both scenarios, >> >> >> >> 1. Single cartridge in single group and it worked, cartridge >> instance starts up and goes active, >> >> 2. Multiple cartridges with startup dependencies : >> >> All instances are started up (in right sequence) and go active >> >> JIRA https://issues.apache.org/jira/browse/STRATOS-1292[image: >> http://wikicentral.cisco.com/images/icons/linkext7.gif] >> <https://issues.apache.org/jira/browse/STRATOS-1292> is being tested and >> verified >> >> >> >> Removing / Un-deploying the application caused an issue as one of the VMs >> did not get terminated (throwing exceptions in the log). I am not sure if >> this is already a known issue related to to the ongoing discussion of >> application removal / un-deployment. >> >> Just in case, I zipped up the logs and attached it to the email thread >> (single-group-multiple-cartridges-startup.zip). Let me know if the >> un-deployment issue is the same already discussed or if this is a new issue >> ? >> >> >> >> Same error occurred in the scenario where multiple cartridges were >> started up (all go active) and then application was removed, I attached the >> log for this scenario as well >> (wscarbon-application-single-group-multiple-cartridges-no-startup-remval-error.log.zip) >> >> >> >> >> >> Thanks >> >> >> >> Martin >> >> >> >> >> >> *From:* Martin Eppel (meppel) >> *Sent:* Monday, March 30, 2015 7:47 AM >> *To:* dev@stratos.apache.org >> *Subject:* RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not >> going active ... was RE: Stratos 4.1.0 : Issue when deploying application >> with single group and 3 cartridges, no dependencies ... >> >> >> >> Thanks Lahiru, >> >> >> >> Will do and let you know >> >> >> >> Martin >> >> >> >> *From:* Lahiru Sandaruwan [mailto:lahi...@wso2.com <lahi...@wso2.com>] >> *Sent:* Monday, March 30, 2015 3:32 AM >> *To:* dev >> *Subject:* Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not >> going active ... was RE: Stratos 4.1.0 : Issue when deploying application >> with single group and 3 cartridges, no dependencies ... >> >> >> >> Hi Martin, >> >> >> >> I was able to reproduce the issue and committed a fix. >> >> Could you please verify? >> >> >> >> Thanks. >> >> >> >> On Sun, Mar 29, 2015 at 5:43 PM, Lahiru Sandaruwan <lahi...@wso2.com> >> wrote: >> >> Hi Martin, >> >> >> >> According to the logs attached, it seems the cartridge agent waits for >> the artifact updated event, and somehow it is not sent. >> >> >> >> Looking into it. >> >> >> >> Thanks. >> >> >> >> On Sat, Mar 28, 2015 at 6:48 AM, Imesh Gunaratne <im...@apache.org> >> wrote: >> >> Hi Martin, >> >> >> >> Thanks for the quick response on this! >> >> I will use these artifacts to re-produce the problem locally and will let >> you know the results soon. >> >> >> >> Thanks >> >> >> >> On Sat, Mar 28, 2015 at 4:19 AM, Martin Eppel (meppel) <mep...@cisco.com> >> wrote: >> >> Hi Imesh, >> >> >> >> As discussed please find attached the “real” artefacts (application, >> cartridge group, cartridge): >> >> >> >> Attached it as well to the jira >> >> >> >> Thanks >> >> >> >> Martin >> >> >> >> >> >> *From:* Martin Eppel (meppel) >> *Sent:* Friday, March 27, 2015 11:37 AM >> *To:* dev@stratos.apache.org >> *Subject:* RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not >> going active ... was RE: Stratos 4.1.0 : Issue when deploying application >> with single group and 3 cartridges, no dependencies ... >> >> >> >> Hi Imesh, >> >> >> >> Originally I tested with 3 cartridges but then realized that it also >> occurs with a single cartridge in a group – see also the JIRA for the >> respective notes and artefacts (single group, single cartridge) >> >> >> >> Thanks >> >> >> >> Martin >> >> >> >> *From:* Imesh Gunaratne [mailto:im...@apache.org <im...@apache.org>] >> *Sent:* Friday, March 27, 2015 11:34 AM >> *To:* dev >> *Subject:* Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not >> going active ... was RE: Stratos 4.1.0 : Issue when deploying application >> with single group and 3 cartridges, no dependencies ... >> >> >> >> Hi Martin, >> >> >> >> Thanks for reporting this problem. You have mentioned that there are >> three cartridges in the application but in the given application.json I >> only can see one cartridge. Are we missing anything here? Will it be >> possible for you to attach the complete set of Stratos artifacts that you >> are using in a zip file? >> >> >> >> Thanks >> >> Imesh >> >> >> >> On Fri, Mar 27, 2015 at 11:51 PM, Martin Eppel (meppel) <mep...@cisco.com> >> wrote: >> >> Forgot to mention, this is the same issue as reported in JIRA >> https://issues.apache.org/jira/browse/STRATOS-1292 >> >> >> >> Thanks >> >> >> >> Martin >> >> >> >> *From:* Martin Eppel (meppel) >> *Sent:* Friday, March 27, 2015 10:52 AM >> *To:* dev@stratos.apache.org >> *Subject:* Issue (?): Stratos 4.1.0 - Cartridges in a group are not >> going active ... was RE: Stratos 4.1.0 : Issue when deploying application >> with single group and 3 cartridges, no dependencies ... >> >> >> >> I retested with a similar scenario (single cartridges in group) with >> debug logs turned on, please see logs attached. >> >> >> >> Cartridge is stuck in “Starting” state >> >> >> >> Thanks >> >> >> >> Martin >> >> >> >> *From:* Martin Eppel (meppel) >> *Sent:* Monday, March 23, 2015 7:15 PM >> *To:* dev@stratos.apache.org >> *Subject:* Stratos 4.1.0 : Issue when deploying application with single >> group and 3 cartridges, no dependencies ... >> >> >> >> Hi, >> >> >> >> I see an issue when I subscribe a single group with 3 cartridges, no >> dependencies – they never go active instead they all are stuck in >> “Starting”. I attached the server log and the cartridge-agent logs >> >> >> >> >> >> Member status: >> >> subscription-c1-c2-c3: applicationInstances 1, groupInstances 1, >> clusterInstances 3, members 3 (Starting 3) >> >> c2: 172.16.2.18/172.28.40.69: status Starting >> >> c3: 172.16.2.19/172.28.40.70: status Starting >> >> c1: 172.16.2.17/172.28.40.68: status Starting >> >> >> >> Let me know if I should go ahead and open a JIRA >> >> >> >> >> >> Thanks >> >> >> >> Martin >> >> >> >> >> >> -- >> >> Imesh Gunaratne >> >> >> >> Technical Lead, WSO2 >> >> Committer & PMC Member, Apache Stratos >> >> >> >> >> >> -- >> >> Imesh Gunaratne >> >> >> >> Technical Lead, WSO2 >> >> Committer & PMC Member, Apache Stratos >> >> >> >> >> >> -- >> >> -- >> Lahiru Sandaruwan >> >> Committer and PMC member, Apache Stratos, >> Senior Software Engineer, >> WSO2 Inc., http://wso2.com >> >> lean.enterprise.middleware >> >> phone: +94773325954 >> email: lahi...@wso2.com blog: http://lahiruwrites.blogspot.com/ >> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146 >> >> >> >> >> >> >> >> -- >> >> -- >> Lahiru Sandaruwan >> >> Committer and PMC member, Apache Stratos, >> Senior Software Engineer, >> WSO2 Inc., http://wso2.com >> >> lean.enterprise.middleware >> >> phone: +94773325954 >> email: lahi...@wso2.com blog: http://lahiruwrites.blogspot.com/ >> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146 >> >> >> >> >> >> >> >> -- >> >> -- >> Lahiru Sandaruwan >> >> Committer and PMC member, Apache Stratos, >> Senior Software Engineer, >> WSO2 Inc., http://wso2.com >> >> lean.enterprise.middleware >> >> phone: +94773325954 >> email: lahi...@wso2.com blog: http://lahiruwrites.blogspot.com/ >> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146 >> >> >> >> >> >> >> >> -- >> >> -- >> Lahiru Sandaruwan >> >> Committer and PMC member, Apache Stratos, >> Senior Software Engineer, >> WSO2 Inc., http://wso2.com >> >> lean.enterprise.middleware >> >> phone: +94773325954 >> email: lahi...@wso2.com blog: http://lahiruwrites.blogspot.com/ >> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146 >> >> >> >> >> >> >> >> -- >> >> -- >> Lahiru Sandaruwan >> >> Committer and PMC member, Apache Stratos, >> Senior Software Engineer, >> WSO2 Inc., http://wso2.com >> >> lean.enterprise.middleware >> >> phone: +94773325954 >> email: lahi...@wso2.com blog: http://lahiruwrites.blogspot.com/ >> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146 >> >> >> > > > > -- > -- > Lahiru Sandaruwan > Committer and PMC member, Apache Stratos, > Senior Software Engineer, > WSO2 Inc., http://wso2.com > lean.enterprise.middleware > > phone: +94773325954 > email: lahi...@wso2.com blog: http://lahiruwrites.blogspot.com/ > linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146 > > -- -- Lahiru Sandaruwan Committer and PMC member, Apache Stratos, Senior Software Engineer, WSO2 Inc., http://wso2.com lean.enterprise.middleware phone: +94773325954 email: lahi...@wso2.com blog: http://lahiruwrites.blogspot.com/ linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
