Hi Martin,
On Wed, Apr 1, 2015 at 12:34 AM, Martin Eppel (meppel) <mep...@cisco.com>
wrote:
> Hi Lahiru,
>
>
>
> I tested the fix and noticed the following issue:
>
>
>
> From the 4 cartridges 3 instances go Active, however one instance got
> stuck in INITIALIZED state.
>
>
>
> One of the things I noticed while the member was stuck in INITIALIZED
> state is that the system received 2 active events for the same membered:
>
>
>
> *TID: [0] [STRATOS] [2015-03-31 18:10:27,645] INFO
> {org.apache.stratos.cloud.controller.messaging.publisher.TopologyEventPublisher}
> - Publishing member activated event: [service-name] c2 [cluster-id]
> subscription-G1-G2-G3.c2-1x1.c2.domain [cluster-instance-id]
> subscription-G1-G2-G3-1 [member-id]
> subscription-G1-G2-G3.c2-1x1.c2.domainf939e8dc-e0cc-408a-8343-ac86e479549f
> [network-partition-id] RegionOne [partition-id] whole-region*
>
> *TID: [0] [STRATOS] [2015-03-31 18:10:31,059] INFO
> {org.apache.stratos.cloud.controller.messaging.publisher.TopologyEventPublisher}
> - Publishing member activated event: [service-name] c2 [cluster-id]
> subscription-G1-G2-G3.c2-1x1.c2.domain [cluster-instance-id]
> subscription-G1-G2-G3-1 [member-id]
> subscription-G1-G2-G3.c2-1x1.c2.domainf939e8dc-e0cc-408a-8343-ac86e479549f
> [network-partition-id] RegionOne [partition-id] whole-region*
>
>
>
> As another note, but I can’t be sure since the log got overwritten is that
> it appeared to me that at one point (while the cartridge was stuck in
> INITIALIZED state is that the cartridge alias of 2 cartridges was the same
> (~71, ~70: 71: CARTRIDGE_ALIAS=c2-1x1), but I can’t say for sure since when
> I double checked a new instance of the cartridge was created (which went
> ACTIVE).
>
Can i assume this occurred when two cartridges use same alias? May be we
have to put a validation on Application as we cannot use same alias in two
cartridges.
If so, could you please try to reproduce this with different aliases?
Thanks.
>
>
> Eventually the system terminated the “faulty” member and the cartridge
> came up and went into active state:
>
>
>
> *“Pending state of member expired, member will be moved to obsolete list.
> [pending member]
> subscription-G1-G2-G3.c2-1x0.c2.domainfe9a30cf-fffc-4a6c-9b3b-f419498895b9*
> ”
>
> …
>
> “*Member activated: [service] c2 [cluster]
> subscription-G1-G2-G3.c2-1x0.c2.domain [member]
> subscription-G1-G2-G3.c2-1x0.c2.domain44a2ae3c-9968-41de-9e10-7212ef3788c2*
> ”
>
>
>
> Cartridge alias after all 4 cartridges went ACTIVE:
>
>
>
> 70: CARTRIDGE_ALIAS=c2-1x0
>
> 71: CARTRIDGE_ALIAS=c2-1x1
>
>
>
> 69: CARTRIDGE_ALIAS=c1-1x0
>
> 68: CARTRIDGE_ALIAS=c1-1x1
>
>
>
> I attached the logs and artefacts
>
>
>
> Thanks
>
>
>
> Martin
>
>
>
> *From:* Lahiru Sandaruwan [mailto:lahi...@wso2.com]
> *Sent:* Tuesday, March 31, 2015 2:24 AM
>
> *To:* dev
> *Subject:* Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not
> going active ... was RE: Stratos 4.1.0 : Issue when deploying application
> with single group and 3 cartridges, no dependencies ...
>
>
>
> Hi Martin,
>
>
>
> I have reproduced the issue and pushed a fix. Please verify.
>
>
>
> Thanks.
>
>
>
> On Tue, Mar 31, 2015 at 8:32 AM, Lahiru Sandaruwan <lahi...@wso2.com>
> wrote:
>
> Hi Martin,
>
>
>
> On Tue, Mar 31, 2015 at 5:41 AM, Martin Eppel (meppel) <mep...@cisco.com>
> wrote:
>
> Hi Lahiru,
>
>
>
> I tested a new scenario, where I have 2 groups with cartridges and no
> start up dependencies and I see again the issue with cartridge instances
> being stuck in “started” state. I also noticed the following log message:
>
> *“Artifact repositories not found for application signup, artifact updated
> event not sent: [application-id] subscription-G1-G2-G3*”.
>
>
>
>
>
> Will have a look.
>
>
>
> Thanks.
>
> Please find attached the artefacts and log messages,
>
>
>
> Thanks
>
>
>
> Martin
>
>
>
> *From:* Martin Eppel (meppel)
> *Sent:* Monday, March 30, 2015 2:36 PM
>
>
> *To:* dev@stratos.apache.org
> *Subject:* RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not
> going active ... was RE: Stratos 4.1.0 : Issue when deploying application
> with single group and 3 cartridges, no dependencies ...
>
>
>
> Hi Lahiru,
>
>
>
> I tested both scenarios,
>
>
>
> 1. Single cartridge in single group and it worked, cartridge
> instance starts up and goes active,
>
> 2. Multiple cartridges with startup dependencies :
>
> All instances are started up (in right sequence) and go active
>
> JIRA https://issues.apache.org/jira/browse/STRATOS-1292[image:
> http://wikicentral.cisco.com/images/icons/linkext7.gif]
> <https://issues.apache.org/jira/browse/STRATOS-1292> is being tested and
> verified
>
>
>
> Removing / Un-deploying the application caused an issue as one of the VMs
> did not get terminated (throwing exceptions in the log). I am not sure if
> this is already a known issue related to to the ongoing discussion of
> application removal / un-deployment.
>
> Just in case, I zipped up the logs and attached it to the email thread
> (single-group-multiple-cartridges-startup.zip). Let me know if the
> un-deployment issue is the same already discussed or if this is a new issue
> ?
>
>
>
> Same error occurred in the scenario where multiple cartridges were started
> up (all go active) and then application was removed, I attached the log for
> this scenario as well
> (wscarbon-application-single-group-multiple-cartridges-no-startup-remval-error.log.zip)
>
>
>
>
>
> Thanks
>
>
>
> Martin
>
>
>
>
>
> *From:* Martin Eppel (meppel)
> *Sent:* Monday, March 30, 2015 7:47 AM
> *To:* dev@stratos.apache.org
> *Subject:* RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not
> going active ... was RE: Stratos 4.1.0 : Issue when deploying application
> with single group and 3 cartridges, no dependencies ...
>
>
>
> Thanks Lahiru,
>
>
>
> Will do and let you know
>
>
>
> Martin
>
>
>
> *From:* Lahiru Sandaruwan [mailto:lahi...@wso2.com <lahi...@wso2.com>]
> *Sent:* Monday, March 30, 2015 3:32 AM
> *To:* dev
> *Subject:* Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not
> going active ... was RE: Stratos 4.1.0 : Issue when deploying application
> with single group and 3 cartridges, no dependencies ...
>
>
>
> Hi Martin,
>
>
>
> I was able to reproduce the issue and committed a fix.
>
> Could you please verify?
>
>
>
> Thanks.
>
>
>
> On Sun, Mar 29, 2015 at 5:43 PM, Lahiru Sandaruwan <lahi...@wso2.com>
> wrote:
>
> Hi Martin,
>
>
>
> According to the logs attached, it seems the cartridge agent waits for the
> artifact updated event, and somehow it is not sent.
>
>
>
> Looking into it.
>
>
>
> Thanks.
>
>
>
> On Sat, Mar 28, 2015 at 6:48 AM, Imesh Gunaratne <im...@apache.org> wrote:
>
> Hi Martin,
>
>
>
> Thanks for the quick response on this!
>
> I will use these artifacts to re-produce the problem locally and will let
> you know the results soon.
>
>
>
> Thanks
>
>
>
> On Sat, Mar 28, 2015 at 4:19 AM, Martin Eppel (meppel) <mep...@cisco.com>
> wrote:
>
> Hi Imesh,
>
>
>
> As discussed please find attached the “real” artefacts (application,
> cartridge group, cartridge):
>
>
>
> Attached it as well to the jira
>
>
>
> Thanks
>
>
>
> Martin
>
>
>
>
>
> *From:* Martin Eppel (meppel)
> *Sent:* Friday, March 27, 2015 11:37 AM
> *To:* dev@stratos.apache.org
> *Subject:* RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not
> going active ... was RE: Stratos 4.1.0 : Issue when deploying application
> with single group and 3 cartridges, no dependencies ...
>
>
>
> Hi Imesh,
>
>
>
> Originally I tested with 3 cartridges but then realized that it also
> occurs with a single cartridge in a group – see also the JIRA for the
> respective notes and artefacts (single group, single cartridge)
>
>
>
> Thanks
>
>
>
> Martin
>
>
>
> *From:* Imesh Gunaratne [mailto:im...@apache.org <im...@apache.org>]
> *Sent:* Friday, March 27, 2015 11:34 AM
> *To:* dev
> *Subject:* Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not
> going active ... was RE: Stratos 4.1.0 : Issue when deploying application
> with single group and 3 cartridges, no dependencies ...
>
>
>
> Hi Martin,
>
>
>
> Thanks for reporting this problem. You have mentioned that there are three
> cartridges in the application but in the given application.json I only can
> see one cartridge. Are we missing anything here? Will it be possible for
> you to attach the complete set of Stratos artifacts that you are using in a
> zip file?
>
>
>
> Thanks
>
> Imesh
>
>
>
> On Fri, Mar 27, 2015 at 11:51 PM, Martin Eppel (meppel) <mep...@cisco.com>
> wrote:
>
> Forgot to mention, this is the same issue as reported in JIRA
> https://issues.apache.org/jira/browse/STRATOS-1292
>
>
>
> Thanks
>
>
>
> Martin
>
>
>
> *From:* Martin Eppel (meppel)
> *Sent:* Friday, March 27, 2015 10:52 AM
> *To:* dev@stratos.apache.org
> *Subject:* Issue (?): Stratos 4.1.0 - Cartridges in a group are not going
> active ... was RE: Stratos 4.1.0 : Issue when deploying application with
> single group and 3 cartridges, no dependencies ...
>
>
>
> I retested with a similar scenario (single cartridges in group) with
> debug logs turned on, please see logs attached.
>
>
>
> Cartridge is stuck in “Starting” state
>
>
>
> Thanks
>
>
>
> Martin
>
>
>
> *From:* Martin Eppel (meppel)
> *Sent:* Monday, March 23, 2015 7:15 PM
> *To:* dev@stratos.apache.org
> *Subject:* Stratos 4.1.0 : Issue when deploying application with single
> group and 3 cartridges, no dependencies ...
>
>
>
> Hi,
>
>
>
> I see an issue when I subscribe a single group with 3 cartridges, no
> dependencies – they never go active instead they all are stuck in
> “Starting”. I attached the server log and the cartridge-agent logs
>
>
>
>
>
> Member status:
>
> subscription-c1-c2-c3: applicationInstances 1, groupInstances 1,
> clusterInstances 3, members 3 (Starting 3)
>
> c2: 172.16.2.18/172.28.40.69: status Starting
>
> c3: 172.16.2.19/172.28.40.70: status Starting
>
> c1: 172.16.2.17/172.28.40.68: status Starting
>
>
>
> Let me know if I should go ahead and open a JIRA
>
>
>
>
>
> Thanks
>
>
>
> Martin
>
>
>
>
>
> --
>
> Imesh Gunaratne
>
>
>
> Technical Lead, WSO2
>
> Committer & PMC Member, Apache Stratos
>
>
>
>
>
> --
>
> Imesh Gunaratne
>
>
>
> Technical Lead, WSO2
>
> Committer & PMC Member, Apache Stratos
>
>
>
>
>
> --
>
> --
> Lahiru Sandaruwan
>
> Committer and PMC member, Apache Stratos,
> Senior Software Engineer,
> WSO2 Inc., http://wso2.com
>
> lean.enterprise.middleware
>
> phone: +94773325954
> email: lahi...@wso2.com blog: http://lahiruwrites.blogspot.com/
> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>
>
>
>
>
>
>
> --
>
> --
> Lahiru Sandaruwan
>
> Committer and PMC member, Apache Stratos,
> Senior Software Engineer,
> WSO2 Inc., http://wso2.com
>
> lean.enterprise.middleware
>
> phone: +94773325954
> email: lahi...@wso2.com blog: http://lahiruwrites.blogspot.com/
> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>
>
>
>
>
>
>
> --
>
> --
> Lahiru Sandaruwan
>
> Committer and PMC member, Apache Stratos,
> Senior Software Engineer,
> WSO2 Inc., http://wso2.com
>
> lean.enterprise.middleware
>
> phone: +94773325954
> email: lahi...@wso2.com blog: http://lahiruwrites.blogspot.com/
> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>
>
>
>
>
>
>
> --
>
> --
> Lahiru Sandaruwan
>
> Committer and PMC member, Apache Stratos,
> Senior Software Engineer,
> WSO2 Inc., http://wso2.com
>
> lean.enterprise.middleware
>
> phone: +94773325954
> email: lahi...@wso2.com blog: http://lahiruwrites.blogspot.com/
> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>
>
>
--
--
Lahiru Sandaruwan
Committer and PMC member, Apache Stratos,
Senior Software Engineer,
WSO2 Inc., http://wso2.com
lean.enterprise.middleware
phone: +94773325954
email: lahi...@wso2.com blog: http://lahiruwrites.blogspot.com/
linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
