One more observation: When removing the application I see the exception below ( although all members are terminated and removed),
TID: [0] [STRATOS] [2015-03-31 19:32:57,142] INFO {org.apache.stratos.cloud.controller.messaging.publisher.TopologyEventPublisher} - Publishing Cluster terminated event: [application-id] subscription-G1-G2-G3 [cluster id] subscription-G1-G2-G3.c1-1x0.c1.domain [instance-id] subscription-G1-G2-G3-1 TID: [0] [STRATOS] [2015-03-31 19:32:57,155] INFO {org.apache.stratos.autoscaler.event.receiver.topology.AutoscalerTopologyEventReceiver} - [ClusterTerminatedEvent] Received: class org.apache.stratos.messaging.event.topology.ClusterInstanceTerminatedEvent TID: [0] [STRATOS] [2015-03-31 19:32:57,155] INFO {org.apache.stratos.autoscaler.status.processor.group.GroupStatusTerminatedProcessor} - sending group instance terminated for [group] multiple-groups-no-startup-G1 [instance] subscription-G1-G2-G3-1 TID: [0] [STRATOS] [2015-03-31 19:32:57,156] ERROR {org.apache.stratos.messaging.message.receiver.topology.TopologyEventMessageDelegator} - Failed to retrieve topology event message java.lang.NullPointerException at org.apache.stratos.autoscaler.event.receiver.topology.AutoscalerTopologyEventReceiver$8.onEvent(AutoscalerTopologyEventReceiver.java:350) at org.apache.stratos.messaging.listener.EventListener.update(EventListener.java:42) at java.util.Observable.notifyObservers(Observable.java:159) at org.apache.stratos.messaging.event.EventObservable.notifyEventListeners(EventObservable.java:51) at org.apache.stratos.messaging.message.processor.topology.ClusterInstanceTerminatedProcessor.doProcess(ClusterInstanceTerminatedProcessor.java:132) at org.apache.stratos.messaging.message.processor.topology.ClusterInstanceTerminatedProcessor.process(ClusterInstanceTerminatedProcessor.java:64) at org.apache.stratos.messaging.message.processor.topology.ClusterRemovedMessageProcessor.process(ClusterRemovedMessageProcessor.java:65) at org.apache.stratos.messaging.message.processor.topology.ClusterInstanceInactivateProcessor.process(ClusterInstanceInactivateProcessor.java:73) at org.apache.stratos.messaging.message.processor.topology.ClusterInstanceActivatedProcessor.process(ClusterInstanceActivatedProcessor.java:73) at org.apache.stratos.messaging.message.processor.topology.ClusterCreatedMessageProcessor.process(ClusterCreatedMessageProcessor.java:65) at org.apache.stratos.messaging.message.processor.topology.ApplicationClustersRemovedMessageProcessor.process(ApplicationClustersRemovedMessageProcessor.java:63) at org.apache.stratos.messaging.message.processor.topology.ApplicationClustersCreatedMessageProcessor.process(ApplicationClustersCreatedMessageProcessor.java:65) at org.apache.stratos.messaging.message.processor.topology.ServiceRemovedMessageProcessor.process(ServiceRemovedMessageProcessor.java:64) at org.apache.stratos.messaging.message.processor.topology.ServiceCreatedMessageProcessor.process(ServiceCreatedMessageProcessor.java:65) at org.apache.stratos.messaging.message.processor.topology.CompleteTopologyMessageProcessor.process(CompleteTopologyMessageProcessor.java:74) at org.apache.stratos.messaging.message.processor.MessageProcessorChain.process(MessageProcessorChain.java:61) at org.apache.stratos.messaging.message.receiver.topology.TopologyEventMessageDelegator.run(TopologyEventMessageDelegator.java:73) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) From: Martin Eppel (meppel) Sent: Tuesday, March 31, 2015 12:04 PM To: dev@stratos.apache.org Subject: RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going active ... was RE: Stratos 4.1.0 : Issue when deploying application with single group and 3 cartridges, no dependencies ... Hi Lahiru, I tested the fix and noticed the following issue: From the 4 cartridges 3 instances go Active, however one instance got stuck in INITIALIZED state. One of the things I noticed while the member was stuck in INITIALIZED state is that the system received 2 active events for the same membered: TID: [0] [STRATOS] [2015-03-31 18:10:27,645] INFO {org.apache.stratos.cloud.controller.messaging.publisher.TopologyEventPublisher} - Publishing member activated event: [service-name] c2 [cluster-id] subscription-G1-G2-G3.c2-1x1.c2.domain [cluster-instance-id] subscription-G1-G2-G3-1 [member-id] subscription-G1-G2-G3.c2-1x1.c2.domainf939e8dc-e0cc-408a-8343-ac86e479549f [network-partition-id] RegionOne [partition-id] whole-region TID: [0] [STRATOS] [2015-03-31 18:10:31,059] INFO {org.apache.stratos.cloud.controller.messaging.publisher.TopologyEventPublisher} - Publishing member activated event: [service-name] c2 [cluster-id] subscription-G1-G2-G3.c2-1x1.c2.domain [cluster-instance-id] subscription-G1-G2-G3-1 [member-id] subscription-G1-G2-G3.c2-1x1.c2.domainf939e8dc-e0cc-408a-8343-ac86e479549f [network-partition-id] RegionOne [partition-id] whole-region As another note, but I can’t be sure since the log got overwritten is that it appeared to me that at one point (while the cartridge was stuck in INITIALIZED state is that the cartridge alias of 2 cartridges was the same (~71, ~70: 71: CARTRIDGE_ALIAS=c2-1x1), but I can’t say for sure since when I double checked a new instance of the cartridge was created (which went ACTIVE). Eventually the system terminated the “faulty” member and the cartridge came up and went into active state: “Pending state of member expired, member will be moved to obsolete list. [pending member] subscription-G1-G2-G3.c2-1x0.c2.domainfe9a30cf-fffc-4a6c-9b3b-f419498895b9” … “Member activated: [service] c2 [cluster] subscription-G1-G2-G3.c2-1x0.c2.domain [member] subscription-G1-G2-G3.c2-1x0.c2.domain44a2ae3c-9968-41de-9e10-7212ef3788c2” Cartridge alias after all 4 cartridges went ACTIVE: 70: CARTRIDGE_ALIAS=c2-1x0 71: CARTRIDGE_ALIAS=c2-1x1 69: CARTRIDGE_ALIAS=c1-1x0 68: CARTRIDGE_ALIAS=c1-1x1 I attached the logs and artefacts Thanks Martin From: Lahiru Sandaruwan [mailto:lahi...@wso2.com] Sent: Tuesday, March 31, 2015 2:24 AM To: dev Subject: Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going active ... was RE: Stratos 4.1.0 : Issue when deploying application with single group and 3 cartridges, no dependencies ... Hi Martin, I have reproduced the issue and pushed a fix. Please verify. Thanks. On Tue, Mar 31, 2015 at 8:32 AM, Lahiru Sandaruwan <lahi...@wso2.com<mailto:lahi...@wso2.com>> wrote: Hi Martin, On Tue, Mar 31, 2015 at 5:41 AM, Martin Eppel (meppel) <mep...@cisco.com<mailto:mep...@cisco.com>> wrote: Hi Lahiru, I tested a new scenario, where I have 2 groups with cartridges and no start up dependencies and I see again the issue with cartridge instances being stuck in “started” state. I also noticed the following log message: “Artifact repositories not found for application signup, artifact updated event not sent: [application-id] subscription-G1-G2-G3”. Will have a look. Thanks. Please find attached the artefacts and log messages, Thanks Martin From: Martin Eppel (meppel) Sent: Monday, March 30, 2015 2:36 PM To: dev@stratos.apache.org<mailto:dev@stratos.apache.org> Subject: RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going active ... was RE: Stratos 4.1.0 : Issue when deploying application with single group and 3 cartridges, no dependencies ... Hi Lahiru, I tested both scenarios, 1. Single cartridge in single group and it worked, cartridge instance starts up and goes active, 2. Multiple cartridges with startup dependencies : All instances are started up (in right sequence) and go active JIRA https://issues.apache.org/jira/browse/STRATOS-1292[http://wikicentral.cisco.com/images/icons/linkext7.gif] is being tested and verified Removing / Un-deploying the application caused an issue as one of the VMs did not get terminated (throwing exceptions in the log). I am not sure if this is already a known issue related to to the ongoing discussion of application removal / un-deployment. Just in case, I zipped up the logs and attached it to the email thread (single-group-multiple-cartridges-startup.zip). Let me know if the un-deployment issue is the same already discussed or if this is a new issue ? Same error occurred in the scenario where multiple cartridges were started up (all go active) and then application was removed, I attached the log for this scenario as well (wscarbon-application-single-group-multiple-cartridges-no-startup-remval-error.log.zip) Thanks Martin From: Martin Eppel (meppel) Sent: Monday, March 30, 2015 7:47 AM To: dev@stratos.apache.org<mailto:dev@stratos.apache.org> Subject: RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going active ... was RE: Stratos 4.1.0 : Issue when deploying application with single group and 3 cartridges, no dependencies ... Thanks Lahiru, Will do and let you know Martin From: Lahiru Sandaruwan [mailto:lahi...@wso2.com] Sent: Monday, March 30, 2015 3:32 AM To: dev Subject: Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going active ... was RE: Stratos 4.1.0 : Issue when deploying application with single group and 3 cartridges, no dependencies ... Hi Martin, I was able to reproduce the issue and committed a fix. Could you please verify? Thanks. On Sun, Mar 29, 2015 at 5:43 PM, Lahiru Sandaruwan <lahi...@wso2.com<mailto:lahi...@wso2.com>> wrote: Hi Martin, According to the logs attached, it seems the cartridge agent waits for the artifact updated event, and somehow it is not sent. Looking into it. Thanks. On Sat, Mar 28, 2015 at 6:48 AM, Imesh Gunaratne <im...@apache.org<mailto:im...@apache.org>> wrote: Hi Martin, Thanks for the quick response on this! I will use these artifacts to re-produce the problem locally and will let you know the results soon. Thanks On Sat, Mar 28, 2015 at 4:19 AM, Martin Eppel (meppel) <mep...@cisco.com<mailto:mep...@cisco.com>> wrote: Hi Imesh, As discussed please find attached the “real” artefacts (application, cartridge group, cartridge): Attached it as well to the jira Thanks Martin From: Martin Eppel (meppel) Sent: Friday, March 27, 2015 11:37 AM To: dev@stratos.apache.org<mailto:dev@stratos.apache.org> Subject: RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going active ... was RE: Stratos 4.1.0 : Issue when deploying application with single group and 3 cartridges, no dependencies ... Hi Imesh, Originally I tested with 3 cartridges but then realized that it also occurs with a single cartridge in a group – see also the JIRA for the respective notes and artefacts (single group, single cartridge) Thanks Martin From: Imesh Gunaratne [mailto:im...@apache.org] Sent: Friday, March 27, 2015 11:34 AM To: dev Subject: Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going active ... was RE: Stratos 4.1.0 : Issue when deploying application with single group and 3 cartridges, no dependencies ... Hi Martin, Thanks for reporting this problem. You have mentioned that there are three cartridges in the application but in the given application.json I only can see one cartridge. Are we missing anything here? Will it be possible for you to attach the complete set of Stratos artifacts that you are using in a zip file? Thanks Imesh On Fri, Mar 27, 2015 at 11:51 PM, Martin Eppel (meppel) <mep...@cisco.com<mailto:mep...@cisco.com>> wrote: Forgot to mention, this is the same issue as reported in JIRA https://issues.apache.org/jira/browse/STRATOS-1292 Thanks Martin From: Martin Eppel (meppel) Sent: Friday, March 27, 2015 10:52 AM To: dev@stratos.apache.org<mailto:dev@stratos.apache.org> Subject: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going active ... was RE: Stratos 4.1.0 : Issue when deploying application with single group and 3 cartridges, no dependencies ... I retested with a similar scenario (single cartridges in group) with debug logs turned on, please see logs attached. Cartridge is stuck in “Starting” state Thanks Martin From: Martin Eppel (meppel) Sent: Monday, March 23, 2015 7:15 PM To: dev@stratos.apache.org<mailto:dev@stratos.apache.org> Subject: Stratos 4.1.0 : Issue when deploying application with single group and 3 cartridges, no dependencies ... Hi, I see an issue when I subscribe a single group with 3 cartridges, no dependencies – they never go active instead they all are stuck in “Starting”. I attached the server log and the cartridge-agent logs Member status: subscription-c1-c2-c3: applicationInstances 1, groupInstances 1, clusterInstances 3, members 3 (Starting 3) c2: 172.16.2.18/172.28.40.69<http://172.16.2.18/172.28.40.69>: status Starting c3: 172.16.2.19/172.28.40.70<http://172.16.2.19/172.28.40.70>: status Starting c1: 172.16.2.17/172.28.40.68<http://172.16.2.17/172.28.40.68>: status Starting Let me know if I should go ahead and open a JIRA Thanks Martin -- Imesh Gunaratne Technical Lead, WSO2 Committer & PMC Member, Apache Stratos -- Imesh Gunaratne Technical Lead, WSO2 Committer & PMC Member, Apache Stratos -- -- Lahiru Sandaruwan Committer and PMC member, Apache Stratos, Senior Software Engineer, WSO2 Inc., http://wso2.com lean.enterprise.middleware phone: +94773325954<tel:%2B94773325954> email: lahi...@wso2.com<mailto:lahi...@wso2.com> blog: http://lahiruwrites.blogspot.com/ linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146 -- -- Lahiru Sandaruwan Committer and PMC member, Apache Stratos, Senior Software Engineer, WSO2 Inc., http://wso2.com lean.enterprise.middleware phone: +94773325954<tel:%2B94773325954> email: lahi...@wso2.com<mailto:lahi...@wso2.com> blog: http://lahiruwrites.blogspot.com/ linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146 -- -- Lahiru Sandaruwan Committer and PMC member, Apache Stratos, Senior Software Engineer, WSO2 Inc., http://wso2.com lean.enterprise.middleware phone: +94773325954<tel:%2B94773325954> email: lahi...@wso2.com<mailto:lahi...@wso2.com> blog: http://lahiruwrites.blogspot.com/ linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146 -- -- Lahiru Sandaruwan Committer and PMC member, Apache Stratos, Senior Software Engineer, WSO2 Inc., http://wso2.com lean.enterprise.middleware phone: +94773325954 email: lahi...@wso2.com<mailto:lahi...@wso2.com> blog: http://lahiruwrites.blogspot.com/ linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146