Hi Martin, I added some info logs in efe67ad99f9fca80d4ba66c01e9b642c57838dc3. Could you please resend the logs with this change applied?. I tried to reproduce this issue but was not success full. Could you please send us debugs logs so we can get a better understanding about the exact issue.
Additionally is this an intermittent issue or occurring frequently? On Fri, Jun 5, 2015 at 11:41 AM, Udara Liyanage <[email protected]> wrote: > Hi, > > This might be possible if AS did not receive member activated event > published by CC. Is it possible to enable debug logs if this is > reproducible. > Or else I can add an INFO logs and commit. > > > On Fri, Jun 5, 2015 at 9:11 AM, Udara Liyanage <[email protected]> wrote: > >> Hi, >> >> >> For the first issue you have mentioned, the particular member is >> activated, but it is still identified as an obsolete member and is being >> marked to be terminated since pending time expired. Does that mean member >> is still in Obsolete list even though it is being activated? >> >> //member started >> TID: [0] [STRATOS] [2015-06-04 19:53:04,706] INFO >> {org.apache.stratos.autoscaler.context.cluster.ClusterContext} - Member >> stat context has been added: [application] g-sc-G12-1 [cluster] >> g-sc-G12-1.c1-0x0.c1.domain [clusterInstanceContext] g-sc-G12-1-1 >> [partitionContext] whole-region [member-id] >> g-sc-G12-1.c1-0x0.c1.domainb0aa0188-49f1-47f6-a040-c2eab4acb5b1 >> >> //member activated >> TID: [0] [STRATOS] [2015-06-04 19:56:00,907] INFO >> {org.apache.stratos.cloud.controller.messaging.publisher.TopologyEventPublisher} >> - Publishing member activated event: [service-name] c1 [cluster-id] >> g-sc-G12-1.c1-0x0.c1.domain [cluster-instance-id] g-sc-G12-1-1 [member-id] >> g-sc-G12-1.c1-0x0.c1.domainb0aa0188-49f1-47f6-a040-c2eab4acb5b1 >> [network-partition-id] RegionOne [partition-id] whole-region >> TID: [0] [STRATOS] [2015-06-04 19:56:00,916] INFO >> {org.apache.stratos.messaging.message.processor.topology.MemberActivatedMessageProcessor} >> - Member activated: [service] c1 [cluster] g-sc-G12-1.c1-0x0.c1.domain >> [member] g-sc-G12-1.c1-0x0.c1.domainb0aa0188-49f1-47f6-a040-c2eab4acb5b1 >> >> //after 15 minutes ---member is still in pending state, pending timeout >> expired >> TID: [0] [STRATOS] [2015-06-04 20:08:04,713] INFO >> {org.apache.stratos.autoscaler.context.partition.ClusterLevelPartitionContext$PendingMemberWatcher} >> - Pending state of member expired, member will be moved to obsolete list. >> [pending member] >> g-sc-G12-1.c1-0x0.c1.domainb0aa0188-49f1-47f6-a040-c2eab4acb5b1 [expiry >> time] 900000 [cluster] g-sc-G12-1.c1-0x0.c1.domain [cluster instance] null >> >> On Fri, Jun 5, 2015 at 5:14 AM, Martin Eppel (meppel) <[email protected]> >> wrote: >> >>> Hi, >>> >>> >>> >>> I am running into a scenario where application un-deployment fails >>> (using stratos with latest commit >>> b1b6bca3f99b6127da24c9af0a6b20faff2907be). >>> >>> >>> >>> For application structure see [1.], (debug enabled) wso2carbon.log, >>> application.json, cartridge-group.json, deployment-policy, auto-scaling >>> policies see attached zip file. >>> >>> >>> >>> *It is noteworthy, that while the application is running the following >>> log statements /exceptions are observed:* >>> >>> >>> >>> *…* >>> >>> *Member is in the wrong list and it is removed from active members list: >>> g-sc-G12-1.c1-0x0.c1.domainb0aa0188-49f1-47f6-a040-c2eab4acb5b1* >>> >>> *…* >>> >>> *TID: [0] [STRATOS] [2015-06-04 20:11:03,425] ERROR >>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Cannot terminate >>> instance* >>> >>> *…* >>> >>> *// **after receiving the application undeploy event:* >>> >>> *[2015-06-04 20:12:39,465] INFO >>> {org.apache.stratos.autoscaler.services.impl.AutoscalerServiceImpl} - >>> Application undeployment process started: [application-id] g-sc-G12-1* >>> >>> *// **a new instance is being started up* >>> >>> *…* >>> >>> *[2015-06-04 20:13:13,445] INFO >>> {org.apache.stratos.cloud.controller.services.impl.InstanceCreator} - >>> Instance started successfully: [cartridge-type] c2 [cluster-id] >>> g-sc-G12-1.c2-1x0.c2.domain [instance-id] >>> RegionOne/5d4699f7-b00b-42eb-b565-b48fc8f20407* >>> >>> >>> >>> *// Also noteworthy seems the following warning which is seen repeatedly >>> in the logs:* >>> >>> *ReadWriteLock} - System warning! Trying to release a lock which has >>> not been taken by the same thread: [lock-name]* >>> >>> >>> >>> >>> >>> [1.] Application structure >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >> >> >> >> -- >> >> Udara Liyanage >> Software Engineer >> WSO2, Inc.: http://wso2.com >> lean. enterprise. middleware >> >> web: http://udaraliyanage.wordpress.com >> phone: +94 71 443 6897 >> > > > > -- > > Udara Liyanage > Software Engineer > WSO2, Inc.: http://wso2.com > lean. enterprise. middleware > > web: http://udaraliyanage.wordpress.com > phone: +94 71 443 6897 > -- Udara Liyanage Software Engineer WSO2, Inc.: http://wso2.com lean. enterprise. middleware web: http://udaraliyanage.wordpress.com phone: +94 71 443 6897
