Hi, I have multiple VMs running on vCloud-Director through Stratos & all of them are in ACTIVE state for more than 12 hours. I unsubscribe one of the VM which is not showing in output of list-subscribed-cartridges but the cartridge-vm is still up & running on vCloud-Director.
Attached is the section of wso2carbon.log after unsubscription of cartridge. Regards Manas
TID: [0] [STRATOS] [2014-11-11 07:39:25,774] INFO {org.wso2.carbon.databridge.core.DataBridge} - admin connected {org.wso2.carbon.databridge.core.DataBridge} TID: [0] [STRATOS] [2014-11-11 07:41:07,991] INFO {org.wso2.carbon.databridge.core.DataBridge} - admin connected {org.wso2.carbon.databridge.core.DataBridge} TID: [0] [STRATOS] [2014-11-11 07:42:17,682] INFO {org.wso2.carbon.databridge.core.DataBridge} - admin connected {org.wso2.carbon.databridge.core.DataBridge} TID: [0] [STRATOS] [2014-11-11 07:43:11,623] INFO {org.wso2.carbon.databridge.core.DataBridge} - admin connected {org.wso2.carbon.databridge.core.DataBridge} TID: [0] [STRATOS] [2014-11-11 07:47:26,159] INFO {org.wso2.carbon.databridge.core.DataBridge} - admin connected {org.wso2.carbon.databridge.core.DataBridge} TID: [0] [STRATOS] [2014-11-11 07:48:36,972] INFO {org.wso2.carbon.databridge.core.DataBridge} - admin connected {org.wso2.carbon.databridge.core.DataBridge} TID: [0] [STRATOS] [2014-11-11 07:48:54,834] INFO {org.wso2.carbon.databridge.core.DataBridge} - admin connected {org.wso2.carbon.databridge.core.DataBridge} TID: [0] [STRATOS] [2014-11-11 07:52:19,528] INFO {org.wso2.carbon.databridge.core.DataBridge} - admin connected {org.wso2.carbon.databridge.core.DataBridge} TID: [0] [STRATOS] [2014-11-11 07:54:11,970] INFO {org.apache.stratos.manager.publisher.InstanceNotificationPublisher} - Publishing Instance Cleanup Event: [cluster] uag.uag.domain {org.apache.stratos.manager.publisher.InstanceNotificationPublisher} TID: [0] [STRATOS] [2014-11-11 07:54:11,973] INFO {org.apache.stratos.manager.behaviour.CartridgeMgtBehaviour} - Instance Cleanup Notification sent to Cluster: uag.uag.domain {org.apache.stratos.manager.behaviour.CartridgeMgtBehaviour} TID: [0] [STRATOS] [2014-11-11 07:54:11,978] INFO {org.apache.stratos.cloud.controller.topic.instance.status.InstanceStatusEventMessageDelegator} - Instance status event message received from queue: org.apache.stratos.messaging.event.instance.status.InstanceMaintenanceModeEvent {org.apache.stratos.cloud.controller.topic.instance.status.InstanceStatusEventMessageDelegator} TID: [0] [STRATOS] [2014-11-11 07:54:11,978] INFO {org.apache.stratos.manager.listener.InstanceStatusListener} - Instance status message received {org.apache.stratos.manager.listener.InstanceStatusListener} TID: [0] [STRATOS] [2014-11-11 07:54:11,980] INFO {org.apache.stratos.manager.listener.InstanceStatusListener} - Event class name: org.apache.stratos.messaging.event.instance.status.InstanceMaintenanceModeEvent {org.apache.stratos.manager.listener.InstanceStatusListener} TID: [0] [STRATOS] [2014-11-11 07:54:11,981] INFO {org.apache.stratos.cloud.controller.topology.TopologyBuilder} - member maintenance mode event adding status started {org.apache.stratos.cloud.controller.topology.TopologyBuilder} TID: [0] [STRATOS] [2014-11-11 07:54:11,986] INFO {org.apache.stratos.manager.listener.InstanceStatusListener} - Instance status message received {org.apache.stratos.manager.listener.InstanceStatusListener} TID: [0] [STRATOS] [2014-11-11 07:54:11,986] INFO {org.apache.stratos.manager.listener.InstanceStatusListener} - Event class name: org.apache.stratos.messaging.event.instance.status.InstanceReadyToShutdownEvent {org.apache.stratos.manager.listener.InstanceStatusListener} TID: [0] [STRATOS] [2014-11-11 07:54:12,010] INFO {org.apache.stratos.cloud.controller.topology.TopologyEventPublisher} - Publishing Maintenance mode event: [service] uag [cluster] uag.uag.domain [network-partition] N1 [partition] RegionOne-Core [member] uag.uag.domain38306e0a-9295-4031-ba2b-825128db9dab [groupId] null {org.apache.stratos.cloud.controller.topology.TopologyEventPublisher} TID: [0] [STRATOS] [2014-11-11 07:54:12,013] INFO {org.apache.stratos.messaging.message.processor.topology.MemberMaintenanceModeProcessor} - Member updated as In_Maintenance: [service] uag [cluster] uag.uag.domain [member] uag.uag.domain38306e0a-9295-4031-ba2b-825128db9dab {org.apache.stratos.messaging.message.processor.topology.MemberMaintenanceModeProcessor} TID: [0] [STRATOS] [2014-11-11 07:54:12,014] WARN {org.apache.stratos.messaging.message.processor.topology.MemberMaintenanceModeProcessor} - Member already updated as In_Maintenance: [service] uag [cluster] uag.uag.domain [member] uag.uag.domain38306e0a-9295-4031-ba2b-825128db9dab {org.apache.stratos.messaging.message.processor.topology.MemberMaintenanceModeProcessor} TID: [0] [STRATOS] [2014-11-11 07:54:12,015] INFO {org.apache.stratos.cloud.controller.topic.instance.status.InstanceStatusEventMessageDelegator} - Instance status event message received from queue: org.apache.stratos.messaging.event.instance.status.InstanceReadyToShutdownEvent {org.apache.stratos.cloud.controller.topic.instance.status.InstanceStatusEventMessageDelegator} TID: [0] [STRATOS] [2014-11-11 07:54:12,015] WARN {org.apache.stratos.messaging.message.processor.topology.MemberMaintenanceModeProcessor} - Member already updated as In_Maintenance: [service] uag [cluster] uag.uag.domain [member] uag.uag.domain38306e0a-9295-4031-ba2b-825128db9dab {org.apache.stratos.messaging.message.processor.topology.MemberMaintenanceModeProcessor} TID: [0] [STRATOS] [2014-11-11 07:54:12,031] INFO {org.apache.stratos.cloud.controller.topology.TopologyBuilder} - Member Ready to shut down event adding status started {org.apache.stratos.cloud.controller.topology.TopologyBuilder} TID: [0] [STRATOS] [2014-11-11 07:54:12,031] INFO {org.apache.stratos.cloud.controller.topology.TopologyEventPublisher} - Publishing cluster maintenance mode event: [service] uag [cluster] uag.uag.domain {org.apache.stratos.cloud.controller.topology.TopologyEventPublisher} TID: [0] [STRATOS] [2014-11-11 07:54:12,034] INFO {org.apache.stratos.messaging.message.processor.topology.ClusterMaintenanceModeMessageProcessor} - Cluster updated as maintenance mode: Cluster [serviceName=uag, clusterId=uag.uag.domain, autoscalePolicyName=economyPolicy, deploymentPolicyName=static-1-Core, hostNames=[uag.qmog.cisco.com], tenantRange=null, isLbCluster=false, properties={}] {org.apache.stratos.messaging.message.processor.topology.ClusterMaintenanceModeMessageProcessor} TID: [0] [STRATOS] [2014-11-11 07:54:12,034] INFO {org.apache.stratos.messaging.message.processor.topology.ClusterMaintenanceModeMessageProcessor} - Cluster updated as maintenance mode: Cluster [serviceName=uag, clusterId=uag.uag.domain, autoscalePolicyName=economyPolicy, deploymentPolicyName=static-1-Core, hostNames=[uag.qmog.cisco.com], tenantRange=null, isLbCluster=false, properties={}] {org.apache.stratos.messaging.message.processor.topology.ClusterMaintenanceModeMessageProcessor} TID: [0] [STRATOS] [2014-11-11 07:54:12,036] INFO {org.apache.stratos.autoscaler.message.receiver.topology.AutoscalerTopologyEventReceiver} - Event received: ClusterMaintenanceModeEvent [serviceName=uag, clusterStatus=In_Maintenance] {org.apache.stratos.autoscaler.message.receiver.topology.AutoscalerTopologyEventReceiver} TID: [0] [STRATOS] [2014-11-11 07:54:12,037] INFO {org.apache.stratos.messaging.message.processor.topology.ClusterMaintenanceModeMessageProcessor} - Cluster updated as maintenance mode: Cluster [serviceName=uag, clusterId=uag.uag.domain, autoscalePolicyName=economyPolicy, deploymentPolicyName=static-1-Core, hostNames=[uag.qmog.cisco.com], tenantRange=null, isLbCluster=false, properties={}] {org.apache.stratos.messaging.message.processor.topology.ClusterMaintenanceModeMessageProcessor} TID: [0] [STRATOS] [2014-11-11 07:54:12,039] INFO {org.apache.stratos.manager.behaviour.CartridgeMgtBehaviour} - Unregistered service cluster, domain uag.uag.domain {org.apache.stratos.manager.behaviour.CartridgeMgtBehaviour} TID: [0] [STRATOS] [2014-11-11 07:54:12,052] INFO {org.apache.stratos.cloud.controller.topology.TopologyEventPublisher} - Publishing member Ready to shut down event: [service] uag [cluster] uag.uag.domain [network-partition] N1 [partition] RegionOne-Core [member] uag.uag.domain38306e0a-9295-4031-ba2b-825128db9dab [groupId] null {org.apache.stratos.cloud.controller.topology.TopologyEventPublisher} TID: [0] [STRATOS] [2014-11-11 07:54:12,054] INFO {org.apache.stratos.messaging.message.processor.topology.MemberReadyToShutdownMessageProcessor} - Member updated as Ready to shutdown: [service] uag [cluster] uag.uag.domain [member] uag.uag.domain38306e0a-9295-4031-ba2b-825128db9dab {org.apache.stratos.messaging.message.processor.topology.MemberReadyToShutdownMessageProcessor} TID: [0] [STRATOS] [2014-11-11 07:54:12,055] WARN {org.apache.stratos.messaging.message.processor.topology.MemberReadyToShutdownMessageProcessor} - Member already updated as Ready to Shutdown: [service] uag [cluster] uag.uag.domain [member] uag.uag.domain38306e0a-9295-4031-ba2b-825128db9dab {org.apache.stratos.messaging.message.processor.topology.MemberReadyToShutdownMessageProcessor} TID: [0] [STRATOS] [2014-11-11 07:54:12,056] WARN {org.apache.stratos.messaging.message.processor.topology.MemberReadyToShutdownMessageProcessor} - Member already updated as Ready to Shutdown: [service] uag [cluster] uag.uag.domain [member] uag.uag.domain38306e0a-9295-4031-ba2b-825128db9dab {org.apache.stratos.messaging.message.processor.topology.MemberReadyToShutdownMessageProcessor} TID: [0] [STRATOS] [2014-11-11 07:54:12,057] INFO {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient} - Terminating instance via cloud controller: [member] uag.uag.domain38306e0a-9295-4031-ba2b-825128db9dab {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient} TID: [0] [STRATOS] [2014-11-11 07:54:12,068] INFO {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} - Starting to terminate an instance with member id : uag.uag.domain38306e0a-9295-4031-ba2b-825128db9dab in partition id: RegionOne-Core of cluster id: uag.uag.domain and of cartridge type: uag {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} TID: [0] [STRATOS] [2014-11-11 07:54:12,069] INFO {org.apache.stratos.autoscaler.message.receiver.topology.AutoscalerTopologyEventReceiver} - Member is terminated and removed from the active members list: [member] uag.uag.domain38306e0a-9295-4031-ba2b-825128db9dab [partition] RegionOne-Core [cluster] uag.uag.domain {org.apache.stratos.autoscaler.message.receiver.topology.AutoscalerTopologyEventReceiver} TID: [0] [STRATOS] [2014-11-11 07:54:12,081] INFO {org.apache.stratos.manager.subscription.utils.CartridgeSubscriptionUtils} - Publishing tenant un-subscribed event: [tenant-id] -1234 [service] uag {org.apache.stratos.manager.subscription.utils.CartridgeSubscriptionUtils} TID: [0] [STRATOS] [2014-11-11 07:54:36,216] INFO {org.wso2.carbon.databridge.core.DataBridge} - admin connected {org.wso2.carbon.databridge.core.DataBridge} TID: [0] [STRATOS] [2014-11-11 07:54:42,041] INFO {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} - Starting to terminate an instance with member id : uag.uag.domain38306e0a-9295-4031-ba2b-825128db9dab in partition id: RegionOne-Core of cluster id: uag.uag.domain and of cartridge type: uag {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} TID: [0] [STRATOS] [2014-11-11 07:54:12,069] INFO {org.apache.stratos.autoscaler.message.receiver.topology.AutoscalerTopologyEventReceiver} - Member is terminated and removed from the active members list: [member] uag.uag.domain38306e0a-9295-4031-ba2b-825128db9dab [partition] RegionOne-Core [cluster] uag.uag.domain {org.apache.stratos.autoscaler.message.receiver.topology.AutoscalerTopologyEventReceiver} TID: [0] [STRATOS] [2014-11-11 07:54:12,081] INFO {org.apache.stratos.manager.subscription.utils.CartridgeSubscriptionUtils} - Publishing tenant un-subscribed event: [tenant-id] -1234 [service] uag {org.apache.stratos.manager.subscription.utils.CartridgeSubscriptionUtils} TID: [0] [STRATOS] [2014-11-11 07:54:36,216] INFO {org.wso2.carbon.databridge.core.DataBridge} - admin connected {org.wso2.carbon.databridge.core.DataBridge} TID: [0] [STRATOS] [2014-11-11 07:54:42,041] INFO {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} - Starting to terminate an instance with member id : uag.uag.domain38306e0a-9295-4031-ba2b-825128db9dab in partition id: RegionOne-Core of cluster id: uag.uag.domain and of cartridge type: uag {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} TID: [0] [STRATOS] [2014-11-11 07:57:33,160] INFO {org.wso2.carbon.databridge.core.DataBridge} - admin connected {org.wso2.carbon.databridge.core.DataBridge} TID: [0] [STRATOS] [2014-11-11 08:00:14,118] ERROR {org.jclouds.http.handlers.BackoffLimitedRetryHandler} - Cannot retry after server error, command has exceeded retry limit 5: [method=public abstract java.util.SortedMap org.jclouds.vcloud.VCloudVersionsApi.getSupportedVersions()[], request=GET https://10.58.6.231/api/versions HTTP/1.1] {org.jclouds.http.handlers.BackoffLimitedRetryHandler} TID: [0] [STRATOS] [2014-11-11 08:06:16,172] ERROR {org.jclouds.http.handlers.BackoffLimitedRetryHandler} - Cannot retry after server error, command has exceeded retry limit 5: [method=public abstract java.util.SortedMap org.jclouds.vcloud.VCloudVersionsApi.getSupportedVersions()[], request=GET https://10.58.6.231/api/versions HTTP/1.1] {org.jclouds.http.handlers.BackoffLimitedRetryHandler}