[jira] [Commented] (CLOUDSTACK-4740) Some vSphere VMs are shutdown when ACS is restarted
[ https://issues.apache.org/jira/browse/CLOUDSTACK-4740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13852807#comment-13852807 ] Damodar Reddy T commented on CLOUDSTACK-4740: - Hi ilya musayev, I tried to re produce this use case with the following set up and not able to re produce it. 1. ESXi 5.1 and ACS 4.3 2. launched 3 to 4 vms 3. Added a fire wall(iptable rule) in MS machine not to access my vSphere host. now restarted the management server and all my VMs(including system vms) are in Running state only. and once I removed the firewall restriction on MS machine all went fine. Is this the correct way to re produce this use case? If not can you please share re producible steps so that I will try once again. If not can I close this defect saying not re producible? Thanks & Regards Damoder > Some vSphere VMs are shutdown when ACS is restarted > --- > > Key: CLOUDSTACK-4740 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4740 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the > default.) > Components: Management Server >Affects Versions: 4.1.0, 4.1.1, 4.2.0, Future > Environment: I'm running ACS 4.1.1 with vSphere 5.1 >Reporter: ilya musayev >Assignee: Damodar Reddy T >Priority: Critical > Labels: management, poweroff > Fix For: 4.3.0 > > > If management server is restarted, when management server starts - it checks > whether the agentState for vSphere VMs and if it does not get a proper > response, it marks them as stopped. > As the result, some of my virtual instances would shutdown. > Attempting to analyze this issue further, here are my findings and errors > seen in the log. > 2013-09-25 14:35:49,928 DEBUG [vmware.resource.VmwareResource] > (AgentTaskPool-1:null) Detecting a new state but couldn't find a old state so > adding it to the changes: i-2-262-acs-docs-fc17 > 2013-09-25 14:35:51,213 DEBUG [agent.transport.Request] > (AgentTaskPool-1:null) Seq -1--1: Startup request from directly connected > host: { Cmd , MgmtId: -1, via: -1, Ver: v1, Flags: 11, > [{"cpus":16,"speed":2199,"memory":68683468800,"dom0MinMemory":0,"poolSync":false,"vms":{"i-8-270-CLOUD411":{"state":"Running"},"r-15-CLOUD41-OLD":{"state":"Stopped"},"v-260-CLOUD411":{"state":"Running"},"i-2-283-vmbld01l-ops-08":{"state":"Running"},"i-2-104-ACS41VM":{"state":"Running"},"--s-1-CLOUD41-OLD":{"state":"Running"},"i-27-280-CLOUD411":{"state":"Running"},"i-2-285-ossec01l-ops-08":{"state":"Running"},"i-2-262-acs-docs-fc17":{"state":"Stopped"},"i-24-265-test3":{"state":"Running"},"cloud01l-ops-08.portal.webmd.com":{"state":"Running"},"i-2-278-demo01t-ops-08":{"state":"Running"},"s-63-CLOUD411":{"state":"Running"},"r-66-CLOUD411":{"state":"Running"},"i-2-281-acs-appliance":{"state":"Running"}},"caps":"hvm","hypervisorType":"VMware","hostDetails":{"com.cloud.network.Networks.RouterPrivateIpStrategy":"DcGlobal","NativeHA":"true"},"hypervisorVersion":"5.0","type":"Routing","dataCenter":"2","pod":"2","cluster":"3","guid":"HostSystem:host-19...@vc00q-ops-08.portal.webmd.com","name":"vmha62d-ops-08.portal.webmd.com","version":"4.1.1-SNAPSHOT","privateIpAddress":"172.25.243.31","privateMacAddress":"68:b5:99:73:0b:c2","privateNetmask":"255.255.255.0","storageIpAddress":"172.25.243.31","storageNetmask":"255.255.255.0","storageMacAddress":"68:b5:99:73:0b:c2","wait":0},{"totalSize":0,"poolInfo":{"uuid":"72c8aedb-58c4-4569-ac51-adc5af770bf6","host":"vmha62d-ops-08.portal.webmd.com","localPath":"","hostPath":"datastore-19718","poolType":"LVM","capacityBytes":141465485312,"availableBytes":140383354880},"resourceType":"STORAGE_POOL","hostDetails":{},"type":"Storage","dataCenter":"2","pod":"2","cluster":"3","guid":"72c8aedb-58c4-4569-ac51-adc5af770bf6","name":"72c8aedb-58c4-4569-ac51-adc5af770bf6","wait":0}] > } > 2013-09-25 14:35:53,614 DEBUG [cloud.vm.VirtualMachineManagerImpl] > (AgentTaskPool-1:null) VM i-2-262-acs-docs-fc17: cs state = Running and > realState = Stopped > 2013-09-25 14:35:53,614 DEBUG [cloud.vm.VirtualMachineManagerImpl] > (AgentTaskPool-1:null) VM i-2-262-acs-docs-fc17: cs state = Running and > realState = Stopped > 2013-09-25 14:35:53,614 INFO [cloud.ha.HighAvailabilityManagerImpl] > (AgentTaskPool-1:null) Skip HA for VMware VM i-2-262-acs-docs-fc17 > 2013-09-25 14:35:53,694 DEBUG [agent.transport.Request] > (AgentTaskPool-1:null) Seq 11-1418264581: Sending { Cmd , MgmtId: > 345049078181, via: 11, Ver: v1, Flags: 100101, > [{"StopCommand":{"isProxy":false,"vmName":"i-2-262-acs-docs-fc17","wait":0}},{"StopCommand":{"isProxy":false,"vmName":"i-2-278-demo01t-ops-08","wait":0}},{"StopCommand":{"isProxy":false,"vmName":"i-2-281-acs-appliance","wait":0}},{"StopCommand
[jira] [Commented] (CLOUDSTACK-4740) Some vSphere VMs are shutdown when ACS is restarted
[ https://issues.apache.org/jira/browse/CLOUDSTACK-4740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13833566#comment-13833566 ] Abhinandan Prateek commented on CLOUDSTACK-4740: Sateesh can you comment ? > Some vSphere VMs are shutdown when ACS is restarted > --- > > Key: CLOUDSTACK-4740 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4740 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the > default.) > Components: Management Server >Affects Versions: 4.1.0, 4.1.1, 4.2.0, Future > Environment: I'm running ACS 4.1.1 with vSphere 5.1 >Reporter: ilya musayev >Assignee: Sateesh Chodapuneedi >Priority: Critical > Labels: management, poweroff > Fix For: 4.3.0 > > > If management server is restarted, when management server starts - it checks > whether the agentState for vSphere VMs and if it does not get a proper > response, it marks them as stopped. > As the result, some of my virtual instances would shutdown. > Attempting to analyze this issue further, here are my findings and errors > seen in the log. > 2013-09-25 14:35:49,928 DEBUG [vmware.resource.VmwareResource] > (AgentTaskPool-1:null) Detecting a new state but couldn't find a old state so > adding it to the changes: i-2-262-acs-docs-fc17 > 2013-09-25 14:35:51,213 DEBUG [agent.transport.Request] > (AgentTaskPool-1:null) Seq -1--1: Startup request from directly connected > host: { Cmd , MgmtId: -1, via: -1, Ver: v1, Flags: 11, > [{"cpus":16,"speed":2199,"memory":68683468800,"dom0MinMemory":0,"poolSync":false,"vms":{"i-8-270-CLOUD411":{"state":"Running"},"r-15-CLOUD41-OLD":{"state":"Stopped"},"v-260-CLOUD411":{"state":"Running"},"i-2-283-vmbld01l-ops-08":{"state":"Running"},"i-2-104-ACS41VM":{"state":"Running"},"--s-1-CLOUD41-OLD":{"state":"Running"},"i-27-280-CLOUD411":{"state":"Running"},"i-2-285-ossec01l-ops-08":{"state":"Running"},"i-2-262-acs-docs-fc17":{"state":"Stopped"},"i-24-265-test3":{"state":"Running"},"cloud01l-ops-08.portal.webmd.com":{"state":"Running"},"i-2-278-demo01t-ops-08":{"state":"Running"},"s-63-CLOUD411":{"state":"Running"},"r-66-CLOUD411":{"state":"Running"},"i-2-281-acs-appliance":{"state":"Running"}},"caps":"hvm","hypervisorType":"VMware","hostDetails":{"com.cloud.network.Networks.RouterPrivateIpStrategy":"DcGlobal","NativeHA":"true"},"hypervisorVersion":"5.0","type":"Routing","dataCenter":"2","pod":"2","cluster":"3","guid":"HostSystem:host-19...@vc00q-ops-08.portal.webmd.com","name":"vmha62d-ops-08.portal.webmd.com","version":"4.1.1-SNAPSHOT","privateIpAddress":"172.25.243.31","privateMacAddress":"68:b5:99:73:0b:c2","privateNetmask":"255.255.255.0","storageIpAddress":"172.25.243.31","storageNetmask":"255.255.255.0","storageMacAddress":"68:b5:99:73:0b:c2","wait":0},{"totalSize":0,"poolInfo":{"uuid":"72c8aedb-58c4-4569-ac51-adc5af770bf6","host":"vmha62d-ops-08.portal.webmd.com","localPath":"","hostPath":"datastore-19718","poolType":"LVM","capacityBytes":141465485312,"availableBytes":140383354880},"resourceType":"STORAGE_POOL","hostDetails":{},"type":"Storage","dataCenter":"2","pod":"2","cluster":"3","guid":"72c8aedb-58c4-4569-ac51-adc5af770bf6","name":"72c8aedb-58c4-4569-ac51-adc5af770bf6","wait":0}] > } > 2013-09-25 14:35:53,614 DEBUG [cloud.vm.VirtualMachineManagerImpl] > (AgentTaskPool-1:null) VM i-2-262-acs-docs-fc17: cs state = Running and > realState = Stopped > 2013-09-25 14:35:53,614 DEBUG [cloud.vm.VirtualMachineManagerImpl] > (AgentTaskPool-1:null) VM i-2-262-acs-docs-fc17: cs state = Running and > realState = Stopped > 2013-09-25 14:35:53,614 INFO [cloud.ha.HighAvailabilityManagerImpl] > (AgentTaskPool-1:null) Skip HA for VMware VM i-2-262-acs-docs-fc17 > 2013-09-25 14:35:53,694 DEBUG [agent.transport.Request] > (AgentTaskPool-1:null) Seq 11-1418264581: Sending { Cmd , MgmtId: > 345049078181, via: 11, Ver: v1, Flags: 100101, > [{"StopCommand":{"isProxy":false,"vmName":"i-2-262-acs-docs-fc17","wait":0}},{"StopCommand":{"isProxy":false,"vmName":"i-2-278-demo01t-ops-08","wait":0}},{"StopCommand":{"isProxy":false,"vmName":"i-2-281-acs-appliance","wait":0}},{"StopCommand":{"isProxy":false,"vmName":"i-2-283-vmbld01l-ops-08","wait":0}},{"StopCommand":{"isProxy":false,"vmName":"i-2-285-ossec01l-ops-08","wait":0}}] > } > 2013-09-25 14:35:53,695 DEBUG [agent.transport.Request] > (AgentTaskPool-1:null) Seq 11-1418264581: Executing: { Cmd , MgmtId: > 345049078181, via: 11, Ver: v1, Flags: 100101, > [{"StopCommand":{"isProxy":false,"vmName":"i-2-262-acs-docs-fc17","wait":0}},{"StopCommand":{"isProxy":false,"vmName":"i-2-278-demo01t-ops-08","wait":0}},{"StopCommand":{"isProxy":false,"vmName":"i-2-281-acs-appliance","wait":0}},{"StopCommand":{"isProxy":false,"vmNa
[jira] [Commented] (CLOUDSTACK-4740) Some vSphere VMs are shutdown when ACS is restarted
[ https://issues.apache.org/jira/browse/CLOUDSTACK-4740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13790239#comment-13790239 ] Abhinandan Prateek commented on CLOUDSTACK-4740: Moving it to Future as there is limited time for 4.2.1. > Some vSphere VMs are shutdown when ACS is restarted > --- > > Key: CLOUDSTACK-4740 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4740 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the > default.) > Components: Management Server >Affects Versions: 4.1.0, 4.1.1, 4.2.0, Future > Environment: I'm running ACS 4.1.1 with vSphere 5.1 >Reporter: ilya musayev >Priority: Critical > Labels: management, poweroff > Fix For: 4.3.0 > > > If management server is restarted, when management server starts - it checks > whether the agentState for vSphere VMs and if it does not get a proper > response, it marks them as stopped. > As the result, some of my virtual instances would shutdown. > Attempting to analyze this issue further, here are my findings and errors > seen in the log. > 2013-09-25 14:35:49,928 DEBUG [vmware.resource.VmwareResource] > (AgentTaskPool-1:null) Detecting a new state but couldn't find a old state so > adding it to the changes: i-2-262-acs-docs-fc17 > 2013-09-25 14:35:51,213 DEBUG [agent.transport.Request] > (AgentTaskPool-1:null) Seq -1--1: Startup request from directly connected > host: { Cmd , MgmtId: -1, via: -1, Ver: v1, Flags: 11, > [{"cpus":16,"speed":2199,"memory":68683468800,"dom0MinMemory":0,"poolSync":false,"vms":{"i-8-270-CLOUD411":{"state":"Running"},"r-15-CLOUD41-OLD":{"state":"Stopped"},"v-260-CLOUD411":{"state":"Running"},"i-2-283-vmbld01l-ops-08":{"state":"Running"},"i-2-104-ACS41VM":{"state":"Running"},"--s-1-CLOUD41-OLD":{"state":"Running"},"i-27-280-CLOUD411":{"state":"Running"},"i-2-285-ossec01l-ops-08":{"state":"Running"},"i-2-262-acs-docs-fc17":{"state":"Stopped"},"i-24-265-test3":{"state":"Running"},"cloud01l-ops-08.portal.webmd.com":{"state":"Running"},"i-2-278-demo01t-ops-08":{"state":"Running"},"s-63-CLOUD411":{"state":"Running"},"r-66-CLOUD411":{"state":"Running"},"i-2-281-acs-appliance":{"state":"Running"}},"caps":"hvm","hypervisorType":"VMware","hostDetails":{"com.cloud.network.Networks.RouterPrivateIpStrategy":"DcGlobal","NativeHA":"true"},"hypervisorVersion":"5.0","type":"Routing","dataCenter":"2","pod":"2","cluster":"3","guid":"HostSystem:host-19...@vc00q-ops-08.portal.webmd.com","name":"vmha62d-ops-08.portal.webmd.com","version":"4.1.1-SNAPSHOT","privateIpAddress":"172.25.243.31","privateMacAddress":"68:b5:99:73:0b:c2","privateNetmask":"255.255.255.0","storageIpAddress":"172.25.243.31","storageNetmask":"255.255.255.0","storageMacAddress":"68:b5:99:73:0b:c2","wait":0},{"totalSize":0,"poolInfo":{"uuid":"72c8aedb-58c4-4569-ac51-adc5af770bf6","host":"vmha62d-ops-08.portal.webmd.com","localPath":"","hostPath":"datastore-19718","poolType":"LVM","capacityBytes":141465485312,"availableBytes":140383354880},"resourceType":"STORAGE_POOL","hostDetails":{},"type":"Storage","dataCenter":"2","pod":"2","cluster":"3","guid":"72c8aedb-58c4-4569-ac51-adc5af770bf6","name":"72c8aedb-58c4-4569-ac51-adc5af770bf6","wait":0}] > } > 2013-09-25 14:35:53,614 DEBUG [cloud.vm.VirtualMachineManagerImpl] > (AgentTaskPool-1:null) VM i-2-262-acs-docs-fc17: cs state = Running and > realState = Stopped > 2013-09-25 14:35:53,614 DEBUG [cloud.vm.VirtualMachineManagerImpl] > (AgentTaskPool-1:null) VM i-2-262-acs-docs-fc17: cs state = Running and > realState = Stopped > 2013-09-25 14:35:53,614 INFO [cloud.ha.HighAvailabilityManagerImpl] > (AgentTaskPool-1:null) Skip HA for VMware VM i-2-262-acs-docs-fc17 > 2013-09-25 14:35:53,694 DEBUG [agent.transport.Request] > (AgentTaskPool-1:null) Seq 11-1418264581: Sending { Cmd , MgmtId: > 345049078181, via: 11, Ver: v1, Flags: 100101, > [{"StopCommand":{"isProxy":false,"vmName":"i-2-262-acs-docs-fc17","wait":0}},{"StopCommand":{"isProxy":false,"vmName":"i-2-278-demo01t-ops-08","wait":0}},{"StopCommand":{"isProxy":false,"vmName":"i-2-281-acs-appliance","wait":0}},{"StopCommand":{"isProxy":false,"vmName":"i-2-283-vmbld01l-ops-08","wait":0}},{"StopCommand":{"isProxy":false,"vmName":"i-2-285-ossec01l-ops-08","wait":0}}] > } > 2013-09-25 14:35:53,695 DEBUG [agent.transport.Request] > (AgentTaskPool-1:null) Seq 11-1418264581: Executing: { Cmd , MgmtId: > 345049078181, via: 11, Ver: v1, Flags: 100101, > [{"StopCommand":{"isProxy":false,"vmName":"i-2-262-acs-docs-fc17","wait":0}},{"StopCommand":{"isProxy":false,"vmName":"i-2-278-demo01t-ops-08","wait":0}},{"StopCommand":{"isProxy":false,"vmName":"i-2-281-acs-appliance","wait":0}},{"StopCommand":{"isProxy":false,"vmName":"i-2-283-v
[jira] [Commented] (CLOUDSTACK-4740) Some vSphere VMs are shutdown when ACS is restarted
[ https://issues.apache.org/jira/browse/CLOUDSTACK-4740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13778166#comment-13778166 ] ilya musayev commented on CLOUDSTACK-4740: -- this usually occurs with VMs that dont have latest/support vmware tools installed i assume or the state has not been recorded in ACS for whatever the reason was. The vmware tools are noted as "Running (3rd-party/Independent)" - that were downloaded of from vmware as part of open-vm-tools project. > Some vSphere VMs are shutdown when ACS is restarted > --- > > Key: CLOUDSTACK-4740 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4740 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the > default.) > Components: Management Server >Affects Versions: 4.1.0, 4.1.1, 4.2.0, Future > Environment: I'm running ACS 4.1.1 with vSphere 5.1 >Reporter: ilya musayev >Priority: Critical > Labels: management, poweroff > Fix For: 4.1.0, 4.1.1, 4.2.0, Future > > > If management server is restarted, when management server starts - it checks > whether the agentState for vSphere VMs and if it does not get a proper > response, it marks them as stopped. > As the result, some of my virtual instances would shutdown. > Attempting to analyze this issue further, here are my findings and errors > seen in the log. > 2013-09-25 14:35:49,928 DEBUG [vmware.resource.VmwareResource] > (AgentTaskPool-1:null) Detecting a new state but couldn't find a old state so > adding it to the changes: i-2-262-acs-docs-fc17 > 2013-09-25 14:35:51,213 DEBUG [agent.transport.Request] > (AgentTaskPool-1:null) Seq -1--1: Startup request from directly connected > host: { Cmd , MgmtId: -1, via: -1, Ver: v1, Flags: 11, > [{"cpus":16,"speed":2199,"memory":68683468800,"dom0MinMemory":0,"poolSync":false,"vms":{"i-8-270-CLOUD411":{"state":"Running"},"r-15-CLOUD41-OLD":{"state":"Stopped"},"v-260-CLOUD411":{"state":"Running"},"i-2-283-vmbld01l-ops-08":{"state":"Running"},"i-2-104-ACS41VM":{"state":"Running"},"--s-1-CLOUD41-OLD":{"state":"Running"},"i-27-280-CLOUD411":{"state":"Running"},"i-2-285-ossec01l-ops-08":{"state":"Running"},"i-2-262-acs-docs-fc17":{"state":"Stopped"},"i-24-265-test3":{"state":"Running"},"cloud01l-ops-08.portal.webmd.com":{"state":"Running"},"i-2-278-demo01t-ops-08":{"state":"Running"},"s-63-CLOUD411":{"state":"Running"},"r-66-CLOUD411":{"state":"Running"},"i-2-281-acs-appliance":{"state":"Running"}},"caps":"hvm","hypervisorType":"VMware","hostDetails":{"com.cloud.network.Networks.RouterPrivateIpStrategy":"DcGlobal","NativeHA":"true"},"hypervisorVersion":"5.0","type":"Routing","dataCenter":"2","pod":"2","cluster":"3","guid":"HostSystem:host-19...@vc00q-ops-08.portal.webmd.com","name":"vmha62d-ops-08.portal.webmd.com","version":"4.1.1-SNAPSHOT","privateIpAddress":"172.25.243.31","privateMacAddress":"68:b5:99:73:0b:c2","privateNetmask":"255.255.255.0","storageIpAddress":"172.25.243.31","storageNetmask":"255.255.255.0","storageMacAddress":"68:b5:99:73:0b:c2","wait":0},{"totalSize":0,"poolInfo":{"uuid":"72c8aedb-58c4-4569-ac51-adc5af770bf6","host":"vmha62d-ops-08.portal.webmd.com","localPath":"","hostPath":"datastore-19718","poolType":"LVM","capacityBytes":141465485312,"availableBytes":140383354880},"resourceType":"STORAGE_POOL","hostDetails":{},"type":"Storage","dataCenter":"2","pod":"2","cluster":"3","guid":"72c8aedb-58c4-4569-ac51-adc5af770bf6","name":"72c8aedb-58c4-4569-ac51-adc5af770bf6","wait":0}] > } > 2013-09-25 14:35:53,614 DEBUG [cloud.vm.VirtualMachineManagerImpl] > (AgentTaskPool-1:null) VM i-2-262-acs-docs-fc17: cs state = Running and > realState = Stopped > 2013-09-25 14:35:53,614 DEBUG [cloud.vm.VirtualMachineManagerImpl] > (AgentTaskPool-1:null) VM i-2-262-acs-docs-fc17: cs state = Running and > realState = Stopped > 2013-09-25 14:35:53,614 INFO [cloud.ha.HighAvailabilityManagerImpl] > (AgentTaskPool-1:null) Skip HA for VMware VM i-2-262-acs-docs-fc17 > 2013-09-25 14:35:53,694 DEBUG [agent.transport.Request] > (AgentTaskPool-1:null) Seq 11-1418264581: Sending { Cmd , MgmtId: > 345049078181, via: 11, Ver: v1, Flags: 100101, > [{"StopCommand":{"isProxy":false,"vmName":"i-2-262-acs-docs-fc17","wait":0}},{"StopCommand":{"isProxy":false,"vmName":"i-2-278-demo01t-ops-08","wait":0}},{"StopCommand":{"isProxy":false,"vmName":"i-2-281-acs-appliance","wait":0}},{"StopCommand":{"isProxy":false,"vmName":"i-2-283-vmbld01l-ops-08","wait":0}},{"StopCommand":{"isProxy":false,"vmName":"i-2-285-ossec01l-ops-08","wait":0}}] > } > 2013-09-25 14:35:53,695 DEBUG [agent.transport.Request] > (AgentTaskPool-1:null) Seq 11-1418264581: Executing: { Cmd , MgmtId: > 345049078181, via: 11, Ver: v1, Flags: 100101, > [{"StopCom