The agent doesn't reconnect if there are stopped VMs
Dear colleagues,
I have a simple setup where the management server (CentOS-6.6 +
ACS-4.5.1) is orchestrating a bunch of KVM hosts (each of them is
running CentOS-6.6 + ACS-4.5.1 as well).
Any host with at least one VM in the Stopped state can't reconnect to
the management server. It has the Alert state and here's what I see in
the management server's log-file:
--- 8 ---
2015-08-18 06:24:46,332 DEBUG [c.c.a.t.Request]
(AgentConnectTaskPool-213:ctx-76903ef6) Seq 0-148: Processing the first command
{ Cmd ,
MgmtId: -1, via: 0, Ver: v1, Flags: 1,
[{com.cloud.agent.api.StartupRoutingCommand:{cpuSockets:1,cpus:48,speed:2299,memory:6743
9632384,dom0MinMemory:805306368,poolSync:false,caps:hvm,snapshot,pool:/root,hypervisorType:KVM,hostDetails:{com.cloud.
network.Networks.RouterPrivateIpStrategy:HostLocal,Host.OS:CentOS,Host.OS.Kernel.Version:2.6.32-504.23.4.el6.x86_64,Host.OS.V
ersion:6.6},hostTags:[],groupDetails:{},type:Routing,dataCenter:6,pod:7,cluster:7,guid:1318c38d-4ed6-3296-a6bd-753676e25ad4-LibvirtComputingResource,name:***.***.***,id:0,version:4.5.1,publicIpAddress:172.27.65.1,publicNetmask:255.255.255.0,publicMacAddress:ec:f4:bb:d6:89:c5,privateIpAddress:172.27.65.1,privateMacAddress:ec:f4:bb:d6:89:c5,privateNetmask:255.255.255.0,storageIpAddress:172.27.65.1,storageNetmask:255.255.255.0,storageMacAddress:ec:f4:bb:d6:89:c5,resourceName:LibvirtComputingResource,gatewayIpAddress:***.***.***.***,wait:0}},{com.cloud.agent.api.StartupStorageCommand:{totalSize:0,poolInfo:{uuid:51670fbd-ece2-4a3e-9971-3928e6576f0e,host:172.27.65.1,localPath:/var/lib/libvirt/images,hostPath:/var/lib/libvirt/images,poolType:Filesystem,capacityBytes:1563804868608,availableBytes:1474368700416},resourceType:STORAGE_POOL,hostDetails:{},type:Storage,dataCenter:6,pod:7,guid:1318c38d-4ed6-3296-a6bd-753676e25ad4-LibvirtComputingResource,name:***.***.***,id:0,version:4.5.1,resourceName:LibvirtComputingResource,wait:0}}]
}
2015-08-18 06:24:46,336 DEBUG [c.c.r.ResourceManagerImpl]
(AgentConnectTaskPool-213:ctx-76903ef6) Dispatching resource state event
CREATE_HOST_VO_FOR_CONNECTED to BaremetalDhcpManagerImpl
2015-08-18 06:24:46,336 DEBUG [c.c.r.ResourceManagerImpl]
(AgentConnectTaskPool-213:ctx-76903ef6) Dispatching resource state event
CREATE_HOST_VO_FOR_CONNECTED to BaremetalPxeManagerImpl
2015-08-18 06:24:46,336 DEBUG [c.c.r.ResourceManagerImpl]
(AgentConnectTaskPool-213:ctx-76903ef6) Dispatching resource state event
CREATE_HOST_VO_FOR_CONNECTED to NetworkUsageManagerImpl
2015-08-18 06:24:46,336 DEBUG [c.c.r.ResourceManagerImpl]
(AgentConnectTaskPool-213:ctx-76903ef6) Dispatching resource state event
CREATE_HOST_VO_FOR_CONNECTED to NuageVspElement
2015-08-18 06:24:46,336 DEBUG [c.c.r.ResourceManagerImpl]
(AgentConnectTaskPool-213:ctx-76903ef6) Dispatching resource state event
CREATE_HOST_VO_FOR_CONNECTED to Ovs
2015-08-18 06:24:46,336 DEBUG [c.c.r.ResourceManagerImpl]
(AgentConnectTaskPool-213:ctx-76903ef6) Dispatching resource state event
CREATE_HOST_VO_FOR_CONNECTED to PaloAltoExternalFirewallElement
2015-08-18 06:24:46,336 DEBUG [c.c.r.ResourceManagerImpl]
(AgentConnectTaskPool-213:ctx-76903ef6) Dispatching resource state event
CREATE_HOST_VO_FOR_CONNECTED to GloboDnsElement
2015-08-18 06:24:46,336 DEBUG [c.c.r.ResourceManagerImpl]
(AgentConnectTaskPool-213:ctx-76903ef6) Dispatching resource state event
CREATE_HOST_VO_FOR_CONNECTED to KvmServerDiscoverer
2015-08-18 06:24:46,362 DEBUG [c.c.r.ResourceState]
(AgentConnectTaskPool-213:ctx-76903ef6) Resource state update: [id = 27; name =
***.***.***; old state = Enabled; event = InternalCreated; new state = Enabled]
2015-08-18 06:24:46,362 DEBUG [c.c.h.Status]
(AgentConnectTaskPool-213:ctx-76903ef6) Transition:[Resource state = Enabled,
Agent event = AgentConnected, Host id = 27, name = ***.***.***]
2015-08-18 06:24:46,365 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
(AgentConnectTaskPool-213:ctx-76903ef6) create ClusteredAgentAttache for 27
2015-08-18 06:24:46,367 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentConnectTaskPool-213:ctx-76903ef6) Sending Connect to listener:
XcpServerDiscoverer
2015-08-18 06:24:46,367 DEBUG [c.c.h.x.d.XcpServerDiscoverer]
(AgentConnectTaskPool-213:ctx-76903ef6) Not XenServer so moving on.
2015-08-18 06:24:46,367 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentConnectTaskPool-213:ctx-76903ef6) Sending Connect to listener:
HypervServerDiscoverer
2015-08-18 06:24:46,367 DEBUG [c.c.h.h.d.HypervServerDiscoverer]
(AgentConnectTaskPool-213:ctx-76903ef6) Not Hyper-V hypervisor, so moving on.
2015-08-18 06:24:46,367 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentConnectTaskPool-213:ctx-76903ef6) Sending Connect to listener:
ClusteredVirtualMachineManagerImpl
2015-08-18 06:24:46,367 DEBUG [c.c.v.VirtualMachineManagerImpl]
(AgentConnectTaskPool-213:ctx-76903ef6) Received startup command from
hypervisor host. host id: 27
2015-08-18 06:24:46,367 INFO [c.c.v.VirtualMachinePowerStateSyncImpl]
(AgentConnectTaskPool-213:ctx-76903ef6) Reset VM power state sync
Re: VM stuck in a failing Host
Yes we could , but I'm not sure if maybe I lost the VM because it is tied to
that host.
Cordialmente,
Jaime Rojas
On 20/08/2015, at 11:47 a.m., Somesh Naidu somesh.na...@citrix.com wrote:
Can you not remove the failed host from CCP and XS cluster?
Regards,
Somesh
-Original Message-
From: Jaime Orlando Rojas Sanchez [mailto:jaime.ro...@kumo.com.co]
Sent: Thursday, August 20, 2015 12:38 PM
To: users@cloudstack.apache.org
Subject: RE: VM stuck in a failing Host
Hello,
state/status of the failed host in CS? = Disconnected
We run the command in the 2 remaining host with no results.
[root@dc1fdtptgcx04 /]# xe vm-list name-label= i-574-14584-VM
[root@dc1fdtptgcx04 /]#
[root@dc1fdtptgcx02 /]# xe vm-list name-label= i-574-14584-VM
[root@dc1fdtptgcx02 /]#
Regards / Cordialmente,
Jaime O. Rojas S.
Technology Manager
jaime.ro...@kumo.com.co
Mobile: +57 301-3382382
Office: +57-1-8766767 x215
-Mensaje original-
De: Somesh Naidu [mailto:somesh.na...@citrix.com]
Enviado el: jueves, 20 de agosto de 2015 11:00 a. m.
Para: users@cloudstack.apache.org
Asunto: RE: VM stuck in a failing Host
Quick question, what is the state/status of the failed host in CS? Also, look
up the particular VM on XS pool (xe vm-list name-label= i-574-14584-VM), what
does it say?
Regards,
Somesh
-Original Message-
From: Jaime Orlando Rojas Sanchez [mailto:jaime.ro...@kumo.com.co]
Sent: Thursday, August 20, 2015 11:37 AM
To: users@cloudstack.apache.org
Subject: RE: VM stuck in a failing Host
Following the logs when I click 'run' in ACS after did the following in the DB
- Change the state to 'stopped'
- Change host ID to a working host
- Change last host ID to a working host
- Check VR is up and running on a working host
-bash-4.1# tail -f management-server.log | grep 14584
2015-08-20 05:45:27,513 DEBUG [cloud.async.AsyncJobManagerImpl]
(catalina-exec-21:null) submit async job-45973 = [
d3ed77b7-534e-4a3b-9038-a66359162087 ], details: AsyncJobVO {id:45973,
userId: 2, accountId: 2, sessionKey: null, instanceType: VirtualMachine,
instanceId: 14584, cmd: org.apache.cloudstack.api.command.user.vm.StopVMCmd,
cmdOriginator: null, cmdInfo:
{response:json,id:98227dc9-682e-4f42-87e1-bd4b8045c7c9,sessionkey:hwnxmM0He9EXs2craugKg3XyWL4\u003d,cmdEventType:VM.STOP,ctxUserId:2,httpmethod:GET,_:1440067422009,ctxAccountId:2,ctxStartEventId:38650949,forced:true},
cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0,
processStatus: 0, resultCode: 0, result: null, initMsid: 139549854171544,
completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
2015-08-20 05:46:33,404 DEBUG [agent.transport.Request]
(Job-Executor-31:job-45974 = [ 9996fc72-fb83-4e5d-94c5-886396dac536 ]) Seq
792-1073807425: Sending { Cmd , MgmtId: 139549854171544, via: 792, Ver: v1,
Flags: 100111,
[{org.apache.cloudstack.storage.command.CopyCommand:{srcTO:{org.apache.cloudstack.storage.to.SnapshotObjectTO:{path:snapshots/574/57764/28586b35-cb45-4565-bd9b-7aa46a2898da,volume:{uuid:a15d0923-0a25-408f-9d10-fd5d47b3fef9,volumeType:ROOT,dataStore:{org.apache.cloudstack.storage.to.PrimaryDataStoreTO:{uuid:3PAR_3000GB_ADV_SATA1,id:211,poolType:PreSetup,host:localhost,path:/3PAR_3000GB_ADV_SATA1,port:0}},name:ROOT-14584,size:107374182400,path:c7a8eebc-7750-455c-804f-64c0d66cb4f4,volumeId:57764,vmName:i-574-14584-VM,accountId:574,format:VHD,id:57764,hypervisorType:XenServer},dataStore:{com.cloud.agent.api.to.NfsTO:{_url:nfs://172.16.4.65/vol/secondary_clpr,_role:Image}},vmName:i-574-14584-VM,name:srvrasautos2_ROOT-14584_20141007233517,hypervisorType:XenServer,id:11831}},destTO:{org.apache.cloudstack.storage.to.TemplateObjectTO:{path:template/tmpl/574/599,uuid:70d21214-33d0-49e0-8b45-c7702b0fe579,id:599,format:RAW,accountId:574,hvm:true,displayText:templateras,imageDataStore:{com.cloud.agent.api.to.NfsTO:{_url:nfs://172.16.4.65/vol/secondary_clpr,_role:Image}},name:248e2097b-4af7-38f7-a851-029ef11f52cc,hypervisorType:XenServer}},executeInSequence:true,wait:10800}}]
}
2015-08-20 05:49:03,212 DEBUG [cloud.async.AsyncJobManagerImpl]
(catalina-exec-1:null) submit async job-45975 = [
8ac23585-989d-4e3d-bcb9-3d3602842b8f ], details: AsyncJobVO {id:45975,
userId: 2, accountId: 2, sessionKey: null, instanceType: VirtualMachine,
instanceId: 14584, cmd: org.apache.cloudstack.api.command.user.vm.StartVMCmd,
cmdOriginator: null, cmdInfo:
{response:json,id:98227dc9-682e-4f42-87e1-bd4b8045c7c9,sessionkey:hwnxmM0He9EXs2craugKg3XyWL4\u003d,cmdEventType:VM.START,ctxUserId:2,httpmethod:GET,_:1440067641673,ctxAccountId:2,ctxStartEventId:38651246},
cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0,
processStatus: 0, resultCode: 0, result: null, initMsid: 139549854171544,
completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
2015-08-20 05:49:05,821 DEBUG
RE: VM stuck in a failing Host
I do not believe that should happen as the VM is primarily tied to the storage.
But just to avoid taking any chances, we could do this:
1. Put the failed host in to maintenance.
2. Set host_id and last_host_id to NULL in vm_instance table for that VM.
3. Make sure the XS pool doesn't have any stale data for this VM. Look for the
VM and associated VDI in the pool one more time. I noticed there is a space
name-label= i-574-14584-VM after the equal operator, remove that space and
try again or use XenCenter.
If #3 doesn't raise any flags, go ahead and try starting the VM again.
There is also another alternative, you could register the root disk of the VM
as a template and launch a new VM from that template.
Regards,
Somesh
-Original Message-
From: Jaime Orlando Rojas Sanchez [mailto:jaime.ro...@kumo.com.co]
Sent: Thursday, August 20, 2015 1:00 PM
To: users@cloudstack.apache.org
Subject: Re: VM stuck in a failing Host
Yes we could , but I'm not sure if maybe I lost the VM because it is tied to
that host.
Cordialmente,
Jaime Rojas
On 20/08/2015, at 11:47 a.m., Somesh Naidu somesh.na...@citrix.com wrote:
Can you not remove the failed host from CCP and XS cluster?
Regards,
Somesh
-Original Message-
From: Jaime Orlando Rojas Sanchez [mailto:jaime.ro...@kumo.com.co]
Sent: Thursday, August 20, 2015 12:38 PM
To: users@cloudstack.apache.org
Subject: RE: VM stuck in a failing Host
Hello,
state/status of the failed host in CS? = Disconnected
We run the command in the 2 remaining host with no results.
[root@dc1fdtptgcx04 /]# xe vm-list name-label= i-574-14584-VM
[root@dc1fdtptgcx04 /]#
[root@dc1fdtptgcx02 /]# xe vm-list name-label= i-574-14584-VM
[root@dc1fdtptgcx02 /]#
Regards / Cordialmente,
Jaime O. Rojas S.
Technology Manager
jaime.ro...@kumo.com.co
Mobile: +57 301-3382382
Office: +57-1-8766767 x215
-Mensaje original-
De: Somesh Naidu [mailto:somesh.na...@citrix.com]
Enviado el: jueves, 20 de agosto de 2015 11:00 a. m.
Para: users@cloudstack.apache.org
Asunto: RE: VM stuck in a failing Host
Quick question, what is the state/status of the failed host in CS? Also, look
up the particular VM on XS pool (xe vm-list name-label= i-574-14584-VM), what
does it say?
Regards,
Somesh
-Original Message-
From: Jaime Orlando Rojas Sanchez [mailto:jaime.ro...@kumo.com.co]
Sent: Thursday, August 20, 2015 11:37 AM
To: users@cloudstack.apache.org
Subject: RE: VM stuck in a failing Host
Following the logs when I click 'run' in ACS after did the following in the DB
- Change the state to 'stopped'
- Change host ID to a working host
- Change last host ID to a working host
- Check VR is up and running on a working host
-bash-4.1# tail -f management-server.log | grep 14584
2015-08-20 05:45:27,513 DEBUG [cloud.async.AsyncJobManagerImpl]
(catalina-exec-21:null) submit async job-45973 = [
d3ed77b7-534e-4a3b-9038-a66359162087 ], details: AsyncJobVO {id:45973,
userId: 2, accountId: 2, sessionKey: null, instanceType: VirtualMachine,
instanceId: 14584, cmd: org.apache.cloudstack.api.command.user.vm.StopVMCmd,
cmdOriginator: null, cmdInfo:
{response:json,id:98227dc9-682e-4f42-87e1-bd4b8045c7c9,sessionkey:hwnxmM0He9EXs2craugKg3XyWL4\u003d,cmdEventType:VM.STOP,ctxUserId:2,httpmethod:GET,_:1440067422009,ctxAccountId:2,ctxStartEventId:38650949,forced:true},
cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0,
processStatus: 0, resultCode: 0, result: null, initMsid: 139549854171544,
completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
2015-08-20 05:46:33,404 DEBUG [agent.transport.Request]
(Job-Executor-31:job-45974 = [ 9996fc72-fb83-4e5d-94c5-886396dac536 ]) Seq
792-1073807425: Sending { Cmd , MgmtId: 139549854171544, via: 792, Ver: v1,
Flags: 100111,
how to disable CPVM http trace and disable sslV3 poodle security issue
hello??We use cloudstack 3.0.2 ,Since CPVM have http trace enable and sslV3 poodle securiy issue?? Does anyone how to disable it?? Some security tools said web proxy console link have php security issue??How to upgrade php for CPVM Thanks
Re: VM stuck in a failing Host
Hello, I already tried he force stop check, unfortunately it didn't work VM
keeps starting in failing host. I am trying to force vm to stop from xen server
with no results.
[root@dc1fdtptgcx02 /]# xe vm-shutdown name-label=i-574-14584-VM --force
You attempted an operation which involves a host which could not be contacted.
host: 53109eef-2f53-4f0e-a763-68817d573bd9 (DC1FDTPTGCX03
Cordialmente,
Jaime Rojas
Sent from my iPhone
On 20/08/2015, at 4:03 p.m., Somesh Naidu
somesh.na...@citrix.commailto:somesh.na...@citrix.com wrote:
Thanks I just realized that the job was a stop command and not start command
that I was looking for.
The stop command failed to stop the VM leaving the VM in Running state:
2015-08-20 05:45:27,513 DEBUG [cloud.async.AsyncJobManagerImpl]
(catalina-exec-21:null) submit async job-45973 = [
d3ed77b7-534e-4a3b-9038-a66359162087 ], details: AsyncJobVO {id:45973, userId:
2, accountId: 2, sessionKey: null, instanceType: VirtualMachine, instanceId:
14584, cmd: org.apache.cloudstack.api.command.user.vm.StopVMCmd, cmdOriginator:
null, cmdInfo:
{response:json,id:98227dc9-682e-4f42-87e1-bd4b8045c7c9,sessionkey:hwnxmM0He9EXs2craugKg3XyWL4\u003d,cmdEventType:VM.STOP,ctxUserId:2,httpmethod:GET,_:1440067422009,ctxAccountId:2,ctxStartEventId:38650949,forced:true},
cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0,
processStatus: 0, resultCode: 0, result: null, initMsid: 139549854171544,
completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
2015-08-20 05:45:27,605 DEBUG [cloud.capacity.CapacityManagerImpl]
(Job-Executor-30:job-45973 = [ d3ed77b7-534e-4a3b-9038-a66359162087 ]) VM state
transitted from :Running to Stopping with event: StopRequestedvm's original
host id: 599 new host id: 599 host id before state transition: 599
2015-08-20 05:45:27,617 WARN [cloud.vm.VirtualMachineManagerImpl]
(Job-Executor-30:job-45973 = [ d3ed77b7-534e-4a3b-9038-a66359162087 ]) Unable
to stop vm, agent unavailable: com.cloud.exception.AgentUnavailableException:
Resource [Host:599] is unreachable: Host 599: Host with specified id is not in
the right state: Disconnected
2015-08-20 05:45:32,654 DEBUG [cloud.capacity.CapacityManagerImpl]
(Job-Executor-30:job-45973 = [ d3ed77b7-534e-4a3b-9038-a66359162087 ]) VM state
transitted from :Stopping to Running with event: OperationFailedvm's original
host id: 599 new host id: 599 host id before state transition: 599
2015-08-20 05:45:33,683 DEBUG [cloud.async.AsyncJobManagerImpl]
(Job-Executor-30:job-45973 = [ d3ed77b7-534e-4a3b-9038-a66359162087 ]) Complete
async job-45973 = [ d3ed77b7-534e-4a3b-9038-a66359162087 ], jobStatus: 2,
resultCode: 530, result: Error Code: 530 Error text: Failed to stop vm
I can't tell why the VM keeps starting on the failed host unless I have a look
at the logs from a start command.
But I think there is a good chance that if you check the Force Stop check box
when stopping that VM and then try to start it again.
Regards,
Somesh
-Original Message-
From: Jaime Orlando Rojas Sanchez [mailto:jaime.ro...@kumo.com.co]
Sent: Thursday, August 20, 2015 4:40 PM
To: users@cloudstack.apache.orgmailto:users@cloudstack.apache.org
Subject: RE: VM stuck in a failing Host
Hello,
I am attaching the logs
http://jrojas-test.s3.kumo.com.co/management-server.log?AWSAccessKeyId=0099c1623d6704f9f5d5Expires=1440711480Signature=CFYwowzsJ3x%2B7XXfpAHDLM%2F022A%3Dx-amz-pt=N2ZiYjcyOTk4MTQ0MDEwMzEwNTQ5MA
We have a second manage server but it doesn't execute any task.
Regards / Cordialmente,
Jaime O. Rojas S.
Technology Manager
jaime.ro...@kumo.com.comailto:jaime.ro...@kumo.com.co
Mobile: +57 301-3382382
Office: +57-1-8766767 x215
-Mensaje original-
De: Somesh Naidu [mailto:somesh.na...@citrix.com]
Enviado el: jueves, 20 de agosto de 2015 2:17 p. m.
Para: users@cloudstack.apache.orgmailto:users@cloudstack.apache.org
Asunto: RE: VM stuck in a failing Host
Jaime,
Those are not complete logs I need. I need complete logs (no grep) between
2015-08-20 05:45:27,513 and when the job (job-45973) finished (I can't tell
from the logs you shared earlier).
Also, do you have more than 1 management server?
Regards,
Somesh
-Original Message-
From: Jaime Orlando Rojas Sanchez [mailto:jaime.ro...@kumo.com.co]
Sent: Thursday, August 20, 2015 2:41 PM
To: users@cloudstack.apache.orgmailto:users@cloudstack.apache.org
Subject: Re: VM stuck in a failing Host
We ran the command without the space with no results.
In the second message I sent, I pasted the logs. You can find it in this mail
chain at the beginning.
Cordialmente,
Jaime Rojas
Sent from my iPhone
On 20/08/2015, at 1:37 p.m., Somesh Naidu
somesh.na...@citrix.commailto:somesh.na...@citrix.commailto:somesh.na...@citrix.com
wrote:
We do not need to take a snapshot. Just extract the VHD from storage (you could
use vdi-copy if it is on block storage) and then register it in CS as a
template from UI.
xe vm-list name-label=
Re: how to disable CPVM http trace and disable sslV3 poodle security issue
You need to upgrade system vm templates , check the https://blogs.apache.org/cloudstack/ for mote details Related info thought might help 1. does changing in httpd.conf reflects the setting for apache2? Use the nmap script as suggested below to identify ssl versions on installed system vm template. Check the suggested change works or not. http://security.stackexchange.com/questions/70733/how-do-i-use-openssl-s-cl ient-to-test-for-absence-of-sslv3-support 2. TLSv1.2 is the latest to be used and suggested default, ssl protocol and the ciphers we use leads to vulnerability, the settings for these as well should be available in similar config file. In our code, search for TLS leads to usage at places, and assumption is that it should negotiate the protocol version from configured and available latest version to least, so if TLSv1.2 is configured on server and client supports it, then it should work. http://www.cisco.com/c/en/us/support/docs/security/email-security-appliance /118518-technote-esa-00.html Regards, Santhosh __ On 8/20/15, 11:43 AM, jerry 1163...@qq.com wrote: hello，We use cloudstack 3.0.2 ,Since CPVM have http trace enable and sslV3 poodle securiy issue， Does anyone how to disable it。 Some security tools said web proxy console link have php security issue，How to upgrade php for CPVM Thanks
Re: duplicate use of tips?
If you are manually assigning the ips better use the ips that are outside the cidr that cloudstack manages. The ips assigned by cloudstack dhcp are as per cloudstack assignments anything that happen outside VR’s dhcp is unknown to cloudstack. On 20-Aug-2015, at 11:29 pm, Frank Louwers fr...@openminds.be wrote: Hi, In a zone with Basic Networking, I’ve assigned a certain netblock x.y.z.0/24 to the Guest network in CloudStack. I have a VM-A that has primary ip address x.y.z.92 and secondary address on the same nic x.y.z.52. For various reasons, *both* ips were configured manually, so not using the VR’s dhcp. A while back, the x.y.z.92 was (manually) deconfigured on VM-A (so VM-A only used x.y.x.52, but both ips are still configured to belong to VM-A in Cloudstack). A few days ago, a new instance (VM-B) was spun up, and CS assigned ip x.y.z.92 to that VM. Why would it do that? Today, a new instance was spun up (VM-C), but CS assigned x.y.z.52 to VM-C… How and why does this happen?! Is this because VM-A does not use dhcp? That might explain the .92 re-assignement, but certainly not the .52 reassignment, as secondary ips don’t use dhcp anyhow… Can anyone tell me what’s going on, and what can be done to prevent his? Running CS 4.4.2 at the moment, considering upgrading to 4.5.1 (or 4.5.2) Regards, Frank Find out more about ShapeBlue and our range of CloudStack related services IaaS Cloud Design Buildhttp://shapeblue.com/iaas-cloud-design-and-build// CSForge – rapid IaaS deployment frameworkhttp://shapeblue.com/csforge/ CloudStack Consultinghttp://shapeblue.com/cloudstack-consultancy/ CloudStack Software Engineeringhttp://shapeblue.com/cloudstack-software-engineering/ CloudStack Infrastructure Supporthttp://shapeblue.com/cloudstack-infrastructure-support/ CloudStack Bootcamp Training Courseshttp://shapeblue.com/cloudstack-training/ This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd or related companies. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error. Shape Blue Ltd is a company incorporated in England Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is a registered trademark.
RE: VM stuck in a failing Host
We do not need to take a snapshot. Just extract the VHD from storage (you could
use vdi-copy if it is on block storage) and then register it in CS as a
template from UI.
xe vm-list name-label= i-574-14584-VM
The command you tried above has space after =. Did you try without the space?
Also, can you share the management server logs that capture the VM start job so
we can take a look?
Regards,
Somesh
-Original Message-
From: Jaime Orlando Rojas Sanchez [mailto:jaime.ro...@kumo.com.co]
Sent: Thursday, August 20, 2015 1:59 PM
To: users@cloudstack.apache.org
Subject: Re: VM stuck in a failing Host
We already did all of that and VM keeps trying to start on failed host.
Template is not possible we tried to take a snapshot but it return an error.
Cordialmente,
Jaime Rojas
On 20/08/2015, at 12:19 p.m., Somesh Naidu somesh.na...@citrix.com wrote:
I do not believe that should happen as the VM is primarily tied to the
storage. But just to avoid taking any chances, we could do this:
1. Put the failed host in to maintenance.
2. Set host_id and last_host_id to NULL in vm_instance table for that VM.
3. Make sure the XS pool doesn't have any stale data for this VM. Look for
the VM and associated VDI in the pool one more time. I noticed there is a
space name-label= i-574-14584-VM after the equal operator, remove that
space and try again or use XenCenter.
If #3 doesn't raise any flags, go ahead and try starting the VM again.
There is also another alternative, you could register the root disk of the VM
as a template and launch a new VM from that template.
Regards,
Somesh
-Original Message-
From: Jaime Orlando Rojas Sanchez [mailto:jaime.ro...@kumo.com.co]
Sent: Thursday, August 20, 2015 1:00 PM
To: users@cloudstack.apache.org
Subject: Re: VM stuck in a failing Host
Yes we could , but I'm not sure if maybe I lost the VM because it is tied to
that host.
Cordialmente,
Jaime Rojas
On 20/08/2015, at 11:47 a.m., Somesh Naidu somesh.na...@citrix.com wrote:
Can you not remove the failed host from CCP and XS cluster?
Regards,
Somesh
-Original Message-
From: Jaime Orlando Rojas Sanchez [mailto:jaime.ro...@kumo.com.co]
Sent: Thursday, August 20, 2015 12:38 PM
To: users@cloudstack.apache.org
Subject: RE: VM stuck in a failing Host
Hello,
state/status of the failed host in CS? = Disconnected
We run the command in the 2 remaining host with no results.
[root@dc1fdtptgcx04 /]# xe vm-list name-label= i-574-14584-VM
[root@dc1fdtptgcx04 /]#
[root@dc1fdtptgcx02 /]# xe vm-list name-label= i-574-14584-VM
[root@dc1fdtptgcx02 /]#
Regards / Cordialmente,
Jaime O. Rojas S.
Technology Manager
jaime.ro...@kumo.com.co
Mobile: +57 301-3382382
Office: +57-1-8766767 x215
-Mensaje original-
De: Somesh Naidu [mailto:somesh.na...@citrix.com]
Enviado el: jueves, 20 de agosto de 2015 11:00 a. m.
Para: users@cloudstack.apache.org
Asunto: RE: VM stuck in a failing Host
Quick question, what is the state/status of the failed host in CS? Also,
look up the particular VM on XS pool (xe vm-list name-label=
i-574-14584-VM), what does it say?
Regards,
Somesh
-Original Message-
From: Jaime Orlando Rojas Sanchez [mailto:jaime.ro...@kumo.com.co]
Sent: Thursday, August 20, 2015 11:37 AM
To: users@cloudstack.apache.org
Subject: RE: VM stuck in a failing Host
Following the logs when I click 'run' in ACS after did the following in the
DB
- Change the state to 'stopped'
- Change host ID to a working host
- Change last host ID to a working host
- Check VR is up and running on a working host
-bash-4.1# tail -f management-server.log | grep 14584
2015-08-20 05:45:27,513 DEBUG [cloud.async.AsyncJobManagerImpl]
(catalina-exec-21:null) submit async job-45973 = [
d3ed77b7-534e-4a3b-9038-a66359162087 ], details: AsyncJobVO {id:45973,
userId: 2, accountId: 2, sessionKey: null, instanceType: VirtualMachine,
instanceId: 14584, cmd: org.apache.cloudstack.api.command.user.vm.StopVMCmd,
cmdOriginator: null, cmdInfo:
{response:json,id:98227dc9-682e-4f42-87e1-bd4b8045c7c9,sessionkey:hwnxmM0He9EXs2craugKg3XyWL4\u003d,cmdEventType:VM.STOP,ctxUserId:2,httpmethod:GET,_:1440067422009,ctxAccountId:2,ctxStartEventId:38650949,forced:true},
cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0,
processStatus: 0, resultCode: 0, result: null, initMsid: 139549854171544,
completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
2015-08-20 05:46:33,404 DEBUG [agent.transport.Request]
(Job-Executor-31:job-45974 = [ 9996fc72-fb83-4e5d-94c5-886396dac536 ]) Seq
792-1073807425: Sending { Cmd , MgmtId: 139549854171544, via: 792, Ver: v1,
Flags: 100111,
Re: VM stuck in a failing Host
We already did all of that and VM keeps trying to start on failed host.
Template is not possible we tried to take a snapshot but it return an error.
Cordialmente,
Jaime Rojas
On 20/08/2015, at 12:19 p.m., Somesh Naidu somesh.na...@citrix.com wrote:
I do not believe that should happen as the VM is primarily tied to the
storage. But just to avoid taking any chances, we could do this:
1. Put the failed host in to maintenance.
2. Set host_id and last_host_id to NULL in vm_instance table for that VM.
3. Make sure the XS pool doesn't have any stale data for this VM. Look for
the VM and associated VDI in the pool one more time. I noticed there is a
space name-label= i-574-14584-VM after the equal operator, remove that
space and try again or use XenCenter.
If #3 doesn't raise any flags, go ahead and try starting the VM again.
There is also another alternative, you could register the root disk of the VM
as a template and launch a new VM from that template.
Regards,
Somesh
-Original Message-
From: Jaime Orlando Rojas Sanchez [mailto:jaime.ro...@kumo.com.co]
Sent: Thursday, August 20, 2015 1:00 PM
To: users@cloudstack.apache.org
Subject: Re: VM stuck in a failing Host
Yes we could , but I'm not sure if maybe I lost the VM because it is tied to
that host.
Cordialmente,
Jaime Rojas
On 20/08/2015, at 11:47 a.m., Somesh Naidu somesh.na...@citrix.com wrote:
Can you not remove the failed host from CCP and XS cluster?
Regards,
Somesh
-Original Message-
From: Jaime Orlando Rojas Sanchez [mailto:jaime.ro...@kumo.com.co]
Sent: Thursday, August 20, 2015 12:38 PM
To: users@cloudstack.apache.org
Subject: RE: VM stuck in a failing Host
Hello,
state/status of the failed host in CS? = Disconnected
We run the command in the 2 remaining host with no results.
[root@dc1fdtptgcx04 /]# xe vm-list name-label= i-574-14584-VM
[root@dc1fdtptgcx04 /]#
[root@dc1fdtptgcx02 /]# xe vm-list name-label= i-574-14584-VM
[root@dc1fdtptgcx02 /]#
Regards / Cordialmente,
Jaime O. Rojas S.
Technology Manager
jaime.ro...@kumo.com.co
Mobile: +57 301-3382382
Office: +57-1-8766767 x215
-Mensaje original-
De: Somesh Naidu [mailto:somesh.na...@citrix.com]
Enviado el: jueves, 20 de agosto de 2015 11:00 a. m.
Para: users@cloudstack.apache.org
Asunto: RE: VM stuck in a failing Host
Quick question, what is the state/status of the failed host in CS? Also,
look up the particular VM on XS pool (xe vm-list name-label=
i-574-14584-VM), what does it say?
Regards,
Somesh
-Original Message-
From: Jaime Orlando Rojas Sanchez [mailto:jaime.ro...@kumo.com.co]
Sent: Thursday, August 20, 2015 11:37 AM
To: users@cloudstack.apache.org
Subject: RE: VM stuck in a failing Host
Following the logs when I click 'run' in ACS after did the following in the
DB
- Change the state to 'stopped'
- Change host ID to a working host
- Change last host ID to a working host
- Check VR is up and running on a working host
-bash-4.1# tail -f management-server.log | grep 14584
2015-08-20 05:45:27,513 DEBUG [cloud.async.AsyncJobManagerImpl]
(catalina-exec-21:null) submit async job-45973 = [
d3ed77b7-534e-4a3b-9038-a66359162087 ], details: AsyncJobVO {id:45973,
userId: 2, accountId: 2, sessionKey: null, instanceType: VirtualMachine,
instanceId: 14584, cmd: org.apache.cloudstack.api.command.user.vm.StopVMCmd,
cmdOriginator: null, cmdInfo:
{response:json,id:98227dc9-682e-4f42-87e1-bd4b8045c7c9,sessionkey:hwnxmM0He9EXs2craugKg3XyWL4\u003d,cmdEventType:VM.STOP,ctxUserId:2,httpmethod:GET,_:1440067422009,ctxAccountId:2,ctxStartEventId:38650949,forced:true},
cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0,
processStatus: 0, resultCode: 0, result: null, initMsid: 139549854171544,
completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
2015-08-20 05:46:33,404 DEBUG [agent.transport.Request]
(Job-Executor-31:job-45974 = [ 9996fc72-fb83-4e5d-94c5-886396dac536 ]) Seq
792-1073807425: Sending { Cmd , MgmtId: 139549854171544, via: 792, Ver: v1,
Flags: 100111,
duplicate use of tips?
Hi, In a zone with Basic Networking, I’ve assigned a certain netblock x.y.z.0/24 to the Guest network in CloudStack. I have a VM-A that has primary ip address x.y.z.92 and secondary address on the same nic x.y.z.52. For various reasons, *both* ips were configured manually, so not using the VR’s dhcp. A while back, the x.y.z.92 was (manually) deconfigured on VM-A (so VM-A only used x.y.x.52, but both ips are still configured to belong to VM-A in Cloudstack). A few days ago, a new instance (VM-B) was spun up, and CS assigned ip x.y.z.92 to that VM. Why would it do that? Today, a new instance was spun up (VM-C), but CS assigned x.y.z.52 to VM-C… How and why does this happen?! Is this because VM-A does not use dhcp? That might explain the .92 re-assignement, but certainly not the .52 reassignment, as secondary ips don’t use dhcp anyhow… Can anyone tell me what’s going on, and what can be done to prevent his? Running CS 4.4.2 at the moment, considering upgrading to 4.5.1 (or 4.5.2) Regards, Frank
Re: VM stuck in a failing Host
We ran the command without the space with no results.
In the second message I sent, I pasted the logs. You can find it in this mail
chain at the beginning.
Cordialmente,
Jaime Rojas
Sent from my iPhone
On 20/08/2015, at 1:37 p.m., Somesh Naidu
somesh.na...@citrix.commailto:somesh.na...@citrix.com wrote:
We do not need to take a snapshot. Just extract the VHD from storage (you could
use vdi-copy if it is on block storage) and then register it in CS as a
template from UI.
xe vm-list name-label= i-574-14584-VM
The command you tried above has space after =. Did you try without the space?
Also, can you share the management server logs that capture the VM start job so
we can take a look?
Regards,
Somesh
-Original Message-
From: Jaime Orlando Rojas Sanchez [mailto:jaime.ro...@kumo.com.co]
Sent: Thursday, August 20, 2015 1:59 PM
To: users@cloudstack.apache.orgmailto:users@cloudstack.apache.org
Subject: Re: VM stuck in a failing Host
We already did all of that and VM keeps trying to start on failed host.
Template is not possible we tried to take a snapshot but it return an error.
Cordialmente,
Jaime Rojas
On 20/08/2015, at 12:19 p.m., Somesh Naidu
somesh.na...@citrix.commailto:somesh.na...@citrix.com wrote:
I do not believe that should happen as the VM is primarily tied to the storage.
But just to avoid taking any chances, we could do this:
1. Put the failed host in to maintenance.
2. Set host_id and last_host_id to NULL in vm_instance table for that VM.
3. Make sure the XS pool doesn't have any stale data for this VM. Look for the
VM and associated VDI in the pool one more time. I noticed there is a space
name-label= i-574-14584-VM after the equal operator, remove that space and
try again or use XenCenter.
If #3 doesn't raise any flags, go ahead and try starting the VM again.
There is also another alternative, you could register the root disk of the VM
as a template and launch a new VM from that template.
Regards,
Somesh
-Original Message-
From: Jaime Orlando Rojas Sanchez [mailto:jaime.ro...@kumo.com.co]
Sent: Thursday, August 20, 2015 1:00 PM
To: users@cloudstack.apache.orgmailto:users@cloudstack.apache.org
Subject: Re: VM stuck in a failing Host
Yes we could , but I'm not sure if maybe I lost the VM because it is tied to
that host.
Cordialmente,
Jaime Rojas
On 20/08/2015, at 11:47 a.m., Somesh Naidu
somesh.na...@citrix.commailto:somesh.na...@citrix.com wrote:
Can you not remove the failed host from CCP and XS cluster?
Regards,
Somesh
-Original Message-
From: Jaime Orlando Rojas Sanchez [mailto:jaime.ro...@kumo.com.co]
Sent: Thursday, August 20, 2015 12:38 PM
To: users@cloudstack.apache.orgmailto:users@cloudstack.apache.org
Subject: RE: VM stuck in a failing Host
Hello,
state/status of the failed host in CS? = Disconnected
We run the command in the 2 remaining host with no results.
[root@dc1fdtptgcx04 /]# xe vm-list name-label= i-574-14584-VM
[root@dc1fdtptgcx04 /]#
[root@dc1fdtptgcx02 /]# xe vm-list name-label= i-574-14584-VM
[root@dc1fdtptgcx02 /]#
Regards / Cordialmente,
Jaime O. Rojas S.
Technology Manager
jaime.ro...@kumo.com.comailto:jaime.ro...@kumo.com.co
Mobile: +57 301-3382382
Office: +57-1-8766767 x215
-Mensaje original-
De: Somesh Naidu [mailto:somesh.na...@citrix.com]
Enviado el: jueves, 20 de agosto de 2015 11:00 a. m.
Para: users@cloudstack.apache.orgmailto:users@cloudstack.apache.org
Asunto: RE: VM stuck in a failing Host
Quick question, what is the state/status of the failed host in CS? Also, look
up the particular VM on XS pool (xe vm-list name-label= i-574-14584-VM), what
does it say?
Regards,
Somesh
-Original Message-
From: Jaime Orlando Rojas Sanchez [mailto:jaime.ro...@kumo.com.co]
Sent: Thursday, August 20, 2015 11:37 AM
To: users@cloudstack.apache.orgmailto:users@cloudstack.apache.org
Subject: RE: VM stuck in a failing Host
Following the logs when I click 'run' in ACS after did the following in the DB
- Change the state to 'stopped'
- Change host ID to a working host
- Change last host ID to a working host
- Check VR is up and running on a working host
-bash-4.1# tail -f management-server.log | grep 14584
2015-08-20 05:45:27,513 DEBUG [cloud.async.AsyncJobManagerImpl]
(catalina-exec-21:null) submit async job-45973 = [
d3ed77b7-534e-4a3b-9038-a66359162087 ], details: AsyncJobVO {id:45973, userId:
2, accountId: 2, sessionKey: null, instanceType: VirtualMachine, instanceId:
14584, cmd: org.apache.cloudstack.api.command.user.vm.StopVMCmd, cmdOriginator:
null, cmdInfo:
{response:json,id:98227dc9-682e-4f42-87e1-bd4b8045c7c9,sessionkey:hwnxmM0He9EXs2craugKg3XyWL4\u003d,cmdEventType:VM.STOP,ctxUserId:2,httpmethod:GET,_:1440067422009,ctxAccountId:2,ctxStartEventId:38650949,forced:true},
cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0,
processStatus: 0, resultCode: 0, result: null, initMsid: 139549854171544,
completeMsid:
Re: The agent doesn't reconnect if there are stopped VMs
Vladimir,
Could you turn up debugging on the agent and post another agent log?
You can do this by running: sed -i 's/INFO/DEBUG/g'
/etc/cloudstack/agent/log4j-cloud.xml
Then restart the agent.
- Si
From: Vladimir Melnik v.mel...@uplink.ua
Sent: Thursday, August 20, 2015 4:36 AM
To: users@cloudstack.apache.org
Subject: The agent doesn't reconnect if there are stopped VMs
Dear colleagues,
I have a simple setup where the management server (CentOS-6.6 +
ACS-4.5.1) is orchestrating a bunch of KVM hosts (each of them is
running CentOS-6.6 + ACS-4.5.1 as well).
Any host with at least one VM in the Stopped state can't reconnect to
the management server. It has the Alert state and here's what I see in
the management server's log-file:
--- 8 ---
2015-08-18 06:24:46,332 DEBUG [c.c.a.t.Request]
(AgentConnectTaskPool-213:ctx-76903ef6) Seq 0-148: Processing the first command
{ Cmd ,
MgmtId: -1, via: 0, Ver: v1, Flags: 1,
[{com.cloud.agent.api.StartupRoutingCommand:{cpuSockets:1,cpus:48,speed:2299,memory:6743
9632384,dom0MinMemory:805306368,poolSync:false,caps:hvm,snapshot,pool:/root,hypervisorType:KVM,hostDetails:{com.cloud.
network.Networks.RouterPrivateIpStrategy:HostLocal,Host.OS:CentOS,Host.OS.Kernel.Version:2.6.32-504.23.4.el6.x86_64,Host.OS.V
ersion:6.6},hostTags:[],groupDetails:{},type:Routing,dataCenter:6,pod:7,cluster:7,guid:1318c38d-4ed6-3296-a6bd-753676e25ad4-LibvirtComputingResource,name:***.***.***,id:0,version:4.5.1,publicIpAddress:172.27.65.1,publicNetmask:255.255.255.0,publicMacAddress:ec:f4:bb:d6:89:c5,privateIpAddress:172.27.65.1,privateMacAddress:ec:f4:bb:d6:89:c5,privateNetmask:255.255.255.0,storageIpAddress:172.27.65.1,storageNetmask:255.255.255.0,storageMacAddress:ec:f4:bb:d6:89:c5,resourceName:LibvirtComputingResource,gatewayIpAddress:***.***.***.***,wait:0}},{com.cloud.agent.api.StartupStorageCommand:{totalSize:0,poolInfo:{uuid:51670fbd-ece2-4a3e-9971-3928e6576f0e,host:172.27.65.1,localPath:/var/lib/libvirt/images,hostPath:/var/lib/libvirt/images,poolType:Filesystem,capacityBytes:1563804868608,availableBytes:1474368700416},resourceType:STORAGE_POOL,hostDetails:{},type:Storage,dataCenter:6,pod:7,guid:1318c38d-4ed6-3296-a6bd-753676e25ad4-LibvirtComputingResource,name:***.***.***,id:0,version:4.5.1,resourceName:LibvirtComputingResource,wait:0}}]
}
2015-08-18 06:24:46,336 DEBUG [c.c.r.ResourceManagerImpl]
(AgentConnectTaskPool-213:ctx-76903ef6) Dispatching resource state event
CREATE_HOST_VO_FOR_CONNECTED to BaremetalDhcpManagerImpl
2015-08-18 06:24:46,336 DEBUG [c.c.r.ResourceManagerImpl]
(AgentConnectTaskPool-213:ctx-76903ef6) Dispatching resource state event
CREATE_HOST_VO_FOR_CONNECTED to BaremetalPxeManagerImpl
2015-08-18 06:24:46,336 DEBUG [c.c.r.ResourceManagerImpl]
(AgentConnectTaskPool-213:ctx-76903ef6) Dispatching resource state event
CREATE_HOST_VO_FOR_CONNECTED to NetworkUsageManagerImpl
2015-08-18 06:24:46,336 DEBUG [c.c.r.ResourceManagerImpl]
(AgentConnectTaskPool-213:ctx-76903ef6) Dispatching resource state event
CREATE_HOST_VO_FOR_CONNECTED to NuageVspElement
2015-08-18 06:24:46,336 DEBUG [c.c.r.ResourceManagerImpl]
(AgentConnectTaskPool-213:ctx-76903ef6) Dispatching resource state event
CREATE_HOST_VO_FOR_CONNECTED to Ovs
2015-08-18 06:24:46,336 DEBUG [c.c.r.ResourceManagerImpl]
(AgentConnectTaskPool-213:ctx-76903ef6) Dispatching resource state event
CREATE_HOST_VO_FOR_CONNECTED to PaloAltoExternalFirewallElement
2015-08-18 06:24:46,336 DEBUG [c.c.r.ResourceManagerImpl]
(AgentConnectTaskPool-213:ctx-76903ef6) Dispatching resource state event
CREATE_HOST_VO_FOR_CONNECTED to GloboDnsElement
2015-08-18 06:24:46,336 DEBUG [c.c.r.ResourceManagerImpl]
(AgentConnectTaskPool-213:ctx-76903ef6) Dispatching resource state event
CREATE_HOST_VO_FOR_CONNECTED to KvmServerDiscoverer
2015-08-18 06:24:46,362 DEBUG [c.c.r.ResourceState]
(AgentConnectTaskPool-213:ctx-76903ef6) Resource state update: [id = 27; name =
***.***.***; old state = Enabled; event = InternalCreated; new state = Enabled]
2015-08-18 06:24:46,362 DEBUG [c.c.h.Status]
(AgentConnectTaskPool-213:ctx-76903ef6) Transition:[Resource state = Enabled,
Agent event = AgentConnected, Host id = 27, name = ***.***.***]
2015-08-18 06:24:46,365 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
(AgentConnectTaskPool-213:ctx-76903ef6) create ClusteredAgentAttache for 27
2015-08-18 06:24:46,367 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentConnectTaskPool-213:ctx-76903ef6) Sending Connect to listener:
XcpServerDiscoverer
2015-08-18 06:24:46,367 DEBUG [c.c.h.x.d.XcpServerDiscoverer]
(AgentConnectTaskPool-213:ctx-76903ef6) Not XenServer so moving on.
2015-08-18 06:24:46,367 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentConnectTaskPool-213:ctx-76903ef6) Sending Connect to listener:
HypervServerDiscoverer
2015-08-18 06:24:46,367 DEBUG [c.c.h.h.d.HypervServerDiscoverer]
(AgentConnectTaskPool-213:ctx-76903ef6) Not Hyper-V hypervisor, so moving on.
2015-08-18 06:24:46,367 DEBUG
Re: The agent doesn't reconnect if there are stopped VMs
Oh, I'm sorry, I should have initially send the DEBUG-log!
Here is an example:
--- 8 ---
2015-07-29 00:53:42,988 INFO [utils.nio.NioClient] (Agent-Selector:null)
Connecting to ***.***.***.***:8250
2015-07-29 00:53:44,254 INFO [utils.nio.NioClient] (Agent-Selector:null) SSL:
Handshake done
2015-07-29 00:53:44,255 INFO [utils.nio.NioClient] (Agent-Selector:null)
Connected to ***.***.***.***:8250
2015-07-29 00:53:44,258 WARN [kvm.resource.LibvirtComputingResource]
(Agent-Handler-1:null) Could not read cpuinfo_max_freq
2015-07-29 00:53:44,266 DEBUG [kvm.resource.LibvirtCapXMLParser]
(Agent-Handler-1:null) Found /usr/libexec/qemu-kvm as a suiteable emulat
or
2015-07-29 00:53:44,266 DEBUG [kvm.resource.LibvirtComputingResource]
(Agent-Handler-1:null) Executing: /bin/bash -c qemu-img --help|grep
convert
2015-07-29 00:53:44,270 DEBUG [kvm.resource.LibvirtComputingResource]
(Agent-Handler-1:null) Execution is successful.
2015-07-29 00:53:44,271 DEBUG [kvm.resource.LibvirtComputingResource]
(Agent-Handler-1:null) convert [-c] [-p] [-f fmt] [-t cache] [-T
src_cache] [-O output_fmt] [-o options] [-S sparse_size] filename [filename2
[...]] output_filename
2015-07-29 00:53:44,271 DEBUG [kvm.resource.LibvirtComputingResource]
(Agent-Handler-1:null) cpus=8, speed=2660, ram=30153224192, dom0ram
=805306368, cpu sockets=1
2015-07-29 00:53:44,272 DEBUG [cloud.resource.ServerResourceBase]
(Agent-Handler-1:null) Parameters for private nic: 172.26.65.1 - 84:2b:
2b:56:d3:d9-255.255.255.0
2015-07-29 00:53:44,272 DEBUG [cloud.resource.ServerResourceBase]
(Agent-Handler-1:null) Parameters for storage nic: 172.26.65.1 - 84:2b:
2b:56:d3:d9-255.255.255.0
2015-07-29 00:53:44,272 DEBUG [cloud.resource.ServerResourceBase]
(Agent-Handler-1:null) Parameters for pubic nic: 172.26.65.1 - 84:2b:2b
:56:d3:d9-255.255.255.0
2015-07-29 00:53:44,272 DEBUG [kvm.resource.LibvirtComputingResource]
(Agent-Handler-1:null) Executing: /usr/share/cloudstack-common/scri
pts/vm/hypervisor/versions.sh
2015-07-29 00:53:44,281 DEBUG [kvm.resource.LibvirtComputingResource]
(Agent-Handler-1:null) Execution is successful.
2015-07-29 00:53:44,282 DEBUG [kvm.resource.LibvirtComputingResource]
(Agent-Handler-1:null) Executing: sudo grep InitiatorName= /etc/isc
si/initiatorname.iscsi
2015-07-29 00:53:44,290 DEBUG [kvm.resource.LibvirtComputingResource]
(Agent-Handler-1:null) Execution is successful.
2015-07-29 00:53:44,290 INFO [kvm.storage.LibvirtStorageAdaptor]
(Agent-Handler-1:null) Attempting to create storage pool 42bed7d9-88ae-
403d-9b53-0b44f31b2312 (Filesystem) in libvirt
2015-07-29 00:53:44,292 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(Agent-Handler-1:null) Found existing defined storage pool 42bed7d9-88a
e-403d-9b53-0b44f31b2312, using it.
2015-07-29 00:53:44,292 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(Agent-Handler-1:null) Trying to fetch storage pool
42bed7d9-88ae-403d-9b53-0b44f31b2312 from libvirt
2015-07-29 00:53:44,811 DEBUG [cloud.agent.Agent] (Agent-Handler-1:null)
Executing: hostname
2015-07-29 00:53:44,813 DEBUG [cloud.agent.Agent] (Agent-Handler-1:null)
Execution is successful.
2015-07-29 00:53:44,834 DEBUG [cloud.agent.Agent] (Agent-Handler-1:null)
Executing: hostname
2015-07-29 00:53:44,836 DEBUG [cloud.agent.Agent] (Agent-Handler-1:null)
Execution is successful.
2015-07-29 00:53:44,838 DEBUG [cloud.agent.Agent] (Agent-Handler-1:null)
Sending Startup: Seq 0-64: { Cmd , MgmtId: -1, via: 0, Ver: v1, Flags: 1,
[{com.cloud.agent.api.StartupRoutingCommand:{cpuSockets:1,cpus:8,speed:2660,memory:30153224192,dom0MinMemory:805306368,poolSync:false,caps:hvm,snapshot,pool:/root,hypervisorType:KVM,hostDetails:{com.cloud.network.Networks.RouterPrivateIpStrategy:HostLocal,Host.OS:CentOS,Host.OS.Kernel.Version:2.6.32-504.16.2.el6.x86_64,Host.OS.Version:6.6},hostTags:[],groupDetails:{},type:Routing,dataCenter:4,pod:5,cluster:5,guid:26e2bf7d-2fcf-3a67-a23d-ce9c09ef2ca5-LibvirtComputingResource,name:***.***.***,id:0,version:4.5.1,iqn:iqn.1994-05.com.redhat:f044a5e741a1,publicIpAddress:172.26.65.1,publicNetmask:255.255.255.0,publicMacAddress:84:2b:2b:56:d3:d9,privateIpAddress:172.26.65.1,privateMacAddress:84:2b:2b:56:d3:d9,privateNetmask:255.255.255.0,storageIpAddress:172.26.65.1,storageNetmask:255.255.255.0,storageMacAddress:84:2b:2b:56:d3:d9,resourceName:LibvirtComputingResource,gatewayIpAddress:103.247.149.1,wait:0}},{com.cloud.agent.api.StartupStorageCommand:{totalSize:0,poolInfo:{uuid:42bed7d9-88ae-403d-9b53-0b44f31b2312,host:172.26.65.1,localPath:/var/lib/libvirt/images,hostPath:/var/lib/libvirt/images,poolType:Filesystem,capacityBytes:913829568512,availableBytes:810211274752},resourceType:STORAGE_POOL,hostDetails:{},type:Storage,dataCenter:4,pod:5,guid:26e2bf7d-2fcf-3a67-a23d-ce9c09ef2ca5-LibvirtComputingResource,name:***.***.***,id:0,version:4.5.1,resourceName:LibvirtComputingResource,wait:0}}]
}
2015-07-29 00:53:44,838 DEBUG [cloud.agent.Agent] (Agent-Handler-1:null)
Startup task created
2015-07-29
RE: VM stuck in a failing Host
Thanks I just realized that the job was a stop command and not start command
that I was looking for.
The stop command failed to stop the VM leaving the VM in Running state:
2015-08-20 05:45:27,513 DEBUG [cloud.async.AsyncJobManagerImpl]
(catalina-exec-21:null) submit async job-45973 = [
d3ed77b7-534e-4a3b-9038-a66359162087 ], details: AsyncJobVO {id:45973, userId:
2, accountId: 2, sessionKey: null, instanceType: VirtualMachine, instanceId:
14584, cmd: org.apache.cloudstack.api.command.user.vm.StopVMCmd, cmdOriginator:
null, cmdInfo:
{response:json,id:98227dc9-682e-4f42-87e1-bd4b8045c7c9,sessionkey:hwnxmM0He9EXs2craugKg3XyWL4\u003d,cmdEventType:VM.STOP,ctxUserId:2,httpmethod:GET,_:1440067422009,ctxAccountId:2,ctxStartEventId:38650949,forced:true},
cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0,
processStatus: 0, resultCode: 0, result: null, initMsid: 139549854171544,
completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
2015-08-20 05:45:27,605 DEBUG [cloud.capacity.CapacityManagerImpl]
(Job-Executor-30:job-45973 = [ d3ed77b7-534e-4a3b-9038-a66359162087 ]) VM state
transitted from :Running to Stopping with event: StopRequestedvm's original
host id: 599 new host id: 599 host id before state transition: 599
2015-08-20 05:45:27,617 WARN [cloud.vm.VirtualMachineManagerImpl]
(Job-Executor-30:job-45973 = [ d3ed77b7-534e-4a3b-9038-a66359162087 ]) Unable
to stop vm, agent unavailable: com.cloud.exception.AgentUnavailableException:
Resource [Host:599] is unreachable: Host 599: Host with specified id is not in
the right state: Disconnected
2015-08-20 05:45:32,654 DEBUG [cloud.capacity.CapacityManagerImpl]
(Job-Executor-30:job-45973 = [ d3ed77b7-534e-4a3b-9038-a66359162087 ]) VM state
transitted from :Stopping to Running with event: OperationFailedvm's original
host id: 599 new host id: 599 host id before state transition: 599
2015-08-20 05:45:33,683 DEBUG [cloud.async.AsyncJobManagerImpl]
(Job-Executor-30:job-45973 = [ d3ed77b7-534e-4a3b-9038-a66359162087 ]) Complete
async job-45973 = [ d3ed77b7-534e-4a3b-9038-a66359162087 ], jobStatus: 2,
resultCode: 530, result: Error Code: 530 Error text: Failed to stop vm
I can't tell why the VM keeps starting on the failed host unless I have a look
at the logs from a start command.
But I think there is a good chance that if you check the Force Stop check box
when stopping that VM and then try to start it again.
Regards,
Somesh
-Original Message-
From: Jaime Orlando Rojas Sanchez [mailto:jaime.ro...@kumo.com.co]
Sent: Thursday, August 20, 2015 4:40 PM
To: users@cloudstack.apache.org
Subject: RE: VM stuck in a failing Host
Hello,
I am attaching the logs
http://jrojas-test.s3.kumo.com.co/management-server.log?AWSAccessKeyId=0099c1623d6704f9f5d5Expires=1440711480Signature=CFYwowzsJ3x%2B7XXfpAHDLM%2F022A%3Dx-amz-pt=N2ZiYjcyOTk4MTQ0MDEwMzEwNTQ5MA
We have a second manage server but it doesn't execute any task.
Regards / Cordialmente,
Jaime O. Rojas S.
Technology Manager
jaime.ro...@kumo.com.co
Mobile: +57 301-3382382
Office: +57-1-8766767 x215
-Mensaje original-
De: Somesh Naidu [mailto:somesh.na...@citrix.com]
Enviado el: jueves, 20 de agosto de 2015 2:17 p. m.
Para: users@cloudstack.apache.org
Asunto: RE: VM stuck in a failing Host
Jaime,
Those are not complete logs I need. I need complete logs (no grep) between
2015-08-20 05:45:27,513 and when the job (job-45973) finished (I can't tell
from the logs you shared earlier).
Also, do you have more than 1 management server?
Regards,
Somesh
-Original Message-
From: Jaime Orlando Rojas Sanchez [mailto:jaime.ro...@kumo.com.co]
Sent: Thursday, August 20, 2015 2:41 PM
To: users@cloudstack.apache.org
Subject: Re: VM stuck in a failing Host
We ran the command without the space with no results.
In the second message I sent, I pasted the logs. You can find it in this mail
chain at the beginning.
Cordialmente,
Jaime Rojas
Sent from my iPhone
On 20/08/2015, at 1:37 p.m., Somesh Naidu
somesh.na...@citrix.commailto:somesh.na...@citrix.com wrote:
We do not need to take a snapshot. Just extract the VHD from storage (you could
use vdi-copy if it is on block storage) and then register it in CS as a
template from UI.
xe vm-list name-label= i-574-14584-VM
The command you tried above has space after =. Did you try without the space?
Also, can you share the management server logs that capture the VM start job so
we can take a look?
Regards,
Somesh
-Original Message-
From: Jaime Orlando Rojas Sanchez [mailto:jaime.ro...@kumo.com.co]
Sent: Thursday, August 20, 2015 1:59 PM
To: users@cloudstack.apache.orgmailto:users@cloudstack.apache.org
Subject: Re: VM stuck in a failing Host
We already did all of that and VM keeps trying to start on failed host.
Template is not possible we tried to take a snapshot but it return an error.
Cordialmente,
Jaime Rojas
On 20/08/2015, at 12:19 p.m., Somesh Naidu
RE: VM stuck in a failing Host
Hello, I am attaching the logs http://jrojas-test.s3.kumo.com.co/management-server.log?AWSAccessKeyId=0099c1623d6704f9f5d5Expires=1440711480Signature=CFYwowzsJ3x%2B7XXfpAHDLM%2F022A%3Dx-amz-pt=N2ZiYjcyOTk4MTQ0MDEwMzEwNTQ5MA We have a second manage server but it doesn't execute any task. Regards / Cordialmente, Jaime O. Rojas S. Technology Manager jaime.ro...@kumo.com.co Mobile: +57 301-3382382 Office: +57-1-8766767 x215 -Mensaje original- De: Somesh Naidu [mailto:somesh.na...@citrix.com] Enviado el: jueves, 20 de agosto de 2015 2:17 p. m. Para: users@cloudstack.apache.org Asunto: RE: VM stuck in a failing Host Jaime, Those are not complete logs I need. I need complete logs (no grep) between 2015-08-20 05:45:27,513 and when the job (job-45973) finished (I can't tell from the logs you shared earlier). Also, do you have more than 1 management server? Regards, Somesh -Original Message- From: Jaime Orlando Rojas Sanchez [mailto:jaime.ro...@kumo.com.co] Sent: Thursday, August 20, 2015 2:41 PM To: users@cloudstack.apache.org Subject: Re: VM stuck in a failing Host We ran the command without the space with no results. In the second message I sent, I pasted the logs. You can find it in this mail chain at the beginning. Cordialmente, Jaime Rojas Sent from my iPhone On 20/08/2015, at 1:37 p.m., Somesh Naidu somesh.na...@citrix.commailto:somesh.na...@citrix.com wrote: We do not need to take a snapshot. Just extract the VHD from storage (you could use vdi-copy if it is on block storage) and then register it in CS as a template from UI. xe vm-list name-label= i-574-14584-VM The command you tried above has space after =. Did you try without the space? Also, can you share the management server logs that capture the VM start job so we can take a look? Regards, Somesh -Original Message- From: Jaime Orlando Rojas Sanchez [mailto:jaime.ro...@kumo.com.co] Sent: Thursday, August 20, 2015 1:59 PM To: users@cloudstack.apache.orgmailto:users@cloudstack.apache.org Subject: Re: VM stuck in a failing Host We already did all of that and VM keeps trying to start on failed host. Template is not possible we tried to take a snapshot but it return an error. Cordialmente, Jaime Rojas On 20/08/2015, at 12:19 p.m., Somesh Naidu somesh.na...@citrix.commailto:somesh.na...@citrix.com wrote: I do not believe that should happen as the VM is primarily tied to the storage. But just to avoid taking any chances, we could do this: 1. Put the failed host in to maintenance. 2. Set host_id and last_host_id to NULL in vm_instance table for that VM. 3. Make sure the XS pool doesn't have any stale data for this VM. Look for the VM and associated VDI in the pool one more time. I noticed there is a space name-label= i-574-14584-VM after the equal operator, remove that space and try again or use XenCenter. If #3 doesn't raise any flags, go ahead and try starting the VM again. There is also another alternative, you could register the root disk of the VM as a template and launch a new VM from that template. Regards, Somesh -Original Message- From: Jaime Orlando Rojas Sanchez [mailto:jaime.ro...@kumo.com.co] Sent: Thursday, August 20, 2015 1:00 PM To: users@cloudstack.apache.orgmailto:users@cloudstack.apache.org Subject: Re: VM stuck in a failing Host Yes we could , but I'm not sure if maybe I lost the VM because it is tied to that host. Cordialmente, Jaime Rojas On 20/08/2015, at 11:47 a.m., Somesh Naidu somesh.na...@citrix.commailto:somesh.na...@citrix.com wrote: Can you not remove the failed host from CCP and XS cluster? Regards, Somesh -Original Message- From: Jaime Orlando Rojas Sanchez [mailto:jaime.ro...@kumo.com.co] Sent: Thursday, August 20, 2015 12:38 PM To: users@cloudstack.apache.orgmailto:users@cloudstack.apache.org Subject: RE: VM stuck in a failing Host Hello, state/status of the failed host in CS? = Disconnected We run the command in the 2 remaining host with no results. [root@dc1fdtptgcx04 /]# xe vm-list name-label= i-574-14584-VM [root@dc1fdtptgcx04 /]# [root@dc1fdtptgcx02 /]# xe vm-list name-label= i-574-14584-VM [root@dc1fdtptgcx02 /]# Regards / Cordialmente, Jaime O. Rojas S. Technology Manager jaime.ro...@kumo.com.comailto:jaime.ro...@kumo.com.co Mobile: +57 301-3382382 Office: +57-1-8766767 x215 -Mensaje original- De: Somesh Naidu [mailto:somesh.na...@citrix.com] Enviado el: jueves, 20 de agosto de 2015 11:00 a. m. Para: users@cloudstack.apache.orgmailto:users@cloudstack.apache.org Asunto: RE: VM stuck in a failing Host Quick question, what is the state/status of the failed host in CS? Also, look up the particular VM on XS pool (xe vm-list name-label= i-574-14584-VM), what does it say? Regards, Somesh -Original Message- From: Jaime Orlando Rojas Sanchez [mailto:jaime.ro...@kumo.com.co] Sent: Thursday, August 20, 2015 11:37 AM To: users@cloudstack.apache.orgmailto:users@cloudstack.apache.org Subject: RE:
VM stuck in a failing Host
Hello, We have a 4.2.1 ACS, running on XenServer 6.2.0, we have a zone with a pool of 3 host, yesterday 1 host crash and OS get corrupted. I think we lost that host and have to reinstall it, but the issue is that we had a couple of VM and VR running on that host. The failing host was the master of the pool, so once it fails all the pool was disconnected, we change the master role and recover pool management from Xencenter and ACS, once we did it a VM moved to the remaining host, all VR and 1 VM kept stuck in failing host. In DB we see the VR and VM running, even if the host was marked as down and maintenance. We changed the VR state to 'stopped' and change de last host ID and Host ID to a working host. Once we did it we were able to destroy the VR and recreate them with successful results, they came up on working host. If we change only the state, the VR couldn't be destroyed. Here we workaround with the 70% of the outage, BUT one VM remain stuck to the host, we change the state, the last host ID, but once we press start, it runs on the failing host and the VM appears as running even if it doesn't. Any suggestion to force the VM to start in a different host and remove it from the failing host? This is a critical VM, we hope somebody else could give us a hand. Regards / Cordialmente, Jaime O. Rojas S. Technology Manager jaime.ro...@kumo.com.comailto:jaime.ro...@kumo.com.co Mobile: +57 301-3382382 Office: +57-1-8766767 x215
RE: The agent doesn't reconnect if there are stopped VMs
2015-08-18 06:24:46,542 DEBUG [c.c.c.CapacityManagerImpl]
(AgentConnectTaskPool-213:ctx-76903ef6) Found 1 VM, not running on host 27
2015-08-18 06:24:46,544 ERROR [c.c.a.m.AgentManagerImpl]
(AgentConnectTaskPool-213:ctx-76903ef6) Monitor ComputeCapacityListener says
there is an error in the connect process for 27 due to null
java.lang.NullPointerException
It will help if you send the complete exception stack for the above NPE.
Regards,
Somesh
-Original Message-
From: Simon Weller [mailto:swel...@ena.com]
Sent: Thursday, August 20, 2015 11:21 AM
To: users@cloudstack.apache.org
Subject: Re: The agent doesn't reconnect if there are stopped VMs
Do you have any dynamic service offerings?
From: Vladimir Melnik v.mel...@uplink.ua
Sent: Thursday, August 20, 2015 8:20 AM
To: users@cloudstack.apache.org
Subject: Re: The agent doesn't reconnect if there are stopped VMs
Oh, I'm sorry, I should have initially send the DEBUG-log!
Here is an example:
--- 8 ---
2015-07-29 00:53:42,988 INFO [utils.nio.NioClient] (Agent-Selector:null)
Connecting to ***.***.***.***:8250
2015-07-29 00:53:44,254 INFO [utils.nio.NioClient] (Agent-Selector:null) SSL:
Handshake done
2015-07-29 00:53:44,255 INFO [utils.nio.NioClient] (Agent-Selector:null)
Connected to ***.***.***.***:8250
2015-07-29 00:53:44,258 WARN [kvm.resource.LibvirtComputingResource]
(Agent-Handler-1:null) Could not read cpuinfo_max_freq
2015-07-29 00:53:44,266 DEBUG [kvm.resource.LibvirtCapXMLParser]
(Agent-Handler-1:null) Found /usr/libexec/qemu-kvm as a suiteable emulat
or
2015-07-29 00:53:44,266 DEBUG [kvm.resource.LibvirtComputingResource]
(Agent-Handler-1:null) Executing: /bin/bash -c qemu-img --help|grep
convert
2015-07-29 00:53:44,270 DEBUG [kvm.resource.LibvirtComputingResource]
(Agent-Handler-1:null) Execution is successful.
2015-07-29 00:53:44,271 DEBUG [kvm.resource.LibvirtComputingResource]
(Agent-Handler-1:null) convert [-c] [-p] [-f fmt] [-t cache] [-T
src_cache] [-O output_fmt] [-o options] [-S sparse_size] filename [filename2
[...]] output_filename
2015-07-29 00:53:44,271 DEBUG [kvm.resource.LibvirtComputingResource]
(Agent-Handler-1:null) cpus=8, speed=2660, ram=30153224192, dom0ram
=805306368, cpu sockets=1
2015-07-29 00:53:44,272 DEBUG [cloud.resource.ServerResourceBase]
(Agent-Handler-1:null) Parameters for private nic: 172.26.65.1 - 84:2b:
2b:56:d3:d9-255.255.255.0
2015-07-29 00:53:44,272 DEBUG [cloud.resource.ServerResourceBase]
(Agent-Handler-1:null) Parameters for storage nic: 172.26.65.1 - 84:2b:
2b:56:d3:d9-255.255.255.0
2015-07-29 00:53:44,272 DEBUG [cloud.resource.ServerResourceBase]
(Agent-Handler-1:null) Parameters for pubic nic: 172.26.65.1 - 84:2b:2b
:56:d3:d9-255.255.255.0
2015-07-29 00:53:44,272 DEBUG [kvm.resource.LibvirtComputingResource]
(Agent-Handler-1:null) Executing: /usr/share/cloudstack-common/scri
pts/vm/hypervisor/versions.sh
2015-07-29 00:53:44,281 DEBUG [kvm.resource.LibvirtComputingResource]
(Agent-Handler-1:null) Execution is successful.
2015-07-29 00:53:44,282 DEBUG [kvm.resource.LibvirtComputingResource]
(Agent-Handler-1:null) Executing: sudo grep InitiatorName= /etc/isc
si/initiatorname.iscsi
2015-07-29 00:53:44,290 DEBUG [kvm.resource.LibvirtComputingResource]
(Agent-Handler-1:null) Execution is successful.
2015-07-29 00:53:44,290 INFO [kvm.storage.LibvirtStorageAdaptor]
(Agent-Handler-1:null) Attempting to create storage pool 42bed7d9-88ae-
403d-9b53-0b44f31b2312 (Filesystem) in libvirt
2015-07-29 00:53:44,292 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(Agent-Handler-1:null) Found existing defined storage pool 42bed7d9-88a
e-403d-9b53-0b44f31b2312, using it.
2015-07-29 00:53:44,292 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(Agent-Handler-1:null) Trying to fetch storage pool
42bed7d9-88ae-403d-9b53-0b44f31b2312 from libvirt
2015-07-29 00:53:44,811 DEBUG [cloud.agent.Agent] (Agent-Handler-1:null)
Executing: hostname
2015-07-29 00:53:44,813 DEBUG [cloud.agent.Agent] (Agent-Handler-1:null)
Execution is successful.
2015-07-29 00:53:44,834 DEBUG [cloud.agent.Agent] (Agent-Handler-1:null)
Executing: hostname
2015-07-29 00:53:44,836 DEBUG [cloud.agent.Agent] (Agent-Handler-1:null)
Execution is successful.
2015-07-29 00:53:44,838 DEBUG [cloud.agent.Agent] (Agent-Handler-1:null)
Sending Startup: Seq 0-64: { Cmd , MgmtId: -1, via: 0, Ver: v1, Flags: 1,
Re: The agent doesn't reconnect if there are stopped VMs
Do you have any dynamic service offerings?
From: Vladimir Melnik v.mel...@uplink.ua
Sent: Thursday, August 20, 2015 8:20 AM
To: users@cloudstack.apache.org
Subject: Re: The agent doesn't reconnect if there are stopped VMs
Oh, I'm sorry, I should have initially send the DEBUG-log!
Here is an example:
--- 8 ---
2015-07-29 00:53:42,988 INFO [utils.nio.NioClient] (Agent-Selector:null)
Connecting to ***.***.***.***:8250
2015-07-29 00:53:44,254 INFO [utils.nio.NioClient] (Agent-Selector:null) SSL:
Handshake done
2015-07-29 00:53:44,255 INFO [utils.nio.NioClient] (Agent-Selector:null)
Connected to ***.***.***.***:8250
2015-07-29 00:53:44,258 WARN [kvm.resource.LibvirtComputingResource]
(Agent-Handler-1:null) Could not read cpuinfo_max_freq
2015-07-29 00:53:44,266 DEBUG [kvm.resource.LibvirtCapXMLParser]
(Agent-Handler-1:null) Found /usr/libexec/qemu-kvm as a suiteable emulat
or
2015-07-29 00:53:44,266 DEBUG [kvm.resource.LibvirtComputingResource]
(Agent-Handler-1:null) Executing: /bin/bash -c qemu-img --help|grep
convert
2015-07-29 00:53:44,270 DEBUG [kvm.resource.LibvirtComputingResource]
(Agent-Handler-1:null) Execution is successful.
2015-07-29 00:53:44,271 DEBUG [kvm.resource.LibvirtComputingResource]
(Agent-Handler-1:null) convert [-c] [-p] [-f fmt] [-t cache] [-T
src_cache] [-O output_fmt] [-o options] [-S sparse_size] filename [filename2
[...]] output_filename
2015-07-29 00:53:44,271 DEBUG [kvm.resource.LibvirtComputingResource]
(Agent-Handler-1:null) cpus=8, speed=2660, ram=30153224192, dom0ram
=805306368, cpu sockets=1
2015-07-29 00:53:44,272 DEBUG [cloud.resource.ServerResourceBase]
(Agent-Handler-1:null) Parameters for private nic: 172.26.65.1 - 84:2b:
2b:56:d3:d9-255.255.255.0
2015-07-29 00:53:44,272 DEBUG [cloud.resource.ServerResourceBase]
(Agent-Handler-1:null) Parameters for storage nic: 172.26.65.1 - 84:2b:
2b:56:d3:d9-255.255.255.0
2015-07-29 00:53:44,272 DEBUG [cloud.resource.ServerResourceBase]
(Agent-Handler-1:null) Parameters for pubic nic: 172.26.65.1 - 84:2b:2b
:56:d3:d9-255.255.255.0
2015-07-29 00:53:44,272 DEBUG [kvm.resource.LibvirtComputingResource]
(Agent-Handler-1:null) Executing: /usr/share/cloudstack-common/scri
pts/vm/hypervisor/versions.sh
2015-07-29 00:53:44,281 DEBUG [kvm.resource.LibvirtComputingResource]
(Agent-Handler-1:null) Execution is successful.
2015-07-29 00:53:44,282 DEBUG [kvm.resource.LibvirtComputingResource]
(Agent-Handler-1:null) Executing: sudo grep InitiatorName= /etc/isc
si/initiatorname.iscsi
2015-07-29 00:53:44,290 DEBUG [kvm.resource.LibvirtComputingResource]
(Agent-Handler-1:null) Execution is successful.
2015-07-29 00:53:44,290 INFO [kvm.storage.LibvirtStorageAdaptor]
(Agent-Handler-1:null) Attempting to create storage pool 42bed7d9-88ae-
403d-9b53-0b44f31b2312 (Filesystem) in libvirt
2015-07-29 00:53:44,292 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(Agent-Handler-1:null) Found existing defined storage pool 42bed7d9-88a
e-403d-9b53-0b44f31b2312, using it.
2015-07-29 00:53:44,292 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(Agent-Handler-1:null) Trying to fetch storage pool
42bed7d9-88ae-403d-9b53-0b44f31b2312 from libvirt
2015-07-29 00:53:44,811 DEBUG [cloud.agent.Agent] (Agent-Handler-1:null)
Executing: hostname
2015-07-29 00:53:44,813 DEBUG [cloud.agent.Agent] (Agent-Handler-1:null)
Execution is successful.
2015-07-29 00:53:44,834 DEBUG [cloud.agent.Agent] (Agent-Handler-1:null)
Executing: hostname
2015-07-29 00:53:44,836 DEBUG [cloud.agent.Agent] (Agent-Handler-1:null)
Execution is successful.
2015-07-29 00:53:44,838 DEBUG [cloud.agent.Agent] (Agent-Handler-1:null)
Sending Startup: Seq 0-64: { Cmd , MgmtId: -1, via: 0, Ver: v1, Flags: 1,
RE: VM stuck in a failing Host
Jaime,
Those are not complete logs I need. I need complete logs (no grep) between
2015-08-20 05:45:27,513 and when the job (job-45973) finished (I can't tell
from the logs you shared earlier).
Also, do you have more than 1 management server?
Regards,
Somesh
-Original Message-
From: Jaime Orlando Rojas Sanchez [mailto:jaime.ro...@kumo.com.co]
Sent: Thursday, August 20, 2015 2:41 PM
To: users@cloudstack.apache.org
Subject: Re: VM stuck in a failing Host
We ran the command without the space with no results.
In the second message I sent, I pasted the logs. You can find it in this mail
chain at the beginning.
Cordialmente,
Jaime Rojas
Sent from my iPhone
On 20/08/2015, at 1:37 p.m., Somesh Naidu
somesh.na...@citrix.commailto:somesh.na...@citrix.com wrote:
We do not need to take a snapshot. Just extract the VHD from storage (you could
use vdi-copy if it is on block storage) and then register it in CS as a
template from UI.
xe vm-list name-label= i-574-14584-VM
The command you tried above has space after =. Did you try without the space?
Also, can you share the management server logs that capture the VM start job so
we can take a look?
Regards,
Somesh
-Original Message-
From: Jaime Orlando Rojas Sanchez [mailto:jaime.ro...@kumo.com.co]
Sent: Thursday, August 20, 2015 1:59 PM
To: users@cloudstack.apache.orgmailto:users@cloudstack.apache.org
Subject: Re: VM stuck in a failing Host
We already did all of that and VM keeps trying to start on failed host.
Template is not possible we tried to take a snapshot but it return an error.
Cordialmente,
Jaime Rojas
On 20/08/2015, at 12:19 p.m., Somesh Naidu
somesh.na...@citrix.commailto:somesh.na...@citrix.com wrote:
I do not believe that should happen as the VM is primarily tied to the storage.
But just to avoid taking any chances, we could do this:
1. Put the failed host in to maintenance.
2. Set host_id and last_host_id to NULL in vm_instance table for that VM.
3. Make sure the XS pool doesn't have any stale data for this VM. Look for the
VM and associated VDI in the pool one more time. I noticed there is a space
name-label= i-574-14584-VM after the equal operator, remove that space and
try again or use XenCenter.
If #3 doesn't raise any flags, go ahead and try starting the VM again.
There is also another alternative, you could register the root disk of the VM
as a template and launch a new VM from that template.
Regards,
Somesh
-Original Message-
From: Jaime Orlando Rojas Sanchez [mailto:jaime.ro...@kumo.com.co]
Sent: Thursday, August 20, 2015 1:00 PM
To: users@cloudstack.apache.orgmailto:users@cloudstack.apache.org
Subject: Re: VM stuck in a failing Host
Yes we could , but I'm not sure if maybe I lost the VM because it is tied to
that host.
Cordialmente,
Jaime Rojas
On 20/08/2015, at 11:47 a.m., Somesh Naidu
somesh.na...@citrix.commailto:somesh.na...@citrix.com wrote:
Can you not remove the failed host from CCP and XS cluster?
Regards,
Somesh
-Original Message-
From: Jaime Orlando Rojas Sanchez [mailto:jaime.ro...@kumo.com.co]
Sent: Thursday, August 20, 2015 12:38 PM
To: users@cloudstack.apache.orgmailto:users@cloudstack.apache.org
Subject: RE: VM stuck in a failing Host
Hello,
state/status of the failed host in CS? = Disconnected
We run the command in the 2 remaining host with no results.
[root@dc1fdtptgcx04 /]# xe vm-list name-label= i-574-14584-VM
[root@dc1fdtptgcx04 /]#
[root@dc1fdtptgcx02 /]# xe vm-list name-label= i-574-14584-VM
[root@dc1fdtptgcx02 /]#
Regards / Cordialmente,
Jaime O. Rojas S.
Technology Manager
jaime.ro...@kumo.com.comailto:jaime.ro...@kumo.com.co
Mobile: +57 301-3382382
Office: +57-1-8766767 x215
-Mensaje original-
De: Somesh Naidu [mailto:somesh.na...@citrix.com]
Enviado el: jueves, 20 de agosto de 2015 11:00 a. m.
Para: users@cloudstack.apache.orgmailto:users@cloudstack.apache.org
Asunto: RE: VM stuck in a failing Host
Quick question, what is the state/status of the failed host in CS? Also, look
up the particular VM on XS pool (xe vm-list name-label= i-574-14584-VM), what
does it say?
Regards,
Somesh
-Original Message-
From: Jaime Orlando Rojas Sanchez [mailto:jaime.ro...@kumo.com.co]
Sent: Thursday, August 20, 2015 11:37 AM
To: users@cloudstack.apache.orgmailto:users@cloudstack.apache.org
Subject: RE: VM stuck in a failing Host
Following the logs when I click 'run' in ACS after did the following in the DB
- Change the state to 'stopped'
- Change host ID to a working host
- Change last host ID to a working host
- Check VR is up and running on a working host
-bash-4.1# tail -f management-server.log | grep 14584
2015-08-20 05:45:27,513 DEBUG [cloud.async.AsyncJobManagerImpl]
(catalina-exec-21:null) submit async job-45973 = [
d3ed77b7-534e-4a3b-9038-a66359162087 ], details: AsyncJobVO {id:45973, userId:
2, accountId: 2, sessionKey: null, instanceType: VirtualMachine, instanceId:
RE: VM stuck in a failing Host
Hello,
state/status of the failed host in CS? = Disconnected
We run the command in the 2 remaining host with no results.
[root@dc1fdtptgcx04 /]# xe vm-list name-label= i-574-14584-VM
[root@dc1fdtptgcx04 /]#
[root@dc1fdtptgcx02 /]# xe vm-list name-label= i-574-14584-VM
[root@dc1fdtptgcx02 /]#
Regards / Cordialmente,
Jaime O. Rojas S.
Technology Manager
jaime.ro...@kumo.com.co
Mobile: +57 301-3382382
Office: +57-1-8766767 x215
-Mensaje original-
De: Somesh Naidu [mailto:somesh.na...@citrix.com]
Enviado el: jueves, 20 de agosto de 2015 11:00 a. m.
Para: users@cloudstack.apache.org
Asunto: RE: VM stuck in a failing Host
Quick question, what is the state/status of the failed host in CS? Also, look
up the particular VM on XS pool (xe vm-list name-label= i-574-14584-VM), what
does it say?
Regards,
Somesh
-Original Message-
From: Jaime Orlando Rojas Sanchez [mailto:jaime.ro...@kumo.com.co]
Sent: Thursday, August 20, 2015 11:37 AM
To: users@cloudstack.apache.org
Subject: RE: VM stuck in a failing Host
Following the logs when I click 'run' in ACS after did the following in the DB
- Change the state to 'stopped'
- Change host ID to a working host
- Change last host ID to a working host
- Check VR is up and running on a working host
-bash-4.1# tail -f management-server.log | grep 14584
2015-08-20 05:45:27,513 DEBUG [cloud.async.AsyncJobManagerImpl]
(catalina-exec-21:null) submit async job-45973 = [
d3ed77b7-534e-4a3b-9038-a66359162087 ], details: AsyncJobVO {id:45973, userId:
2, accountId: 2, sessionKey: null, instanceType: VirtualMachine, instanceId:
14584, cmd: org.apache.cloudstack.api.command.user.vm.StopVMCmd, cmdOriginator:
null, cmdInfo:
{response:json,id:98227dc9-682e-4f42-87e1-bd4b8045c7c9,sessionkey:hwnxmM0He9EXs2craugKg3XyWL4\u003d,cmdEventType:VM.STOP,ctxUserId:2,httpmethod:GET,_:1440067422009,ctxAccountId:2,ctxStartEventId:38650949,forced:true},
cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0,
processStatus: 0, resultCode: 0, result: null, initMsid: 139549854171544,
completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
2015-08-20 05:46:33,404 DEBUG [agent.transport.Request]
(Job-Executor-31:job-45974 = [ 9996fc72-fb83-4e5d-94c5-886396dac536 ]) Seq
792-1073807425: Sending { Cmd , MgmtId: 139549854171544, via: 792, Ver: v1,
Flags: 100111,
[{org.apache.cloudstack.storage.command.CopyCommand:{srcTO:{org.apache.cloudstack.storage.to.SnapshotObjectTO:{path:snapshots/574/57764/28586b35-cb45-4565-bd9b-7aa46a2898da,volume:{uuid:a15d0923-0a25-408f-9d10-fd5d47b3fef9,volumeType:ROOT,dataStore:{org.apache.cloudstack.storage.to.PrimaryDataStoreTO:{uuid:3PAR_3000GB_ADV_SATA1,id:211,poolType:PreSetup,host:localhost,path:/3PAR_3000GB_ADV_SATA1,port:0}},name:ROOT-14584,size:107374182400,path:c7a8eebc-7750-455c-804f-64c0d66cb4f4,volumeId:57764,vmName:i-574-14584-VM,accountId:574,format:VHD,id:57764,hypervisorType:XenServer},dataStore:{com.cloud.agent.api.to.NfsTO:{_url:nfs://172.16.4.65/vol/secondary_clpr,_role:Image}},vmName:i-574-14584-VM,name:srvrasautos2_ROOT-14584_20141007233517,hypervisorType:XenServer,id:11831}},destTO:{org.apache.cloudstack.storage.to.TemplateObjectTO:{path:template/tmpl/574/599,uuid:70d21214-33d0-49e0-8b45-c7702b0fe579,id:599,format:RAW,accountId:574,hvm:true,displayText:templateras,imageDataStore:{com.cloud.agent.api.to.NfsTO:{_url:nfs://172.16.4.65/vol/secondary_clpr,_role:Image}},name:248e2097b-4af7-38f7-a851-029ef11f52cc,hypervisorType:XenServer}},executeInSequence:true,wait:10800}}]
}
2015-08-20 05:49:03,212 DEBUG [cloud.async.AsyncJobManagerImpl]
(catalina-exec-1:null) submit async job-45975 = [
8ac23585-989d-4e3d-bcb9-3d3602842b8f ], details: AsyncJobVO {id:45975, userId:
2, accountId: 2, sessionKey: null, instanceType: VirtualMachine, instanceId:
14584, cmd: org.apache.cloudstack.api.command.user.vm.StartVMCmd,
cmdOriginator: null, cmdInfo:
{response:json,id:98227dc9-682e-4f42-87e1-bd4b8045c7c9,sessionkey:hwnxmM0He9EXs2craugKg3XyWL4\u003d,cmdEventType:VM.START,ctxUserId:2,httpmethod:GET,_:1440067641673,ctxAccountId:2,ctxStartEventId:38651246},
cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0,
processStatus: 0, resultCode: 0, result: null, initMsid: 139549854171544,
completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
2015-08-20 05:49:05,821 DEBUG [cloud.network.NetworkManagerImpl]
(Job-Executor-32:job-45975 = [ 8ac23585-989d-4e3d-bcb9-3d3602842b8f ]) Asking
VirtualRouter to prepare for
Nic[194602-14584-5d82a92d-b828-45bc-882a-b5ce17401812-172.16.100.244]
2015-08-20 05:49:08,947 DEBUG [cloud.network.NetworkManagerImpl]
(Job-Executor-32:job-45975 = [ 8ac23585-989d-4e3d-bcb9-3d3602842b8f ]) Asking
VirtualRouter to prepare for Nic[194621-14584-null-172.16.180.35]
2015-08-20 05:49:10,181 DEBUG [cloud.storage.VolumeManagerImpl]
(Job-Executor-32:job-45975 = [ 8ac23585-989d-4e3d-bcb9-3d3602842b8f ]) No need
to recreate the
RE: VM stuck in a failing Host
Can you not remove the failed host from CCP and XS cluster?
Regards,
Somesh
-Original Message-
From: Jaime Orlando Rojas Sanchez [mailto:jaime.ro...@kumo.com.co]
Sent: Thursday, August 20, 2015 12:38 PM
To: users@cloudstack.apache.org
Subject: RE: VM stuck in a failing Host
Hello,
state/status of the failed host in CS? = Disconnected
We run the command in the 2 remaining host with no results.
[root@dc1fdtptgcx04 /]# xe vm-list name-label= i-574-14584-VM
[root@dc1fdtptgcx04 /]#
[root@dc1fdtptgcx02 /]# xe vm-list name-label= i-574-14584-VM
[root@dc1fdtptgcx02 /]#
Regards / Cordialmente,
Jaime O. Rojas S.
Technology Manager
jaime.ro...@kumo.com.co
Mobile: +57 301-3382382
Office: +57-1-8766767 x215
-Mensaje original-
De: Somesh Naidu [mailto:somesh.na...@citrix.com]
Enviado el: jueves, 20 de agosto de 2015 11:00 a. m.
Para: users@cloudstack.apache.org
Asunto: RE: VM stuck in a failing Host
Quick question, what is the state/status of the failed host in CS? Also, look
up the particular VM on XS pool (xe vm-list name-label= i-574-14584-VM), what
does it say?
Regards,
Somesh
-Original Message-
From: Jaime Orlando Rojas Sanchez [mailto:jaime.ro...@kumo.com.co]
Sent: Thursday, August 20, 2015 11:37 AM
To: users@cloudstack.apache.org
Subject: RE: VM stuck in a failing Host
Following the logs when I click 'run' in ACS after did the following in the DB
- Change the state to 'stopped'
- Change host ID to a working host
- Change last host ID to a working host
- Check VR is up and running on a working host
-bash-4.1# tail -f management-server.log | grep 14584
2015-08-20 05:45:27,513 DEBUG [cloud.async.AsyncJobManagerImpl]
(catalina-exec-21:null) submit async job-45973 = [
d3ed77b7-534e-4a3b-9038-a66359162087 ], details: AsyncJobVO {id:45973, userId:
2, accountId: 2, sessionKey: null, instanceType: VirtualMachine, instanceId:
14584, cmd: org.apache.cloudstack.api.command.user.vm.StopVMCmd, cmdOriginator:
null, cmdInfo:
{response:json,id:98227dc9-682e-4f42-87e1-bd4b8045c7c9,sessionkey:hwnxmM0He9EXs2craugKg3XyWL4\u003d,cmdEventType:VM.STOP,ctxUserId:2,httpmethod:GET,_:1440067422009,ctxAccountId:2,ctxStartEventId:38650949,forced:true},
cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0,
processStatus: 0, resultCode: 0, result: null, initMsid: 139549854171544,
completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
2015-08-20 05:46:33,404 DEBUG [agent.transport.Request]
(Job-Executor-31:job-45974 = [ 9996fc72-fb83-4e5d-94c5-886396dac536 ]) Seq
792-1073807425: Sending { Cmd , MgmtId: 139549854171544, via: 792, Ver: v1,
Flags: 100111,
[{org.apache.cloudstack.storage.command.CopyCommand:{srcTO:{org.apache.cloudstack.storage.to.SnapshotObjectTO:{path:snapshots/574/57764/28586b35-cb45-4565-bd9b-7aa46a2898da,volume:{uuid:a15d0923-0a25-408f-9d10-fd5d47b3fef9,volumeType:ROOT,dataStore:{org.apache.cloudstack.storage.to.PrimaryDataStoreTO:{uuid:3PAR_3000GB_ADV_SATA1,id:211,poolType:PreSetup,host:localhost,path:/3PAR_3000GB_ADV_SATA1,port:0}},name:ROOT-14584,size:107374182400,path:c7a8eebc-7750-455c-804f-64c0d66cb4f4,volumeId:57764,vmName:i-574-14584-VM,accountId:574,format:VHD,id:57764,hypervisorType:XenServer},dataStore:{com.cloud.agent.api.to.NfsTO:{_url:nfs://172.16.4.65/vol/secondary_clpr,_role:Image}},vmName:i-574-14584-VM,name:srvrasautos2_ROOT-14584_20141007233517,hypervisorType:XenServer,id:11831}},destTO:{org.apache.cloudstack.storage.to.TemplateObjectTO:{path:template/tmpl/574/599,uuid:70d21214-33d0-49e0-8b45-c7702b0fe579,id:599,format:RAW,accountId:574,hvm:true,displayText:templateras,imageDataStore:{com.cloud.agent.api.to.NfsTO:{_url:nfs://172.16.4.65/vol/secondary_clpr,_role:Image}},name:248e2097b-4af7-38f7-a851-029ef11f52cc,hypervisorType:XenServer}},executeInSequence:true,wait:10800}}]
}
2015-08-20 05:49:03,212 DEBUG [cloud.async.AsyncJobManagerImpl]
(catalina-exec-1:null) submit async job-45975 = [
8ac23585-989d-4e3d-bcb9-3d3602842b8f ], details: AsyncJobVO {id:45975, userId:
2, accountId: 2, sessionKey: null, instanceType: VirtualMachine, instanceId:
14584, cmd: org.apache.cloudstack.api.command.user.vm.StartVMCmd,
cmdOriginator: null, cmdInfo:
{response:json,id:98227dc9-682e-4f42-87e1-bd4b8045c7c9,sessionkey:hwnxmM0He9EXs2craugKg3XyWL4\u003d,cmdEventType:VM.START,ctxUserId:2,httpmethod:GET,_:1440067641673,ctxAccountId:2,ctxStartEventId:38651246},
cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0,
processStatus: 0, resultCode: 0, result: null, initMsid: 139549854171544,
completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
2015-08-20 05:49:05,821 DEBUG [cloud.network.NetworkManagerImpl]
(Job-Executor-32:job-45975 = [ 8ac23585-989d-4e3d-bcb9-3d3602842b8f ]) Asking
VirtualRouter to prepare for
Nic[194602-14584-5d82a92d-b828-45bc-882a-b5ce17401812-172.16.100.244]
2015-08-20 05:49:08,947 DEBUG [cloud.network.NetworkManagerImpl]
RE: VM stuck in a failing Host
Quick question, what is the state/status of the failed host in CS? Also, look
up the particular VM on XS pool (xe vm-list name-label= i-574-14584-VM), what
does it say?
Regards,
Somesh
-Original Message-
From: Jaime Orlando Rojas Sanchez [mailto:jaime.ro...@kumo.com.co]
Sent: Thursday, August 20, 2015 11:37 AM
To: users@cloudstack.apache.org
Subject: RE: VM stuck in a failing Host
Following the logs when I click 'run' in ACS after did the following in the DB
- Change the state to 'stopped'
- Change host ID to a working host
- Change last host ID to a working host
- Check VR is up and running on a working host
-bash-4.1# tail -f management-server.log | grep 14584
2015-08-20 05:45:27,513 DEBUG [cloud.async.AsyncJobManagerImpl]
(catalina-exec-21:null) submit async job-45973 = [
d3ed77b7-534e-4a3b-9038-a66359162087 ], details: AsyncJobVO {id:45973, userId:
2, accountId: 2, sessionKey: null, instanceType: VirtualMachine, instanceId:
14584, cmd: org.apache.cloudstack.api.command.user.vm.StopVMCmd, cmdOriginator:
null, cmdInfo:
{response:json,id:98227dc9-682e-4f42-87e1-bd4b8045c7c9,sessionkey:hwnxmM0He9EXs2craugKg3XyWL4\u003d,cmdEventType:VM.STOP,ctxUserId:2,httpmethod:GET,_:1440067422009,ctxAccountId:2,ctxStartEventId:38650949,forced:true},
cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0,
processStatus: 0, resultCode: 0, result: null, initMsid: 139549854171544,
completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
2015-08-20 05:46:33,404 DEBUG [agent.transport.Request]
(Job-Executor-31:job-45974 = [ 9996fc72-fb83-4e5d-94c5-886396dac536 ]) Seq
792-1073807425: Sending { Cmd , MgmtId: 139549854171544, via: 792, Ver: v1,
Flags: 100111,
[{org.apache.cloudstack.storage.command.CopyCommand:{srcTO:{org.apache.cloudstack.storage.to.SnapshotObjectTO:{path:snapshots/574/57764/28586b35-cb45-4565-bd9b-7aa46a2898da,volume:{uuid:a15d0923-0a25-408f-9d10-fd5d47b3fef9,volumeType:ROOT,dataStore:{org.apache.cloudstack.storage.to.PrimaryDataStoreTO:{uuid:3PAR_3000GB_ADV_SATA1,id:211,poolType:PreSetup,host:localhost,path:/3PAR_3000GB_ADV_SATA1,port:0}},name:ROOT-14584,size:107374182400,path:c7a8eebc-7750-455c-804f-64c0d66cb4f4,volumeId:57764,vmName:i-574-14584-VM,accountId:574,format:VHD,id:57764,hypervisorType:XenServer},dataStore:{com.cloud.agent.api.to.NfsTO:{_url:nfs://172.16.4.65/vol/secondary_clpr,_role:Image}},vmName:i-574-14584-VM,name:srvrasautos2_ROOT-14584_20141007233517,hypervisorType:XenServer,id:11831}},destTO:{org.apache.cloudstack.storage.to.TemplateObjectTO:{path:template/tmpl/574/599,uuid:70d21214-33d0-49e0-8b45-c7702b0fe579,id:599,format:RAW,accountId:574,hvm:true,displayText:templateras,imageDataStore:{com.cloud.agent.api.to.NfsTO:{_url:nfs://172.16.4.65/vol/secondary_clpr,_role:Image}},name:248e2097b-4af7-38f7-a851-029ef11f52cc,hypervisorType:XenServer}},executeInSequence:true,wait:10800}}]
}
2015-08-20 05:49:03,212 DEBUG [cloud.async.AsyncJobManagerImpl]
(catalina-exec-1:null) submit async job-45975 = [
8ac23585-989d-4e3d-bcb9-3d3602842b8f ], details: AsyncJobVO {id:45975, userId:
2, accountId: 2, sessionKey: null, instanceType: VirtualMachine, instanceId:
14584, cmd: org.apache.cloudstack.api.command.user.vm.StartVMCmd,
cmdOriginator: null, cmdInfo:
{response:json,id:98227dc9-682e-4f42-87e1-bd4b8045c7c9,sessionkey:hwnxmM0He9EXs2craugKg3XyWL4\u003d,cmdEventType:VM.START,ctxUserId:2,httpmethod:GET,_:1440067641673,ctxAccountId:2,ctxStartEventId:38651246},
cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0,
processStatus: 0, resultCode: 0, result: null, initMsid: 139549854171544,
completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
2015-08-20 05:49:05,821 DEBUG [cloud.network.NetworkManagerImpl]
(Job-Executor-32:job-45975 = [ 8ac23585-989d-4e3d-bcb9-3d3602842b8f ]) Asking
VirtualRouter to prepare for
Nic[194602-14584-5d82a92d-b828-45bc-882a-b5ce17401812-172.16.100.244]
2015-08-20 05:49:08,947 DEBUG [cloud.network.NetworkManagerImpl]
(Job-Executor-32:job-45975 = [ 8ac23585-989d-4e3d-bcb9-3d3602842b8f ]) Asking
VirtualRouter to prepare for Nic[194621-14584-null-172.16.180.35]
2015-08-20 05:49:10,181 DEBUG [cloud.storage.VolumeManagerImpl]
(Job-Executor-32:job-45975 = [ 8ac23585-989d-4e3d-bcb9-3d3602842b8f ]) No need
to recreate the volume: Vol[13270|vm=14584|DATADISK], since it already has a
pool assigned: 208, adding disk to VM
2015-08-20 05:49:10,184 DEBUG [cloud.storage.VolumeManagerImpl]
(Job-Executor-32:job-45975 = [ 8ac23585-989d-4e3d-bcb9-3d3602842b8f ]) No need
to recreate the volume: Vol[57764|vm=14584|ROOT], since it already has a pool
assigned: 211, adding disk to VM
2015-08-20 05:49:10,271 DEBUG [agent.transport.Request]
(Job-Executor-32:job-45975 = [ 8ac23585-989d-4e3d-bcb9-3d3602842b8f ]) Seq
595-838074847: Sending { Cmd , MgmtId: 139549854171544, via: 595, Ver: v1,
Flags: 100111,
