RE: The agent doesn't reconnect if there are stopped VMs
If you mean offerings with custom parameters - yes, I use them There is a possibility that this might be affecting this. I don't have more information in the log-file Can you open the mgmt server log from 2015-08-18, search for 2015-08-18 06:24:46,544, the next few lines should contain the java exception stack which should point us to the method-file-line_number that will help us understand what is causing the NPE. Regards, Somesh -Original Message- From: Vladimir Melnik [mailto:v.mel...@uplink.ua] Sent: Friday, August 21, 2015 2:42 AM To: users@cloudstack.apache.org Subject: Re: The agent doesn't reconnect if there are stopped VMs I'll gladly send it, but where to find it? I don't have more information in the log-file :( On Thu, Aug 20, 2015 at 03:49:25PM +, Somesh Naidu wrote: 2015-08-18 06:24:46,542 DEBUG [c.c.c.CapacityManagerImpl] (AgentConnectTaskPool-213:ctx-76903ef6) Found 1 VM, not running on host 27 2015-08-18 06:24:46,544 ERROR [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-213:ctx-76903ef6) Monitor ComputeCapacityListener says there is an error in the connect process for 27 due to null java.lang.NullPointerException It will help if you send the complete exception stack for the above NPE. Regards, Somesh -Original Message- From: Simon Weller [mailto:swel...@ena.com] Sent: Thursday, August 20, 2015 11:21 AM To: users@cloudstack.apache.org Subject: Re: The agent doesn't reconnect if there are stopped VMs Do you have any dynamic service offerings? From: Vladimir Melnik v.mel...@uplink.ua Sent: Thursday, August 20, 2015 8:20 AM To: users@cloudstack.apache.org Subject: Re: The agent doesn't reconnect if there are stopped VMs Oh, I'm sorry, I should have initially send the DEBUG-log! Here is an example: --- 8 --- 2015-07-29 00:53:42,988 INFO [utils.nio.NioClient] (Agent-Selector:null) Connecting to ***.***.***.***:8250 2015-07-29 00:53:44,254 INFO [utils.nio.NioClient] (Agent-Selector:null) SSL: Handshake done 2015-07-29 00:53:44,255 INFO [utils.nio.NioClient] (Agent-Selector:null) Connected to ***.***.***.***:8250 2015-07-29 00:53:44,258 WARN [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:null) Could not read cpuinfo_max_freq 2015-07-29 00:53:44,266 DEBUG [kvm.resource.LibvirtCapXMLParser] (Agent-Handler-1:null) Found /usr/libexec/qemu-kvm as a suiteable emulat or 2015-07-29 00:53:44,266 DEBUG [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:null) Executing: /bin/bash -c qemu-img --help|grep convert 2015-07-29 00:53:44,270 DEBUG [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:null) Execution is successful. 2015-07-29 00:53:44,271 DEBUG [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:null) convert [-c] [-p] [-f fmt] [-t cache] [-T src_cache] [-O output_fmt] [-o options] [-S sparse_size] filename [filename2 [...]] output_filename 2015-07-29 00:53:44,271 DEBUG [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:null) cpus=8, speed=2660, ram=30153224192, dom0ram =805306368, cpu sockets=1 2015-07-29 00:53:44,272 DEBUG [cloud.resource.ServerResourceBase] (Agent-Handler-1:null) Parameters for private nic: 172.26.65.1 - 84:2b: 2b:56:d3:d9-255.255.255.0 2015-07-29 00:53:44,272 DEBUG [cloud.resource.ServerResourceBase] (Agent-Handler-1:null) Parameters for storage nic: 172.26.65.1 - 84:2b: 2b:56:d3:d9-255.255.255.0 2015-07-29 00:53:44,272 DEBUG [cloud.resource.ServerResourceBase] (Agent-Handler-1:null) Parameters for pubic nic: 172.26.65.1 - 84:2b:2b :56:d3:d9-255.255.255.0 2015-07-29 00:53:44,272 DEBUG [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:null) Executing: /usr/share/cloudstack-common/scri pts/vm/hypervisor/versions.sh 2015-07-29 00:53:44,281 DEBUG [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:null) Execution is successful. 2015-07-29 00:53:44,282 DEBUG [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:null) Executing: sudo grep InitiatorName= /etc/isc si/initiatorname.iscsi 2015-07-29 00:53:44,290 DEBUG [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:null) Execution is successful. 2015-07-29 00:53:44,290 INFO [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) Attempting to create storage pool 42bed7d9-88ae- 403d-9b53-0b44f31b2312 (Filesystem) in libvirt 2015-07-29 00:53:44,292 DEBUG [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) Found existing defined storage pool 42bed7d9-88a e-403d-9b53-0b44f31b2312, using it. 2015-07-29 00:53:44,292 DEBUG [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) Trying to fetch storage pool 42bed7d9-88ae-403d-9b53-0b44f31b2312 from libvirt 2015-07-29 00:53:44,811 DEBUG [cloud.agent.Agent] (Agent-Handler-1:null) Executing: hostname 2015-07-29 00:53:44,813 DEBUG [cloud.agent.Agent] (Agent-Handler-1:null) Execution is successful. 2015-07-29 00:53:44,834
RE: VM stuck in a failing Host *RESOLVED
Hello, finally we fixed the problem: 1. From CS state VM was change to Stopped from DB, last_host_ID was marked as NULL 2. Then on Xenservers we did a. Auto power on for VM : xe vm-param-set uuid=[UUID] other-config:auto_poweron=true b. Pool DB sync. xe pool-sync-database c. Forced Power off xe vm-reset-powerstate uuid=[UUID] -force d. Check VM state xe vm-list name-label=i-574-14584-VM (state: healt) 3. From CS it was turned on and it started in a different host. Not sure exactly which command made the difference on XS but it was clear that once the VM was stuck it turned a XS issue further than CS handled. Regards / Cordialmente, Jaime O. Rojas S. Technology Manager jaime.ro...@kumo.com.co Mobile: +57 301-3382382 Office: +57-1-8766767 x215 -Mensaje original- De: Jaime Orlando Rojas Sanchez [mailto:jaime.ro...@kumo.com.co] Enviado el: jueves, 20 de agosto de 2015 7:07 p. m. Para: users@cloudstack.apache.org Asunto: Re: VM stuck in a failing Host Hello, I already tried he force stop check, unfortunately it didn't work VM keeps starting in failing host. I am trying to force vm to stop from xen server with no results. [root@dc1fdtptgcx02 /]# xe vm-shutdown name-label=i-574-14584-VM --force You attempted an operation which involves a host which could not be contacted. host: 53109eef-2f53-4f0e-a763-68817d573bd9 (DC1FDTPTGCX03 Cordialmente, Jaime Rojas Sent from my iPhone On 20/08/2015, at 4:03 p.m., Somesh Naidu somesh.na...@citrix.commailto:somesh.na...@citrix.com wrote: Thanks I just realized that the job was a stop command and not start command that I was looking for. The stop command failed to stop the VM leaving the VM in Running state: 2015-08-20 05:45:27,513 DEBUG [cloud.async.AsyncJobManagerImpl] (catalina-exec-21:null) submit async job-45973 = [ d3ed77b7-534e-4a3b-9038-a66359162087 ], details: AsyncJobVO {id:45973, userId: 2, accountId: 2, sessionKey: null, instanceType: VirtualMachine, instanceId: 14584, cmd: org.apache.cloudstack.api.command.user.vm.StopVMCmd, cmdOriginator: null, cmdInfo: {response:json,id:98227dc9-682e-4f42-87e1-bd4b8045c7c9,sessionkey:hwnxmM0He9EXs2craugKg3XyWL4\u003d,cmdEventType:VM.STOP,ctxUserId:2,httpmethod:GET,_:1440067422009,ctxAccountId:2,ctxStartEventId:38650949,forced:true}, cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0, processStatus: 0, resultCode: 0, result: null, initMsid: 139549854171544, completeMsid: null, lastUpdated: null, lastPolled: null, created: null} 2015-08-20 05:45:27,605 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-30:job-45973 = [ d3ed77b7-534e-4a3b-9038-a66359162087 ]) VM state transitted from :Running to Stopping with event: StopRequestedvm's original host id: 599 new host id: 599 host id before state transition: 599 2015-08-20 05:45:27,617 WARN [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-30:job-45973 = [ d3ed77b7-534e-4a3b-9038-a66359162087 ]) Unable to stop vm, agent unavailable: com.cloud.exception.AgentUnavailableException: Resource [Host:599] is unreachable: Host 599: Host with specified id is not in the right state: Disconnected 2015-08-20 05:45:32,654 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-30:job-45973 = [ d3ed77b7-534e-4a3b-9038-a66359162087 ]) VM state transitted from :Stopping to Running with event: OperationFailedvm's original host id: 599 new host id: 599 host id before state transition: 599 2015-08-20 05:45:33,683 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-30:job-45973 = [ d3ed77b7-534e-4a3b-9038-a66359162087 ]) Complete async job-45973 = [ d3ed77b7-534e-4a3b-9038-a66359162087 ], jobStatus: 2, resultCode: 530, result: Error Code: 530 Error text: Failed to stop vm I can't tell why the VM keeps starting on the failed host unless I have a look at the logs from a start command. But I think there is a good chance that if you check the Force Stop check box when stopping that VM and then try to start it again. Regards, Somesh -Original Message- From: Jaime Orlando Rojas Sanchez [mailto:jaime.ro...@kumo.com.co] Sent: Thursday, August 20, 2015 4:40 PM To: users@cloudstack.apache.orgmailto:users@cloudstack.apache.org Subject: RE: VM stuck in a failing Host Hello, I am attaching the logs http://jrojas-test.s3.kumo.com.co/management-server.log?AWSAccessKeyId=0099c1623d6704f9f5d5Expires=1440711480Signature=CFYwowzsJ3x%2B7XXfpAHDLM%2F022A%3Dx-amz-pt=N2ZiYjcyOTk4MTQ0MDEwMzEwNTQ5MA We have a second manage server but it doesn't execute any task. Regards / Cordialmente, Jaime O. Rojas S. Technology Manager jaime.ro...@kumo.com.comailto:jaime.ro...@kumo.com.co Mobile: +57 301-3382382 Office: +57-1-8766767 x215 -Mensaje original- De: Somesh Naidu [mailto:somesh.na...@citrix.com] Enviado el: jueves, 20 de agosto de 2015 2:17 p. m. Para: users@cloudstack.apache.orgmailto:users@cloudstack.apache.org Asunto: RE: VM stuck in a failing Host Jaime, Those are not complete logs I
RE: duplicate use of tips?
Frank, That’s not really true: This is from the “nics” table in the Database: What about the removed column? Also, did you check the table user_ip_address table for the IPs? Are there multiple entries for those? If not then it should not be possible for mgmt server to assign one IP to two VMs at the same time unless the record is incorrectly not being marked as Allocated while assignment. Regards, Somesh -Original Message- From: Frank Louwers [mailto:fr...@openminds.be] Sent: Friday, August 21, 2015 3:30 AM To: users@cloudstack.apache.org Subject: Re: duplicate use of tips? On 21 Aug 2015 at 04:44:13, Abhinandan Prateek (abhinandan.prat...@shapeblue.com) wrote: If you are manually assigning the ips better use the ips that are outside the cidr that cloudstack manages. The ips assigned by cloudstack dhcp are as per cloudstack assignments anything that happen outside VR’s dhcp is unknown to cloudstack. That’s not really true: This is from the “nics” table in the Database: +---+-+--+--++--+ | id | instance_id | state | ip4_address | network_id | secondary_ip | +---+-+--+--++--+ | 72 | 58 | Reserved | x.y.x.92 | 204 | 1 | | 19092 | 4929 | Reserved | x.y.z.92 | 204 | 0 | +---+-+--+--++———+ On 20-Aug-2015, at 11:29 pm, Frank Louwers fr...@openminds.be wrote: Hi, In a zone with Basic Networking, I’ve assigned a certain netblock x.y.z.0/24 to the Guest network in CloudStack. I have a VM-A that has primary ip address x.y.z.92 and secondary address on the same nic x.y.z.52. For various reasons, *both* ips were configured manually, so not using the VR’s dhcp. A while back, the x.y.z.92 was (manually) deconfigured on VM-A (so VM-A only used x.y.x.52, but both ips are still configured to belong to VM-A in Cloudstack). A few days ago, a new instance (VM-B) was spun up, and CS assigned ip x.y.z.92 to that VM. Why would it do that? Today, a new instance was spun up (VM-C), but CS assigned x.y.z.52 to VM-C… How and why does this happen?! Is this because VM-A does not use dhcp? That might explain the .92 re-assignement, but certainly not the .52 reassignment, as secondary ips don’t use dhcp anyhow… Can anyone tell me what’s going on, and what can be done to prevent his? Running CS 4.4.2 at the moment, considering upgrading to 4.5.1 (or 4.5.2) Regards, Frank Find out more about ShapeBlue and our range of CloudStack related services IaaS Cloud Design Buildhttp://shapeblue.com/iaas-cloud-design-and-build// CSForge – rapid IaaS deployment frameworkhttp://shapeblue.com/csforge/ CloudStack Consultinghttp://shapeblue.com/cloudstack-consultancy/ CloudStack Software Engineeringhttp://shapeblue.com/cloudstack-software-engineering/ CloudStack Infrastructure Supporthttp://shapeblue.com/cloudstack-infrastructure-support/ CloudStack Bootcamp Training Courseshttp://shapeblue.com/cloudstack-training/ This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd or related companies. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error. Shape Blue Ltd is a company incorporated in England Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is a registered trademark.
Re: The agent doesn't reconnect if there are stopped VMs
I'll gladly send it, but where to find it? I don't have more information in the log-file :( On Thu, Aug 20, 2015 at 03:49:25PM +, Somesh Naidu wrote: 2015-08-18 06:24:46,542 DEBUG [c.c.c.CapacityManagerImpl] (AgentConnectTaskPool-213:ctx-76903ef6) Found 1 VM, not running on host 27 2015-08-18 06:24:46,544 ERROR [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-213:ctx-76903ef6) Monitor ComputeCapacityListener says there is an error in the connect process for 27 due to null java.lang.NullPointerException It will help if you send the complete exception stack for the above NPE. Regards, Somesh -Original Message- From: Simon Weller [mailto:swel...@ena.com] Sent: Thursday, August 20, 2015 11:21 AM To: users@cloudstack.apache.org Subject: Re: The agent doesn't reconnect if there are stopped VMs Do you have any dynamic service offerings? From: Vladimir Melnik v.mel...@uplink.ua Sent: Thursday, August 20, 2015 8:20 AM To: users@cloudstack.apache.org Subject: Re: The agent doesn't reconnect if there are stopped VMs Oh, I'm sorry, I should have initially send the DEBUG-log! Here is an example: --- 8 --- 2015-07-29 00:53:42,988 INFO [utils.nio.NioClient] (Agent-Selector:null) Connecting to ***.***.***.***:8250 2015-07-29 00:53:44,254 INFO [utils.nio.NioClient] (Agent-Selector:null) SSL: Handshake done 2015-07-29 00:53:44,255 INFO [utils.nio.NioClient] (Agent-Selector:null) Connected to ***.***.***.***:8250 2015-07-29 00:53:44,258 WARN [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:null) Could not read cpuinfo_max_freq 2015-07-29 00:53:44,266 DEBUG [kvm.resource.LibvirtCapXMLParser] (Agent-Handler-1:null) Found /usr/libexec/qemu-kvm as a suiteable emulat or 2015-07-29 00:53:44,266 DEBUG [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:null) Executing: /bin/bash -c qemu-img --help|grep convert 2015-07-29 00:53:44,270 DEBUG [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:null) Execution is successful. 2015-07-29 00:53:44,271 DEBUG [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:null) convert [-c] [-p] [-f fmt] [-t cache] [-T src_cache] [-O output_fmt] [-o options] [-S sparse_size] filename [filename2 [...]] output_filename 2015-07-29 00:53:44,271 DEBUG [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:null) cpus=8, speed=2660, ram=30153224192, dom0ram =805306368, cpu sockets=1 2015-07-29 00:53:44,272 DEBUG [cloud.resource.ServerResourceBase] (Agent-Handler-1:null) Parameters for private nic: 172.26.65.1 - 84:2b: 2b:56:d3:d9-255.255.255.0 2015-07-29 00:53:44,272 DEBUG [cloud.resource.ServerResourceBase] (Agent-Handler-1:null) Parameters for storage nic: 172.26.65.1 - 84:2b: 2b:56:d3:d9-255.255.255.0 2015-07-29 00:53:44,272 DEBUG [cloud.resource.ServerResourceBase] (Agent-Handler-1:null) Parameters for pubic nic: 172.26.65.1 - 84:2b:2b :56:d3:d9-255.255.255.0 2015-07-29 00:53:44,272 DEBUG [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:null) Executing: /usr/share/cloudstack-common/scri pts/vm/hypervisor/versions.sh 2015-07-29 00:53:44,281 DEBUG [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:null) Execution is successful. 2015-07-29 00:53:44,282 DEBUG [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:null) Executing: sudo grep InitiatorName= /etc/isc si/initiatorname.iscsi 2015-07-29 00:53:44,290 DEBUG [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:null) Execution is successful. 2015-07-29 00:53:44,290 INFO [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) Attempting to create storage pool 42bed7d9-88ae- 403d-9b53-0b44f31b2312 (Filesystem) in libvirt 2015-07-29 00:53:44,292 DEBUG [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) Found existing defined storage pool 42bed7d9-88a e-403d-9b53-0b44f31b2312, using it. 2015-07-29 00:53:44,292 DEBUG [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) Trying to fetch storage pool 42bed7d9-88ae-403d-9b53-0b44f31b2312 from libvirt 2015-07-29 00:53:44,811 DEBUG [cloud.agent.Agent] (Agent-Handler-1:null) Executing: hostname 2015-07-29 00:53:44,813 DEBUG [cloud.agent.Agent] (Agent-Handler-1:null) Execution is successful. 2015-07-29 00:53:44,834 DEBUG [cloud.agent.Agent] (Agent-Handler-1:null) Executing: hostname 2015-07-29 00:53:44,836 DEBUG [cloud.agent.Agent] (Agent-Handler-1:null) Execution is successful. 2015-07-29 00:53:44,838 DEBUG [cloud.agent.Agent] (Agent-Handler-1:null) Sending Startup: Seq 0-64: { Cmd , MgmtId: -1, via: 0, Ver: v1, Flags: 1,
Re: duplicate use of tips?
On 21 Aug 2015 at 04:44:13, Abhinandan Prateek (abhinandan.prat...@shapeblue.com) wrote: If you are manually assigning the ips better use the ips that are outside the cidr that cloudstack manages. The ips assigned by cloudstack dhcp are as per cloudstack assignments anything that happen outside VR’s dhcp is unknown to cloudstack. That’s not really true: This is from the “nics” table in the Database: +---+-+--+--++--+ | id | instance_id | state | ip4_address | network_id | secondary_ip | +---+-+--+--++--+ | 72 | 58 | Reserved | x.y.x.92 | 204 | 1 | | 19092 | 4929 | Reserved | x.y.z.92 | 204 | 0 | +---+-+--+--++———+ On 20-Aug-2015, at 11:29 pm, Frank Louwers fr...@openminds.be wrote: Hi, In a zone with Basic Networking, I’ve assigned a certain netblock x.y.z.0/24 to the Guest network in CloudStack. I have a VM-A that has primary ip address x.y.z.92 and secondary address on the same nic x.y.z.52. For various reasons, *both* ips were configured manually, so not using the VR’s dhcp. A while back, the x.y.z.92 was (manually) deconfigured on VM-A (so VM-A only used x.y.x.52, but both ips are still configured to belong to VM-A in Cloudstack). A few days ago, a new instance (VM-B) was spun up, and CS assigned ip x.y.z.92 to that VM. Why would it do that? Today, a new instance was spun up (VM-C), but CS assigned x.y.z.52 to VM-C… How and why does this happen?! Is this because VM-A does not use dhcp? That might explain the .92 re-assignement, but certainly not the .52 reassignment, as secondary ips don’t use dhcp anyhow… Can anyone tell me what’s going on, and what can be done to prevent his? Running CS 4.4.2 at the moment, considering upgrading to 4.5.1 (or 4.5.2) Regards, Frank Find out more about ShapeBlue and our range of CloudStack related services IaaS Cloud Design Buildhttp://shapeblue.com/iaas-cloud-design-and-build// CSForge – rapid IaaS deployment frameworkhttp://shapeblue.com/csforge/ CloudStack Consultinghttp://shapeblue.com/cloudstack-consultancy/ CloudStack Software Engineeringhttp://shapeblue.com/cloudstack-software-engineering/ CloudStack Infrastructure Supporthttp://shapeblue.com/cloudstack-infrastructure-support/ CloudStack Bootcamp Training Courseshttp://shapeblue.com/cloudstack-training/ This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd or related companies. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error. Shape Blue Ltd is a company incorporated in England Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is a registered trademark.
Re: The agent doesn't reconnect if there are stopped VMs
If you mean dynamic scaleable offerings - no I don't have any. If you mean offerings with custom parameters - yes, I use them. Thanks! On Thu, Aug 20, 2015 at 03:21:14PM +, Simon Weller wrote: Do you have any dynamic service offerings? From: Vladimir Melnik v.mel...@uplink.ua Sent: Thursday, August 20, 2015 8:20 AM To: users@cloudstack.apache.org Subject: Re: The agent doesn't reconnect if there are stopped VMs Oh, I'm sorry, I should have initially send the DEBUG-log! Here is an example: --- 8 --- 2015-07-29 00:53:42,988 INFO [utils.nio.NioClient] (Agent-Selector:null) Connecting to ***.***.***.***:8250 2015-07-29 00:53:44,254 INFO [utils.nio.NioClient] (Agent-Selector:null) SSL: Handshake done 2015-07-29 00:53:44,255 INFO [utils.nio.NioClient] (Agent-Selector:null) Connected to ***.***.***.***:8250 2015-07-29 00:53:44,258 WARN [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:null) Could not read cpuinfo_max_freq 2015-07-29 00:53:44,266 DEBUG [kvm.resource.LibvirtCapXMLParser] (Agent-Handler-1:null) Found /usr/libexec/qemu-kvm as a suiteable emulat or 2015-07-29 00:53:44,266 DEBUG [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:null) Executing: /bin/bash -c qemu-img --help|grep convert 2015-07-29 00:53:44,270 DEBUG [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:null) Execution is successful. 2015-07-29 00:53:44,271 DEBUG [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:null) convert [-c] [-p] [-f fmt] [-t cache] [-T src_cache] [-O output_fmt] [-o options] [-S sparse_size] filename [filename2 [...]] output_filename 2015-07-29 00:53:44,271 DEBUG [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:null) cpus=8, speed=2660, ram=30153224192, dom0ram =805306368, cpu sockets=1 2015-07-29 00:53:44,272 DEBUG [cloud.resource.ServerResourceBase] (Agent-Handler-1:null) Parameters for private nic: 172.26.65.1 - 84:2b: 2b:56:d3:d9-255.255.255.0 2015-07-29 00:53:44,272 DEBUG [cloud.resource.ServerResourceBase] (Agent-Handler-1:null) Parameters for storage nic: 172.26.65.1 - 84:2b: 2b:56:d3:d9-255.255.255.0 2015-07-29 00:53:44,272 DEBUG [cloud.resource.ServerResourceBase] (Agent-Handler-1:null) Parameters for pubic nic: 172.26.65.1 - 84:2b:2b :56:d3:d9-255.255.255.0 2015-07-29 00:53:44,272 DEBUG [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:null) Executing: /usr/share/cloudstack-common/scri pts/vm/hypervisor/versions.sh 2015-07-29 00:53:44,281 DEBUG [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:null) Execution is successful. 2015-07-29 00:53:44,282 DEBUG [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:null) Executing: sudo grep InitiatorName= /etc/isc si/initiatorname.iscsi 2015-07-29 00:53:44,290 DEBUG [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:null) Execution is successful. 2015-07-29 00:53:44,290 INFO [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) Attempting to create storage pool 42bed7d9-88ae- 403d-9b53-0b44f31b2312 (Filesystem) in libvirt 2015-07-29 00:53:44,292 DEBUG [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) Found existing defined storage pool 42bed7d9-88a e-403d-9b53-0b44f31b2312, using it. 2015-07-29 00:53:44,292 DEBUG [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) Trying to fetch storage pool 42bed7d9-88ae-403d-9b53-0b44f31b2312 from libvirt 2015-07-29 00:53:44,811 DEBUG [cloud.agent.Agent] (Agent-Handler-1:null) Executing: hostname 2015-07-29 00:53:44,813 DEBUG [cloud.agent.Agent] (Agent-Handler-1:null) Execution is successful. 2015-07-29 00:53:44,834 DEBUG [cloud.agent.Agent] (Agent-Handler-1:null) Executing: hostname 2015-07-29 00:53:44,836 DEBUG [cloud.agent.Agent] (Agent-Handler-1:null) Execution is successful. 2015-07-29 00:53:44,838 DEBUG [cloud.agent.Agent] (Agent-Handler-1:null) Sending Startup: Seq 0-64: { Cmd , MgmtId: -1, via: 0, Ver: v1, Flags: 1,