Hello, VR is shutting down after deploy and no network assigned when trying to create shared guest network with real white IP's
L2 Switch Force10 s60 with PVLAN [image: Inline image 1] 2017-10-29 22:32:22,058 DEBUG [c.c.v.VirtualMachineManagerImpl] (API-Job-Executor-37:ctx-9cc3690d job-348 ctx-20fa9ee9) (logid:2a620d26) Allocating entries for VM: VM[DomainRouter|r-65-VM] 2017-10-29 22:32:22,063 DEBUG [c.c.v.VirtualMachineManagerImpl] (API-Job-Executor-37:ctx-9cc3690d job-348 ctx-20fa9ee9) (logid:2a620d26) Allocating nics for VM[DomainRouter|r-65-VM] 2017-10-29 22:32:22,066 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-37:ctx-9cc3690d job-348 ctx-20fa9ee9) (logid:2a620d26) Allocating nic for vm VM[DomainRouter|r-65-VM] in network Ntwk[220|Guest|7] with requested profile NicProfile[0-0-null-null-null 2017-10-29 22:32:22,092 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-37:ctx-9cc3690d job-348 ctx-20fa9ee9) (logid:2a620d26) Allocating nic for vm VM[DomainRouter|r-65-VM] in network Ntwk[202|Control|3] with requested profile null 2017-10-29 22:32:22,110 DEBUG [c.c.v.VirtualMachineManagerImpl] (API-Job-Executor-37:ctx-9cc3690d job-348 ctx-20fa9ee9) (logid:2a620d26) Allocating disks for VM[DomainRouter|r-65-VM] 2017-10-29 22:32:22,120 DEBUG [c.c.v.VirtualMachineManagerImpl] (API-Job-Executor-37:ctx-9cc3690d job-348 ctx-20fa9ee9) (logid:2a620d26) Allocation completed for VM: VM[DomainRouter|r-65-VM] 2017-10-29 22:32:22,145 DEBUG [c.c.n.r.NetworkHelperImpl] (API-Job-Executor-37:ctx-9cc3690d job-348 ctx-20fa9ee9) (logid:2a620d26) Starting router VM[DomainRouter|r-65-VM] 2017-10-29 22:32:23,755 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-63:ctx-001ec8b4 job-348/job-351 ctx-4a25af66) (logid:2a620d26) Successfully transitioned to start state for VM[DomainRouter|r-65-VM] reservation id = 5afbcfcf-d768-4c70-8af1-acf7c8238a3d 2017-10-29 22:32:23,947 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-63:ctx-001ec8b4 job-348/job-351 ctx-4a25af66) (logid:2a620d26) Deployment found - P0=VM[DomainRouter|r-65-VM], P0=Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))] : Dest[Zone(1)-Pod(1)-Cluster(1)-Host(1)-Storage(Volume(66|ROOT-->Pool(2))] 2017-10-29 22:32:24,088 DEBUG [c.c.n.g.PodBasedNetworkGuru] (Work-Job-Executor-63:ctx-001ec8b4 job-348/job-351 ctx-4a25af66) (logid:2a620d26) Allocated a nic NicProfile[133-65-5afbcfcf-d768-4c70-8af1-acf7c8238a3d-172.23.35.247-null for VM[DomainRouter|r-65-VM] 2017-10-29 22:32:24,115 DEBUG [o.a.c.e.o.VolumeOrchestrator] (Work-Job-Executor-63:ctx-001ec8b4 job-348/job-351 ctx-4a25af66) (logid:2a620d26) Checking if we need to prepare 1 volumes for VM[DomainRouter|r-65-VM] 2017-10-29 22:32:24,207 DEBUG [c.c.a.t.Request] (Work-Job-Executor-63:ctx-001ec8b4 job-348/job-351 ctx-4a25af66) (logid:2a620d26) Seq 1-1437774181038032715: Sending { Cmd , MgmtId: 52240608294, via: 1(172.23.32.1), Ver: v1, Flags: 100111, [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"055468adeeda31e283b0b01fc0631e93","origUrl":" http://packages.shapeblue.com.s3-eu-west-1.amazonaws.com/systemvmtemplate/4.10/dnsmasq/systemvm64template-4.10-vmware.ova","uuid":"2c2b054b-8be2-4e1f-9537-83e852ba9b1c","id":8,"format":"OVA","accountId":1,"checksum":"970bfb070a80bd74820881d8149643c1","hvm":false,"displayText":"SystemVM Template (vSphere)","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"9646e689ebf53959acd3001b2686a95b","id":2,"poolType":"VMFS","host":"VMFS datastore: datastore-183","path":"datastore-183","port":0,"url":"VMFS://VMFS datastore: datastore-183/datastore-183/?ROLE=Primary&STOREUUID=9646e689ebf53959acd3001b2686a95b","isManaged":false}},"name":"routing-8","hypervisorType":"VMware"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"95b692c6-194e-4d92-a756-b5f7697e5418","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"9646e689ebf53959acd3001b2686a95b","id":2,"poolType":"VMFS","host":"VMFS datastore: datastore-183","path":"datastore-183","port":0,"url":"VMFS://VMFS datastore: datastore-183/datastore-183/?ROLE=Primary&STOREUUID=9646e689ebf53959acd3001b2686a95b","fullCloneFlag":true,"isManaged":false}},"name":"ROOT-65","size":3355443200,"volumeId":66,"vmName":"r-65-VM","accountId":1,"format":"OVA","provisioningType":"THIN","id":66,"deviceId":0,"hypervisorType":"VMware"}},"executeInSequence":true,"options":{},"options2":{},"wait":0}}] } 2017-10-29 22:32:24,208 DEBUG [c.c.a.t.Request] (Work-Job-Executor-63:ctx-001ec8b4 job-348/job-351 ctx-4a25af66) (logid:2a620d26) Seq 1-1437774181038032715: Executing: { Cmd , MgmtId: 52240608294, via: 1(172.23.32.1), Ver: v1, Flags: 100111, [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"055468adeeda31e283b0b01fc0631e93","origUrl":" http://packages.shapeblue.com.s3-eu-west-1.amazonaws.com/systemvmtemplate/4.10/dnsmasq/systemvm64template-4.10-vmware.ova","uuid":"2c2b054b-8be2-4e1f-9537-83e852ba9b1c","id":8,"format":"OVA","accountId":1,"checksum":"970bfb070a80bd74820881d8149643c1","hvm":false,"displayText":"SystemVM Template (vSphere)","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"9646e689ebf53959acd3001b2686a95b","id":2,"poolType":"VMFS","host":"VMFS datastore: datastore-183","path":"datastore-183","port":0,"url":"VMFS://VMFS datastore: datastore-183/datastore-183/?ROLE=Primary&STOREUUID=9646e689ebf53959acd3001b2686a95b","isManaged":false}},"name":"routing-8","hypervisorType":"VMware"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"95b692c6-194e-4d92-a756-b5f7697e5418","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"9646e689ebf53959acd3001b2686a95b","id":2,"poolType":"VMFS","host":"VMFS datastore: datastore-183","path":"datastore-183","port":0,"url":"VMFS://VMFS datastore: datastore-183/datastore-183/?ROLE=Primary&STOREUUID=9646e689ebf53959acd3001b2686a95b","fullCloneFlag":true,"isManaged":false}},"name":"ROOT-65","size":3355443200,"volumeId":66,"vmName":"r-65-VM","accountId":1,"format":"OVA","provisioningType":"THIN","id":66,"deviceId":0,"hypervisorType":"VMware"}},"executeInSequence":true,"options":{},"options2":{},"wait":0}}] } 2017-10-29 22:32:37,697 DEBUG [c.c.a.t.Request] (Work-Job-Executor-63:ctx-001ec8b4 job-348/job-351 ctx-4a25af66) (logid:2a620d26) Seq 1-1437774181038032716: Sending { Cmd , MgmtId: 52240608294, via: 1(172.23.32.1), Ver: v1, Flags: 100111, [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"055468adeeda31e283b0b01fc0631e93","origUrl":" http://packages.shapeblue.com.s3-eu-west-1.amazonaws.com/systemvmtemplate/4.10/dnsmasq/systemvm64template-4.10-vmware.ova","uuid":"2c2b054b-8be2-4e1f-9537-83e852ba9b1c","id":8,"format":"OVA","accountId":1,"checksum":"970bfb070a80bd74820881d8149643c1","hvm":false,"displayText":"SystemVM Template (vSphere)","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"9646e689ebf53959acd3001b2686a95b","id":2,"poolType":"VMFS","host":"VMFS datastore: datastore-183","path":"datastore-183","port":0,"url":"VMFS://VMFS datastore: datastore-183/datastore-183/?ROLE=Primary&STOREUUID=9646e689ebf53959acd3001b2686a95b","isManaged":false}},"name":"routing-8","hypervisorType":"VMware"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"95b692c6-194e-4d92-a756-b5f7697e5418","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"9646e689ebf53959acd3001b2686a95b","id":2,"poolType":"VMFS","host":"VMFS datastore: datastore-183","path":"datastore-183","port":0,"url":"VMFS://VMFS datastore: datastore-183/datastore-183/?ROLE=Primary&STOREUUID=9646e689ebf53959acd3001b2686a95b","fullCloneFlag":true,"isManaged":false}},"name":"ROOT-65","size":3355443200,"volumeId":66,"vmName":"r-65-VM","accountId":1,"format":"OVA","provisioningType":"THIN","id":66,"deviceId":0,"hypervisorType":"VMware"}},"executeInSequence":true,"options":{},"options2":{},"wait":0}}] } 2017-10-29 22:32:37,698 DEBUG [c.c.a.t.Request] (Work-Job-Executor-63:ctx-001ec8b4 job-348/job-351 ctx-4a25af66) (logid:2a620d26) Seq 1-1437774181038032716: Executing: { Cmd , MgmtId: 52240608294, via: 1(172.23.32.1), Ver: v1, Flags: 100111, [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"055468adeeda31e283b0b01fc0631e93","origUrl":" http://packages.shapeblue.com.s3-eu-west-1.amazonaws.com/systemvmtemplate/4.10/dnsmasq/systemvm64template-4.10-vmware.ova","uuid":"2c2b054b-8be2-4e1f-9537-83e852ba9b1c","id":8,"format":"OVA","accountId":1,"checksum":"970bfb070a80bd74820881d8149643c1","hvm":false,"displayText":"SystemVM Template (vSphere)","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"9646e689ebf53959acd3001b2686a95b","id":2,"poolType":"VMFS","host":"VMFS datastore: datastore-183","path":"datastore-183","port":0,"url":"VMFS://VMFS datastore: datastore-183/datastore-183/?ROLE=Primary&STOREUUID=9646e689ebf53959acd3001b2686a95b","isManaged":false}},"name":"routing-8","hypervisorType":"VMware"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"95b692c6-194e-4d92-a756-b5f7697e5418","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"9646e689ebf53959acd3001b2686a95b","id":2,"poolType":"VMFS","host":"VMFS datastore: datastore-183","path":"datastore-183","port":0,"url":"VMFS://VMFS datastore: datastore-183/datastore-183/?ROLE=Primary&STOREUUID=9646e689ebf53959acd3001b2686a95b","fullCloneFlag":true,"isManaged":false}},"name":"ROOT-65","size":3355443200,"volumeId":66,"vmName":"r-65-VM","accountId":1,"format":"OVA","provisioningType":"THIN","id":66,"deviceId":0,"hypervisorType":"VMware"}},"executeInSequence":true,"options":{},"options2":{},"wait":0}}] } 2017-10-29 22:32:51,186 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (Work-Job-Executor-63:ctx-001ec8b4 job-348/job-351 ctx-4a25af66) (logid:2a620d26) Boot Args for VM[DomainRouter|r-65-VM]: template=domP name=r-65-VM eth0ip=x.x.x.x eth0mask=255.255.255.128 gateway=x.x.x.x domain=cs1cloud.internal cidrsize=25 dhcprange=x.x.x.x eth1ip=172.23.35.247 eth1mask=255.255.252.0 mgmtcidr=172.23.0.0/28 localgw=172.23.35.254 type=dhcpsrvr disable_rp_filter=true extra_pubnics=5 dns1=8.8.8.8 dns2=8.8.4.4 baremetalnotificationsecuritykey=achUFWnqv46cuEUS-DBbGRVSaFiGglCTEj1x0DODY5aD5VTbZ7uPSQIkzA3SChlM4S90SDNJhqVibyOj_tuU6g baremetalnotificationapikey=3CETJDRI4gDssIQbHWkbuOHz1KF_6c_vo22C83XKhIofwJIXNWVwfZqURo8u1EnIRV622fEpdxe_Hhudmj7iYQ host=172.23.0.1 port=8080 2017-10-29 22:32:51,286 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (Work-Job-Executor-63:ctx-001ec8b4 job-348/job-351 ctx-4a25af66) (logid:2a620d26) Found 0 firewall Egress rule(s) to apply as a part of domR VM[DomainRouter|r-65-VM] start. 2017-10-29 22:32:51,302 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (Work-Job-Executor-63:ctx-001ec8b4 job-348/job-351 ctx-4a25af66) (logid:2a620d26) Creating monitoring services on VM[DomainRouter|r-65-VM] start... 2017-10-29 22:32:51,313 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (Work-Job-Executor-63:ctx-001ec8b4 job-348/job-351 ctx-4a25af66) (logid:2a620d26) Reapplying dhcp entries as a part of domR VM[DomainRouter|r-65-VM] start... 2017-10-29 22:32:51,318 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (Work-Job-Executor-63:ctx-001ec8b4 job-348/job-351 ctx-4a25af66) (logid:2a620d26) Reapplying vm data (userData and metaData) entries as a part of domR VM[DomainRouter|r-65-VM] start... 2017-10-29 22:32:51,336 DEBUG [c.c.a.t.Request] (Work-Job-Executor-63:ctx-001ec8b4 job-348/job-351 ctx-4a25af66) (logid:2a620d26) Seq 1-1437774181038032717: Sending { Cmd , MgmtId: 52240608294, via: 1(172.23.32.1), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StartCommand":{"vm":{"id":65,"name":"r-65-VM","bootloader":"HVM","type":"DomainRouter","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":268435456,"maxRam":268435456,"hostName":"r-65-VM","arch":"x86_64","os":"Debian GNU/Linux 5.0 (64-bit)","platformEmulator":"debian5Guest","bootArgs":" template=domP name=r-65-VM eth0ip=x.x.x.x eth0mask=255.255.255.128 gateway=x.x.x.x domain=cs1cloud.internal cidrsize=25 dhcprange=x.x.x.x eth1ip=172.23.35.247 eth1mask=255.255.252.0 mgmtcidr=172.23.0.0/28 localgw=172.23.35.254 type=dhcpsrvr disable_rp_filter=true extra_pubnics=5 dns1=8.8.8.8 dns2=8.8.4.4 baremetalnotificationsecuritykey=achUFWnqv46cuEUS-DBbGRVSaFiGglCTEj1x0DODY5aD5VTbZ7uPSQIkzA3SChlM4S90SDNJhqVibyOj_tuU6g baremetalnotificationapikey=3CETJDRI4gDssIQbHWkbuOHz1KF_6c_vo22C83XKhIofwJIXNWVwfZqURo8u1EnIRV622fEpdxe_Hhudmj7iYQ host=172.23.0.1 port=8080 nic_macs=1e:00:d9:00:07:87|02:00:1d:7c:00:3f","enableHA":false,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"aJX71-6nnBG9BTsDXVdnvA","params":{"vmware.reserve.cpu":"false","nicAdapter":"Vmxnet3","vmware.reserve.mem":"false"},"uuid":"84272205-4d07-4fb7-8e9e-59e200d24284","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"95b692c6-194e-4d92-a756-b5f7697e5418","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"9646e689ebf53959acd3001b2686a95b","id":2,"poolType":"VMFS","host":"VMFS datastore: datastore-183","path":"datastore-183","port":0,"url":"VMFS://VMFS datastore: datastore-183/datastore-183/?ROLE=Primary&STOREUUID=9646e689ebf53959acd3001b2686a95b","isManaged":false}},"name":"ROOT-65","size":3355443200,"path":"ROOT-65","volumeId":66,"vmName":"r-65-VM","accountId":1,"format":"OVA","provisioningType":"THIN","id":66,"deviceId":0,"bytesReadRate":0,"bytesWriteRate":0,"iopsReadRate":0,"iopsWriteRate":0,"hypervisorType":"VMware"}},"diskSeq":0,"path":"ROOT-65","type":"ROOT","_details":{"storageHost":"VMFS datastore: datastore-183","managed":"false","storagePort":"0","volumeSize":"3355443200"}}],"nics":[{"deviceId":0,"networkRateMbps":200,"defaultNic":true,"pxeDisable":true,"nicUuid":"3eba4f6b-c4ad-4062-9586-2a05b60dd980","uuid":"4a656a92-41d4-4c57-9afb-fee75c22426d","ip":"x.x.x.x","netmask":"255.255.255.128","gateway":"x.x.x.x","mac":"1e:00:d9:00:07:87","dns1":"8.8.8.8","dns2":"8.8.4.4","broadcastType":"Pvlan","type":"Guest","broadcastUri":"pvlan://300-i301","isolationUri":"vlan://300","isSecurityGroupEnabled":false,"name":"DSwitch,,vmwaredvs"},{"deviceId":1,"networkRateMbps":-1,"defaultNic":false,"pxeDisable":true,"nicUuid":"9dc65aa7-8ed3-4fea-b973-54a8176c6023","uuid":"e3c49b15-6ba5-4440-9094-d2dab25ec2e2","ip":"172.23.35.247","netmask":"255.255.252.0","gateway":"172.23.35.254","mac":"02:00:1d:7c:00:3f","broadcastType":"Native","type":"Control","isSecurityGroupEnabled":false,"name":"vSwitch0,9,vmwaresvs"}],"guestOsDetails":{}},"hostIp":"172.23.32.1","executeInSequence":true,"wait":0}},{"com.cloud.agent.api.check.CheckSshCommand":{"ip":"172.23.35.247","port":3922,"interval":6,"retries":100,"name":"r-65-VM","wait":0}},{"com.cloud.agent.api.GetDomRVersionCmd":{"accessDetails":{" router.name ":"r-65-VM","router.ip":"172.23.35.247"},"wait":0}},{},{"com.cloud.agent.api.routing.AggregationControlCommand":{"action":"Start","accessDetails":{" router.name ":"r-65-VM","router.ip":"172.23.35.247","router.guest.ip":"x.x.x.x"},"wait":0}},{"com.cloud.agent.api.routing.SetMonitorServiceCommand":{"services":[{"id":0,"service":"dhcp","processname":"dnsmasq","serviceName":"dnsmasq","servicePath":"/var/run/dnsmasq/dnsmasq.pid","pidFile":"/var/run/dnsmasq/dnsmasq.pid","isDefault":false},{"id":0,"service":"ssh","processname":"sshd","serviceName":"ssh","servicePath":"/var/run/sshd.pid","pidFile":"/var/run/sshd.pid","isDefault":true},{"id":0,"service":"webserver","processname":"apache2","serviceName":"apache2","servicePath":"/var/run/apache2.pid","pidFile":"/var/run/apache2.pid","isDefault":true}],"accessDetails":{" router.name ":"r-65-VM","router.ip":"172.23.35.247","router.guest.ip":"x.x.x.x"},"wait":0}},{"com.cloud.agent.api.routing.AggregationControlCommand":{"action":"Finish","accessDetails":{" router.name":"r-65-VM","router.ip":"172.23.35.247","router.guest.ip":"x.x.x.x"},"wait":0}}] } 2017-10-29 22:32:51,337 DEBUG [c.c.a.t.Request] (Work-Job-Executor-63:ctx-001ec8b4 job-348/job-351 ctx-4a25af66) (logid:2a620d26) Seq 1-1437774181038032717: Executing: { Cmd , MgmtId: 52240608294, via: 1(172.23.32.1), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StartCommand":{"vm":{"id":65,"name":"r-65-VM","bootloader":"HVM","type":"DomainRouter","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":268435456,"maxRam":268435456,"hostName":"r-65-VM","arch":"x86_64","os":"Debian GNU/Linux 5.0 (64-bit)","platformEmulator":"debian5Guest","bootArgs":" template=domP name=r-65-VM eth0ip=x.x.x.x eth0mask=255.255.255.128 gateway=x.x.x.x domain=cs1cloud.internal cidrsize=25 dhcprange=x.x.x.x eth1ip=172.23.35.247 eth1mask=255.255.252.0 mgmtcidr=172.23.0.0/28 localgw=172.23.35.254 type=dhcpsrvr disable_rp_filter=true extra_pubnics=5 dns1=8.8.8.8 dns2=8.8.4.4 baremetalnotificationsecuritykey=achUFWnqv46cuEUS-DBbGRVSaFiGglCTEj1x0DODY5aD5VTbZ7uPSQIkzA3SChlM4S90SDNJhqVibyOj_tuU6g baremetalnotificationapikey=3CETJDRI4gDssIQbHWkbuOHz1KF_6c_vo22C83XKhIofwJIXNWVwfZqURo8u1EnIRV622fEpdxe_Hhudmj7iYQ host=172.23.0.1 port=8080 nic_macs=1e:00:d9:00:07:87|02:00:1d:7c:00:3f","enableHA":false,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"aJX71-6nnBG9BTsDXVdnvA","params":{"vmware.reserve.cpu":"false","nicAdapter":"Vmxnet3","vmware.reserve.mem":"false"},"uuid":"84272205-4d07-4fb7-8e9e-59e200d24284","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"95b692c6-194e-4d92-a756-b5f7697e5418","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"9646e689ebf53959acd3001b2686a95b","id":2,"poolType":"VMFS","host":"VMFS datastore: datastore-183","path":"datastore-183","port":0,"url":"VMFS://VMFS datastore: datastore-183/datastore-183/?ROLE=Primary&STOREUUID=9646e689ebf53959acd3001b2686a95b","isManaged":false}},"name":"ROOT-65","size":3355443200,"path":"ROOT-65","volumeId":66,"vmName":"r-65-VM","accountId":1,"format":"OVA","provisioningType":"THIN","id":66,"deviceId":0,"bytesReadRate":0,"bytesWriteRate":0,"iopsReadRate":0,"iopsWriteRate":0,"hypervisorType":"VMware"}},"diskSeq":0,"path":"ROOT-65","type":"ROOT","_details":{"storageHost":"VMFS datastore: datastore-183","managed":"false","storagePort":"0","volumeSize":"3355443200"}}],"nics":[{"deviceId":0,"networkRateMbps":200,"defaultNic":true,"pxeDisable":true,"nicUuid":"3eba4f6b-c4ad-4062-9586-2a05b60dd980","uuid":"4a656a92-41d4-4c57-9afb-fee75c22426d","ip":"x.x.x.x","netmask":"255.255.255.128","gateway":"x.x.x.x","mac":"1e:00:d9:00:07:87","dns1":"8.8.8.8","dns2":"8.8.4.4","broadcastType":"Pvlan","type":"Guest","broadcastUri":"pvlan://300-i301","isolationUri":"vlan://300","isSecurityGroupEnabled":false,"name":"DSwitch,,vmwaredvs"},{"deviceId":1,"networkRateMbps":-1,"defaultNic":false,"pxeDisable":true,"nicUuid":"9dc65aa7-8ed3-4fea-b973-54a8176c6023","uuid":"e3c49b15-6ba5-4440-9094-d2dab25ec2e2","ip":"172.23.35.247","netmask":"255.255.252.0","gateway":"172.23.35.254","mac":"02:00:1d:7c:00:3f","broadcastType":"Native","type":"Control","isSecurityGroupEnabled":false,"name":"vSwitch0,9,vmwaresvs"}],"guestOsDetails":{}},"hostIp":"172.23.32.1","executeInSequence":true,"wait":0}},{"com.cloud.agent.api.check.CheckSshCommand":{"ip":"172.23.35.247","port":3922,"interval":6,"retries":100,"name":"r-65-VM","wait":0}},{"com.cloud.agent.api.GetDomRVersionCmd":{"accessDetails":{" router.name ":"r-65-VM","router.ip":"172.23.35.247"},"wait":0}},{},{"com.cloud.agent.api.routing.AggregationControlCommand":{"action":"Start","accessDetails":{" router.name ":"r-65-VM","router.ip":"172.23.35.247","router.guest.ip":"x.x.x.x"},"wait":0}},{"com.cloud.agent.api.routing.SetMonitorServiceCommand":{"services":[{"id":0,"service":"dhcp","processname":"dnsmasq","serviceName":"dnsmasq","servicePath":"/var/run/dnsmasq/dnsmasq.pid","pidFile":"/var/run/dnsmasq/dnsmasq.pid","isDefault":false},{"id":0,"service":"ssh","processname":"sshd","serviceName":"ssh","servicePath":"/var/run/sshd.pid","pidFile":"/var/run/sshd.pid","isDefault":true},{"id":0,"service":"webserver","processname":"apache2","serviceName":"apache2","servicePath":"/var/run/apache2.pid","pidFile":"/var/run/apache2.pid","isDefault":true}],"accessDetails":{" router.name ":"r-65-VM","router.ip":"172.23.35.247","router.guest.ip":"x.x.x.x"},"wait":0}},{"com.cloud.agent.api.routing.AggregationControlCommand":{"action":"Finish","accessDetails":{" router.name":"r-65-VM","router.ip":"172.23.35.247","router.guest.ip":"x.x.x.x"},"wait":0}}] } 2017-10-29 22:32:51,339 INFO [c.c.h.v.r.VmwareResource] (DirectAgent-109:ctx-f31e88c7 172.23.32.1, job-348/job-351, cmd: StartCommand) (logid:2a620d26) Executing resource StartCommand: {"vm":{"id":65,"name":"r-65-VM","bootloader":"HVM","type":"DomainRouter","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":268435456,"maxRam":268435456,"hostName":"r-65-VM","arch":"x86_64","os":"Debian GNU/Linux 5.0 (64-bit)","platformEmulator":"debian5Guest","bootArgs":" template=domP name=r-65-VM eth0ip=x.x.x.x eth0mask=255.255.255.128 gateway=x.x.x.x domain=cs1cloud.internal cidrsize=25 dhcprange=x.x.x.x eth1ip=172.23.35.247 eth1mask=255.255.252.0 mgmtcidr=172.23.0.0/28 localgw=172.23.35.254 type=dhcpsrvr disable_rp_filter=true extra_pubnics=5 dns1=8.8.8.8 dns2=8.8.4.4 baremetalnotificationsecuritykey=achUFWnqv46cuEUS-DBbGRVSaFiGglCTEj1x0DODY5aD5VTbZ7uPSQIkzA3SChlM4S90SDNJhqVibyOj_tuU6g baremetalnotificationapikey=3CETJDRI4gDssIQbHWkbuOHz1KF_6c_vo22C83XKhIofwJIXNWVwfZqURo8u1EnIRV622fEpdxe_Hhudmj7iYQ host=172.23.0.1 port=8080 nic_macs=1e:00:d9:00:07:87|02:00:1d:7c:00:3f","enableHA":false,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"aJX71-6nnBG9BTsDXVdnvA","params":{"vmware.reserve.cpu":"false","nicAdapter":"Vmxnet3","vmware.reserve.mem":"false"},"uuid":"84272205-4d07-4fb7-8e9e-59e200d24284","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"95b692c6-194e-4d92-a756-b5f7697e5418","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"9646e689ebf53959acd3001b2686a95b","id":2,"poolType":"VMFS","host":"VMFS datastore: datastore-183","path":"datastore-183","port":0,"url":"VMFS://VMFS datastore: datastore-183/datastore-183/?ROLE=Primary&STOREUUID=9646e689ebf53959acd3001b2686a95b","isManaged":false}},"name":"ROOT-65","size":3355443200,"path":"ROOT-65","volumeId":66,"vmName":"r-65-VM","accountId":1,"format":"OVA","provisioningType":"THIN","id":66,"deviceId":0,"bytesReadRate":0,"bytesWriteRate":0,"iopsReadRate":0,"iopsWriteRate":0,"hypervisorType":"VMware"}},"diskSeq":0,"path":"ROOT-65","type":"ROOT","_details":{"storageHost":"VMFS datastore: datastore-183","managed":"false","storagePort":"0","volumeSize":"3355443200"}}],"nics":[{"deviceId":0,"networkRateMbps":200,"defaultNic":true,"pxeDisable":true,"nicUuid":"3eba4f6b-c4ad-4062-9586-2a05b60dd980","uuid":"4a656a92-41d4-4c57-9afb-fee75c22426d","ip":"x.x.x.x","netmask":"255.255.255.128","gateway":"x.x.x.x","mac":"1e:00:d9:00:07:87","dns1":"8.8.8.8","dns2":"8.8.4.4","broadcastType":"Pvlan","type":"Guest","broadcastUri":"pvlan://300-i301","isolationUri":"vlan://300","isSecurityGroupEnabled":false,"name":"DSwitch,,vmwaredvs"},{"deviceId":1,"networkRateMbps":-1,"defaultNic":false,"pxeDisable":true,"nicUuid":"9dc65aa7-8ed3-4fea-b973-54a8176c6023","uuid":"e3c49b15-6ba5-4440-9094-d2dab25ec2e2","ip":"172.23.35.247","netmask":"255.255.252.0","gateway":"172.23.35.254","mac":"02:00:1d:7c:00:3f","broadcastType":"Native","type":"Control","isSecurityGroupEnabled":false,"name":"vSwitch0,9,vmwaresvs"}],"guestOsDetails":{}},"hostIp":"172.23.32.1","executeInSequence":true,"wait":0} 2017-10-29 22:32:51,394 DEBUG [c.c.h.v.m.HostMO] (DirectAgent-109:ctx-f31e88c7 172.23.32.1, job-348/job-351, cmd: StartCommand) (logid:2a620d26) find VM r-65-VM on host 2017-10-29 22:32:51,394 INFO [c.c.h.v.m.HostMO] (DirectAgent-109:ctx-f31e88c7 172.23.32.1, job-348/job-351, cmd: StartCommand) (logid:2a620d26) VM r-65-VM not found in host cache 2017-10-29 22:32:51,515 INFO [c.c.h.v.m.DatastoreMO] (DirectAgent-109:ctx-f31e88c7 172.23.32.1, job-348/job-351, cmd: StartCommand) (logid:2a620d26) Folder r-65-VM does not exist on datastore 2017-10-29 22:32:51,521 INFO [c.c.h.v.m.DatastoreMO] (DirectAgent-109:ctx-f31e88c7 172.23.32.1, job-348/job-351, cmd: StartCommand) (logid:2a620d26) Search file r-65-VM.vmx on [LocalSSD-32-1] 2017-10-29 22:32:51,568 INFO [c.c.h.v.m.DatastoreMO] (DirectAgent-109:ctx-f31e88c7 172.23.32.1, job-348/job-351, cmd: StartCommand) (logid:2a620d26) File [LocalSSD-32-1] r-65-VM.vmx does not exist on datastore 2017-10-29 22:32:51,568 INFO [c.c.h.v.m.DatastoreMO] (DirectAgent-109:ctx-f31e88c7 172.23.32.1, job-348/job-351, cmd: StartCommand) (logid:2a620d26) Searching file r-65-VM.vmx in [LocalSSD-32-1] 2017-10-29 22:32:53,153 DEBUG [c.c.h.v.m.HostMO] (DirectAgent-109:ctx-f31e88c7 172.23.32.1, job-348/job-351, cmd: StartCommand) (logid:2a620d26) find VM r-65-VM on host 2017-10-29 22:32:53,154 INFO [c.c.h.v.m.HostMO] (DirectAgent-109:ctx-f31e88c7 172.23.32.1, job-348/job-351, cmd: StartCommand) (logid:2a620d26) VM r-65-VM not found in host cache 2017-10-29 22:32:53,190 DEBUG [c.c.h.v.m.HostMO] (DirectAgent-109:ctx-f31e88c7 172.23.32.1, job-348/job-351, cmd: StartCommand) (logid:2a620d26) find VM r-65-VM on host 2017-10-29 22:32:53,190 DEBUG [c.c.h.v.m.HostMO] (DirectAgent-109:ctx-f31e88c7 172.23.32.1, job-348/job-351, cmd: StartCommand) (logid:2a620d26) VM r-65-VM found in host cache 2017-10-29 22:32:53,636 INFO [c.c.h.v.m.DatastoreMO] (DirectAgent-109:ctx-f31e88c7 172.23.32.1, job-348/job-351, cmd: StartCommand) (logid:2a620d26) Folder r-65-VM exists on datastore 2017-10-29 22:32:53,745 INFO [c.c.s.r.VmwareStorageLayoutHelper] (DirectAgent-109:ctx-f31e88c7 172.23.32.1, job-348/job-351, cmd: StartCommand) (logid:2a620d26) sync [LocalSSD-32-1] ROOT-65-flat.vmdk->[LocalSSD-32-1] r-65-VM/ROOT-65-flat.vmdk 2017-10-29 22:32:53,985 INFO [c.c.s.r.VmwareStorageLayoutHelper] (DirectAgent-109:ctx-f31e88c7 172.23.32.1, job-348/job-351, cmd: StartCommand) (logid:2a620d26) sync [LocalSSD-32-1] ROOT-65.vmdk->[LocalSSD-32-1] r-65-VM/ROOT-65.vmdk 2017-10-29 22:32:54,072 INFO [c.c.h.v.m.DatastoreMO] (DirectAgent-109:ctx-f31e88c7 172.23.32.1, job-348/job-351, cmd: StartCommand) (logid:2a620d26) Search file ROOT-65.vmdk on [LocalSSD-32-1] r-65-VM 2017-10-29 22:32:54,122 INFO [c.c.h.v.m.DatastoreMO] (DirectAgent-109:ctx-f31e88c7 172.23.32.1, job-348/job-351, cmd: StartCommand) (logid:2a620d26) File [LocalSSD-32-1] r-65-VM/ROOT-65.vmdk exists on datastore 2017-10-29 22:32:54,135 DEBUG [c.c.h.v.r.VmwareResource] (DirectAgent-109:ctx-f31e88c7 172.23.32.1, job-348/job-351, cmd: StartCommand) (logid:2a620d26) Prepare volume at new device {"capacityInKB":0,"key":-2,"backing":{"diskMode":"persistent","fileName":"[LocalSSD-32-1] r-65-VM/ROOT-65.vmdk","datastore":{"value":"datastore-183","type":"Datastore"}},"connectable":{"startConnected":true,"allowGuestControl":false,"connected":true},"controllerKey":1000,"unitNumber":0} 2017-10-29 22:32:54,135 DEBUG [c.c.h.v.r.VmwareResource] (DirectAgent-109:ctx-f31e88c7 172.23.32.1, job-348/job-351, cmd: StartCommand) (logid:2a620d26) VM r-65-VM will be started with NIC device type: Vmxnet3 2017-10-29 22:32:55,044 DEBUG [c.c.h.v.m.HostMO] (DirectAgent-109:ctx-f31e88c7 172.23.32.1, job-348/job-351, cmd: StartCommand) (logid:2a620d26) find VM r-65-VM on host 2017-10-29 22:32:55,044 DEBUG [c.c.h.v.m.HostMO] (DirectAgent-109:ctx-f31e88c7 172.23.32.1, job-348/job-351, cmd: StartCommand) (logid:2a620d26) VM r-65-VM found in host cache 2017-10-29 22:32:55,072 INFO [c.c.h.v.r.VmwareResource] (DirectAgent-109:ctx-f31e88c7 172.23.32.1, job-348/job-351, cmd: StartCommand) (logid:2a620d26) Configure VNC port for VM r-65-VM, port: 5957, host: 172.23.32.1 2017-10-29 22:32:55,539 DEBUG [c.c.h.v.m.ClusterMO] (DirectAgent-109:ctx-f31e88c7 172.23.32.1, job-348/job-351, cmd: StartCommand) (logid:2a620d26) Couldn't set restart priority for VM: r-65-VM, HA disabled in the cluster 2017-10-29 22:32:55,546 INFO [c.c.h.v.m.VirtualMachineMO] (DirectAgent-109:ctx-f31e88c7 172.23.32.1, job-348/job-351, cmd: StartCommand) (logid:2a620d26) Test against disk backing : [LocalSSD-32-1] r-65-VM/ROOT-65.vmdk 2017-10-29 22:32:55,546 INFO [c.c.h.v.m.VirtualMachineMO] (DirectAgent-109:ctx-f31e88c7 172.23.32.1, job-348/job-351, cmd: StartCommand) (logid:2a620d26) Disk backing : [LocalSSD-32-1] r-65-VM/ROOT-65.vmdk matches ==> scsi0:0 2017-10-29 22:33:34,098 DEBUG [c.c.h.v.r.VmwareResource] (DirectAgent-109:ctx-f31e88c7 172.23.32.1, job-348/job-351, cmd: CheckSshCommand) (logid:2a620d26) Ping command port succeeded for vm r-65-VM 2017-10-29 22:33:34,098 DEBUG [c.c.h.v.r.VmwareResource] (DirectAgent-109:ctx-f31e88c7 172.23.32.1, job-348/job-351, cmd: CheckSshCommand) (logid:2a620d26) Execute network usage setup command on r-65-VM 2017-10-29 22:33:35,303 DEBUG [c.c.a.t.Request] (DirectAgent-109:ctx-f31e88c7) (logid:2a620d26) Seq 1-1437774181038032717: Processing: { Ans: , MgmtId: 52240608294, via: 1(172.23.32.1), Ver: v1, Flags: 10, [{"com.cloud.agent.api.StartAnswer":{"vm":{"id":65,"name":"r-65-VM","bootloader":"HVM","type":"DomainRouter","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":268435456,"maxRam":268435456,"hostName":"r-65-VM","arch":"x86_64","os":"Debian GNU/Linux 5.0 (64-bit)","platformEmulator":"debian5Guest","bootArgs":" template=domP name=r-65-VM eth0ip=x.x.x.x eth0mask=255.255.255.128 gateway=x.x.x.x domain=cs1cloud.internal cidrsize=25 dhcprange=x.x.x.x eth1ip=172.23.35.247 eth1mask=255.255.252.0 mgmtcidr=172.23.0.0/28 localgw=172.23.35.254 type=dhcpsrvr disable_rp_filter=true extra_pubnics=5 dns1=8.8.8.8 dns2=8.8.4.4 baremetalnotificationsecuritykey=achUFWnqv46cuEUS-DBbGRVSaFiGglCTEj1x0DODY5aD5VTbZ7uPSQIkzA3SChlM4S90SDNJhqVibyOj_tuU6g baremetalnotificationapikey=3CETJDRI4gDssIQbHWkbuOHz1KF_6c_vo22C83XKhIofwJIXNWVwfZqURo8u1EnIRV622fEpdxe_Hhudmj7iYQ host=172.23.0.1 port=8080 nic_macs=1e:00:d9:00:07:87|02:00:1d:7c:00:3f","enableHA":false,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"aJX71-6nnBG9BTsDXVdnvA","params":{"vmware.reserve.cpu":"false","nicAdapter":"Vmxnet3","vmware.reserve.mem":"false"},"uuid":"84272205-4d07-4fb7-8e9e-59e200d24284","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"95b692c6-194e-4d92-a756-b5f7697e5418","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"9646e689ebf53959acd3001b2686a95b","id":2,"poolType":"VMFS","host":"VMFS datastore: datastore-183","path":"datastore-183","port":0,"url":"VMFS://VMFS datastore: datastore-183/datastore-183/?ROLE=Primary&STOREUUID=9646e689ebf53959acd3001b2686a95b","isManaged":false}},"name":"ROOT-65","size":3355443200,"path":"ROOT-65","volumeId":66,"vmName":"r-65-VM","accountId":1,"chainInfo":"{\"diskDeviceBusName\":\"scsi0:0\",\"diskChain\":[\"[LocalSSD-32-1] r-65-VM/ROOT-65.vmdk\"]}","format":"OVA","provisioningType":"THIN","id":66,"deviceId":0,"bytesReadRate":0,"bytesWriteRate":0,"iopsReadRate":0,"iopsWriteRate":0,"hypervisorType":"VMware"}},"diskSeq":0,"path":"ROOT-65","type":"ROOT","_details":{"storageHost":"VMFS datastore: datastore-183","managed":"false","storagePort":"0","volumeSize":"3355443200"}}],"nics":[{"deviceId":0,"networkRateMbps":200,"defaultNic":true,"pxeDisable":true,"nicUuid":"3eba4f6b-c4ad-4062-9586-2a05b60dd980","uuid":"4a656a92-41d4-4c57-9afb-fee75c22426d","ip":"x.x.x.x","netmask":"255.255.255.128","gateway":"x.x.x.x","mac":"1e:00:d9:00:07:87","dns1":"8.8.8.8","dns2":"8.8.4.4","broadcastType":"Pvlan","type":"Guest","broadcastUri":"pvlan://300-i301","isolationUri":"vlan://300","isSecurityGroupEnabled":false,"name":"DSwitch,,vmwaredvs"},{"deviceId":1,"networkRateMbps":-1,"defaultNic":false,"pxeDisable":true,"nicUuid":"9dc65aa7-8ed3-4fea-b973-54a8176c6023","uuid":"e3c49b15-6ba5-4440-9094-d2dab25ec2e2","ip":"172.23.35.247","netmask":"255.255.252.0","gateway":"172.23.35.254","mac":"02:00:1d:7c:00:3f","broadcastType":"Native","type":"Control","isSecurityGroupEnabled":false,"name":"vSwitch0,9,vmwaresvs"}],"guestOsDetails":{}},"_iqnToPath":{},"result":true,"wait":0}},{"com.cloud.agent.api.check.CheckSshAnswer":{"result":true,"wait":0}},{"com.cloud.agent.api.GetDomRVersionAnswer":{"templateVersion":"Cloudstack Release 4.10.0 Mon Oct 9 06:45:25 UTC 2017","scriptsVersion":"5b88a56dc2eafdc55729d8b4c57a4c19\n","result":true,"details":"Cloudstack Release 4.10.0 Mon Oct 9 06:45:25 UTC 2017&5b88a56dc2eafdc55729d8b4c57a4c19\n","wait":0}},{"com.cloud.agent.api.NetworkUsageAnswer":{"routerName":"r-65-VM","bytesSent":0,"bytesReceived":0,"result":true,"details":"","wait":0}},{"com.cloud.agent.api.Answer":{"result":true,"details":"Command aggregation started","wait":0}},{"com.cloud.agent.api.Answer":{"result":true,"wait":0}},{"com.cloud.agent.api.Answer":{"result":true,"details":"Command aggregation finished","wait":0}}] } 2017-10-29 22:33:35,338 INFO [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-63:ctx-001ec8b4 job-348/job-351 ctx-4a25af66) (logid:2a620d26) The guru did not like the answers so stopping VM[DomainRouter|r-65-VM] 2017-10-29 22:33:35,344 DEBUG [c.c.a.t.Request] (Work-Job-Executor-63:ctx-001ec8b4 job-348/job-351 ctx-4a25af66) (logid:2a620d26) Seq 1-1437774181038032721: Sending { Cmd , MgmtId: 52240608294, via: 1(172.23.32.1), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"checkBeforeCleanup":false,"forceStop":false,"vmName":"r-65-VM","executeInSequence":true,"wait":0}}] } 2017-10-29 22:33:35,344 DEBUG [c.c.a.t.Request] (Work-Job-Executor-63:ctx-001ec8b4 job-348/job-351 ctx-4a25af66) (logid:2a620d26) Seq 1-1437774181038032721: Executing: { Cmd , MgmtId: 52240608294, via: 1(172.23.32.1), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"checkBeforeCleanup":false,"forceStop":false,"vmName":"r-65-VM","executeInSequence":true,"wait":0}}] } 2017-10-29 22:33:35,344 INFO [c.c.h.v.r.VmwareResource] (DirectAgent-55:ctx-b5ec23f7 172.23.32.1, job-348/job-351, cmd: StopCommand) (logid:2a620d26) Executing resource StopCommand: {"isProxy":false,"checkBeforeCleanup":false,"forceStop":false,"vmName":"r-65-VM","executeInSequence":true,"wait":0} 2017-10-29 22:33:35,357 DEBUG [c.c.h.v.m.HostMO] (DirectAgent-55:ctx-b5ec23f7 172.23.32.1, job-348/job-351, cmd: StopCommand) (logid:2a620d26) find VM r-65-VM on host 2017-10-29 22:33:35,357 INFO [c.c.h.v.m.HostMO] (DirectAgent-55:ctx-b5ec23f7 172.23.32.1, job-348/job-351, cmd: StopCommand) (logid:2a620d26) VM r-65-VM not found in host cache 2017-10-29 22:33:38,732 DEBUG [c.c.a.t.Request] (DirectAgent-55:ctx-b5ec23f7) (logid:2a620d26) Seq 1-1437774181038032721: Processing: { Ans: , MgmtId: 52240608294, via: 1(172.23.32.1), Ver: v1, Flags: 10, [{"com.cloud.agent.api.StopAnswer":{"result":true,"details":"Stop VM r-65-VM Succeed","wait":0}}] } 2017-10-29 22:33:38,732 DEBUG [c.c.a.m.AgentManagerImpl] (Work-Job-Executor-63:ctx-001ec8b4 job-348/job-351 ctx-4a25af66) (logid:2a620d26) Details from executing class com.cloud.agent.api.StopCommand: Stop VM r-65-VM Succeed 2017-10-29 22:33:38,732 ERROR [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-63:ctx-001ec8b4 job-348/job-351 ctx-4a25af66) (logid:2a620d26) Failed to start instance VM[DomainRouter|r-65-VM] 2017-10-29 22:33:38,738 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-63:ctx-001ec8b4 job-348/job-351 ctx-4a25af66) (logid:2a620d26) Cleaning up resources for the vm VM[DomainRouter|r-65-VM] in Starting state 2017-10-29 22:33:38,747 DEBUG [c.c.a.t.Request] (Work-Job-Executor-63:ctx-001ec8b4 job-348/job-351 ctx-4a25af66) (logid:2a620d26) Seq 1-1437774181038032722: Sending { Cmd , MgmtId: 52240608294, via: 1(172.23.32.1), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"checkBeforeCleanup":false,"forceStop":false,"vmName":"r-65-VM","executeInSequence":true,"wait":0}}] } 2017-10-29 22:33:38,747 DEBUG [c.c.a.t.Request] (Work-Job-Executor-63:ctx-001ec8b4 job-348/job-351 ctx-4a25af66) (logid:2a620d26) Seq 1-1437774181038032722: Executing: { Cmd , MgmtId: 52240608294, via: 1(172.23.32.1), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"checkBeforeCleanup":false,"forceStop":false,"vmName":"r-65-VM","executeInSequence":true,"wait":0}}] } 2017-10-29 22:33:38,747 INFO [c.c.h.v.r.VmwareResource] (DirectAgent-437:ctx-3589fc15 172.23.32.1, job-348/job-351, cmd: StopCommand) (logid:2a620d26) Executing resource StopCommand: {"isProxy":false,"checkBeforeCleanup":false,"forceStop":false,"vmName":"r-65-VM","executeInSequence":true,"wait":0} 2017-10-29 22:33:38,759 DEBUG [c.c.h.v.m.HostMO] (DirectAgent-437:ctx-3589fc15 172.23.32.1, job-348/job-351, cmd: StopCommand) (logid:2a620d26) find VM r-65-VM on host 2017-10-29 22:33:38,759 INFO [c.c.h.v.m.HostMO] (DirectAgent-437:ctx-3589fc15 172.23.32.1, job-348/job-351, cmd: StopCommand) (logid:2a620d26) VM r-65-VM not found in host cache 2017-10-29 22:33:38,817 INFO [c.c.h.v.r.VmwareResource] (DirectAgent-437:ctx-3589fc15 172.23.32.1, job-348/job-351, cmd: StopCommand) (logid:2a620d26) VM r-65-VM is already in stopped state 2017-10-29 22:33:38,817 DEBUG [c.c.a.t.Request] (DirectAgent-437:ctx-3589fc15) (logid:2a620d26) Seq 1-1437774181038032722: Processing: { Ans: , MgmtId: 52240608294, via: 1(172.23.32.1), Ver: v1, Flags: 10, [{"com.cloud.agent.api.StopAnswer":{"result":true,"details":"VM r-65-VM is already in stopped state","wait":0}}] } 2017-10-29 22:33:38,827 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (Work-Job-Executor-63:ctx-001ec8b4 job-348/job-351 ctx-4a25af66) (logid:2a620d26) Successfully updated user statistics as a part of domR VM[DomainRouter|r-65-VM] reboot/stop 2017-10-29 22:33:38,887 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-63:ctx-001ec8b4 job-348/job-351 ctx-4a25af66) (logid:2a620d26) Successfully released network resources for the vm VM[DomainRouter|r-65-VM] 2017-10-29 22:33:38,887 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-63:ctx-001ec8b4 job-348/job-351 ctx-4a25af66) (logid:2a620d26) Successfully cleanued up resources for the vm VM[DomainRouter|r-65-VM] in Starting state 2017-10-29 22:34:30,828 INFO [c.c.v.VirtualMachineManagerImpl] (DirectAgentCronJob-177:ctx-029c04e0) (logid:ab00ad7c) VM r-65-VM is at Stopped and we received a power-off report while there is no pending jobs on it 2017-10-29 22:34:30,828 ERROR [c.c.v.VirtualMachineManagerImpl] (DirectAgentCronJob-177:ctx-029c04e0) (logid:ab00ad7c) Invalid answer received in response to a StopCommand for r-65-VM 2017-10-29 22:35:30,833 INFO [c.c.v.VirtualMachineManagerImpl] (DirectAgentCronJob-177:ctx-a684943b) (logid:ae01d056) VM r-65-VM is at Stopped and we received a power-off report while there is no pending jobs on it 2017-10-29 22:35:30,833 ERROR [c.c.v.VirtualMachineManagerImpl] (DirectAgentCronJob-177:ctx-a684943b) (logid:ae01d056) Invalid answer received in response to a StopCommand for r-65-VM 2017-10-29 22:36:30,833 INFO [c.c.v.VirtualMachineManagerImpl] (DirectAgentCronJob-177:ctx-92f13e34) (logid:3e2a44f0) VM r-65-VM is at Stopped and we received a power-off report while there is no pending jobs on it 2017-10-29 22:36:30,834 ERROR [c.c.v.VirtualMachineManagerImpl] (DirectAgentCronJob-177:ctx-92f13e34) (logid:3e2a44f0) Invalid answer received in response to a StopCommand for r-65-VM