Frank Maximus created CLOUDSTACK-9530: -----------------------------------------
Summary: Unable to create VPC: VR creation fails, because execution GetDomRVersionCmd gets incomplete data. Key: CLOUDSTACK-9530 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-9530 Project: CloudStack Issue Type: Bug Security Level: Public (Anyone can view this level - this is the default.) Components: SystemVM, VMware Affects Versions: 4.9.0 Environment: 4.9, Nuage VSP, vmware ESXi Reporter: Frank Maximus Priority: Critical Create VPC often fails, as VR fails to execute the start commands. The failing command is GetDomRVersionCmd. The result only contains the contents of /etc/cloudstack-release, and the ampersand, but is missing the cloud-scripts-signature hash. {noformat:title=log} 2016-09-30 10:21:38,170 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-295:ctx-8f044370) (logid:2a47dd8b) Seq 5-4349351340133057061: Executing request 2016-09-30 10:21:38,328 DEBUG [c.c.h.v.r.VmwareResource] (DirectAgent-291:ctx-d8e41e01 10.31.56.167, job-335/job-336, cmd: GetDomRVersionCmd) (logid:2e69f441) get_template_version.sh execution result: true 2016-09-30 10:21:38,328 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-291:ctx-d8e41e01) (logid:2e69f441) Seq 4-1897985767959953665: Cancelling because one of the answers is false and it is stop on error. 2016-09-30 10:21:38,328 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-291:ctx-d8e41e01) (logid:2e69f441) Seq 4-1897985767959953665: Response Received: 2016-09-30 10:21:38,331 DEBUG [c.c.a.t.Request] (DirectAgent-291:ctx-d8e41e01) (logid:2e69f441) Seq 4-1897985767959953665: Processing: { Ans: , MgmtId: 275619427423232, via: 4(10.31.56.167), Ver: v1, Flags: 110, [{"com.cloud.agent.api.StartAnswer":{"vm":{"id":41,"name":"r-41-VM","bootloader":"HVM","type":"DomainRouter","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":268435456,"maxRam":268435456,"hostName":"r-41-VM","arch":"x86_64","os":"Debian GNU/Linux 5.0 (64-bit)","platformEmulator":"debian5Guest","bootArgs":" vpccidr=10.1.1.0/24 domain=cs2cloud.internal dns1=8.8.8.8 privategateway=None template=domP name=r-41-VM eth0ip=10.31.39.41 eth0mask=255.255.224.0 mgmtcidr=10.31.32.0/19 localgw=10.31.32.1 type=vpcrouter disable_rp_filter=true extra_pubnics=2 baremetalnotificationsecuritykey=VQmAnS2RuCEhPskNakZ46gJM7dfrq52xuZarIrZz4Al-O7GNEThU_v4N0JGf9MH09U7RgXT_MFOG8VtEVDqYYw baremetalnotificationapikey=jUNK98yQ4sBOuNNq9H7sDxfm9JcWSfjCnoFTHEdTaC1W2LRc_RhzBMouEHR9p3MLlsCbCQWeE25V20Mx1CNmiA host=10.31.56.204 port=8080 nic_macs=02:00:16:ef:00:1c","enableHA":true,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"NuVaajLlBO0gl-3jx3-GPw","params":{"nicAdapter":"E1000","vmware.reserve.cpu":"false","vmware.reserve.mem":"false"},"uuid":"1698edae-9c1e-4d89-952c-1bc331c7c08e","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"528f5286-735d-4fc6-bcea-163f30cd4b6a","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1bbabb20-3924-3667-b53d-54be05ffbea9","id":1,"poolType":"NetworkFilesystem","host":"10.31.56.204","path":"/export/primary","port":2049,"url":"NetworkFilesystem://10.31.56.204/export/primary/?ROLE=Primary&STOREUUID=1bbabb20-3924-3667-b53d-54be05ffbea9"}},"name":"ROOT-41","size":3145728000,"path":"ROOT-41-000001","volumeId":41,"vmName":"r-41-VM","accountId":2,"chainInfo":"{\"diskDeviceBusName\":\"scsi0:0\",\"diskChain\":[\"[1bbabb2039243667b53d54be05ffbea9] r-41-VM/ROOT-41-000001.vmdk\"]}","format":"OVA","provisioningType":"THIN","id":41,"deviceId":0,"bytesReadRate":0,"bytesWriteRate":0,"iopsReadRate":0,"iopsWriteRate":0,"hypervisorType":"VMware"}},"diskSeq":0,"path":"ROOT-41-000001","type":"ROOT","_details":{"managed":"false","storagePort":"2049","storageHost":"10.31.56.204","volumeSize":"3145728000"}}],"nics":[{"deviceId":0,"networkRateMbps":-1,"defaultNic":false,"pxeDisable":true,"nicUuid":"b960d1e4-d319-46f3-ba2f-8b70ae3365c0","uuid":"95bebc47-c4df-4273-91d2-12d2c4f6e352","ip":"10.31.39.41","netmask":"255.255.224.0","gateway":"10.31.32.1","mac":"02:00:16:ef:00:1c","broadcastType":"Native","type":"Control","isSecurityGroupEnabled":false,"name":"vSwitch0"}]},"_iqnToPath":{},"result":true,"wait":0}},{"com.cloud.agent.api.check.CheckSshAnswer":{"result":true,"wait":0}},{"com.cloud.agent.api.GetDomRVersionAnswer":{"result":false,"details":"Cloudstack Release 4.6.0 Mon Nov 9 11:21:06 UTC 2015&","wait":0}}] } 2016-09-30 10:21:38,332 DEBUG [c.c.a.m.AgentAttache] (DirectAgent-291:ctx-d8e41e01) (logid:2e69f441) Seq 4-1897985767959953665: No more commands found 2016-09-30 10:21:38,332 DEBUG [c.c.a.t.Request] (Work-Job-Executor-19:ctx-0154027f job-335/job-336 ctx-c08234bb) (logid:2e69f441) Seq 4-1897985767959953665: Received: { Ans: , MgmtId: 275619427423232, via: 4(10.31.56.167), Ver: v1, Flags: 110, { StartAnswer, CheckSshAnswer, GetDomRVersionAnswer } } 2016-09-30 10:21:38,344 WARN [o.a.c.alerts] (Work-Job-Executor-19:ctx-0154027f job-335/job-336 ctx-c08234bb) (logid:2e69f441) alertType:: 9 // dataCenterId:: 1 // podId:: 1 // clusterId:: null // message:: Command: com.cloud.agent.api.GetDomRVersionCommand failed while starting virtual router 2016-09-30 10:21:38,350 ERROR [c.c.n.r.VirtualNetworkApplianceManagerImpl] (Work-Job-Executor-19:ctx-0154027f job-335/job-336 ctx-c08234bb) (logid:2e69f441) Cloudstack Release 4.6.0 Mon Nov 9 11:21:06 UTC 2015& 2016-09-30 10:21:38,350 WARN [c.c.n.r.VirtualNetworkApplianceManagerImpl] (Work-Job-Executor-19:ctx-0154027f job-335/job-336 ctx-c08234bb) (logid:2e69f441) Command: com.cloud.agent.api.GetDomRVersionCommand failed while starting virtual router 2016-09-30 10:21:38,350 INFO [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-19:ctx-0154027f job-335/job-336 ctx-c08234bb) (logid:2e69f441) The guru did not like the answers so stopping VM[DomainRouter|r-41-VM] {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)