davidjumani opened a new issue #3978: systemvms do not come up on vmware URL: https://github.com/apache/cloudstack/issues/3978 When deploying cloudstack on VMware, systemvms are unable to come up and are continually redeployed. This appears to be due to a null pointer exception. When it is build on commit `dc225de8115a5f03b7eb8959c7e726943dfb2be8`, it all works fine and the systemvms come up as usual ##### ISSUE TYPE * Bug Report ##### COMPONENT NAME VmwareResource ##### CLOUDSTACK VERSION master (4.14) ##### CONFIGURATION Just on bringing up the management server for the first time ##### OS / ENVIRONMENT Identified on vmware 65 ##### SUMMARY The systemvms do not come up but get endlessly recreated, ##### STEPS TO REPRODUCE Just build the master as of commit 36efbfcfd1db812dfbe2b005108f0d0a90170fcf and deploy on VMware ##### EXPECTED RESULTS The systemvms fail to come up and the following log errors ~~~ 2020-03-18 09:23:49,203 INFO [c.c.h.v.r.VmwareResource] (DirectAgent-22:ctx-4d5ca7b7 10.2.3.170, job-38/job-44, cmd: StartCommand) (logid:69db3762) Executing resource StartCommand: {"vm":{"id":6,"name":"v-6-VM","bootloader":"HVM","state":"Starting","type":"ConsoleProxy","cpus":1,"minSpeed":250,"maxSpeed":500,"minRam":1073741824,"maxRam":1073741824,"hostName":"v-6-VM","arch":"x86_64","os":"Other Linux (64-bit)","platformEmulator":"otherLinux64Guest","bootArgs":" template=domP type=consoleproxy host=10.2.3.243 port=8250 name=v-6-VM premium=true zone=1 pod=1 guid=Proxy.6 proxy_vm=6 disable_rp_filter=true eth2ip=10.1.36.162 eth2mask=255.255.224.0 gateway=10.1.63.254 eth0ip=0.0.0.0 eth0mask=0.0.0.0 eth1ip=10.2.8.128 eth1mask=255.255.0.0 mgmtcidr=10.2.0.0/16 localgw=10.2.254.254 internaldns1=8.8.8.8 internaldns2=8.8.4.4 dns1=8.8.8.8 dns2=8.8.4.4","enableHA":false,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"z2udCq7b4hpXI87OGEwccA","params":{"cpuOvercommitRatio":"2.0","vmware.reserve.mem":"false","vmware.reserve.cpu":"false","memoryOvercommitRatio":"1.0","nicAdapter":"E1000"},"uuid":"38badf91-b9cf-4fdd-88a8-3fd94add7a5e","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"5aac8e4b-156c-4382-97bb-7e832c2d5a46","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"d8d96070-6f33-3574-ac36-ed3f3349b304","id":2,"poolType":"NetworkFilesystem","host":"10.2.0.16","path":"/acs/primary/trl-221-v-M7-djumani/trl-221-v-M7-djumani-esxi-pri2","port":2049,"url":"NetworkFilesystem://10.2.0.16/acs/primary/trl-221-v-M7-djumani/trl-221-v-M7-djumani-esxi-pri2/?ROLE=Primary&STOREUUID=d8d96070-6f33-3574-ac36-ed3f3349b304","isManaged":false}},"name":"ROOT-6","size":0,"path":"ROOT-6","volumeId":6,"vmName":"v-6-VM","accountId":1,"format":"OVA","provisioningType":"THIN","id":6,"deviceId":0,"bytesReadRate":0,"bytesWriteRate":0,"iopsReadRate":0,"iopsWriteRate":0,"hypervisorType":"VMware","directDownload":false}},"diskSeq":0,"path":"ROOT-6","type":"ROOT","_details":{"storageHost":"10.2.0.16","managed":"false","storagePort":"2049","volumeSize":"0"}}],"nics":[{"deviceId":2,"networkRateMbps":-1,"defaultNic":true,"pxeDisable":true,"nicUuid":"f22aeb91-8721-4b3d-80c7-6812041717dd","details":{"MacAddressChanges":"true","PromiscuousMode":"false","ForgedTransmits":"true"},"dpdkEnabled":false,"uuid":"f1e62679-7d86-43d5-95e5-12e2ffcead7f","ip":"10.1.36.162","netmask":"255.255.224.0","gateway":"10.1.63.254","mac":"1e:00:36:00:00:02","dns1":"8.8.8.8","dns2":"8.8.4.4","broadcastType":"Vlan","type":"Public","broadcastUri":"vlan://7","isolationUri":"vlan://7","isSecurityGroupEnabled":false,"name":"vSwitch1,,vmwaresvs"},{"deviceId":0,"networkRateMbps":-1,"defaultNic":false,"pxeDisable":true,"nicUuid":"1094b7d3-4be5-424a-ae88-1c4da8b8e39d","details":{"MacAddressChanges":"true","PromiscuousMode":"false","ForgedTransmits":"true"},"dpdkEnabled":false,"uuid":"b6614297-1f5b-4030-869e-60284abf0ce3","mac":"02:00:5a:ad:00:06","broadcastType":"LinkLocal","type":"Control","isSecurityGroupEnabled":false,"name":"vSwitch0,,vmwaresvs"},{"deviceId":1,"networkRateMbps":-1,"defaultNic":false,"pxeDisable":true,"nicUuid":"f978470b-9f73-4f7e-ba9f-5aa3477b1fba","details":{"MacAddressChanges":"true","PromiscuousMode":"false","ForgedTransmits":"true"},"dpdkEnabled":false,"uuid":"b55693c5-f807-464e-80d3-a0224f1dc930","ip":"10.2.8.128","netmask":"255.255.0.0","gateway":"10.2.254.254","mac":"1e:00:05:00:00:1c","broadcastType":"Native","type":"Management","isSecurityGroupEnabled":false,"name":"vSwitch0,,vmwaresvs"}],"guestOsDetails":{},"extraConfig":{}},"hostIp":"10.2.3.170","executeInSequence":false,"wait":0} 2020-03-18 09:23:49,342 DEBUG [c.c.h.v.r.VmwareResource] (DirectAgent-22:ctx-4d5ca7b7 10.2.3.170, job-38/job-44, cmd: StartCommand) (logid:69db3762) Using mapping name : OTHER_LINUX_64_GUEST 2020-03-18 09:23:49,420 DEBUG [c.c.h.v.m.HostMO] (DirectAgent-22:ctx-4d5ca7b7 10.2.3.170, job-38/job-44, cmd: StartCommand) (logid:69db3762) find VM v-6-VM on host 2020-03-18 09:23:49,420 INFO [c.c.h.v.m.HostMO] (DirectAgent-22:ctx-4d5ca7b7 10.2.3.170, job-38/job-44, cmd: StartCommand) (logid:69db3762) VM v-6-VM not found in host cache 2020-03-18 09:23:49,420 DEBUG [c.c.h.v.m.HostMO] (DirectAgent-22:ctx-4d5ca7b7 10.2.3.170, job-38/job-44, cmd: StartCommand) (logid:69db3762) load VM cache on host 2020-03-18 09:23:49,613 INFO [c.c.h.v.r.VmwareResource] (DirectAgent-22:ctx-4d5ca7b7 10.2.3.170, job-38/job-44, cmd: StartCommand) (logid:69db3762) Found vm v-6-VM at other host, relocate to 10.2.3.170 2020-03-18 09:23:50,712 DEBUG [c.c.h.v.m.HypervisorHostHelper] (DirectAgent-37:ctx-0d022993 10.2.3.147, job-37/job-43, cmd: StartCommand) (logid:bd8aa2d5) Prepare network on other host, vlan: 7, host: 10.2.3.170 2020-03-18 09:23:50,936 INFO [c.c.h.v.m.HypervisorHostHelper] (DirectAgent-37:ctx-0d022993 10.2.3.147, job-37/job-43, cmd: StartCommand) (logid:bd8aa2d5) Network cloud.public.7.0.1-vSwitch1 is ready on vSwitch vSwitch1 2020-03-18 09:23:50,936 INFO [c.c.h.v.m.HypervisorHostHelper] (DirectAgent-37:ctx-0d022993 10.2.3.147, job-37/job-43, cmd: StartCommand) (logid:bd8aa2d5) Network cloud.public.7.0.1-vSwitch1 is ready on vSwitch vSwitch1 2020-03-18 09:23:50,936 INFO [c.c.h.v.r.VmwareResource] (DirectAgent-37:ctx-0d022993 10.2.3.147, job-37/job-43, cmd: StartCommand) (logid:bd8aa2d5) Preparing NIC device on network cloud.public.7.0.1-vSwitch1 2020-03-18 09:23:50,936 DEBUG [c.c.h.v.r.VmwareResource] (DirectAgent-37:ctx-0d022993 10.2.3.147, job-37/job-43, cmd: StartCommand) (logid:bd8aa2d5) Prepare NIC at new device {"operation":"ADD","device":{"addressType":"Manual","macAddress":"1e:00:4f:00:00:01","key":-5,"backing":{"network":{"value":"network-63","type":"Network"},"deviceName":"cloud.public.7.0.1-vSwitch1"},"connectable":{"startConnected":true,"allowGuestControl":true,"connected":true}}} 2020-03-18 09:23:50,936 DEBUG [c.c.h.v.m.HostMO] (DirectAgent-37:ctx-0d022993 10.2.3.147, job-37/job-43, cmd: StartCommand) (logid:bd8aa2d5) find VM s-5-VM on host 2020-03-18 09:23:50,936 DEBUG [c.c.h.v.m.HostMO] (DirectAgent-37:ctx-0d022993 10.2.3.147, job-37/job-43, cmd: StartCommand) (logid:bd8aa2d5) VM s-5-VM found in host cache 2020-03-18 09:23:50,968 INFO [c.c.h.v.r.VmwareResource] (DirectAgent-37:ctx-0d022993 10.2.3.147, job-37/job-43, cmd: StartCommand) (logid:bd8aa2d5) Configure VNC port for VM s-5-VM, port: 5947, host: 10.2.3.147 2020-03-18 09:23:50,969 INFO [c.c.h.v.u.VmwareHelper] (DirectAgent-37:ctx-0d022993 10.2.3.147, job-37/job-43, cmd: StartCommand) (logid:bd8aa2d5) [ignored]failed to get message for exception: null 2020-03-18 09:23:50,969 WARN [c.c.h.v.r.VmwareResource] (DirectAgent-37:ctx-0d022993 10.2.3.147, job-37/job-43, cmd: StartCommand) (logid:bd8aa2d5) StartCommand failed due to Exception: java.lang.NullPointerException Message: null java.lang.NullPointerException at com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:2289) at com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:531) at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315) at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) 2020-03-18 09:23:50,970 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-37:ctx-0d022993) (logid:bd8aa2d5) Seq 2-7490612080223977492: Cancelling because one of the answers is false and it is stop on error. 2020-03-18 09:23:50,970 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-37:ctx-0d022993) (logid:bd8aa2d5) Seq 2-7490612080223977492: Response Received: ~~~ guessing this is because the bootmode is null, from the VirtualMachineTO object This is soon followed by logs about how there are insufficient resources to bring up the vm
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
