Hi Folks, I've added a host to the CloudStack Management server but now I'm getting NPE's when the Management server tries to create the System VM's:
2018-10-18 06:20:26,052 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-40:ctx-ac017e0e job-1248/job-1621) (logid:b6f3dd96) Done executing com.cloud.vm.VmWorkStart for job-1621 2018-10-18 06:20:26,054 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-40:ctx-ac017e0e job-1248/job-1621) (logid:b6f3dd96) Remove job-1621 from job monitoring 2018-10-18 06:20:26,073 WARN [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-70aba7ab) (logid:f3c51c51) Exception while trying to start secondary storage vm java.lang.NullPointerException at com.cloud.network.guru.DirectPodBasedNetworkGuru$2.doInTransactionWithoutResult(DirectPodBasedNetworkGuru.java:183) at com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(TransactionCallbackWithExceptionNoReturn.java:25) at com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(TransactionCallbackWithExceptionNoReturn.java:21) at com.cloud.utils.db.Transaction.execute(Transaction.java:40) at com.cloud.network.guru.DirectPodBasedNetworkGuru.getIp(DirectPodBasedNetworkGuru.java:173) at com.cloud.network.guru.DirectPodBasedNetworkGuru.reserve(DirectPodBasedNetworkGuru.java:161) at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepareNic(NetworkOrchestrator.java:1594) at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepare(NetworkOrchestrator.java:1565) at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1111) at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:4930) at sun.reflect.GeneratedMethodAccessor155.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107) at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5093) at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581) at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2018-10-18 06:20:26,076 INFO [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-70aba7ab) (logid:f3c51c51) Unable to start secondary storage vm for standby capacity, vm id : 346, will recycle it and start a new one 2018-10-18 06:20:26,076 DEBUG [c.c.a.SecondaryStorageVmAlertAdapter] (secstorage-1:ctx-70aba7ab) (logid:f3c51c51) received secondary storage vm alert 2018-10-18 06:20:26,076 DEBUG [c.c.a.SecondaryStorageVmAlertAdapter] (secstorage-1:ctx-70aba7ab) (logid:f3c51c51) Secondary Storage Vm creation failure, zone: lab 2018-10-18 06:20:26,077 WARN [o.a.c.alerts] (secstorage-1:ctx-70aba7ab) (logid:f3c51c51) AlertType:: 19 | dataCenterId:: 1 | podId:: null | clusterId:: null | message:: Secondary Storage Vm creation failure. zone: lab, error details: null 2018-10-18 06:20:26,095 INFO [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:ctx-70aba7ab) (logid:f3c51c51) Primary secondary storage is not even started, wait until next turn Based on the error, it sounds like there's an issue with the SSVM being able to communicate with secondary storage, but I don't know where to look for additional details about what's going on, or how the SSVM's actually work. I need more information. I've been stumbling around in the dark and making sure I have secondary storage added without errors to the cloudstack management server, I've configured the SSVM template... not sure what else could be wrong or where else to look. Could someone point me in the right direction on where to look to get what's actually failing here? Thank you in advance! Kind regards, Jordan Michaels Vivio Technologies