Looking closer at secondary storage, this appears to be the issue. Thank you Boris.
I've added both a 'local' NFS share (from when I installed it on the single server, 192.168.253.2) and a remote NFS share (192.168.253.30) as secondary storage. They both show up just fine in the CloudStack GUI and when I mount the remote NFS share to a temp directory (/secondary) on the management server, it shows up just fine, so it doesn't appear to be an issue with NFS: [root@local secondary]# mount -t nfs 192.168.253.30:/mnt/secondary /secondary [root@local secondary]# df -h Filesystem Size Used Avail Use% Mounted on 192.168.253.30:/mnt/secondary 923G 5.7G 917G 1% /secondary Again, I'm not actually trying to use /secondary, I'm just demonstrating that it mounts fine from the management server and that the space is available. However, the CloudStack GUI doesn't show any actual space in secondary storage - despite two servers being configured as secondary storage. Screen shots attached. How would I go about further troubleshooting this issue? Thank you very much for any direction you can offer. -- Kind regards, Jordan Michaels Vivio Technologies ----- Original Message ----- From: "Boris Stoyanov" <boris.stoya...@shapeblue.com> To: "users" <users@cloudstack.apache.org>, "jordan" <jor...@viviotech.net> Cc: "Simon Weller" <swel...@ena.com> Sent: Thursday, 18 October, 2018 22:20:06 Subject: Re: Where to look when your SSVM creation NPE's? 2018-10-18 06:20:26,076 INFO [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-70aba7ab) (logid:f3c51c51) Unable to start secondary storage vm for standby capacity, vm id : 346, will recycle it and start a new one 2018-10-18 06:20:26,095 INFO [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:ctx-70aba7ab) (logid:f3c51c51) Primary secondary storage is not even started, wait until next turn Looks like you have an issue with secondary storage, can you check status/add it again and troubleshoot it? Bobby. boris.stoya...@shapeblue.com www.shapeblue.com Amadeus House, Floral Street, London WC2E 9DPUK @shapeblue > On 19 Oct 2018, at 0:23, Jordan Michaels <mic...@viviotech.net> wrote: > > Hi Si, > > Thanks so much for the follow-up! > >> Could you tell us a bit more about your setup? > > I've got a little lab setup, with a dedicated router, switch, and several > 1U's that I can test with. Will eventually set up a little micro-environment > with VLAN's and such but for now just trying to get things working - which is > turning out to be much more difficult than it sounds. > >> Hypervisor Type? > > KVM for now > >> Network design (basic or advanced)? > > Basic; using a single 192.168.253.0/24 range for the various networks similar > to what's shown here: > http://docs.cloudstack.apache.org/en/4.11.1.0/installguide/hypervisor/kvm.html#configuring-the-networking > > I've got the two bridges set up with the management IP assigned to cloudbr0 > and no ip assigned to cloudbr1. > >> Primary storage type? > > I've tried NFS and local, though the result is the same regardless. > >> CloudStack version? > > 4.11 > >> OS being used? > > CentOS 7 > > More than just getting it working, I'd like to know where to find out what > error is actually occurring here. I want to understand the details. > > I'm guessing the NPE is because the API was expecting to get something back > from the storage server and it didn't? Is the command that was sent logged > anywhere? What about the result? If the command that was sent to the storage > server threw an error, is there a way that I can see what the error was > specifically? > > Thanks so much for your time. I really appreciate it. > > -- > Kind regards, > Jordan Michaels > Vivio Technologies > > ----- Original Message ----- > From: "Simon Weller" <swel...@ena.com> > To: "users" <users@cloudstack.apache.org>, "jordan" <jor...@viviotech.net> > Sent: Thursday, 18 October, 2018 13:14:41 > Subject: Re: Where to look when your SSVM creation NPE's? > > Jordan, > > > Could you tell us a bit more about your setup? > > Hypervisor Type? Network design (basic or advanced)? Primary storage type? > CloudStack version? OS being used? > > - Si > ________________________________ > From: Jordan Michaels <mic...@viviotech.net> > Sent: Thursday, October 18, 2018 2:26 PM > To: users > Subject: Where to look when your SSVM creation NPE's? > > Hi Folks, > > I've added a host to the CloudStack Management server but now I'm getting > NPE's when the Management server tries to create the System VM's: > > 2018-10-18 06:20:26,052 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > (Work-Job-Executor-40:ctx-ac017e0e job-1248/job-1621) (logid:b6f3dd96) Done > executing com.cloud.vm.VmWorkStart for job-1621 > 2018-10-18 06:20:26,054 INFO [o.a.c.f.j.i.AsyncJobMonitor] > (Work-Job-Executor-40:ctx-ac017e0e job-1248/job-1621) (logid:b6f3dd96) Remove > job-1621 from job monitoring > 2018-10-18 06:20:26,073 WARN [o.a.c.s.SecondaryStorageManagerImpl] > (secstorage-1:ctx-70aba7ab) (logid:f3c51c51) Exception while trying to start > secondary storage vm > java.lang.NullPointerException > at > com.cloud.network.guru.DirectPodBasedNetworkGuru$2.doInTransactionWithoutResult(DirectPodBasedNetworkGuru.java:183) > at > com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(TransactionCallbackWithExceptionNoReturn.java:25) > at > com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(TransactionCallbackWithExceptionNoReturn.java:21) > at com.cloud.utils.db.Transaction.execute(Transaction.java:40) > at > com.cloud.network.guru.DirectPodBasedNetworkGuru.getIp(DirectPodBasedNetworkGuru.java:173) > at > com.cloud.network.guru.DirectPodBasedNetworkGuru.reserve(DirectPodBasedNetworkGuru.java:161) > at > org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepareNic(NetworkOrchestrator.java:1594) > at > org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepare(NetworkOrchestrator.java:1565) > at > com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1111) > at > com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:4930) > at sun.reflect.GeneratedMethodAccessor155.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107) > at > com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5093) > at > com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102) > at > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > at > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > 2018-10-18 06:20:26,076 INFO [o.a.c.s.SecondaryStorageManagerImpl] > (secstorage-1:ctx-70aba7ab) (logid:f3c51c51) Unable to start secondary > storage vm for standby capacity, vm id : 346, will recycle it and start a new > one > 2018-10-18 06:20:26,076 DEBUG [c.c.a.SecondaryStorageVmAlertAdapter] > (secstorage-1:ctx-70aba7ab) (logid:f3c51c51) received secondary storage vm > alert > 2018-10-18 06:20:26,076 DEBUG [c.c.a.SecondaryStorageVmAlertAdapter] > (secstorage-1:ctx-70aba7ab) (logid:f3c51c51) Secondary Storage Vm creation > failure, zone: lab > 2018-10-18 06:20:26,077 WARN [o.a.c.alerts] (secstorage-1:ctx-70aba7ab) > (logid:f3c51c51) AlertType:: 19 | dataCenterId:: 1 | podId:: null | > clusterId:: null | message:: Secondary Storage Vm creation failure. zone: > lab, error details: null > 2018-10-18 06:20:26,095 INFO [o.a.c.s.PremiumSecondaryStorageManagerImpl] > (secstorage-1:ctx-70aba7ab) (logid:f3c51c51) Primary secondary storage is not > even started, wait until next turn > > Based on the error, it sounds like there's an issue with the SSVM being able > to communicate with secondary storage, but I don't know where to look for > additional details about what's going on, or how the SSVM's actually work. I > need more information. I've been stumbling around in the dark and making sure > I have secondary storage added without errors to the cloudstack management > server, I've configured the SSVM template... not sure what else could be > wrong or where else to look. > > Could someone point me in the right direction on where to look to get what's > actually failing here? > > Thank you in advance! > > Kind regards, > Jordan Michaels > Vivio Technologies
