Re: Where to look when your SSVM creation NPE's?
Looking closer at secondary storage, this appears to be the issue. Thank you Boris. I've added both a 'local' NFS share (from when I installed it on the single server, 192.168.253.2) and a remote NFS share (192.168.253.30) as secondary storage. They both show up just fine in the CloudStack GUI and when I mount the remote NFS share to a temp directory (/secondary) on the management server, it shows up just fine, so it doesn't appear to be an issue with NFS: [root@local secondary]# mount -t nfs 192.168.253.30:/mnt/secondary /secondary [root@local secondary]# df -h Filesystem Size Used Avail Use% Mounted on 192.168.253.30:/mnt/secondary 923G 5.7G 917G 1% /secondary Again, I'm not actually trying to use /secondary, I'm just demonstrating that it mounts fine from the management server and that the space is available. However, the CloudStack GUI doesn't show any actual space in secondary storage - despite two servers being configured as secondary storage. Screen shots attached. How would I go about further troubleshooting this issue? Thank you very much for any direction you can offer. -- Kind regards, Jordan Michaels Vivio Technologies - Original Message - From: "Boris Stoyanov" To: "users" , "jordan" Cc: "Simon Weller" Sent: Thursday, 18 October, 2018 22:20:06 Subject: Re: Where to look when your SSVM creation NPE's? 2018-10-18 06:20:26,076 INFO [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-70aba7ab) (logid:f3c51c51) Unable to start secondary storage vm for standby capacity, vm id : 346, will recycle it and start a new one 2018-10-18 06:20:26,095 INFO [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:ctx-70aba7ab) (logid:f3c51c51) Primary secondary storage is not even started, wait until next turn Looks like you have an issue with secondary storage, can you check status/add it again and troubleshoot it? Bobby. boris.stoya...@shapeblue.com www.shapeblue.com Amadeus House, Floral Street, London WC2E 9DPUK @shapeblue > On 19 Oct 2018, at 0:23, Jordan Michaels wrote: > > Hi Si, > > Thanks so much for the follow-up! > >> Could you tell us a bit more about your setup? > > I've got a little lab setup, with a dedicated router, switch, and several > 1U's that I can test with. Will eventually set up a little micro-environment > with VLAN's and such but for now just trying to get things working - which is > turning out to be much more difficult than it sounds. > >> Hypervisor Type? > > KVM for now > >> Network design (basic or advanced)? > > Basic; using a single 192.168.253.0/24 range for the various networks similar > to what's shown here: > http://docs.cloudstack.apache.org/en/4.11.1.0/installguide/hypervisor/kvm.html#configuring-the-networking > > I've got the two bridges set up with the management IP assigned to cloudbr0 > and no ip assigned to cloudbr1. > >> Primary storage type? > > I've tried NFS and local, though the result is the same regardless. > >> CloudStack version? > > 4.11 > >> OS being used? > > CentOS 7 > > More than just getting it working, I'd like to know where to find out what > error is actually occurring here. I want to understand the details. > > I'm guessing the NPE is because the API was expecting to get something back > from the storage server and it didn't? Is the command that was sent logged > anywhere? What about the result? If the command that was sent to the storage > server threw an error, is there a way that I can see what the error was > specifically? > > Thanks so much for your time. I really appreciate it. > > -- > Kind regards, > Jordan Michaels > Vivio Technologies > > - Original Message - > From: "Simon Weller" > To: "users" , "jordan" > Sent: Thursday, 18 October, 2018 13:14:41 > Subject: Re: Where to look when your SSVM creation NPE's? > > Jordan, > > > Could you tell us a bit more about your setup? > > Hypervisor Type? Network design (basic or advanced)? Primary storage type? > CloudStack version? OS being used? > > - Si > > From: Jordan Michaels > Sent: Thursday, October 18, 2018 2:26 PM > To: users > Subject: Where to look when your SSVM creation NPE's? > > Hi Folks, > > I've added a host to the CloudStack Management server but now I'm getting > NPE's when the Management server tries to create the System VM's: > > 2018-10-18 06:20:26,052 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > (Work-Job-Executor-40:ctx-ac017e0e job-1248/job-1621) (logid:b6f3dd96) Done > executing com.cloud.vm.VmWorkStart for job-1621 > 2018-10-18 06:20:26,054 INFO [o.a.c.f.
Re: Where to look when your SSVM creation NPE's?
TBH, I'm not sure. Probably the latest. Whatever is delivered when you add the CloudStack repo. I built this instance ~3-4 days ago. -- Kind regards, Jordan Michaels Vivio Technologies - Original Message - From: "Rafael Weingärtner" To: "users" , "jordan" Cc: "Simon Weller" Sent: Friday, 19 October, 2018 02:49:32 Subject: Re: Where to look when your SSVM creation NPE's? 4.11.0.0? -- Rafael Weingärtner
Re: Where to look when your SSVM creation NPE's?
4.11.0.0? On Thu, Oct 18, 2018 at 7:44 PM Jordan Michaels wrote: > Hi Si, > > Thanks so much for the follow-up! > > > Could you tell us a bit more about your setup? > > I've got a little lab setup, with a dedicated router, switch, and several > 1U's that I can test with. Will eventually set up a little > micro-environment with VLAN's and such but for now just trying to get > things working - which is turning out to be much more difficult than it > sounds. > > > Hypervisor Type? > > KVM for now > > > Network design (basic or advanced)? > > Basic; using a single 192.168.253.0/24 range for the various networks > similar to what's shown here: > > http://docs.cloudstack.apache.org/en/4.11.1.0/installguide/hypervisor/kvm.html#configuring-the-networking > > I've got the two bridges set up with the management IP assigned to > cloudbr0 and no ip assigned to cloudbr1. > > > Primary storage type? > > I've tried NFS and local, though the result is the same regardless. > > > CloudStack version? > > 4.11 > > > OS being used? > > CentOS 7 > > More than just getting it working, I'd like to know where to find out what > error is actually occurring here. I want to understand the details. > > I'm guessing the NPE is because the API was expecting to get something > back from the storage server and it didn't? Is the command that was sent > logged anywhere? What about the result? If the command that was sent to the > storage server threw an error, is there a way that I can see what the error > was specifically? > > Thanks so much for your time. I really appreciate it. > > -- > Kind regards, > Jordan Michaels > Vivio Technologies > > - Original Message - > From: "Simon Weller" > To: "users" , "jordan" > Sent: Thursday, 18 October, 2018 13:14:41 > Subject: Re: Where to look when your SSVM creation NPE's? > > Jordan, > > > Could you tell us a bit more about your setup? > > Hypervisor Type? Network design (basic or advanced)? Primary storage type? > CloudStack version? OS being used? > > - Si > > From: Jordan Michaels > Sent: Thursday, October 18, 2018 2:26 PM > To: users > Subject: Where to look when your SSVM creation NPE's? > > Hi Folks, > > I've added a host to the CloudStack Management server but now I'm getting > NPE's when the Management server tries to create the System VM's: > > 2018-10-18 06:20:26,052 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > (Work-Job-Executor-40:ctx-ac017e0e job-1248/job-1621) (logid:b6f3dd96) Done > executing com.cloud.vm.VmWorkStart for job-1621 > 2018-10-18 06:20:26,054 INFO [o.a.c.f.j.i.AsyncJobMonitor] > (Work-Job-Executor-40:ctx-ac017e0e job-1248/job-1621) (logid:b6f3dd96) > Remove job-1621 from job monitoring > 2018-10-18 06:20:26,073 WARN [o.a.c.s.SecondaryStorageManagerImpl] > (secstorage-1:ctx-70aba7ab) (logid:f3c51c51) Exception while trying to > start secondary storage vm > java.lang.NullPointerException > at > com.cloud.network.guru.DirectPodBasedNetworkGuru$2.doInTransactionWithoutResult(DirectPodBasedNetworkGuru.java:183) > at > com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(TransactionCallbackWithExceptionNoReturn.java:25) > at > com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(TransactionCallbackWithExceptionNoReturn.java:21) > at com.cloud.utils.db.Transaction.execute(Transaction.java:40) > at > com.cloud.network.guru.DirectPodBasedNetworkGuru.getIp(DirectPodBasedNetworkGuru.java:173) > at > com.cloud.network.guru.DirectPodBasedNetworkGuru.reserve(DirectPodBasedNetworkGuru.java:161) > at > org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepareNic(NetworkOrchestrator.java:1594) > at > org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepare(NetworkOrchestrator.java:1565) > at > com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:) > at > com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:4930) > at sun.reflect.GeneratedMethodAccessor155.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107) > at > com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5093) > at > com.cloud.vm.VmWo
Re: Where to look when your SSVM creation NPE's?
2018-10-18 06:20:26,076 INFO [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-70aba7ab) (logid:f3c51c51) Unable to start secondary storage vm for standby capacity, vm id : 346, will recycle it and start a new one 2018-10-18 06:20:26,095 INFO [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:ctx-70aba7ab) (logid:f3c51c51) Primary secondary storage is not even started, wait until next turn Looks like you have an issue with secondary storage, can you check status/add it again and troubleshoot it? Bobby. boris.stoya...@shapeblue.com www.shapeblue.com Amadeus House, Floral Street, London WC2E 9DPUK @shapeblue > On 19 Oct 2018, at 0:23, Jordan Michaels wrote: > > Hi Si, > > Thanks so much for the follow-up! > >> Could you tell us a bit more about your setup? > > I've got a little lab setup, with a dedicated router, switch, and several > 1U's that I can test with. Will eventually set up a little micro-environment > with VLAN's and such but for now just trying to get things working - which is > turning out to be much more difficult than it sounds. > >> Hypervisor Type? > > KVM for now > >> Network design (basic or advanced)? > > Basic; using a single 192.168.253.0/24 range for the various networks similar > to what's shown here: > http://docs.cloudstack.apache.org/en/4.11.1.0/installguide/hypervisor/kvm.html#configuring-the-networking > > I've got the two bridges set up with the management IP assigned to cloudbr0 > and no ip assigned to cloudbr1. > >> Primary storage type? > > I've tried NFS and local, though the result is the same regardless. > >> CloudStack version? > > 4.11 > >> OS being used? > > CentOS 7 > > More than just getting it working, I'd like to know where to find out what > error is actually occurring here. I want to understand the details. > > I'm guessing the NPE is because the API was expecting to get something back > from the storage server and it didn't? Is the command that was sent logged > anywhere? What about the result? If the command that was sent to the storage > server threw an error, is there a way that I can see what the error was > specifically? > > Thanks so much for your time. I really appreciate it. > > -- > Kind regards, > Jordan Michaels > Vivio Technologies > > - Original Message - > From: "Simon Weller" > To: "users" , "jordan" > Sent: Thursday, 18 October, 2018 13:14:41 > Subject: Re: Where to look when your SSVM creation NPE's? > > Jordan, > > > Could you tell us a bit more about your setup? > > Hypervisor Type? Network design (basic or advanced)? Primary storage type? > CloudStack version? OS being used? > > - Si > > From: Jordan Michaels > Sent: Thursday, October 18, 2018 2:26 PM > To: users > Subject: Where to look when your SSVM creation NPE's? > > Hi Folks, > > I've added a host to the CloudStack Management server but now I'm getting > NPE's when the Management server tries to create the System VM's: > > 2018-10-18 06:20:26,052 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > (Work-Job-Executor-40:ctx-ac017e0e job-1248/job-1621) (logid:b6f3dd96) Done > executing com.cloud.vm.VmWorkStart for job-1621 > 2018-10-18 06:20:26,054 INFO [o.a.c.f.j.i.AsyncJobMonitor] > (Work-Job-Executor-40:ctx-ac017e0e job-1248/job-1621) (logid:b6f3dd96) Remove > job-1621 from job monitoring > 2018-10-18 06:20:26,073 WARN [o.a.c.s.SecondaryStorageManagerImpl] > (secstorage-1:ctx-70aba7ab) (logid:f3c51c51) Exception while trying to start > secondary storage vm > java.lang.NullPointerException >at > com.cloud.network.guru.DirectPodBasedNetworkGuru$2.doInTransactionWithoutResult(DirectPodBasedNetworkGuru.java:183) >at > com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(TransactionCallbackWithExceptionNoReturn.java:25) >at > com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(TransactionCallbackWithExceptionNoReturn.java:21) >at com.cloud.utils.db.Transaction.execute(Transaction.java:40) >at > com.cloud.network.guru.DirectPodBasedNetworkGuru.getIp(DirectPodBasedNetworkGuru.java:173) >at > com.cloud.network.guru.DirectPodBasedNetworkGuru.reserve(DirectPodBasedNetworkGuru.java:161) >at > org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepareNic(NetworkOrchestrator.java:1594) >at > org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepare(NetworkOrchestrator.java:1565) >at > com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMac
Re: Where to look when your SSVM creation NPE's?
Hi Si, Thanks so much for the follow-up! > Could you tell us a bit more about your setup? I've got a little lab setup, with a dedicated router, switch, and several 1U's that I can test with. Will eventually set up a little micro-environment with VLAN's and such but for now just trying to get things working - which is turning out to be much more difficult than it sounds. > Hypervisor Type? KVM for now > Network design (basic or advanced)? Basic; using a single 192.168.253.0/24 range for the various networks similar to what's shown here: http://docs.cloudstack.apache.org/en/4.11.1.0/installguide/hypervisor/kvm.html#configuring-the-networking I've got the two bridges set up with the management IP assigned to cloudbr0 and no ip assigned to cloudbr1. > Primary storage type? I've tried NFS and local, though the result is the same regardless. > CloudStack version? 4.11 > OS being used? CentOS 7 More than just getting it working, I'd like to know where to find out what error is actually occurring here. I want to understand the details. I'm guessing the NPE is because the API was expecting to get something back from the storage server and it didn't? Is the command that was sent logged anywhere? What about the result? If the command that was sent to the storage server threw an error, is there a way that I can see what the error was specifically? Thanks so much for your time. I really appreciate it. -- Kind regards, Jordan Michaels Vivio Technologies - Original Message - From: "Simon Weller" To: "users" , "jordan" Sent: Thursday, 18 October, 2018 13:14:41 Subject: Re: Where to look when your SSVM creation NPE's? Jordan, Could you tell us a bit more about your setup? Hypervisor Type? Network design (basic or advanced)? Primary storage type? CloudStack version? OS being used? - Si From: Jordan Michaels Sent: Thursday, October 18, 2018 2:26 PM To: users Subject: Where to look when your SSVM creation NPE's? Hi Folks, I've added a host to the CloudStack Management server but now I'm getting NPE's when the Management server tries to create the System VM's: 2018-10-18 06:20:26,052 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-40:ctx-ac017e0e job-1248/job-1621) (logid:b6f3dd96) Done executing com.cloud.vm.VmWorkStart for job-1621 2018-10-18 06:20:26,054 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-40:ctx-ac017e0e job-1248/job-1621) (logid:b6f3dd96) Remove job-1621 from job monitoring 2018-10-18 06:20:26,073 WARN [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-70aba7ab) (logid:f3c51c51) Exception while trying to start secondary storage vm java.lang.NullPointerException at com.cloud.network.guru.DirectPodBasedNetworkGuru$2.doInTransactionWithoutResult(DirectPodBasedNetworkGuru.java:183) at com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(TransactionCallbackWithExceptionNoReturn.java:25) at com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(TransactionCallbackWithExceptionNoReturn.java:21) at com.cloud.utils.db.Transaction.execute(Transaction.java:40) at com.cloud.network.guru.DirectPodBasedNetworkGuru.getIp(DirectPodBasedNetworkGuru.java:173) at com.cloud.network.guru.DirectPodBasedNetworkGuru.reserve(DirectPodBasedNetworkGuru.java:161) at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepareNic(NetworkOrchestrator.java:1594) at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepare(NetworkOrchestrator.java:1565) at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:) at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:4930) at sun.reflect.GeneratedMethodAccessor155.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107) at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5093) at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581) at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) at org.apache.cloudstack.managed.context.impl.DefaultManagedCont
Re: Where to look when your SSVM creation NPE's?
Can you please make sure that the Agent is up and running - check agent logs etc ? This has nothing to do with "SSVM cant talk to Secondary Storage" (actually this can be checked/tested with existing script from inside SSVM, once SSVM has been started) It might be capacity issue (though it's usually reported nicely) or perhaps KVM host can't access Primary Storage. You did preseed systemVM template as in the guide, right ? Please share versions, as Simon suggested. Thx On Thu, 18 Oct 2018 at 22:15, Simon Weller wrote: > Jordan, > > > Could you tell us a bit more about your setup? > > Hypervisor Type? Network design (basic or advanced)? Primary storage type? > CloudStack version? OS being used? > > - Si > > From: Jordan Michaels > Sent: Thursday, October 18, 2018 2:26 PM > To: users > Subject: Where to look when your SSVM creation NPE's? > > Hi Folks, > > I've added a host to the CloudStack Management server but now I'm getting > NPE's when the Management server tries to create the System VM's: > > 2018-10-18 06:20:26,052 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > (Work-Job-Executor-40:ctx-ac017e0e job-1248/job-1621) (logid:b6f3dd96) Done > executing com.cloud.vm.VmWorkStart for job-1621 > 2018-10-18 06:20:26,054 INFO [o.a.c.f.j.i.AsyncJobMonitor] > (Work-Job-Executor-40:ctx-ac017e0e job-1248/job-1621) (logid:b6f3dd96) > Remove job-1621 from job monitoring > 2018-10-18 06:20:26,073 WARN [o.a.c.s.SecondaryStorageManagerImpl] > (secstorage-1:ctx-70aba7ab) (logid:f3c51c51) Exception while trying to > start secondary storage vm > java.lang.NullPointerException > at > com.cloud.network.guru.DirectPodBasedNetworkGuru$2.doInTransactionWithoutResult(DirectPodBasedNetworkGuru.java:183) > at > com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(TransactionCallbackWithExceptionNoReturn.java:25) > at > com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(TransactionCallbackWithExceptionNoReturn.java:21) > at com.cloud.utils.db.Transaction.execute(Transaction.java:40) > at > com.cloud.network.guru.DirectPodBasedNetworkGuru.getIp(DirectPodBasedNetworkGuru.java:173) > at > com.cloud.network.guru.DirectPodBasedNetworkGuru.reserve(DirectPodBasedNetworkGuru.java:161) > at > org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepareNic(NetworkOrchestrator.java:1594) > at > org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepare(NetworkOrchestrator.java:1565) > at > com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:) > at > com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:4930) > at sun.reflect.GeneratedMethodAccessor155.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107) > at > com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5093) > at > com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102) > at > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > at > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > 2018-10-18 06:20:26,076 INFO [o.a.c.s.SecondaryStorageManagerImpl] > (secstorage-1:ctx-70aba7ab) (logid:f3c51c51) Unable to start secondary > storage vm for standby capacity, vm id : 346, will recycle it and start a > new one > 2018-10-18 06:20:26,076 DEBUG [c.c.a.SecondaryStorageVmAlertAdapter] > (secstorage-1:ctx-70aba7ab) (logid:f3c51c51) received secondary storage vm > alert > 2018-10-18 06:20:26,076 DEBUG [c.c.a.SecondaryStorageVmAlertAdapter] >
Re: Where to look when your SSVM creation NPE's?
Jordan, Could you tell us a bit more about your setup? Hypervisor Type? Network design (basic or advanced)? Primary storage type? CloudStack version? OS being used? - Si From: Jordan Michaels Sent: Thursday, October 18, 2018 2:26 PM To: users Subject: Where to look when your SSVM creation NPE's? Hi Folks, I've added a host to the CloudStack Management server but now I'm getting NPE's when the Management server tries to create the System VM's: 2018-10-18 06:20:26,052 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-40:ctx-ac017e0e job-1248/job-1621) (logid:b6f3dd96) Done executing com.cloud.vm.VmWorkStart for job-1621 2018-10-18 06:20:26,054 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-40:ctx-ac017e0e job-1248/job-1621) (logid:b6f3dd96) Remove job-1621 from job monitoring 2018-10-18 06:20:26,073 WARN [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-70aba7ab) (logid:f3c51c51) Exception while trying to start secondary storage vm java.lang.NullPointerException at com.cloud.network.guru.DirectPodBasedNetworkGuru$2.doInTransactionWithoutResult(DirectPodBasedNetworkGuru.java:183) at com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(TransactionCallbackWithExceptionNoReturn.java:25) at com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(TransactionCallbackWithExceptionNoReturn.java:21) at com.cloud.utils.db.Transaction.execute(Transaction.java:40) at com.cloud.network.guru.DirectPodBasedNetworkGuru.getIp(DirectPodBasedNetworkGuru.java:173) at com.cloud.network.guru.DirectPodBasedNetworkGuru.reserve(DirectPodBasedNetworkGuru.java:161) at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepareNic(NetworkOrchestrator.java:1594) at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepare(NetworkOrchestrator.java:1565) at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:) at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:4930) at sun.reflect.GeneratedMethodAccessor155.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107) at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5093) at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581) at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2018-10-18 06:20:26,076 INFO [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-70aba7ab) (logid:f3c51c51) Unable to start secondary storage vm for standby capacity, vm id : 346, will recycle it and start a new one 2018-10-18 06:20:26,076 DEBUG [c.c.a.SecondaryStorageVmAlertAdapter] (secstorage-1:ctx-70aba7ab) (logid:f3c51c51) received secondary storage vm alert 2018-10-18 06:20:26,076 DEBUG [c.c.a.SecondaryStorageVmAlertAdapter] (secstorage-1:ctx-70aba7ab) (logid:f3c51c51) Secondary Storage Vm creation failure, zone: lab 2018-10-18 06:20:26,077 WARN [o.a.c.alerts] (secstorage-1:ctx-70aba7ab) (logid:f3c51c51) AlertType:: 19 | dataCenterId:: 1 | podId:: null | clusterId:: null | message:: Secondary Storage Vm creation failure. zone: lab, error details: null 2018-10-18 06:20:26,095 INFO [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:ctx-70aba7ab) (logid:f3c51c51) Primary secondary storage is not even started, wait until next turn Based on the error, it sounds like there's an issue with the SSVM being able to communicate with secondary storage, but I don't know where to
