Hi Marty, It looks like the issue is not caused by server capacity, but storage.
2024-07-09 06:57:20,304 DEBUG [o.a.c.s.a.AbstractStoragePoolAllocator] (Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492 ctx-440bf369) (logid:c56ddd5b) ClusterScopeStoragePoolAllocator is returning [0] suitable storage pools [[]]. 2024-07-09 06:57:20,306 DEBUG [o.a.c.s.a.ZoneWideStoragePoolAllocator] (Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492 ctx-440bf369) (logid:c56ddd5b) Could not find any zone wide storage pool that matched with any of the following tags [[]]. 2024-07-09 06:57:20,308 DEBUG [o.a.c.s.a.AbstractStoragePoolAllocator] (Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492 ctx-440bf369) (logid:c56ddd5b) ZoneWideStoragePoolAllocator is returning [0] suitable storage pools [[]]. 2024-07-09 06:57:20,308 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492 ctx-440bf369) (logid:c56ddd5b) No suitable pools found for volume: Vol[1000|name=ROOT-598|vm=598|ROOT] under cluster: 3 2024-07-09 06:57:20,308 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492 ctx-440bf369) (logid:c56ddd5b) No suitable pools found 2024-07-09 06:57:20,308 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492 ctx-440bf369) (logid:c56ddd5b) No suitable storagePools found under this Cluster: 3 On Tue, Jul 9, 2024 at 9:27 AM Marty Godsey <[email protected]> wrote: > > I am also getting this which is strange: Whey does it say there is not enough > capacity. > > 2024-07-09 01:08:22,201 DEBUG [c.c.c.CapacityManagerImpl] > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b) > (logid:7fa21822) VM instance > {"id":466,"instanceName":"v-466-VM","type":"ConsoleProxy","uuid":"94f2f2cc-15c9-41a7-a678-a9d1cd65d14a"} > state transited from [Starting] to [Starting] with event [OperationRetry]. > VM's original host: null, new host: Host > {"id":5,"name":"csc-n1","type":"Routing","uuid":"52950eed-9931-410e-a482-7df361b14a50"}, > host before state transition: null > 2024-07-09 01:08:22,214 DEBUG [c.c.c.CapacityManagerImpl] > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b) > (logid:7fa21822) Hosts's actual total CPU: 193600 and CPU after applying > overprovisioning: 387200 > 2024-07-09 01:08:22,214 DEBUG [c.c.c.CapacityManagerImpl] > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b) > (logid:7fa21822) We are allocating VM, increasing the used capacity of this > host:5 > 2024-07-09 01:08:22,214 DEBUG [c.c.c.CapacityManagerImpl] > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b) > (logid:7fa21822) Current Used CPU: 41400 , Free CPU:345800 ,Requested CPU: 500 > 2024-07-09 01:08:22,214 DEBUG [c.c.c.CapacityManagerImpl] > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b) > (logid:7fa21822) Current Used RAM: (49.00 GB) 52613349376 , Free RAM:(687.04 > GB) 737701920768 ,Requested RAM: (1.00 GB) 1073741824 > 2024-07-09 01:08:22,214 DEBUG [c.c.c.CapacityManagerImpl] > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b) > (logid:7fa21822) CPU STATS after allocation: for host: 5, old used: 41400, > old reserved: 0, actual total: 193600, total with overprovisioning: 387200; > new used:41900, reserved:0; requested cpu:500,alloc_from_last:false > 2024-07-09 01:08:22,214 DEBUG [c.c.c.CapacityManagerImpl] > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b) > (logid:7fa21822) RAM STATS after allocation: for host: 5, old used: (49.00 > GB) 52613349376, old reserved: (0 bytes) 0, total: (736.04 GB) 790315270144; > new used: (50.00 GB) 53687091200, reserved: (0 bytes) 0; requested mem: (1.00 > GB) 1073741824,alloc_from_last:false > 2024-07-09 01:08:22,217 DEBUG [c.c.c.CapacityManagerImpl] > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b) > (logid:7fa21822) Host: 5 has cpu capability (cpu:88, speed:2200) to support > requested CPU: 1 and requested speed: 500 > 2024-07-09 01:08:22,217 DEBUG [c.c.c.CapacityManagerImpl] > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b) > (logid:7fa21822) Checking if host: 5 has enough capacity for requested CPU: > 500 and requested RAM: (1.00 GB) 1073741824 , cpuOverprovisioningFactor: 2.0 > 2024-07-09 01:08:22,218 DEBUG [c.c.c.CapacityManagerImpl] > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b) > (logid:7fa21822) Hosts's actual total CPU: 193600 and CPU after applying > overprovisioning: 387200 > 2024-07-09 01:08:22,218 DEBUG [c.c.c.CapacityManagerImpl] > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b) > (logid:7fa21822) We need to allocate to the last host again, so checking if > there is enough reserved capacity > 2024-07-09 01:08:22,219 DEBUG [c.c.c.CapacityManagerImpl] > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b) > (logid:7fa21822) Reserved CPU: 0 , Requested CPU: 500 > 2024-07-09 01:08:22,219 DEBUG [c.c.c.CapacityManagerImpl] > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b) > (logid:7fa21822) Reserved RAM: (0 bytes) 0 , Requested RAM: (1.00 GB) > 1073741824 > 2024-07-09 01:08:22,219 DEBUG [c.c.c.CapacityManagerImpl] > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b) > (logid:7fa21822) STATS: Failed to alloc resource from host: 5 reservedCpu: 0, > requested cpu: 500, reservedMem: (0 bytes) 0, requested mem: (1.00 GB) > 1073741824 > 2024-07-09 01:08:22,219 DEBUG [c.c.c.CapacityManagerImpl] > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b) > (logid:7fa21822) Host does not have enough reserved CPU available, cannot > allocate to this host. > 2024-07-09 01:08:22,219 DEBUG [c.c.c.CapacityManagerImpl] > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b) > (logid:7fa21822) Checking if host: 5 has enough capacity for requested CPU: > 500 and requested RAM: (1.00 GB) 1073741824 , cpuOverprovisioningFactor: 2.0 > 2024-07-09 01:08:22,220 DEBUG [c.c.c.CapacityManagerImpl] > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b) > (logid:7fa21822) Hosts's actual total CPU: 193600 and CPU after applying > overprovisioning: 387200 > 2024-07-09 01:08:22,220 DEBUG [c.c.c.CapacityManagerImpl] > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b) > (logid:7fa21822) Free CPU: 345800 , Requested CPU: 500 > 2024-07-09 01:08:22,220 DEBUG [c.c.c.CapacityManagerImpl] > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b) > (logid:7fa21822) Free RAM: (687.04 GB) 737701920768 , Requested RAM: (1.00 > GB) 1073741824 > 2024-07-09 01:08:22,220 DEBUG [c.c.c.CapacityManagerImpl] > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b) > (logid:7fa21822) Host has enough CPU and RAM available > 2024-07-09 01:08:22,220 DEBUG [c.c.c.CapacityManagerImpl] > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b) > (logid:7fa21822) STATS: Can alloc CPU from host: 5, used: 41400, reserved: 0, > actual total: 193600, total with overprovisioning: 387200; requested > cpu:500,alloc_from_last_host?:false ,considerReservedCapacity?: true > 2024-07-09 01:08:22,220 DEBUG [c.c.c.CapacityManagerImpl] > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b) > (logid:7fa21822) STATS: Can alloc MEM from host: 5, used: (49.00 GB) > 52613349376, reserved: (0 bytes) 0, total: (736.04 GB) 790315270144; > requested mem: (1.00 GB) 1073741824, alloc_from_last_host?: false , > considerReservedCapacity?: true > 2024-07-09 01:08:22,234 DEBUG [o.a.c.e.o.NetworkOrchestrator] > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b) > (logid:7fa21822) Network id=208 is already implemented > 2024-07-09 01:08:22,281 DEBUG [o.a.c.e.o.NetworkOrchestrator] > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b) > (logid:7fa21822) Network id=210 is already implemented > 2024-07-09 01:08:22,451 DEBUG [o.a.c.h.HAManagerImpl] > (BackgroundTaskPollManager-4:ctx-3dafbb92) (logid:72e1d259) HA health check > task is running... > 2024-07-09 01:08:22,527 DEBUG [c.c.n.g.ControlNetworkGuru] > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b) > (logid:7fa21822) Reserved NIC for v-466-VM [ipv4:169.254.17.78 > netmask:255.255.0.0 gateway:169.254.0.1] > 2024-07-09 01:08:22,544 DEBUG [o.a.c.e.o.NetworkOrchestrator] > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b) > (logid:7fa21822) Network id=209 is already implemented > 2024-07-09 01:08:22,564 DEBUG [c.c.d.d.DataCenterIpAddressDaoImpl] > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b) > (logid:7fa21822) Releasing ip address for instance=1462 > 2024-07-09 01:08:22,574 DEBUG [c.c.n.g.PodBasedNetworkGuru] > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b) > (logid:7fa21822) Allocated a nic NicProfile > {"broadcastUri":null,"iPv4Address":"10.61.0.91","id":1462,"reservationId":"cc4156ed-873a-4485-a9a8-ff30725ea984","vmId":466} > for VM instance > {"id":466,"instanceName":"v-466-VM","type":"ConsoleProxy","uuid":"94f2f2cc-15c9-41a7-a678-a9d1cd65d14a"} > 2024-07-09 01:08:22,608 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl] > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b) > (logid:7fa21822) template 316 is already in store:6, type:Image > > From: Rohit Yadav <[email protected]> > Date: Tuesday, July 9, 2024 at 12:44 AM > To: [email protected] <[email protected]> > Subject: Re: Unable to create new VMs or Console Proxy > WARNING: This email originated from outside of the organization. Do not click > links or open attachments unless you recognize the sender and know the > content is safe. > > > Hi Marty, > > Can you check and ensure all your CloudStack hosts (management server, kvm > hosts if applicable and ssvm, cpvm) are on the same version (4.19.0.2) & > restarted post update/upgrade. > > Once you've done this, can you share your management server logs where you > see the exceptions/errors around VM deployment. You may also want to check if > you've any capacity issues, or using any offering that's tagged (host or > storage tags) for which you don't have available capacity? > > > > Regards. > > > > > ________________________________ > From: Marty Godsey <[email protected]> > Sent: Tuesday, July 9, 2024 07:29 > To: [email protected] <[email protected]> > Subject: Re: Unable to create new VMs or Console Proxy > > Ok, figured, out the service offerings issues, I rebooted the database and it > came back. I still can not create VMs. > > It is stating that the storage is suitable, but it is in avoid set. I have no > idea why it is in avoid set since all hosts and the Cloudstack management > server can get to it. > > Where would I check this? > > From: Marty Godsey <[email protected]> > Date: Monday, July 8, 2024 at 9:19 PM > To: [email protected] <[email protected]> > Subject: Unable to create new VMs or Console Proxy > WARNING: This email originated from outside of the organization. Do not click > links or open attachments unless you recognize the sender and know the > content is safe. > > > Question. All of a sudden, I cannot create new VMs, and the console proxy > also can’t be created. As I spent the past hour looking, I have found that in > the console, when I go to offerings for compute, disk, or system, I get a > 404, and it says it can’t find anything for domain -1. But when I look in the > database, I see the offerings. I took this opportunity to update to 4.19.0.2 > since I was already on 4.19.0.1. At first I thought it was storage but it is > not. Running VMS and secStorage and VRs are all running fine. > > Where should I look from here? Nothing was changed for the root domain.
