CloudStack management server as a VM on KVM hypervisor
Hello, We need to know about below two possibilities for setting up the cloud stack environment. 1. Can we setup the 'Cloudstack Management Server' as a VM hosted on the same set of KVM hypervisor cluster that we need to manage in CloudStack? 2. How do we recover the cloud stack management server in that case if the KVM Server hosting the VM for management server crashes? Regards, Kapil Bhuskute
KVM VM consoles connection timing out due to authentication failure
Hello, We have a POD setup with CloudStack 4.19 latest version and all schema with a Zone, POD, and cluster has been setup using a Advance Shared networking Architecture. We have a couple of VMs spun up on this environment now and were trying to access its VM console via VNC port. However, it has been observed that the VM console connections are getting timed out due to failed authentication errors observed on the vmconsoleproxy System VM logs. VM Console not working . We have mysql ssl enabled and console proxy ssl is disabled for now. Failed to connect/access token expired. While checking logs on CCVM ..we see vnc auth failed. 2024-02-13 16:12:12,926 INFO [vnc.security.VncTLSSecurity] (Thread-86:null) Processing VNC TLS security 2024-02-13 16:12:12,930 INFO [utils.nio.Link] (Thread-86:null) Conf file found: /usr/local/cloud/systemvm/conf/agent.properties 2024-02-13 16:12:12,964 INFO [vnc.security.VncAuthSecurity] (Thread-83:null) Finished VNCAuth security 2024-02-13 16:12:12,966 ERROR [consoleproxy.vnc.NoVncClient] (Thread-83:null) Connection to VNC server failed: wrong password. 2024-02-13 16:12:12,966 ERROR [consoleproxy.vnc.NoVncClient] (Thread-83:null) Connection to VNC server failed: wrong password. - Reason: Authentication failed 2024-02-13 16:12:13,164 INFO [vnc.security.VncAuthSecurity] (Thread-86:null) VNC server requires password authentication 2024-02-13 16:12:13,184 INFO [vnc.security.VncAuthSecurity] (Thread-86:null) Finished VNCAuth security Kindly suggest if anyone is aware of any fix for this. Regards, Kapil B
RE: Secondary Storage VM not getting started
Hello Daan,
What exactly did you mean by initiating the secondary storage?
We have a secondary storage path and server name mentioned in the Cloudstack
management server wizard to add it, and that got added in the UI.
It is a nfs export hosted on the same cloudstack server and is already
available with all required NFS permissions(rw,no_root_squash,sync).
Is there any manual step required to initiate it?
Kapil Bhuskute
-Original Message-
From: Daan Hoogland
Sent: Monday, February 19, 2024 1:55 PM
To: users@cloudstack.apache.org
Subject: Re: Secondary Storage VM not getting started
CAUTION: This Email is from an EXTERNAL source. Ensure you trust this
sender before clicking on any links or attachments.
Kapil,
did you initiate your secondary storage (at
kvm01.p01.ops.sjc01.qualys.com:/mnt/data)?
and did you export the storage for it from the nfs server?
The error says there is a problem there.
On Mon, Feb 19, 2024 at 7:35 AM Kapil Bhuskute
wrote:
>
> The secondary storage VM is not able to get deployed in CloudStack 4.19 with
> below errors.
>
> Feb 16 13:25:08 qcs01 java[417597]: INFO [o.a.c.e.o.VolumeOrchestrator]
> (secstorage-1:ctx-adb3bb9a ctx-2cb81e1f) (logid:67e774fb) Adding disk object
> [ROOT-6258] to VM [VM instance {"id":6258,"instanceName"
> :"s-6258-VM","type":"SecondaryStorageVm","uuid":"a9520deb-ca26-4a6f-bc
> 40-871841d05c28"}] Feb 16 13:25:11 qcs01 java[417597]: ERROR
> [o.a.c.e.o.VolumeOrchestrator] (Work-Job-Executor-2:ctx-511f3201
> job-9178/job-13001 ctx-8a4381d2) (logid:cbcf72ec) Unable to create volume
> [{"name":"ROOT-6258","uui d":"8444daf7-4145-4a40-8789-c623c5a5ad7c"}] due to
> [com.cloud.utils.exception.CloudRuntimeException:
> org.libvirt.LibvirtException: internal error: Child process (/usr/bin/mount
> -o nodev,nosuid,noexec kvm01.
> p01.ops.sjc01.qualys.com:/mnt/data/template/tmpl/1/222
> /mnt/63da2d8b-fe6a-36ab-93c5-49867865f5b7) unexpected exit status 32:
> mount.nfs: mounting
> kvm01.p01.ops.sjc01.qualys.com:/mnt/data/template/tmpl/1/222
> failed, reason given by server: No such file or directory Feb 16
> 13:25:11 qcs01 java[417597]:
> com.cloud.exception.StorageUnavailableException: Resource
> [StoragePool:5] is unreachable: Unable to create volume
> [{"name":"ROOT-6258","uuid":"8444daf7-4145-4a40-8789-c62
> 3c5a5ad7c"}] due to [com.cloud.utils.exception.CloudRuntimeException:
> org.libvirt.LibvirtException: internal error: Child process
> (/usr/bin/mount -o nodev,nosuid,noexec
> kvm01.p01.ops.sjc01.qualys.com:/mnt/d
> ata/template/tmpl/1/222 /mnt/63da2d8b-fe6a-36ab-93c5-49867865f5b7) unexpected
> exit status 32: mount.nfs: mounting
> kvm01.p01.ops.sjc01.qualys.com:/mnt/data/template/tmpl/1/222 failed, reason
> given by server:
> No such file or directory
> Feb 16 13:25:41 qcs01 java[417597]: ERROR
> [o.a.c.e.o.VolumeOrchestrator] (Work-Job-Executor-8:ctx-5417b759
> job-9178/job-13003 ctx-f6230d28) (logid:cbcf72ec) Unable to create volume
> [{"name":"ROOT-6258","uui d":"8444daf7-4145-4a40-8789-c623c5a5ad7c"}] due to
> [com.cloud.utils.exception.CloudRuntimeException:
> org.libvirt.LibvirtException: internal error: Child process (/usr/bin/mount
> -o nodev,nosuid,noexec kvm01.
> p01.ops.sjc01.qualys.com:/mnt/data/template/tmpl/1/222
> /mnt/63da2d8b-fe6a-36ab-93c5-49867865f5b7) unexpected exit status 32:
> mount.nfs: mounting
> kvm01.p01.ops.sjc01.qualys.com:/mnt/data/template/tmpl/1/222
> failed, reason given by server: No such file or directory Feb 16
> 13:25:41 qcs01 java[417597]:
> com.cloud.exception.StorageUnavailableException: Resource
> [StoragePool:5] is unreachable: Unable to create volume
> [{"name":"ROOT-6258","uuid":"8444daf7-4145-4a40-8789-c62
> 3c5a5ad7c"}] due to [com.cloud.utils.exception.CloudRuntimeException:
> org.libvirt.LibvirtException: internal error: Child process
> (/usr/bin/mount -o nodev,nosuid,noexec
> kvm01.p01.ops.sjc01.qualys.com:/mnt/d
> ata/template/tmpl/1/222 /mnt/63da2d8b-fe6a-36ab-93c5-49867865f5b7) unexpected
> exit status 32: mount.nfs: mounting
> kvm01.p01.ops.sjc01.qualys.com:/mnt/data/template/tmpl/1/222 failed, reason
> given by server:
> No such file or directory
>
> Regards,
> Kapil Bhuskute
--
Daan
Secondary Storage VM not getting started
The secondary storage VM is not able to get deployed in CloudStack 4.19 with
below errors.
Feb 16 13:25:08 qcs01 java[417597]: INFO [o.a.c.e.o.VolumeOrchestrator]
(secstorage-1:ctx-adb3bb9a ctx-2cb81e1f) (logid:67e774fb) Adding disk object
[ROOT-6258] to VM [VM instance {"id":6258,"instanceName"
:"s-6258-VM","type":"SecondaryStorageVm","uuid":"a9520deb-ca26-4a6f-bc40-871841d05c28"}]
Feb 16 13:25:11 qcs01 java[417597]: ERROR [o.a.c.e.o.VolumeOrchestrator]
(Work-Job-Executor-2:ctx-511f3201 job-9178/job-13001 ctx-8a4381d2)
(logid:cbcf72ec) Unable to create volume [{"name":"ROOT-6258","uui
d":"8444daf7-4145-4a40-8789-c623c5a5ad7c"}] due to
[com.cloud.utils.exception.CloudRuntimeException: org.libvirt.LibvirtException:
internal error: Child process (/usr/bin/mount -o nodev,nosuid,noexec kvm01.
p01.ops.sjc01.qualys.com:/mnt/data/template/tmpl/1/222
/mnt/63da2d8b-fe6a-36ab-93c5-49867865f5b7) unexpected exit status 32:
mount.nfs: mounting kvm01.p01.ops.sjc01.qualys.com:/mnt/data/template/tmpl/1/222
failed, reason given by server: No such file or directory
Feb 16 13:25:11 qcs01 java[417597]:
com.cloud.exception.StorageUnavailableException: Resource [StoragePool:5] is
unreachable: Unable to create volume
[{"name":"ROOT-6258","uuid":"8444daf7-4145-4a40-8789-c62
3c5a5ad7c"}] due to [com.cloud.utils.exception.CloudRuntimeException:
org.libvirt.LibvirtException: internal error: Child process (/usr/bin/mount -o
nodev,nosuid,noexec kvm01.p01.ops.sjc01.qualys.com:/mnt/d
ata/template/tmpl/1/222 /mnt/63da2d8b-fe6a-36ab-93c5-49867865f5b7) unexpected
exit status 32: mount.nfs: mounting
kvm01.p01.ops.sjc01.qualys.com:/mnt/data/template/tmpl/1/222 failed, reason
given by server:
No such file or directory
Feb 16 13:25:41 qcs01 java[417597]: ERROR [o.a.c.e.o.VolumeOrchestrator]
(Work-Job-Executor-8:ctx-5417b759 job-9178/job-13003 ctx-f6230d28)
(logid:cbcf72ec) Unable to create volume [{"name":"ROOT-6258","uui
d":"8444daf7-4145-4a40-8789-c623c5a5ad7c"}] due to
[com.cloud.utils.exception.CloudRuntimeException: org.libvirt.LibvirtException:
internal error: Child process (/usr/bin/mount -o nodev,nosuid,noexec kvm01.
p01.ops.sjc01.qualys.com:/mnt/data/template/tmpl/1/222
/mnt/63da2d8b-fe6a-36ab-93c5-49867865f5b7) unexpected exit status 32:
mount.nfs: mounting kvm01.p01.ops.sjc01.qualys.com:/mnt/data/template/tmpl/1/222
failed, reason given by server: No such file or directory
Feb 16 13:25:41 qcs01 java[417597]:
com.cloud.exception.StorageUnavailableException: Resource [StoragePool:5] is
unreachable: Unable to create volume
[{"name":"ROOT-6258","uuid":"8444daf7-4145-4a40-8789-c62
3c5a5ad7c"}] due to [com.cloud.utils.exception.CloudRuntimeException:
org.libvirt.LibvirtException: internal error: Child process (/usr/bin/mount -o
nodev,nosuid,noexec kvm01.p01.ops.sjc01.qualys.com:/mnt/d
ata/template/tmpl/1/222 /mnt/63da2d8b-fe6a-36ab-93c5-49867865f5b7) unexpected
exit status 32: mount.nfs: mounting
kvm01.p01.ops.sjc01.qualys.com:/mnt/data/template/tmpl/1/222 failed, reason
given by server:
No such file or directory
Regards,
Kapil Bhuskute
Unable to delete the template from the CloudStack 4.19
We are unable to delete a system template from the CloudStack 4.19 management
server from UI. The cloudmonkey API command is also failing with below errors
to delete the template:
(localcloud) > delete template zoneid=cdf98dec-7061-442c-8466-cea05aa3f9b3
id=9c55e8b4-570b-47c7-b634-94cfa1148943
jobresultcode = 431
created = 2024-02-19T05:53:07+
domainid = 93097518-c82c-11ee-8d79-246e96679ca0
userid = e928c912-c835-11ee-8d79-246e96679ca0
jobresulttype = object
jobinstancetype = Template
completed = 2024-02-19T05:53:07+
accountid = e927c982-c835-11ee-8d79-246e96679ca0
jobinstanceid = 9c55e8b4-570b-47c7-b634-94cfa1148943
account = admin
cmd = org.apache.cloudstack.api.command.user.template.DeleteTemplateCmd
jobstatus = 2
jobprocstatus = 0
jobresult = {
"errorcode": 431,
"errortext": "The DomR template cannot be deleted."
}
jobid = 56b29187-e556-4750-b351-8017b8cee423
domainpath = ROOT
Error: async API failed for job 56b29187-e556-4750-b351-8017b8cee423
(localcloud) >
Kindly suggest.
Kapil Bhuskute
RE: Unable to start cloudstack.management server after initial installation successfully completed.
Yes, I have followed all steps accordingly to the management server installation document says. And the database server is up and running fine. [root@qcs01 ~]# systemctl status mysqld.service ● mysqld.service - MySQL 8.0 database server Loaded: loaded (/usr/lib/systemd/system/mysqld.service; enabled; vendor preset: disabled) Active: active (running) since Sat 2024-02-10 14:36:47 UTC; 2h 32min ago Main PID: 11969 (mysqld) Status: "Server is operational" Tasks: 46 (limit: 3355442) Memory: 859.1M CGroup: /system.slice/mysqld.service └─11969 /usr/libexec/mysqld --basedir=/usr Feb 10 14:36:43 systemd[1]: Starting MySQL 8.0 database server... Feb 10 14:36:43 mysql-prepare-db-dir[11883]: Initializing MySQL database Feb 10 14:36:47 systemd[1]: Started MySQL 8.0 database server. [root@qcs01 ~]# And one more thing I forgot to mention is that we have FIPS enabled on the OEL8 server here. Does that need any additional configurations before the management server is started? Kindly confirm and share if any additional config is required... Kapil Bhuskute -Original Message- From: Pearl d'Silva Sent: Saturday, February 10, 2024 10:29 PM To: users@cloudstack.apache.org Subject: Re: Unable to start cloudstack.management server after initial installation successfully completed. CAUTION: This Email is from an EXTERNAL source. Ensure you trust this sender before clicking on any links or attachments. Hi Kapil, Can you please check if your database server is running - i.e., you've done all the steps mentioned here:https://urldefense.com/v3/__https://docs.cloudstack.apache.org/en/latest/installguide/management-server/index.html*install-the-database-on-the-management-server-node__;Iw!!Pw1rFClp!tqkNYBskb445HaLvcbQ3ACNCj44tYHqz8tRRA9ZHpdOobhELSD7slGjSqcRdp_QRIx8aDAG9eXOAoanofuXWGgxsQi4$ Thanks, Pearl ____ From: Kapil Bhuskute Sent: February 10, 2024 11:36 AM To: users@cloudstack.apache.org Subject: Unable to start cloudstack.management server after initial installation successfully completed. Hello, I have successfully followed all the management server installation steps for the first time. I see the cloudstack.management service is also showing started and active, but with some errors causing the UI to give below 503 error. Can someone please help? ● cloudstack-management.service - CloudStack Management Server Loaded: loaded (/usr/lib/systemd/system/cloudstack-management.service; enabled; vendor preset: disabled) Active: active (running) since Sat 2024-02-10 15:55:09 UTC; 11min ago Main PID: 13630 (java) Tasks: 102 (limit: 3355442) Memory: 870.5M CGroup: /system.slice/cloudstack-management.service └─13630 /usr/bin/java -Djava.security.properties=/etc/cloudstack/management/java.security.ciphers -Djava.awt.headless=true -Dcom.sun.management.jmxremote=fa> Feb 10 15:55:23 java[13630]: INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Loading module context [nested-cloudstack> Feb 10 15:55:23 java[13630]: WARN [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Failed to start module [nested-cloudstack> Feb 10 15:55:23 java[13630]: INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Loading module context [redfish] from URL> Feb 10 15:55:23 java[13630]: INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Loading module context [redfish] from URL> Feb 10 15:55:23 java[13630]: INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Loading module context [redfish] from URL> Feb 10 15:55:23 java[13630]: INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Loading module context [redfish] from URL> Feb 10 15:55:23 java[13630]: INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Loading module context [redfish] from URL> Feb 10 15:55:23 java[13630]: INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Loading module context [redfish] from URL> Feb 10 15:55:23 java[13630]: INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Loading module context [redfish] from URL> Feb 10 15:55:23 java[13630]: WARN [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Failed to start module [redfish] due to: > ~ The management logs are showing below messages: [root@qcs01 ~]# grep -i -E 'error|exception' /var/log/cloudstack/management/management-server.log 2024-02-10 15:55:15,811 INFO [c.c.u.d.T.Transaction] (main:null) (logid:) Using the following URI to connect to cloud database [jdbc:mysql://localhost:3306/cloud?autoReconnect=true=517=true=sql_mode='STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION'=UTC]. 2024-02-10 15:55:19,952 ERROR [c.c.u.d.Merovingian2] (main:null) (logid:) Unable to get a new db connection
Unable to start cloudstack.management server after initial installation successfully completed.
nable to initialize a connection to the database for locking purposes Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.cloud.server.LockControllerListener]: Constructor threw exception; nested exception is com.cloud.utils.exception.CloudRuntimeException: Unable to initialize a connection to the database for locking purposes Caused by: com.cloud.utils.exception.CloudRuntimeException: Unable to initialize a connection to the database for locking purposes Caused by: java.sql.SQLNonTransientConnectionException: Could not create connection to database server. Attempted reconnect 3 times. Giving up. at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:110) at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97) at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:89) Caused by: com.mysql.cj.exceptions.SSLParamsException: Cannot open file:NONE [NONE (No such file or directory)] Caused by: java.io.FileNotFoundException: NONE (No such file or directory) 2024-02-10 15:55:20,736 WARN [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Failed to start module [core] due to: [Error creating bean with name 'org.springframework.beans.factory.config.PropertyPlaceholderConfigurer#0' defined in URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.19.0.0.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml]: Cannot resolve reference to bean 'DefaultConfigResources' while setting bean property 'locations'; nested exception is org.springframework.beans.factory.NoSuchBeanDefinitionException: No bean named 'DefaultConfigResources' available]. org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.springframework.beans.factory.config.PropertyPlaceholderConfigurer#0' defined in URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.19.0.0.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml]: Cannot resolve reference to bean 'DefaultConfigResources' while setting bean property 'locations'; nested exception is org.springframework.beans.factory.NoSuchBeanDefinitionException: No bean named 'DefaultConfigResources' available * Kapil Bhuskute
VM spin fails with "com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM"
Hello Wei,
I have ensured the systemvm template is also in ready state now. And tried to
re-spin another instance, but it is still complaining for the same reason. And
the Instance is going to Error state again.
Logs indicate the same InsufficientServerCapacityException errors:
[HOST]# egrep
"job-2072|job-2073|9d85fa79-da41-4090-9411-54520792f89e|i-2-390-VM"
/var/log/cloudstack/management/management-server.log
2024-01-30 16:32:01,766 INFO [c.c.v.VirtualMachineManagerImpl]
(qtp1789718525-14144:ctx-b7c95aac ctx-d6bcf85f) (logid:b68ac14f) allocating
virtual machine from template:62077549-9c1f-442a-ad97-ce42cd7c63fe with
hostname:i-2-390-VM and 1 networks
2024-01-30 16:32:01,767 DEBUG [c.c.v.VirtualMachineManagerImpl]
(qtp1789718525-14144:ctx-b7c95aac ctx-d6bcf85f) (logid:b68ac14f) Allocating
entries for VM: VM instance
{"id":390,"instanceName":"i-2-390-VM","type":"User","uuid":"98e8bfaa-4bd2-4e23-94ca-b1005a9cf4bc"}
2024-01-30 16:32:01,768 DEBUG [c.c.v.VirtualMachineManagerImpl]
(qtp1789718525-14144:ctx-b7c95aac ctx-d6bcf85f) (logid:b68ac14f) Allocating
nics for VM instance
{"id":390,"instanceName":"i-2-390-VM","type":"User","uuid":"98e8bfaa-4bd2-4e23-94ca-b1005a9cf4bc"}
2024-01-30 16:32:01,769 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(qtp1789718525-14144:ctx-b7c95aac ctx-d6bcf85f) (logid:b68ac14f) Allocating nic
for vm VM instance
{"id":390,"instanceName":"i-2-390-VM","type":"User","uuid":"98e8bfaa-4bd2-4e23-94ca-b1005a9cf4bc"}
in network Network {"id": 247, "name": "10.252.1.x", "uuid":
"f2971bd7-4635-48f2-8b05-75b5d5a4705c", "networkofferingid": 8} with requested
profile NicProfile
{"broadcastUri":null,"iPv4Address":null,"id":0,"reservationId":null,"vmId":0}
2024-01-30 16:32:01,791 DEBUG [c.c.v.VirtualMachineManagerImpl]
(qtp1789718525-14144:ctx-b7c95aac ctx-d6bcf85f) (logid:b68ac14f) Allocating
disks for VM instance
{"id":390,"instanceName":"i-2-390-VM","type":"User","uuid":"98e8bfaa-4bd2-4e23-94ca-b1005a9cf4bc"}
2024-01-30 16:32:01,791 INFO [o.a.c.e.o.VolumeOrchestrator]
(qtp1789718525-14144:ctx-b7c95aac ctx-d6bcf85f ctx-f0fd01cd) (logid:b68ac14f)
Adding disk object [ROOT-390] to VM [VM instance
{"id":390,"instanceName":"i-2-390-VM","type":"User","uuid":"98e8bfaa-4bd2-4e23-94ca-b1005a9cf4bc"}]
2024-01-30 16:32:01,800 DEBUG [c.c.v.VirtualMachineManagerImpl]
(qtp1789718525-14144:ctx-b7c95aac ctx-d6bcf85f) (logid:b68ac14f) Allocation
completed for VM: VM instance
{"id":390,"instanceName":"i-2-390-VM","type":"User","uuid":"98e8bfaa-4bd2-4e23-94ca-b1005a9cf4bc"}
2024-01-30 16:32:01,801 DEBUG [c.c.v.UserVmManagerImpl]
(qtp1789718525-14144:ctx-b7c95aac ctx-d6bcf85f) (logid:b68ac14f) Successfully
allocated DB entry for VM instance
{"id":390,"instanceName":"i-2-390-VM","type":"User","uuid":"98e8bfaa-4bd2-4e23-94ca-b1005a9cf4bc"}
2024-01-30 16:32:01,819 INFO [o.a.c.f.j.i.AsyncJobMonitor]
(API-Job-Executor-9:ctx-a5c8e722 job-2072) (logid:493589da) Add job-2072 into
job monitoring
2024-01-30 16:32:01,824 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(qtp1789718525-14144:ctx-b7c95aac ctx-d6bcf85f) (logid:b68ac14f) submit async
job-2072, details: AsyncJobVO: {id:2072, userId: 2, accountId: 2, instanceType:
VirtualMachine, instanceId: 390, cmd:
org.apache.cloudstack.api.command.admin.vm.DeployVMCmdByAdmin, cmdInfo:
{"iptonetworklist[0].networkid":"f2971bd7-4635-48f2-8b05-75b5d5a4705c","hostid":"5e6f1b73-5f0a-4ab1-b424-42dc16fc9810","httpmethod":"GET","ctxAccountId":"2","uuid":"98e8bfaa-4bd2-4e23-94ca-b1005a9cf4bc","cmdEventType":"VM.CREATE","bootmode":"LEGACY","iothreadsenabled":"false","rootdisksize":"50","ctxStartEventId":"3911","id":"390","ctxDetails":"{\"interface
com.cloud.offering.ServiceOffering\":\"abb92858-9eb9-45bb-8cce-9778113fe54f\",\"interface
com.cloud.host.Host\":\"5e6f1b73-5f0a-4ab1-b424-42dc16fc9810\",\"interface
com.cloud.dc.DataCenter\":\"bef7ac55-3bb8-4211-bbc7-1e708bb36083\",\"interface
com.cloud.org.Cluster\":\"35c1a5be-5609-4a39-bcf8-1940c0fc3e64\",\"interface
com.cloud.dc.Pod\":\"364522fe-d989-4cee-bc1c-7489aa54f113\",\"interface
com.cloud.template.VirtualMachineTemplate\":\"62077549-9c1f-442a-ad97-ce42cd7c63fe\",\"interface
com.cloud.vm.VirtualMachine\":\"98e8bfaa-4bd2-4e23-94ca-b1005a9cf4bc\"}","dynamicscalingenabled":"true","keypairs":"","group":"App","keyboard":"us","boottype":"BIOS","clusterid":"35c1a5be-5609-4a39-bcf8-1940c0fc3e64","templateid":"62077549-9c1f-442a-ad97-ce42cd7c63fe","startvm":"true","serviceofferingid":"abb92858-9eb9-45bb-8cce-9778113fe54f","response":"json","ctxUserId":"2","displayname":"testnode01","name":"testnode01","zoneid":"bef7ac55-3bb8-4211-bbc7-1e708bb36083","podid":"364522fe-d989-4cee-bc1c-7489aa54f113","affinitygroupids":""},
cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result:
null, initMsid: 229310345246465, completeMsid: null, lastUpdated: null,
lastPolled: null, created: null, removed: null}
2024-01-30 16:32:01,826 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-9:ctx-a5c8e722
VM spin fails with "com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM"
We have been using the CloudStack version 4.18.1.0 with KVM hosts with a NFS
primary storage pool and a secondary NFS storage pool.
We have created a shared guest network with "DefaultSharedNetworkOffering"
network offering with a fixed IP range to spin VMs using a standard linux image.
However, the VM spin keeps failing with errors and the instance goes to 'Error'
state and the ROOT disk volume for the instance also gets destroyed. And no VR
gets created.
Please find the detailed errors we see while trying to spin the instance using
the Shared network.
Error
Unable to start a VM [cf941e56-f29d-4442-947a-cd18f5b5d733] due to [Unable to
create a deployment for VM instance
{"id":388,"instanceName":"i-2-388-VM","type":"User","uuid":"cf941e56-f29d-4442-947a-cd18f5b5d733"}].
The /var/log/messages show below logs:
Jan 30 11:54:41 vhs04 java: ERROR [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-154:ctx-612d5ad2 job-2070/job-2071 ctx-2570ea30)
(logid:7b4f2d1b) Invocation exception, caused by:
com.cloud.exception.InsufficientServerCapacityException: Unable to create a
deployment for VM instance
{"id":388,"instanceName":"i-2-388-VM","type":"User","uuid":"cf941e56-f29d-4442-947a-cd18f5b5d733"}Scope=interface
com.cloud.dc.DataCenter; id=1
Jan 30 11:54:41 vhs04 java: INFO [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-154:ctx-612d5ad2 job-2070/job-2071 ctx-2570ea30)
(logid:7b4f2d1b) Rethrow exception
com.cloud.exception.InsufficientServerCapacityException: Unable to create a
deployment for VM instance
{"id":388,"instanceName":"i-2-388-VM","type":"User","uuid":"cf941e56-f29d-4442-947a-cd18f5b5d733"}Scope=interface
com.cloud.dc.DataCenter; id=1
Jan 30 11:54:41 vhs04 java: ERROR [c.c.v.VmWorkJobDispatcher]
(Work-Job-Executor-154:ctx-612d5ad2 job-2070/job-2071) (logid:7b4f2d1b) Unable
to complete AsyncJobVO: {id:2071, userId: 2, accountId: 2, instanceType: null,
instanceId: null, cmd: com.cloud.vm.VmWorkStart, cmdInfo:
rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9pZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAACAAIBhHQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAXBzcgAOamF2YS5sYW5nLkxvbmc7i-SQzI8j3wIAAUoABXZhbHVleHIAEGphdmEubGFuZy5OdW1iZXKGrJUdC5TgiwIAAHhwAAFxAH4ACnBwcHEAfgAKcHNyABFqYXZhLnV0aWwuSGFzaE1hcAUH2sHDFmDRAwACRgAKbG9hZEZhY3RvckkACXRocmVzaG9sZHhwP0x3CBABdAAKVm1QYXNzd29yZHQAHHJPMEFCWFFBRG5OaGRtVmtYM0JoYzNOM2IzSmt4cA,
cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result:
null, initMsid: 229310345246465, completeMsid: null, lastUpdated: null,
lastPolled: null, created: Tue Jan 30 11:54:40 UTC 2024, removed: null}, job
origin:2070
Jan 30 11:54:41 vhs04 java:
com.cloud.exception.InsufficientServerCapacityException: Unable to create a
deployment for VM instance
{"id":388,"instanceName":"i-2-388-VM","type":"User","uuid":"cf941e56-f29d-4442-947a-cd18f5b5d733"}Scope=interface
com.cloud.dc.DataCenter; id=1
Jan 30 11:54:41 vhs04 java: at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1226)
Jan 30 11:54:41 vhs04 java: at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5412)
Jan 30 11:54:41 vhs04 java: at
jdk.internal.reflect.GeneratedMethodAccessor882.invoke(Unknown Source)
Jan 30 11:54:41 vhs04 java: at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Jan 30 11:54:41 vhs04 java: at
java.base/java.lang.reflect.Method.invoke(Method.java:566)
Jan 30 11:54:41 vhs04 java: at
com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
Jan 30 11:54:41 vhs04 java: at
com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5536)
Jan 30 11:54:41 vhs04 java: at
com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
Jan 30 11:54:41 vhs04 java: at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
Jan 30 11:54:41 vhs04 java: at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
Jan 30 11:54:41 vhs04 java: at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
Jan 30 11:54:41 vhs04 java: at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
Jan 30 11:54:41 vhs04 java: at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
Jan
