I added 3 Monitors: 20XX:YYYY:ZZZZ:LLLL::OO:24, 20XX:YYYY:ZZZZ:LLLL::OO26, 20XX:YYYY:ZZZZ:LLLL::OO:28
and you can see on the logs the 20XX is truncated in some cases and in other cases the last HEX digits. 2025-04-01 10:52:24,628 WARN [utils.script.Script] (agentRequest-Handler-4:[]) (logid:d64e452c) Process [11662] for command [qemu-img convert -O raw -U --image-opts driver=qcow2,file.filename=/mnt/a1aa3257-b554-3896-a168-a593ebde9994/5ccb81a1-26ec-4d57-a02a-37f81e09be08.qcow2 rbd:3cephUserandPool/f32a0f81-5661-41ac-832f-f5dfffa8b1e0:mon_host=20XX:YYYY:ZZZZ:LLLL::OO:24\;20XX:YYYY:ZZZZ:LLLL::OO:26\;20XX:YYYY:ZZZZ:LLLL::OOauth_supported=cephx:id=3cephUserandPool:key=AQD8V3xnRzFJGxAA35C9hbvpDOBbNaaETZXMdw==:rbd_default_format=2:client_mount_timeout=30 ] encountered the error: [qemu-img: rbd:3cephUserandPool/f32a0f81-5661-41ac-832f-f5dfffa8b1e0:mon_host=20XX:YYYY:ZZZZ:LLLL::OO:24\;20XX:YYYY:ZZZZ:LLLL::OO:26\;20XX:YYYY:ZZZZ:LLLL::OOauth_supported=cephx:id=3cephUserandPool:key=AQD8V3xnRzFJGxAA35C9hbvpDOBbNaaETZXMdw==:rbd_default_format=2:client_mount_timeout=30: error while converting raw: invalid conf option YYY:ZZZZ:LLLL::OO:24;20XX:YYYY:ZZZZ:LLLL::OO:26;20XX:YYYY:ZZZZ:LLLL::OOauth_supported: No such file or directory]. 2025-04-01 10:52:24,628 ERROR [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:[]) (logid:d64e452c) Failed to convert from /mnt/a1aa3257-b554-3896-a168-a593ebde9994/5ccb81a1-26ec-4d57-a02a-37f81e09be08.qcow2 to rbd:3cephUserandPool/f32a0f81-5661-41ac-832f-f5dfffa8b1e0:mon_host=20XX:YYYY:ZZZZ:LLLL::OO:24\;20XX:YYYY:ZZZZ:LLLL::OO:26\;20XX:YYYY:ZZZZ:LLLL::OOauth_supported=cephx:id=3cephUserandPool:key=AQD8V3xnRzFJGxAA35C9hbvpDOBbNaaETZXMdw==:rbd_default_format=2:client_mount_timeout=30 the error was: qemu-img: rbd:3cephUserandPool/f32a0f81-5661-41ac-832f-f5dfffa8b1e0:mon_host=20XX:YYYY:ZZZZ:LLLL::OO:24\;20XX:YYYY:ZZZZ:LLLL::OO:26\;20XX:YYYY:ZZZZ:LLLL::OOauth_supported=cephx:id=3cephUserandPool:key=AQD8V3xnRzFJGxAA35C9hbvpDOBbNaaETZXMdw==:rbd_default_format=2:client_mount_timeout=30: error while converting raw: invalid conf option YYY:ZZZZ:LLLL::OO:24;20XX:YYYY:ZZZZ:LLLL::OO:26;20XX:YYYY:ZZZZ:LLLL::OOauth_supported: No such file or directory I just checked the ceph common 19.2.1-1 I did not read anything about needing to run virsh to define the secret for the hypervisor to connect to the CEPH cluster. Is my understanding that the Cloudstack will take of configuring the necessary files on the KVM to connect to the RDB CEPH, including the secret should be take care by the Cloudstack https://docs.ceph.com/en/latest/rbd/rbd-cloudstack/ If I have to follow the Openstack steps please let me know. Please tell is something we can fix Tata Y. > On Apr 1, 2025, at 6:13 AM, Wido den Hollander <w...@denhollander.io> wrote: > > > > Op 01-04-2025 om 12:04 schreef Chi vediamo: >> Hello team, >> I am having the following problem, The Libvirt is unable to convert from >> Secondary Storage to CEPH RBD. >> CEPH is purely IPV6, The RBD was created, There are no error on the CEPH >> side. > > error while converting raw: invalid conf option > 550:5607:fff0::22:24;20XX:YYYY:ZZZZ:LLLL::OO:26;20XX:YYYY:ZZZZ:LLLL::OO:auth_supported: > No such file or directory]. > > - What OS? Do you have the Ceph client packages installed on the hypervisor? > - What did you provide as Monitor address to CloudStack? > > Wido > >> The Initial VM Creation starts and then stops, they never come enabled, >> the initial VMs are deleted and the process start in a loop. >> Libvirt is up and running but throwing the following errors: >> root@kvmvcompatl2001:~# systemctl status libvertd >> Unit libvertd.service could not be found. >> root@kvmvcompatl2001:~# systemctl status libvirtd >> ● libvirtd.service - Virtualization daemon >> Loaded: loaded (/lib/systemd/system/libvirtd.service; enabled; vendor >> preset: enabled) >> Active: active (running) since Tue 2025-04-01 01:19:22 UTC; 8h ago >> Docs: man:libvirtd(8) >> https://libvirt.org >> Main PID: 2271 (libvirtd) >> Tasks: 19 (limit: 32768) >> Memory: 98.5M >> CPU: 9.064s >> CGroup: /system.slice/libvirtd.service >> └─2271 /usr/sbin/libvirtd --listen >> Apr 01 02:32:57 kvmvcompatl2001 libvirtd[2271]: invalid argument: >> Connections from inside daemon must be direct >> Apr 01 02:32:57 kvmvcompatl2001 libvirtd[2271]: End of file while reading >> data: Input/output error >> Apr 01 02:33:02 kvmvcompatl2001 libvirtd[2271]: invalid argument: >> Connections from inside daemon must be direct >> Apr 01 02:33:02 kvmvcompatl2001 libvirtd[2271]: End of file while reading >> data: Input/output error >> Apr 01 02:33:27 kvmvcompatl2001 libvirtd[2271]: invalid argument: >> Connections from inside daemon must be direct >> Apr 01 02:33:27 kvmvcompatl2001 libvirtd[2271]: End of file while reading >> data: Input/output error >> Apr 01 02:33:32 kvmvcompatl2001 libvirtd[2271]: invalid argument: >> Connections from inside daemon must be direct >> the agent.log is showing the following in a loop: >> 2025-04-01 02:33:54,480 DEBUG [kvm.storage.LibvirtStorageAdaptor] >> (agentRequest-Handler-4:[]) (logid:d64e452c) Starting copy from source image >> /mnt/a1aa3257-b554-3896-a168-a593ebde9994/5ccb81a1-26ec-4d57-a02a-37f81e09be08.qcow2 >> to RBD image 3cephUserandPool/f32a0f81-5661-41ac-832f-f5dfffa8b1e0 >> 2025-04-01 02:33:54,480 DEBUG [utils.script.Script] >> (agentRequest-Handler-4:[]) (logid:d64e452c) Executing command [qemu-img >> convert -O raw -U --image-opts >> driver=qcow2,file.filename=/mnt/a1aa3257-b554-3896-a168-a593ebde9994/5ccb81a1-26ec-4d57-a02a-37f81e09be08.qcow2 >> >> rbd:3cephUserandPool/f32a0f81-5661-41ac-832f-f5dfffa8b1e0:mon_host=20XX:YYYY:ZZZZ:LLLL::OO:24;20XX:YYYY:ZZZZ:LLLL::OO:26;20XX:YYYY:ZZZZ:LLLL::OO:auth_supported=cephx:id=3cephUserandPool:key=KEYGENERATEDBYCEPH:rbd_default_format=2:client_mount_timeout=30 >> ]. >> 2025-04-01 02:33:54,532 WARN [utils.script.Script] >> (agentRequest-Handler-4:[]) (logid:d64e452c) Execution of process [7500] for >> command [qemu-img convert -O raw -U --image-opts >> driver=qcow2,file.filename=/mnt/a1aa3257-b554-3896-a168-a593ebde9994/5ccb81a1-26ec-4d57-a02a-37f81e09be08.qcow2 >> >> rbd:3cephUserandPool/f32a0f81-5661-41ac-832f-f5dfffa8b1e0:mon_host=20XX:YYYY:ZZZZ:LLLL::OO:24;20XX:YYYY:ZZZZ:LLLL::OO:26;20XX:YYYY:ZZZZ:LLLL::OO:auth_supported=cephx:id=3cephUserandPool:key=KEYGENERATEDBYCEPH:rbd_default_format=2:client_mount_timeout=30 >> ] failed. >> 2025-04-01 02:33:54,532 DEBUG [utils.script.Script] >> (agentRequest-Handler-4:[]) (logid:d64e452c) Exit value of process [7500] >> for command [qemu-img convert -O raw -U --image-opts >> driver=qcow2,file.filename=/mnt/a1aa3257-b554-3896-a168-a593ebde9994/5ccb81a1-26ec-4d57-a02a-37f81e09be08.qcow2 >> >> rbd:3cephUserandPool/f32a0f81-5661-41ac-832f-f5dfffa8b1e0:mon_host=20XX:YYYY:ZZZZ:LLLL::OO:24;20XX:YYYY:ZZZZ:LLLL::OO:26;20XX:YYYY:ZZZZ:LLLL::OO:auth_supported=cephx:id=3cephUserandPool:key=KEYGENERATEDBYCEPH:rbd_default_format=2:client_mount_timeout=30 >> ] is [1]. >> 2025-04-01 02:33:54,532 WARN [utils.script.Script] >> (agentRequest-Handler-4:[]) (logid:d64e452c) Process [7500] for command >> [qemu-img convert -O raw -U --image-opts >> driver=qcow2,file.filename=/mnt/a1aa3257-b554-3896-a168-a593ebde9994/5ccb81a1-26ec-4d57-a02a-37f81e09be08.qcow2 >> >> rbd:3cephUserandPool/f32a0f81-5661-41ac-832f-f5dfffa8b1e0:mon_host=20XX:YYYY:ZZZZ:LLLL::OO:24;20XX:YYYY:ZZZZ:LLLL::OO:26;20XX:YYYY:ZZZZ:LLLL::OO:auth_supported=cephx:id=3cephUserandPool:key=KEYGENERATEDBYCEPH:rbd_default_format=2:client_mount_timeout=30 >> ] encountered the error: [qemu-img: >> rbd:3cephUserandPool/f32a0f81-5661-41ac-832f-f5dfffa8b1e0:mon_host=20XX:YYYY:ZZZZ:LLLL::OO:24;20XX:YYYY:ZZZZ:LLLL::OO:26;20XX:YYYY:ZZZZ:LLLL::OO:auth_supported=cephx:id=3cephUserandPool:key=KEYGENERATEDBYCEPH:rbd_default_format=2:client_mount_timeout=30: >> error while converting raw: invalid conf option >> 550:5607:fff0::22:24;20XX:YYYY:ZZZZ:LLLL::OO:26;20XX:YYYY:ZZZZ:LLLL::OO:auth_supported: >> No such file or directory]. >> 2025-04-01 02:33:54,532 ERROR [kvm.storage.LibvirtStorageAdaptor] >> (agentRequest-Handler-4:[]) (logid:d64e452c) Failed to convert from >> /mnt/a1aa3257-b554-3896-a168-a593ebde9994/5ccb81a1-26ec-4d57-a02a-37f81e09be08.qcow2 >> to >> rbd:3cephUserandPool/f32a0f81-5661-41ac-832f-f5dfffa8b1e0:mon_host=20XX:YYYY:ZZZZ:LLLL::OO:24;20XX:YYYY:ZZZZ:LLLL::OO:26;20XX:YYYY:ZZZZ:LLLL::OO:auth_supported=cephx:id=3cephUserandPool:key=KEYGENERATEDBYCEPH:rbd_default_format=2:client_mount_timeout=30 >> the error was: qemu-img: >> rbd:3cephUserandPool/f32a0f81-5661-41ac-832f-f5dfffa8b1e0:mon_host=20XX:YYYY:ZZZZ:LLLL::OO:24;20XX:YYYY:ZZZZ:LLLL::OO:26;20XX:YYYY:ZZZZ:LLLL::OO:auth_supported=cephx:id=3cephUserandPool:key=KEYGENERATEDBYCEPH:rbd_default_format=2:client_mount_timeout=30: >> error while converting raw: invalid conf option >> 550:5607:fff0::22:24;20XX:YYYY:ZZZZ:LLLL::OO:26;20XX:YYYY:ZZZZ:LLLL::OO:auth_supported: >> No such file or directory >> 2025-04-01 02:33:54,532 INFO [kvm.storage.LibvirtStorageAdaptor] >> (agentRequest-Handler-4:[]) (logid:d64e452c) Attempting to remove storage >> pool a1aa3257-b554-3896-a168-a593ebde9994 from libvirt >> 2025-04-01 02:33:54,532 DEBUG [kvm.resource.LibvirtConnection] >> (agentRequest-Handler-4:[]) (logid:d64e452c) Looking for libvirtd connection >> at: qemu:///system >> 2025-04-01 02:33:54,534 INFO [kvm.storage.LibvirtStorageAdaptor] >> (agentRequest-Handler-4:[]) (logid:d64e452c) Storage pool >> a1aa3257-b554-3896-a168-a593ebde9994 has no corresponding secret. Not >> removing any secret. >> 2025-04-01 02:33:54,577 INFO [kvm.storage.LibvirtStorageAdaptor] >> (agentRequest-Handler-4:[]) (logid:d64e452c) Storage pool >> a1aa3257-b554-3896-a168-a593ebde9994 was successfully removed from libvirt. >> 2025-04-01 02:33:54,579 DEBUG [cloud.agent.Agent] >> (agentRequest-Handler-4:[]) (logid:d64e452c) Seq 3-8729383452727574659: { >> Ans: , MgmtId: 250977680725600, via: 3, Ver: v1, Flags: 110, >> [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":"false","details":"com.cloud.utils.exception.CloudRuntimeException: >> Failed to copy >> /mnt/a1aa3257-b554-3896-a168-a593ebde9994/5ccb81a1-26ec-4d57-a02a-37f81e09be08.qcow2 >> to >> f32a0f81-5661-41ac-832f-f5dfffa8b1e0","wait":"0","bypassHostMaintenance":"false"}}] >> } >> 2025-04-01 02:33:54,621 DEBUG [cloud.agent.Agent] >> (agentRequest-Handler-3:[]) (logid:d64e452c) Request:Seq >> 3-8729383452727574660: { Cmd , MgmtId: 250977680725600, via: 3, Ver: v1, >> Flags: 100011, >> [{"com.cloud.agent.api.StopCommand":{"isProxy":"false","checkBeforeCleanup":"false","controlIp":"169.254.138.135","forceStop":"false","volumesToDisconnect":[],"vmName":"v-172-VM","executeInSequence":"false","wait":"0","bypassHostMaintenance":"false"}}] >> } >> 2025-04-01 02:33:54,621 DEBUG [cloud.agent.Agent] >> (agentRequest-Handler-3:[]) (logid:d64e452c) Processing command: >> com.cloud.agent.api.StopCommand >> 2025-04-01 02:33:54,621 DEBUG [resource.wrapper.LibvirtStopCommandWrapper] >> (agentRequest-Handler-3:[]) (logid:d64e452c) backing up the cmdline >> 2025-04-01 02:33:57,691 DEBUG [resource.wrapper.LibvirtStopCommandWrapper] >> (agentRequest-Handler-3:[]) (logid:d64e452c) Failed to backup cmdline file >> due to There was a problem while connecting to 169.254.138.135:3922 >> Error logs showing: >> libvirt: Remote Driver error : invalid argument: Connections from inside >> daemon must be direct >> libvirt: QEMU Driver error : Domain not found: no domain with matching name >> 'v-172-VM' >> libvirt: QEMU Driver error : Domain not found: no domain with matching name >> 'v-172-VM' >> libvirt: QEMU Driver error : Domain not found: no domain with matching name >> 'v-172-VM' >> libvirt: QEMU Driver error : Domain not found: no domain with matching name >> 'v-172-VM' >> libvirt: QEMU Driver error : Domain not found: no domain with matching name >> 'v-172-VM' >> libvirt: Storage Driver error : Storage pool not found: no storage pool with >> matching uuid 'a1aa3257-b554-3896-a168-a593ebde9994' >> libvirt: Secrets Driver error : Secret not found: no secret with matching >> uuid 'a1aa3257-b554-3896-a168-a593ebde9994' >> Please kindly guide us. >> Thank you >> Tata Y. >
