Hi CloudStack Community,
I am facing an issue with Secondary Storage VM ( s-*-VM ) deployment in Apache
CloudStack 4. 21. 0 on Rocky Linux 9 KVM hypervisors when integrating with Ceph
Ingress NFSv4 proxy. Environment Details:
*
CloudStack Version: 4.21.0.0
*
Management Server: Rocky Linux 9
*
Hypervisors: KVM on Rocky Linux 9 (modular libvirt / monolithic libvirtd with
TCP 16509 )
*
Primary Storage: Ceph RBD ( titan-vm pool, TitanFlash )
*
Secondary Storage: Ceph NFS Ganesha exported via Ceph Ingress VIP (
10.0.25.9:/secondary )
Observed Behavior:
1.
When using Ceph Ingress VIP ( 10.0.25.9:/secondary ):
*
The Secondary Storage VM ( s-2-VM / s-4-VM ) fails during orchestration and
goes into Stopped state.
*
On the KVM agent side, the VM creation hangs or triggers a NullPointerException
/ interface plug failure:
Caught: java.lang. NullPointerException: Cannot invoke
"java.net.URI.getScheme()" because "uri" is null
at com.cloud.network.Networks$ BroadcastDomainType.
getSchemeValue(Networks.java: 180)
at com.cloud.hypervisor.kvm. resource.BridgeVifDriver.plug(
BridgeVifDriver.java:258)
at com.cloud.hypervisor.kvm. resource. LibvirtComputingResource. createVif(
LibvirtComputingResource.java: 3816)
*
Management server reports:
ERROR [c.c.v. ClusteredVirtualMachineManager Impl] Unable to orchestrate start
VM instance due to [Unable to get answer that is of class com.cloud.agent.api.
StartAnswer].
*
On Ceph RBD, leftover images and exclusive locks are retained, causing
subsequent attempts to fail opening the disk device.
2.
When using Direct Node IP ( 10.0.25.20:/secondary ):
*
Mounts without issues, system VMs start up cleanly, and StartAnswer: true is
returned.
Questions:
1.
Is Ceph Ingress VIP (Keepalived + HAProxy on port 2049) supported natively for
Secondary Storage NFS mounts, or does the lack of RPC portmapper / HAProxy TCP
session reset break the KVM agent's NFSv4 state handling?
2.
Are there recommended NFS mount options ( secstorage.nfs.version , proto=tcp ,
sync , etc.) or specific HAProxy timeout settings when using an Ingress VIP for
Secondary Storage in CloudStack 4.21?
3.
What is the recommended production architecture for providing high-availability
to Ceph NFS Secondary Storage without falling into session/lock drops on
failover?
Any insights, recommendations, or architecture best practices from the
community would be greatly appreciated.
Best regards,
Md Titu Hossain
Cloud & Systems Infrastructure Architect