kiranchavala opened a new issue, #14263:
URL: https://github.com/apache/cloudstack/issues/14263
### problem
_No response_
### versions
ACS 4.23.x
ACS 4.22.x
ACS 4.20.X
2 kvm host
2 pirmary storages ( nfs and cluster scoped)
### The steps to reproduce the bug
1. Confirm the pool is mounted on the KVM host and tracked in the DB:
# KVM host
df -hT | grep <pool-uuid>
virsh pool-list --all | grep <pool-uuid>
management server DB
mysql> SELECT r.host_id, r.pool_id FROM storage_pool_host_ref r JOIN
storage_pool p ON p.id = r.pool_id WHERE p.uuid = '<pool-uuid>';
2. A running VM or system VM with a disk on the pool.
3. Make the management server disconnect the host from the pool. Use either
option:
**Option A: disable the pool and reconnect the host**
```bash
cmk update configuration name=mount.disabled.storage.pool value=false
clusterid=<cluster-id>
cmk update storagepool id=<pool-id> enabled=false
cmk reconnect host id=<host-id>
```
**Option B: storage access group mismatch**
```bash
cmk configure storageaccess storageid=<pool-id> storageaccessgroups=groupA
cmk configure storageaccess hostid=<host-id> storageaccessgroups=groupB
cmk reconnect host id=<host-id>
```
4. Check the agent log on the KVM host
grep -E "DeleteStoragePoolCommand|umount|device is busy|Succeeded in
unmounting|Failed to unmount" \
/var/log/cloudstack/agent/agent.log | grep -E "<pool-uuid>|busy" | tail
-20
You see `umount.nfs4: ... device is busy` twice, followed by `Succeeded in
unmounting /mnt/<pool-uuid>`.
Logs
```
2026-09-29 05:23:06,828 DEBUG [cloud.agent.Agent]
(AgentRequest-Handler-1:null) (logid:50042c6e) Processing command:
com.cloud.agent.api.DeleteStoragePoolCommand
2026-09-29 05:23:06,828 DEBUG [utils.script.Script]
(AgentRequest-Handler-1:null) (logid:50042c6e) Executing: /bin/bash -c umount
/mnt/0861c431-b02f-3a66-9be5-10587d4f38e5
2026-09-29 05:23:06,844 DEBUG [utils.script.Script]
(AgentRequest-Handler-1:null) (logid:50042c6e) umount.nfs4:
/mnt/0861c431-b02f-3a66-9be5-10587d4f38e5: device is busy
2026-09-29 05:23:06,844 INFO [kvm.storage.LibvirtStorageAdaptor]
(AgentRequest-Handler-1:null) (logid:50042c6e) Attempting to remove storage
pool 0861c431-b02f-3a66-9be5-10587d4f38e5 from libvirt
2026-09-29 05:23:06,846 INFO [kvm.storage.LibvirtStorageAdaptor]
(AgentRequest-Handler-1:null) (logid:50042c6e) Storage pool
0861c431-b02f-3a66-9be5-10587d4f38e5 has no corresponding secret. Not removing
any secret.
2026-09-29 05:23:06,875 ERROR [kvm.storage.LibvirtStorageAdaptor]
(AgentRequest-Handler-1:null) (logid:50042c6e) deleteStoragePool removed pool
from libvirt, but libvirt had trouble unmounting the pool. Trying umount
location /mnt/0861c431-b02f-3a66-9be5-10587d4f38e5again in a few seconds
2026-09-29 05:23:06,875 DEBUG [utils.script.Script]
(AgentRequest-Handler-1:null) (logid:50042c6e) Executing: /bin/bash -c sleep 5
&& umount /mnt/0861c431-b02f-3a66-9be5-10587d4f38e5
2026-09-29 05:23:11,896 DEBUG [utils.script.Script]
(AgentRequest-Handler-1:null) (logid:50042c6e) umount.nfs4:
/mnt/0861c431-b02f-3a66-9be5-10587d4f38e5: device is busy
2026-09-29 05:23:11,897 ERROR [kvm.storage.LibvirtStorageAdaptor]
(AgentRequest-Handler-1:null) (logid:50042c6e) Succeeded in unmounting
/mnt/0861c431-b02f-3a66-9be5-10587d4f38e5
```
5. Check the host state:
```bash
df -hT | grep <pool-uuid> # still mounted
virsh pool-list --all | grep <pool-uuid> # gone from libvirt
6. Check the DB (same query as step 1). The `storage_pool_host_ref` row for
this host and pool has been deleted, and no "Unable to detach storage pool"
alert was raised (`cmk list alerts`).
### What to do about it?
Expected
- The agent checks the `umount` exit code, e.g. with
`Script.runSimpleBashScriptForExitValue(...) == 0`.
- On failure, `DeleteStoragePoolCommand` returns a failed answer, so the
management server keeps the `storage_pool_host_ref` row and raises the existing
"Unable to detach storage pool ... from the host ..." alert.
Actual
- The agent logs "Succeeded in unmounting" and returns success.
- The DB row is removed, the libvirt pool is removed, and the NFS mount
stays on the host.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
