nvazquez commented on issue #6804:
URL: https://github.com/apache/cloudstack/issues/6804#issuecomment-1275027686
Hi guys, just 2 cents on this issue - I could also reproduce it using NFS
storage and qemu-kvm packages instead of qemu-kvm-ev after upgrading from
4.16.1.1:
First attempt:
````
2022-10-11 17:15:43,266 DEBUG [c.c.a.t.Request]
(API-Job-Executor-1:ctx-891d3141 job-53 ctx-e5d41064) (logid:972cc1b3) Seq
3-368169269537538057: Sending { Cmd , MgmtId: 167781181, via: 3(s-2-VM), Ver:
v1, Flags: 100011,
[{"org.apache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/2/5/60e97a3c-d66f-44ee-bdef-c765d0895070","volume":{"uuid":"9f44a962-c4de-4298-ae0d-323a92556a02","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"db33bd91-fc29-3d98-886e-1ca265d5ad71","name":"ref-trl-3728-k-Mr8-nicolas-vazquez-kvm-pri1","id":"1","poolType":"NetworkFilesystem","host":"10.0.32.4","path":"/acs/primary/ref-trl-3728-k-Mr8-nicolas-vazquez/ref-trl-3728-k-Mr8-nicolas-vazquez-kvm-pri1","port":"2049","url":"NetworkFilesystem://10.0.32.4/acs/primary/ref-trl-3728-k-Mr8-nicolas-vazquez/ref-trl-3728-k-Mr8-nicolas-vazquez-kvm-pri1/?ROLE=Primary&STOREUUID=db33bd91-fc29-3d98-886e-1ca
265d5ad71","isManaged":"false"}},"name":"ROOT-5","size":"(8.00 GB)
8589934592","path":"9f44a962-c4de-4298-ae0d-323a92556a02","volumeId":"5","vmName":"i-2-5-VM","accountId":"2","format":"QCOW2","provisioningType":"THIN","poolId":"1","id":"5","deviceId":"0","hypervisorType":"KVM","directDownload":"false","deployAsIs":"false"},"dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"NFS://10.0.32.4/acs/secondary/ref-trl-3728-k-Mr8-nicolas-vazquez/ref-trl-3728-k-Mr8-nicolas-vazquez-sec1","_role":"Image"}},"vmName":"i-2-5-VM","name":"volsnap3","hypervisorType":"KVM","id":"4","quiescevm":"false","physicalSize":"0"}},"wait":"0","bypassHostMaintenance":"false"}}]
}
2022-10-11 17:15:43,363 DEBUG [c.c.a.ApiServlet]
(qtp396883763-19:ctx-7c4c567b) (logid:5a61ae94) ===START=== 10.0.3.251 -- GET
jobId=972cc1b3-cfab-48c9-b7a2-f317c8a117e4&command=queryAsyncJobResult&response=json
2022-10-11 17:15:43,375 DEBUG [c.c.a.ApiServer]
(qtp396883763-19:ctx-7c4c567b ctx-2e032dfb) (logid:5a61ae94) CIDRs from which
account 'Acct[a7b00689-497a-11ed-8541-1e00100003d0-admin] -- Account {"id": 2,
"name": "admin", "uuid": "a7b00689-497a-11ed-8541-1e00100003d0"}' is allowed to
perform API calls: 0.0.0.0/0,::/0
2022-10-11 17:15:43,400 DEBUG [c.c.a.ApiServlet]
(qtp396883763-19:ctx-7c4c567b ctx-2e032dfb) (logid:5a61ae94) ===END===
10.0.3.251 -- GET
jobId=972cc1b3-cfab-48c9-b7a2-f317c8a117e4&command=queryAsyncJobResult&response=json
2022-10-11 17:15:44,117 DEBUG [o.a.c.h.HAManagerImpl]
(BackgroundTaskPollManager-6:ctx-0e27659d) (logid:c2272697) HA health check
task is running...
2022-10-11 17:15:45,079 DEBUG [c.c.a.t.Request]
(AgentManager-Handler-1:null) (logid:) Seq 3-368169269537538057: Processing: {
Ans: , MgmtId: 167781181, via: 3, Ver: v1, Flags: 10,
[{"com.cloud.agent.api.Answer":{"result":"true","wait":"0","bypassHostMaintenance":"false"}}]
}
2022-10-11 17:15:45,080 DEBUG [c.c.a.t.Request]
(API-Job-Executor-1:ctx-891d3141 job-53 ctx-e5d41064) (logid:972cc1b3) Seq
3-368169269537538057: Received: { Ans: , MgmtId: 167781181, via: 3(s-2-VM),
Ver: v1, Flags: 10, { Answer } }
2022-10-11 17:15:45,097 DEBUG [o.a.c.s.s.DefaultSnapshotStrategy]
(API-Job-Executor-1:ctx-891d3141 job-53 ctx-e5d41064) (logid:972cc1b3) Snapshot
{"uuid":"efe1e43e-1819-43ff-9fa6-268c571ce5ce","name":"volsnap3","volumeId":5}
was deleted on secondary storage {uuid: "f618d895-4b01-4b26-bf08-f2e142074341",
name:
"NFS://10.0.32.4/acs/secondary/ref-trl-3728-k-Mr8-nicolas-vazquez/ref-trl-3728-k-Mr8-nicolas-vazquez-sec1"}.
2022-10-11 17:15:45,137 DEBUG [o.a.c.s.s.DefaultSnapshotStrategy]
(API-Job-Executor-1:ctx-891d3141 job-53 ctx-e5d41064) (logid:972cc1b3) Deleting
SnapshotTO[datastore=PrimaryDataStoreTO[uuid=db33bd91-fc29-3d98-886e-1ca265d5ad71|name=ref-trl-3728-k-Mr8-nicolas-vazquez-kvm-pri1|id=1|pooltype=NetworkFilesystem]|volume=volumeTO[uuid=9f44a962-c4de-4298-ae0d-323a92556a02|path=9f44a962-c4de-4298-ae0d-323a92556a02|datastore=PrimaryDataStoreTO[uuid=db33bd91-fc29-3d98-886e-1ca265d5ad71|name=ref-trl-3728-k-Mr8-nicolas-vazquez-kvm-pri1|id=1|pooltype=NetworkFilesystem]]|path/mnt/db33bd91-fc29-3d98-886e-1ca265d5ad71/9f44a962-c4de-4298-ae0d-323a92556a02/60e97a3c-d66f-44ee-bdef-c765d0895070]
chain of snapshots.
2022-10-11 17:15:45,138 DEBUG [o.a.c.s.s.DefaultSnapshotStrategy]
(API-Job-Executor-1:ctx-891d3141 job-53 ctx-e5d41064) (logid:972cc1b3) Snapshot
[SnapshotTO[datastore=PrimaryDataStoreTO[uuid=db33bd91-fc29-3d98-886e-1ca265d5ad71|name=ref-trl-3728-k-Mr8-nicolas-vazquez-kvm-pri1|id=1|pooltype=NetworkFilesystem]|volume=volumeTO[uuid=9f44a962-c4de-4298-ae0d-323a92556a02|path=9f44a962-c4de-4298-ae0d-323a92556a02|datastore=PrimaryDataStoreTO[uuid=db33bd91-fc29-3d98-886e-1ca265d5ad71|name=ref-trl-3728-k-Mr8-nicolas-vazquez-kvm-pri1|id=1|pooltype=NetworkFilesystem]]|path/mnt/db33bd91-fc29-3d98-886e-1ca265d5ad71/9f44a962-c4de-4298-ae0d-323a92556a02/60e97a3c-d66f-44ee-bdef-c765d0895070]]
does not have children; therefore, we will delete it and its parents.
2022-10-11 17:15:45,186 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
(API-Job-Executor-1:ctx-891d3141 job-53 ctx-e5d41064) (logid:972cc1b3)
getCommandHostDelegation: class
org.apache.cloudstack.storage.command.DeleteCommand
2022-10-11 17:15:45,186 DEBUG [c.c.h.XenServerGuru]
(API-Job-Executor-1:ctx-891d3141 job-53 ctx-e5d41064) (logid:972cc1b3) We are
returning the default host to execute commands because the command is not of
Copy type.
2022-10-11 17:15:45,191 DEBUG [c.c.a.t.Request]
(API-Job-Executor-1:ctx-891d3141 job-53 ctx-e5d41064) (logid:972cc1b3) Seq
1-7690177838711832591: Sending { Cmd , MgmtId: 167781181, via:
1(ref-trl-3728-k-Mr8-nicolas-vazquez-kvm1), Ver: v1, Flags: 100011,
[{"org.apache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"/mnt/db33bd91-fc29-3d98-886e-1ca265d5ad71/9f44a962-c4de-4298-ae0d-323a92556a02/60e97a3c-d66f-44ee-bdef-c765d0895070","volume":{"uuid":"9f44a962-c4de-4298-ae0d-323a92556a02","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"db33bd91-fc29-3d98-886e-1ca265d5ad71","name":"ref-trl-3728-k-Mr8-nicolas-vazquez-kvm-pri1","id":"1","poolType":"NetworkFilesystem","host":"10.0.32.4","path":"/acs/primary/ref-trl-3728-k-Mr8-nicolas-vazquez/ref-trl-3728-k-Mr8-nicolas-vazquez-kvm-pri1","port":"2049","url":"NetworkFilesystem://10.0.32.4/acs/primary/ref-trl-3728-k-Mr8-nicolas-vazq
uez/ref-trl-3728-k-Mr8-nicolas-vazquez-kvm-pri1/?ROLE=Primary&STOREUUID=db33bd91-fc29-3d98-886e-1ca265d5ad71","isManaged":"false"}},"name":"ROOT-5","size":"(8.00
GB)
8589934592","path":"9f44a962-c4de-4298-ae0d-323a92556a02","volumeId":"5","vmName":"i-2-5-VM","accountId":"2","format":"QCOW2","provisioningType":"THIN","poolId":"1","id":"5","deviceId":"0","hypervisorType":"KVM","directDownload":"false","deployAsIs":"false"},"dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"db33bd91-fc29-3d98-886e-1ca265d5ad71","name":"ref-trl-3728-k-Mr8-nicolas-vazquez-kvm-pri1","id":"1","poolType":"NetworkFilesystem","host":"10.0.32.4","path":"/acs/primary/ref-trl-3728-k-Mr8-nicolas-vazquez/ref-trl-3728-k-Mr8-nicolas-vazquez-kvm-pri1","port":"2049","url":"NetworkFilesystem://10.0.32.4/acs/primary/ref-trl-3728-k-Mr8-nicolas-vazquez/ref-trl-3728-k-Mr8-nicolas-vazquez-kvm-pri1/?ROLE=Primary&STOREUUID=db33bd91-fc29-3d98-886e-1ca265d5ad71","isManaged":"false"}},"vmName":"i-2-5-VM"
,"name":"volsnap3","hypervisorType":"KVM","id":"4","quiescevm":"false","physicalSize":"0"}},"wait":"0","bypassHostMaintenance":"false"}}]
}
2022-10-11 17:15:45,218 DEBUG [c.c.a.t.Request]
(AgentManager-Handler-2:null) (logid:) Seq 1-7690177838711832591: Processing:
{ Ans: , MgmtId: 167781181, via: 1, Ver: v1, Flags: 10,
[{"com.cloud.agent.api.Answer":{"result":"false","details":"Failed to remove
snapshot
60e97a3c-d66f-44ee-bdef-c765d0895070","wait":"0","bypassHostMaintenance":"false"}}]
}
2022-10-11 17:15:45,220 DEBUG [c.c.a.t.Request]
(API-Job-Executor-1:ctx-891d3141 job-53 ctx-e5d41064) (logid:972cc1b3) Seq
1-7690177838711832591: Received: { Ans: , MgmtId: 167781181, via:
1(ref-trl-3728-k-Mr8-nicolas-vazquez-kvm1), Ver: v1, Flags: 10, { Answer } }
2022-10-11 17:15:45,221 DEBUG [o.a.c.s.s.SnapshotServiceImpl]
(API-Job-Executor-1:ctx-891d3141 job-53 ctx-e5d41064) (logid:972cc1b3) delete
snapshot failedFailed to remove snapshot 60e97a3c-d66f-44ee-bdef-c765d0895070
2022-10-11 17:15:45,233 ERROR [o.a.c.s.s.DefaultSnapshotStrategy]
(API-Job-Executor-1:ctx-891d3141 job-53 ctx-e5d41064) (logid:972cc1b3) Failed
to delete snapshot
[SnapshotTO[datastore=PrimaryDataStoreTO[uuid=db33bd91-fc29-3d98-886e-1ca265d5ad71|name=ref-trl-3728-k-Mr8-nicolas-vazquez-kvm-pri1|id=1|pooltype=NetworkFilesystem]|volume=volumeTO[uuid=9f44a962-c4de-4298-ae0d-323a92556a02|path=9f44a962-c4de-4298-ae0d-323a92556a02|datastore=PrimaryDataStoreTO[uuid=db33bd91-fc29-3d98-886e-1ca265d5ad71|name=ref-trl-3728-k-Mr8-nicolas-vazquez-kvm-pri1|id=1|pooltype=NetworkFilesystem]]|path/mnt/db33bd91-fc29-3d98-886e-1ca265d5ad71/9f44a962-c4de-4298-ae0d-323a92556a02/60e97a3c-d66f-44ee-bdef-c765d0895070]]
on storage [primary storage {uuid: "db33bd91-fc29-3d98-886e-1ca265d5ad71",
name: "ref-trl-3728-k-Mr8-nicolas-vazquez-kvm-pri1"}] due to [Failed to remove
snapshot 60e97a3c-d66f-44ee-bdef-c765d0895070].
com.cloud.utils.exception.CloudRuntimeException: Failed to remove snapshot
60e97a3c-d66f-44ee-bdef-c765d0895070
at
org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.deleteSnapshot(SnapshotServiceImpl.java:419)
at
org.apache.cloudstack.storage.snapshot.DefaultSnapshotStrategy.deleteSnapshotChain(DefaultSnapshotStrategy.java:218)
at
org.apache.cloudstack.storage.snapshot.DefaultSnapshotStrategy.deleteSnapshotInfo(DefaultSnapshotStrategy.java:331)
at
org.apache.cloudstack.storage.snapshot.DefaultSnapshotStrategy.deleteSnapshotInfos(DefaultSnapshotStrategy.java:306)
at
org.apache.cloudstack.storage.snapshot.DefaultSnapshotStrategy.destroySnapshotEntriesAndFiles(DefaultSnapshotStrategy.java:285)
at
org.apache.cloudstack.storage.snapshot.DefaultSnapshotStrategy.deleteSnapshot(DefaultSnapshotStrategy.java:277)
at
com.cloud.storage.snapshot.SnapshotManagerImpl.deleteSnapshot(SnapshotManagerImpl.java:607)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
````
Second attempt:
````
2022-10-11 17:15:56,162 DEBUG [c.c.a.ApiServlet]
(qtp396883763-18:ctx-83fe3d1e ctx-5f0c22aa) (logid:9a0603b7) ===END===
10.0.3.251 -- GET
id=efe1e43e-1819-43ff-9fa6-268c571ce5ce&command=deleteSnapshot&response=json
2022-10-11 17:15:56,164 INFO [o.a.c.f.j.i.AsyncJobMonitor]
(API-Job-Executor-2:ctx-88542e84 job-56) (logid:a4c66950) Add job-56 into job
monitoring
2022-10-11 17:15:56,169 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-2:ctx-88542e84 job-56) (logid:954427aa) Executing AsyncJobVO:
{id:56, userId: 2, accountId: 2, instanceType: Snapshot, instanceId: 4, cmd:
org.apache.cloudstack.api.command.user.snapshot.DeleteSnapshotCmd, cmdInfo:
{"response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId":"127","id":"efe1e43e-1819-43ff-9fa6-268c571ce5ce","ctxDetails":"{\"interface
com.cloud.storage.Snapshot\":\"efe1e43e-1819-43ff-9fa6-268c571ce5ce\"}","ctxAccountId":"2","uuid":"efe1e43e-1819-43ff-9fa6-268c571ce5ce","cmdEventType":"SNAPSHOT.DELETE"},
cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result:
null, initMsid: 167781181, completeMsid: null, lastUpdated: null, lastPolled:
null, created: null, removed: null}
2022-10-11 17:15:56,194 DEBUG [o.a.c.s.s.StorPoolSnapshotStrategy]
(API-Job-Executor-2:ctx-88542e84 job-56 ctx-b34fc88d) (logid:954427aa)
StorpoolSnapshotStrategy.canHandle: snapshot=volsnap3,
uuid=efe1e43e-1819-43ff-9fa6-268c571ce5ce, op=DELETE
2022-10-11 17:15:56,202 DEBUG [o.a.c.s.s.DefaultSnapshotStrategy]
(API-Job-Executor-2:ctx-88542e84 job-56 ctx-b34fc88d) (logid:954427aa) Could
not find Snapshot
{"uuid":"efe1e43e-1819-43ff-9fa6-268c571ce5ce","name":"volsnap3","volumeId":5}
entry on a secondary storage. Skipping deletion on secondary storage.
2022-10-11 17:15:56,202 DEBUG [o.a.c.s.s.DefaultSnapshotStrategy]
(API-Job-Executor-2:ctx-88542e84 job-56 ctx-b34fc88d) (logid:954427aa) Could
not find Snapshot
{"uuid":"efe1e43e-1819-43ff-9fa6-268c571ce5ce","name":"volsnap3","volumeId":5}
entry on a primary storage. Skipping deletion on primary storage.
2022-10-11 17:15:56,218 DEBUG [c.c.r.ResourceLimitManagerImpl]
(API-Job-Executor-2:ctx-88542e84 job-56 ctx-b34fc88d) (logid:954427aa) Updating
resource Type = snapshot count for Account = 2 Operation = decreasing Amount = 1
2022-10-11 17:15:56,239 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-2:ctx-88542e84 job-56 ctx-b34fc88d) (logid:954427aa) Complete
async job-56, jobStatus: SUCCEEDED, resultCode: 0, result:
org.apache.cloudstack.api.response.SuccessResponse/null/{"success":"true"}
2022-10-11 17:15:56,240 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-2:ctx-88542e84 job-56 ctx-b34fc88d) (logid:954427aa) Publish
async job-56 complete on message bus
2022-10-11 17:15:56,240 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-2:ctx-88542e84 job-56 ctx-b34fc88d) (logid:954427aa) Wake up
jobs related to job-56
2022-10-11 17:15:56,240 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-2:ctx-88542e84 job-56 ctx-b34fc88d) (logid:954427aa) Update
db status for job-56
2022-10-11 17:15:56,241 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-2:ctx-88542e84 job-56 ctx-b34fc88d) (logid:954427aa) Wake up
jobs joined with job-56 and disjoin all subjobs created from job- 56
2022-10-11 17:15:56,245 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-2:ctx-88542e84 job-56) (logid:954427aa) Done executing
org.apache.cloudstack.api.command.user.snapshot.DeleteSnapshotCmd for job-56
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
