HiManaged to solve the last issue, seems I deleted an image file that caused this behaviour. Will log ticket on GitHub on initial issue.
On 12/20/22 16:13, Wei ZHOU wrote:
The logs shows com.cloud.utils.exception.CloudRuntimeException: Failed to copy /mnt/14205cea-e7ab-39c7-b8d1-0cea6cf65976/10666dea-0eae-3a43-a673-b07c587bf86b.qcow2 to fa27384f-e4c8-4c34-8243-c32c7107b739 Any error log on the host (id=36) ? -Wei On Tuesday, 20 December 2022, Granwille Strauss<granwi...@namhost.com> wrote:Hi Guys I have not logged this on Github yet because I am still experiencing after affects caused by this bug. After we concluded that the image was indeed deleted. I "forced" deleted the template via CM and now tried to deploy a new template from a volume snapshot. This was successfully done thus far. Now when I want to boot a new VM using the new template I just created it keeps failing and I am seeing errors such as: Caused by: com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM instance {id: "109", name: "i-2-109-VM", uuid: "43e31cda-4c8d-4c7d-8518-2a20d3b47d81", type="User"}Scope=interface com.cloud.dc.DataCenter; id=16 2022-12-20 15:06:45,646 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-24:ctx-15a7865f job-2860/job-2861 ctx-a0108f4c) (logid:769d88da) DeploymentPlan has host_id specified, choosing this host and making no checks on this host: 36 2022-12-20 15:06:45,647 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-24:ctx-15a7865f job-2860/job-2861 ctx-a0108f4c) (logid:769d88da) The specified host is in avoid set 2022-12-20 15:06:45,647 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-24:ctx-15a7865f job-2860/job-2861 ctx-a0108f4c) (logid:769d88da) Cannot deploy to specified host, returning. 2022-12-20 15:06:45,655 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-24:ctx-15a7865f job-2860/job-2861 ctx-a0108f4c) (logid:769d88da) VM instance {id: "109", name: "i-2-109-VM", uuid: "43e31cda-4c8d-4c7d-8518-2a20d3b47d81", type="User"} state transited from [Starting] to [Stopped] with event [OperationFailed]. VM's original host: null, new host: null, host before state transition: Host {"id": "36", "name": "hostname", "uuid": "35877892-f831-4dee-b6e2-19ed651f1299", "type"="Routing"} I am trying to deploy a new VM from a template created from a snapshot that exists in a different zone/host. I coped the template to my new zone and now trying to deploy said VM but seeing this failures. Attached is the full log during VM creation. Any ideas what can be causing this and potential options I can try to solve? On 12/20/22 12:58, Granwille Strauss wrote: Hi Wei I will log on Github. In the meantime, I processed "forced" delete for template so that I can get it removed from CM. Is it safe to remove images stored in primary, which are not related to VMs, since you mentioned templates are also stored in primary? On 12/20/22 12:48, Wei ZHOU wrote: This seems to be a bug. Please file a github issue. If the template is still used by active vms, you can find the template on primary storage. -Wei On Tue, 20 Dec 2022 at 07:50, Granwille Strauss<granwi...@namhost.com.invalid> <granwi...@namhost.com.invalid> wrote: Hi I can confirm it is indeed deleted. So in short, the error that said something in the lines of You cannot delete template because there are active VMs associated with it, means nothing. It still deleted it anyway. I believe this needs to be looked at. On 12/20/22 08:45, jordan j wrote: Here is the issue, ACS cannot find the template volume to copy - d2c4feca-da5c-44b3-8ecc-40a01fd6c243 2022-12-20 07:59:52,453 DEBUG [c.c.a.t.Request] (Work-Job-Executor-7:ctx-5298589d job-2783/job-2784 ctx-5325e481) (logid:4c740aa8) Seq 36-1710804908447415280: Sending { Cmd , MgmtId: 66988330791812, via: 36(athena03.namhost.com), Ver: v1, Flags: 100111, [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"d2c4feca-da5c-44b3-8ecc-40a01fd6c243","uuid":"d2c4feca-da5c-44b3-8ecc-40a01fd6c243","id":"212","format":"QCOW2","accountId":"2","checksum":"{SHA-512}fbb1808d2823bfad0ac8c17ddc1038871b4678ed326b91ac4949376dd9823da0153faa031ad00ed3ad9772d78982d8fd2bdf3816b1f2d894d10fa5a72fb5d638","hvm":"true","displayText":"Almalinux 8 cPanel Template","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"3a808086-b3bd-3dda-b27d-18ab9f0059f4","name":"Prime","id":"25","poolType":"NetworkFilesystem","host":"192.168.50.2","path":"/mnt/dataprime","port":"2049","url":"NetworkFilesystem://192.168.50.2/mnt/dataprime/?ROLE=Primary&STOREUUID=3a808086-b3bd-3dda-b27d-18ab9f0059f4","isManaged":"false"}},"name":"5225a781-17b1-4cc6-b112-0e110e2ca4e5","size":"(60,00 GB) 64424509440","hypervisorType":"KVM","bootable":"false","uniqueName":"5225a781-17b1-4cc6-b112-0e110e2ca4e5","directDownload":"false","deployAsIs":"false"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"88695a0f-ac1e-42b8-8eb8-61b258e9e064","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"3a808086-b3bd-3dda-b27d-18ab9f0059f4","name":"Prime","id":"25","poolType":"NetworkFilesystem","host":"192.168.50.2","path":"/mnt/dataprime","port":"2049","url":"NetworkFilesystem://192.168.50.2/mnt/dataprime/?ROLE=Primary&STOREUUID=3a808086-b3bd-3dda-b27d-18ab9f0059f4","isManaged":"false"}},"name":"ROOT-100","size":"(60,00 GB) 64424509440","volumeId":"104","vmName":"i-2-100-VM","accountId":"2","format":"QCOW2","provisioningType":"THIN","poolId":"25","id":"104","deviceId":"0","cacheMode":"NONE","hypervisorType":"KVM","directDownload":"false","deployAsIs":"false"}},"executeInSequence":"true","options":{},"options2":{},"wait":"0","bypassHostMaintenance":"false"}}] } 2022-12-20 08:02:11,514 WARN [o.a.c.f.j.i.AsyncJobMonitor] (Timer-0:ctx-6616ece4) (logid:5c6d1e6b) Task (job-2783) has been pending for 141 seconds 2022-12-20 08:03:11,515 WARN [o.a.c.f.j.i.AsyncJobMonitor] (Timer-0:ctx-60f0ac1b) (logid:5e5388e3) Task (job-2783) has been pending for 201 seconds 2022-12-20 08:04:11,514 WARN [o.a.c.f.j.i.AsyncJobMonitor] (Timer-0:ctx-2e7c936a) (logid:a3027c48) Task (job-2783) has been pending for 261 seconds 2022-12-20 08:04:55,237 DEBUG [c.c.a.t.Request] (Work-Job-Executor-7:ctx-5298589d job-2783/job-2784 ctx-5325e481) (logid:4c740aa8) Seq 36-1710804908447415280: Received: { Ans: , MgmtId: 66988330791812, via: 36(athena03.namhost.com), Ver: v1, Flags: 110, { CopyCmdAnswer } } 2022-12-20 08:04:55,241 DEBUG [o.a.c.e.o.VolumeOrchestrator] (Work-Job-Executor-7:ctx-5298589d job-2783/job-2784 ctx-5325e481) (logid:4c740aa8) Unable to create Vol[104|vm=100|ROOT]:com.cloud.utils.exception.CloudRuntimeException: Can't find volume:d2c4feca-da5c-44b3-8ecc-40a01fd6c243 If other templates can be used from the same SR this means it is accessible but the template file is not there anymore. To look for it you can login on the secondary storage and look at a path similar to this <mount dir>/../templates. inside should be 1 folder for each template, it will include XML meta file and volume file. The volume, if exists should be named by the ID d2c4feca-da5c-44b3-8ecc-40a01fd6c243. if no such is present than indeed it is deleted. Regards, Jordan On Tue, Dec 20, 2022 at 7:23 AM Granwille Strauss<granwi...@namhost.com> <granwi...@namhost.com> wrote: Hi Jordan Please see log attached. Can you confirm what's the best way to confirm if a template exists on secondary storage? On 12/20/22 08:15, jordan j wrote: Hey Granwille Strauss, Can you upload a full log somewhere? This error usually means there is another underlying issue even though it says Network/interface scope. Deployment for VM instance {id: "100", name: "i-2-100-VM", uuid: "e0611ff1-cab5-48d0-ac4e-09fe0f034b19", type="User"}Scope=interface com.cloud.dc.DataCenter; id=16 Can you also verify the template file is intact on the secondary server? Regards, Jordan On Tue, Dec 20, 2022 at 8:09 AM Granwille Strauss<granwi...@namhost.com.invalid> <granwi...@namhost.com.invalid> <granwi...@namhost.com.invalid> <granwi...@namhost.com.invalid> wrote: Hi Guys I accidentality "deleted" a wrong template. Fortunately, the delete was not successful and gave an error stating that it cannot be deleted because there are active VMs associated with it. However, since I did this, I can now no longer deploy new VMs using said template. Creating a VM now fails since I did this, see attached screenshot. And this is the management log entry: 2022-12-20 08:04:55,618 INFO [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-7:ctx-5298589d job-2783/job-2784 ctx-5325e481) (logid:4c740aa8) Rethrow exception com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM instance {id: "100", name: "i-2-100-VM", uuid: "e0611ff1-cab5-48d0-ac4e-09fe0f034b19", type="User"}Scope=interface com.cloud.dc.DataCenter; id=16 2022-12-20 08:04:55,618 DEBUG [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-7:ctx-5298589d job-2783/job-2784) (logid:4c740aa8) Done with run of VM work job: com.cloud.vm.VmWorkStart for VM 100, job origin: 2783 2022-12-20 08:04:55,618 ERROR [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-7:ctx-5298589d job-2783/job-2784) (logid:4c740aa8) Unable to complete AsyncJobVO: {id:2784, userId: 2, accountId: 2, instanceType: null, instanceId: null, cmd: com.cloud.vm.VmWorkStart, cmdInfo: rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9pZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAACAAAAAAAAAAIAAAAAAAAAZHQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAEHBzcgAOamF2YS5sYW5nLkxvbmc7i-SQzI8j3wIAAUoABXZhbHVleHIAEGphdmEubGFuZy5OdW1iZXKGrJUdC5TgiwIAAHhwAAAAAAAAAA1zcQB-AAgAAAAAAAAAJHBwcHNxAH4ACAAAAAAAAAAPcHNyABFqYXZhLnV0aWwuSGFzaE1hcAUH2sHDFmDRAwACRgAKbG9hZEZhY3RvckkACXRocmVzaG9sZHhwP0AAAAAAAAx3CAAAABAAAAABdAAKVm1QYXNzd29yZHQAHHJPMEFCWFFBRG5OaGRtVmtYM0JoYzNOM2IzSmt4cA, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 66988330791812, completeMsid: null, lastUpdated: null, lastPolled: null, created: Tue Dec 20 07:59:49 SAST 2022, removed: null}, job origin:2783 com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM instance {id: "100", name: "i-2-100-VM", uuid: "e0611ff1-cab5-48d0-ac4e-09fe0f034b19", type="User"}Scope=interface com.cloud.dc.DataCenter; id=16 This seems to ONLY happen with this one template that I deleted, I can deploy VMs successfully from other templates but this one that was "deleted", even though it was NOT deleted. So it seems the template broke somehow because now no volume is created for new VMs when using this template. I am using 4.17.1.0. Is this a known issue? -- Regards / Groete<https://www.namhost.com> <https://www.namhost.com> <https://www.namhost.com> <https://www.namhost.com> Granwille Strauss // Senior Systems Admin *e:*granwi...@namhost.com *m:* +264 81 323 1260 <+264813231260> *w:*www.namhost.com<https://www.facebook.com/namhost> <https://www.facebook.com/namhost> <https://www.facebook.com/namhost> <https://www.facebook.com/namhost> <https://twitter.com/namhost> <https://twitter.com/namhost> <https://twitter.com/namhost> <https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> Namhost Internet Services (Pty) Ltd, 24 Black Eagle Rd, Hermanus<https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy [image: Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> -- Regards / Groete <https://www.namhost.com> <https://www.namhost.com> Granwille Strauss // Senior Systems Admin *e:*granwi...@namhost.com *m:* +264 81 323 1260 <+264813231260> *w:*www.namhost.com <https://www.facebook.com/namhost> <https://www.facebook.com/namhost> <https://twitter.com/namhost> <https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> Namhost Internet Services (Pty) Ltd, 24 Black Eagle Rd, Hermanus<https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy [image: Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> -- Regards / Groete <https://www.namhost.com> <https://www.namhost.com> Granwille Strauss // Senior Systems Admin *e:*granwi...@namhost.com *m:* +264 81 323 1260 <+264813231260> *w:*www.namhost.com <https://www.facebook.com/namhost> <https://www.facebook.com/namhost> <https://twitter.com/namhost> <https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> Namhost Internet Services (Pty) Ltd, 24 Black Eagle Rd, Hermanus<https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy [image: Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> -- Regards / Groete <https://www.namhost.com> Granwille Strauss // Senior Systems Admin *e:*granwi...@namhost.com *m:* +264 81 323 1260 <+264813231260> *w:*www.namhost.com <https://www.facebook.com/namhost> <https://twitter.com/namhost> <https://www.instagram.com/namhostinternetservices/> <https://www.linkedin.com/company/namhos> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> Namhost Internet Services (Pty) Ltd, 24 Black Eagle Rd, Hermanus<https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/ privacy-policy [image: Powered by AdSigner] <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> -- Regards / Groete <https://www.namhost.com> Granwille Strauss // Senior Systems Admin *e:*granwi...@namhost.com *m:* +264 81 323 1260 <+264813231260> *w:*www.namhost.com <https://www.facebook.com/namhost> <https://twitter.com/namhost> <https://www.instagram.com/namhostinternetservices/> <https://www.linkedin.com/company/namhos> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> Namhost Internet Services (Pty) Ltd, 24 Black Eagle Rd, Hermanus<https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/ privacy-policy [image: Powered by AdSigner] <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
-- Regards / Groete <https://www.namhost.com> Granwille Strauss // Senior Systems Admin *e:* granwi...@namhost.com *m:* +264 81 323 1260 <tel:+264813231260> *w:* www.namhost.com <https://www.namhost.com/> <https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> Namhost Internet Services (Pty) Ltd, 24 Black Eagle Rd, Hermanus, 7210, RSAThe content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go to https://www.namhost.com/privacy-policy
Powered by AdSigner <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
smime.p7s
Description: S/MIME Cryptographic Signature
