The logs shows com.cloud.utils.exception.CloudRuntimeException: Failed to copy /mnt/14205cea-e7ab-39c7-b8d1-0cea6cf65976/10666dea-0eae-3a43-a673-b07c587bf86b.qcow2 to fa27384f-e4c8-4c34-8243-c32c7107b739
Any error log on the host (id=36) ? -Wei On Tuesday, 20 December 2022, Granwille Strauss <granwi...@namhost.com> wrote: > Hi Guys > > I have not logged this on Github yet because I am still experiencing after > affects caused by this bug. After we concluded that the image was indeed > deleted. I "forced" deleted the template via CM and now tried to deploy a > new template from a volume snapshot. This was successfully done thus far. > Now when I want to boot a new VM using the new template I just created it > keeps failing and I am seeing errors such as: > > > Caused by: com.cloud.exception.InsufficientServerCapacityException: > Unable to create a deployment for VM instance {id: "109", name: > "i-2-109-VM", uuid: "43e31cda-4c8d-4c7d-8518-2a20d3b47d81", > type="User"}Scope=interface com.cloud.dc.DataCenter; id=16 > > > 2022-12-20 15:06:45,646 DEBUG [c.c.d.DeploymentPlanningManagerImpl] > (Work-Job-Executor-24:ctx-15a7865f job-2860/job-2861 ctx-a0108f4c) > (logid:769d88da) DeploymentPlan has host_id specified, choosing this host > and making no checks on this host: 36 > 2022-12-20 15:06:45,647 DEBUG [c.c.d.DeploymentPlanningManagerImpl] > (Work-Job-Executor-24:ctx-15a7865f job-2860/job-2861 ctx-a0108f4c) > (logid:769d88da) The specified host is in avoid set > 2022-12-20 15:06:45,647 DEBUG [c.c.d.DeploymentPlanningManagerImpl] > (Work-Job-Executor-24:ctx-15a7865f job-2860/job-2861 ctx-a0108f4c) > (logid:769d88da) Cannot deploy to specified host, returning. > 2022-12-20 15:06:45,655 DEBUG [c.c.c.CapacityManagerImpl] > (Work-Job-Executor-24:ctx-15a7865f job-2860/job-2861 ctx-a0108f4c) > (logid:769d88da) VM instance {id: "109", name: "i-2-109-VM", uuid: > "43e31cda-4c8d-4c7d-8518-2a20d3b47d81", type="User"} state transited from > [Starting] to [Stopped] with event [OperationFailed]. VM's original host: > null, new host: null, host before state transition: Host {"id": "36", > "name": "hostname", "uuid": "35877892-f831-4dee-b6e2-19ed651f1299", > "type"="Routing"} > > > I am trying to deploy a new VM from a template created from a snapshot > that exists in a different zone/host. I coped the template to my new zone > and now trying to deploy said VM but seeing this failures. Attached is the > full log during VM creation. > Any ideas what can be causing this and potential options I can try to > solve? > > On 12/20/22 12:58, Granwille Strauss wrote: > > Hi Wei > > I will log on Github. In the meantime, I processed "forced" delete for > template so that I can get it removed from CM. Is it safe to remove images > stored in primary, which are not related to VMs, since you mentioned > templates are also stored in primary? > On 12/20/22 12:48, Wei ZHOU wrote: > > This seems to be a bug. Please file a github issue. > > If the template is still used by active vms, you can find the template on > primary storage. > > -Wei > > On Tue, 20 Dec 2022 at 07:50, Granwille > Strauss<granwi...@namhost.com.invalid> <granwi...@namhost.com.invalid> wrote: > > > Hi > > I can confirm it is indeed deleted. So in short, the error that said > something in the lines of You cannot delete template because there are > active VMs associated with it, means nothing. It still deleted it anyway. I > believe this needs to be looked at. > On 12/20/22 08:45, jordan j wrote: > > Here is the issue, ACS cannot find the template volume to copy - > d2c4feca-da5c-44b3-8ecc-40a01fd6c243 > > 2022-12-20 07:59:52,453 DEBUG [c.c.a.t.Request] > (Work-Job-Executor-7:ctx-5298589d job-2783/job-2784 ctx-5325e481) > (logid:4c740aa8) Seq 36-1710804908447415280: Sending { Cmd , MgmtId: > 66988330791812, via: 36(athena03.namhost.com), Ver: v1, Flags: 100111, > [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"d2c4feca-da5c-44b3-8ecc-40a01fd6c243","uuid":"d2c4feca-da5c-44b3-8ecc-40a01fd6c243","id":"212","format":"QCOW2","accountId":"2","checksum":"{SHA-512}fbb1808d2823bfad0ac8c17ddc1038871b4678ed326b91ac4949376dd9823da0153faa031ad00ed3ad9772d78982d8fd2bdf3816b1f2d894d10fa5a72fb5d638","hvm":"true","displayText":"Almalinux > 8 cPanel > Template","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"3a808086-b3bd-3dda-b27d-18ab9f0059f4","name":"Prime","id":"25","poolType":"NetworkFilesystem","host":"192.168.50.2","path":"/mnt/dataprime","port":"2049","url":"NetworkFilesystem://192.168.50.2/mnt/dataprime/?ROLE=Primary&STOREUUID=3a808086-b3bd-3dda-b27d-18ab9f0059f4","isManaged":"false"}},"name":"5225a781-17b1-4cc6-b112-0e110e2ca4e5","size":"(60,00 > GB) > 64424509440","hypervisorType":"KVM","bootable":"false","uniqueName":"5225a781-17b1-4cc6-b112-0e110e2ca4e5","directDownload":"false","deployAsIs":"false"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"88695a0f-ac1e-42b8-8eb8-61b258e9e064","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"3a808086-b3bd-3dda-b27d-18ab9f0059f4","name":"Prime","id":"25","poolType":"NetworkFilesystem","host":"192.168.50.2","path":"/mnt/dataprime","port":"2049","url":"NetworkFilesystem://192.168.50.2/mnt/dataprime/?ROLE=Primary&STOREUUID=3a808086-b3bd-3dda-b27d-18ab9f0059f4","isManaged":"false"}},"name":"ROOT-100","size":"(60,00 > GB) > 64424509440","volumeId":"104","vmName":"i-2-100-VM","accountId":"2","format":"QCOW2","provisioningType":"THIN","poolId":"25","id":"104","deviceId":"0","cacheMode":"NONE","hypervisorType":"KVM","directDownload":"false","deployAsIs":"false"}},"executeInSequence":"true","options":{},"options2":{},"wait":"0","bypassHostMaintenance":"false"}}] > } > 2022-12-20 08:02:11,514 WARN [o.a.c.f.j.i.AsyncJobMonitor] > (Timer-0:ctx-6616ece4) (logid:5c6d1e6b) Task (job-2783) has been pending for > 141 seconds > 2022-12-20 08:03:11,515 WARN [o.a.c.f.j.i.AsyncJobMonitor] > (Timer-0:ctx-60f0ac1b) (logid:5e5388e3) Task (job-2783) has been pending for > 201 seconds > 2022-12-20 08:04:11,514 WARN [o.a.c.f.j.i.AsyncJobMonitor] > (Timer-0:ctx-2e7c936a) (logid:a3027c48) Task (job-2783) has been pending for > 261 seconds > 2022-12-20 08:04:55,237 DEBUG [c.c.a.t.Request] > (Work-Job-Executor-7:ctx-5298589d job-2783/job-2784 ctx-5325e481) > (logid:4c740aa8) Seq 36-1710804908447415280: Received: { Ans: , MgmtId: > 66988330791812, via: 36(athena03.namhost.com), Ver: v1, Flags: 110, { > CopyCmdAnswer } } > 2022-12-20 08:04:55,241 DEBUG [o.a.c.e.o.VolumeOrchestrator] > (Work-Job-Executor-7:ctx-5298589d job-2783/job-2784 ctx-5325e481) > (logid:4c740aa8) Unable to create > Vol[104|vm=100|ROOT]:com.cloud.utils.exception.CloudRuntimeException: Can't > find volume:d2c4feca-da5c-44b3-8ecc-40a01fd6c243 > > > If other templates can be used from the same SR this means it is > accessible but the template file is not there anymore. > To look for it you can login on the secondary storage and look at a path > similar to this <mount dir>/../templates. > inside should be 1 folder for each template, it will include XML meta file > and volume file. The volume, if exists should be named by the ID > d2c4feca-da5c-44b3-8ecc-40a01fd6c243. > > if no such is present than indeed it is deleted. > > Regards, > Jordan > > On Tue, Dec 20, 2022 at 7:23 AM Granwille Strauss <granwi...@namhost.com> > <granwi...@namhost.com> > wrote: > > > Hi Jordan > > Please see log attached. Can you confirm what's the best way to confirm > if a template exists on secondary storage? > On 12/20/22 08:15, jordan j wrote: > > Hey Granwille Strauss, > > Can you upload a full log somewhere? > This error usually means there is another underlying issue even though it > says Network/interface scope. > Deployment for VM instance {id: "100", name: "i-2-100-VM", uuid: > "e0611ff1-cab5-48d0-ac4e-09fe0f034b19", type="User"}Scope=interface > com.cloud.dc.DataCenter; id=16 > > Can you also verify the template file is intact on the secondary server? > > Regards, > Jordan > > On Tue, Dec 20, 2022 at 8:09 AM Granwille > Strauss<granwi...@namhost.com.invalid> <granwi...@namhost.com.invalid> > <granwi...@namhost.com.invalid> <granwi...@namhost.com.invalid> wrote: > > > Hi Guys > > I accidentality "deleted" a wrong template. Fortunately, the delete was > not successful and gave an error stating that it cannot be deleted because > there are active VMs associated with it. However, since I did this, I can > now no longer deploy new VMs using said template. Creating a VM now fails > since I did this, see attached screenshot. > > And this is the management log entry: > > 2022-12-20 08:04:55,618 INFO [c.c.v.VmWorkJobHandlerProxy] > (Work-Job-Executor-7:ctx-5298589d job-2783/job-2784 ctx-5325e481) > (logid:4c740aa8) Rethrow exception > com.cloud.exception.InsufficientServerCapacityException: Unable to create a > deployment for VM instance {id: "100", name: "i-2-100-VM", uuid: > "e0611ff1-cab5-48d0-ac4e-09fe0f034b19", type="User"}Scope=interface > com.cloud.dc.DataCenter; id=16 > 2022-12-20 08:04:55,618 DEBUG [c.c.v.VmWorkJobDispatcher] > (Work-Job-Executor-7:ctx-5298589d job-2783/job-2784) (logid:4c740aa8) Done > with run of VM work job: com.cloud.vm.VmWorkStart for VM 100, job origin: > 2783 > 2022-12-20 08:04:55,618 ERROR [c.c.v.VmWorkJobDispatcher] > (Work-Job-Executor-7:ctx-5298589d job-2783/job-2784) (logid:4c740aa8) > Unable to complete AsyncJobVO: {id:2784, userId: 2, accountId: 2, > instanceType: null, instanceId: null, cmd: com.cloud.vm.VmWorkStart, > cmdInfo: > rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9pZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAACAAAAAAAAAAIAAAAAAAAAZHQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAEHBzcgAOamF2YS5sYW5nLkxvbmc7i-SQzI8j3wIAAUoABXZhbHVleHIAEGphdmEubGFuZy5OdW1iZXKGrJUdC5TgiwIAAHhwAAAAAAAAAA1zcQB-AAgAAAAAAAAAJHBwcHNxAH4ACAAAAAAAAAAPcHNyABFqYXZhLnV0aWwuSGFzaE1hcAUH2sHDFmDRAwACRgAKbG9hZEZhY3RvckkACXRocmVzaG9sZHhwP0AAAAAAAAx3CAAAABAAAAABdAAKVm1QYXNzd29yZHQAHHJPMEFCWFFBRG5OaGRtVmtYM0JoYzNOM2IzSmt4cA, > cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, > result: null, initMsid: 66988330791812, completeMsid: null, lastUpdated: > null, lastPolled: null, created: Tue Dec 20 07:59:49 SAST 2022, removed: > null}, job origin:2783 > com.cloud.exception.InsufficientServerCapacityException: Unable to create > a deployment for VM instance {id: "100", name: "i-2-100-VM", uuid: > "e0611ff1-cab5-48d0-ac4e-09fe0f034b19", type="User"}Scope=interface > com.cloud.dc.DataCenter; id=16 > > This seems to ONLY happen with this one template that I deleted, I can > deploy VMs successfully from other templates but this one that was > "deleted", even though it was NOT deleted. So it seems the template broke > somehow because now no volume is created for new VMs when using this > template. I am using 4.17.1.0. Is this a known issue? > -- > Regards / Groete<https://www.namhost.com> <https://www.namhost.com> > <https://www.namhost.com> <https://www.namhost.com> Granwille Strauss // > Senior Systems Admin > > *e:* granwi...@namhost.com > *m:* +264 81 323 1260 <+264813231260> > *w:* www.namhost.com<https://www.facebook.com/namhost> > <https://www.facebook.com/namhost> <https://www.facebook.com/namhost> > <https://www.facebook.com/namhost> <https://twitter.com/namhost> > <https://twitter.com/namhost> <https://twitter.com/namhost> > <https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/> > <https://www.instagram.com/namhostinternetservices/> > <https://www.instagram.com/namhostinternetservices/> > <https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos> > <https://www.linkedin.com/company/namhos> > <https://www.linkedin.com/company/namhos> > <https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> > <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> > <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> > <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> > <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> > > <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> > > <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> > > <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> > > Namhost Internet Services (Pty) Ltd, > 24 Black Eagle Rd, Hermanus > <https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, > 7210, RSA > > > > The content of this message is confidential. If you have received it by > mistake, please inform us by email reply and then delete the message. It is > forbidden to copy, forward, or in any way reveal the contents of this > message to anyone without our explicit consent. The integrity and security > of this email cannot be guaranteed over the Internet. Therefore, the sender > will not be held liable for any damage caused by the message. For our full > privacy policy and disclaimers, please go > tohttps://www.namhost.com/privacy-policy > > [image: Powered by > AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> > > <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> > > <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> > > <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> > > -- > Regards / Groete > <https://www.namhost.com> <https://www.namhost.com> Granwille Strauss // > Senior Systems Admin > > *e:* granwi...@namhost.com > *m:* +264 81 323 1260 <+264813231260> > *w:* www.namhost.com > <https://www.facebook.com/namhost> <https://www.facebook.com/namhost> > <https://twitter.com/namhost> > <https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/> > > <https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos> > > <https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> > <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> > > <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> > > <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> > > Namhost Internet Services (Pty) Ltd, > 24 Black Eagle Rd, Hermanus > <https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, > 7210, RSA > > > > The content of this message is confidential. If you have received it by > mistake, please inform us by email reply and then delete the message. It is > forbidden to copy, forward, or in any way reveal the contents of this > message to anyone without our explicit consent. The integrity and security > of this email cannot be guaranteed over the Internet. Therefore, the sender > will not be held liable for any damage caused by the message. For our full > privacy policy and disclaimers, please go > tohttps://www.namhost.com/privacy-policy > > [image: Powered by > AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> > > <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> > > -- > Regards / Groete > <https://www.namhost.com> <https://www.namhost.com> Granwille Strauss // > Senior Systems Admin > > *e:* granwi...@namhost.com > *m:* +264 81 323 1260 <+264813231260> > *w:* www.namhost.com > <https://www.facebook.com/namhost> <https://www.facebook.com/namhost> > <https://twitter.com/namhost> > <https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/> > > <https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos> > > <https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> > <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> > > <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> > > <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> > > Namhost Internet Services (Pty) Ltd, > 24 Black Eagle Rd, Hermanus > <https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, > 7210, RSA > > > > The content of this message is confidential. If you have received it by > mistake, please inform us by email reply and then delete the message. It is > forbidden to copy, forward, or in any way reveal the contents of this > message to anyone without our explicit consent. The integrity and security > of this email cannot be guaranteed over the Internet. Therefore, the sender > will not be held liable for any damage caused by the message. For our full > privacy policy and disclaimers, please go > tohttps://www.namhost.com/privacy-policy > > [image: Powered by > AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> > > <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> > > -- > Regards / Groete > > <https://www.namhost.com> Granwille Strauss // Senior Systems Admin > > *e:* granwi...@namhost.com > *m:* +264 81 323 1260 <+264813231260> > *w:* www.namhost.com > > <https://www.facebook.com/namhost> <https://twitter.com/namhost> > <https://www.instagram.com/namhostinternetservices/> > <https://www.linkedin.com/company/namhos> > <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> > > > <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> > > Namhost Internet Services (Pty) Ltd, > > 24 Black Eagle Rd, Hermanus > <https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, > 7210, RSA > > > > The content of this message is confidential. If you have received it by > mistake, please inform us by email reply and then delete the message. It is > forbidden to copy, forward, or in any way reveal the contents of this > message to anyone without our explicit consent. The integrity and security > of this email cannot be guaranteed over the Internet. Therefore, the sender > will not be held liable for any damage caused by the message. For our full > privacy policy and disclaimers, please go to https://www.namhost.com/ > privacy-policy > > [image: Powered by AdSigner] > <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> > > -- > Regards / Groete > > <https://www.namhost.com> Granwille Strauss // Senior Systems Admin > > *e:* granwi...@namhost.com > *m:* +264 81 323 1260 <+264813231260> > *w:* www.namhost.com > > <https://www.facebook.com/namhost> <https://twitter.com/namhost> > <https://www.instagram.com/namhostinternetservices/> > <https://www.linkedin.com/company/namhos> > <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> > > > <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> > > Namhost Internet Services (Pty) Ltd, > > 24 Black Eagle Rd, Hermanus > <https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, > 7210, RSA > > > > The content of this message is confidential. If you have received it by > mistake, please inform us by email reply and then delete the message. It is > forbidden to copy, forward, or in any way reveal the contents of this > message to anyone without our explicit consent. The integrity and security > of this email cannot be guaranteed over the Internet. Therefore, the sender > will not be held liable for any damage caused by the message. For our full > privacy policy and disclaimers, please go to https://www.namhost.com/ > privacy-policy > > [image: Powered by AdSigner] > <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> >