Hi I can confirm that I am also having this issue on 4.11.1.0. To be honest, this issue was always present for me as far as I remember.
Kind of an urgent issue to fix to stop people running out of space. Cheers ----- Original Message ----- > From: "Rafael Weingärtner" <rafaelweingart...@gmail.com> > To: "users" <users@cloudstack.apache.org> > Sent: Wednesday, 10 October, 2018 16:24:54 > Subject: Re: VM Snapshot not removed from primary SR > Well, I am almost sure I have seen a PR fixing something like you describe, > I thought that it went into 4.11.1.0.However, only with a more in depth > debugging I would be able to confirm your problem. > > Are you seeing any unexpected exception in your log files? > > On Wed, Oct 10, 2018 at 11:54 AM Sami Rajala (FAPPS) <sami.raj...@fapps.com> > wrote: > >> Hi, >> >> CS running on 4.11.1.0 and does not work. >> >> BR >> -sami >> >> >> 10.10.2018 16.49, "Rafael Weingärtner" <rafaelweingart...@gmail.com>: >> >> >If I am not mistaken, this has already been fixed in 4.11.1.0 >> > >> >On Wed, Oct 10, 2018 at 10:31 AM Sami Rajala <sami.raj...@fapps.com> >> >wrote: >> > >> >> Hi, >> >> >> >> >> >> I have got zero advice for this. >> >> >> >> "I have had weird issue since I updated CS from 4.9 to 4.10. >> >> CS does not delete/remove temporary snapshot from primary SR and I have >> >> had to removed it by manual time to time, before 30 snapshot limit has >> >> reached. >> >> + doesn't CS not update secondary_storege count on snapshot_count table” >> >> >> >> >> >> >> >> >> >> Now I have a situation where Xenserver has 19 snapshot on primary SR for >> >> each VM + resource count for ”secondary_storege” in database increase >> >>all >> >> the time per account. >> >> ACS storage cleaner removed old snapshots from secondary when they >> >>getting >> >> old, but thats all, not decrease resource count for ”secondary_storege” >> >> with value of deleted snapshot ( or not remove snapshot from primary SR >> >> after snapshot has created and copied to secondary ) >> >> >> >> I have still delete primary snapshot by manual + give a new value for >> >> ”secondary_storege” before they reach limit. >> >> >> >> has anyone got this kind of issue? any fix >> >> >> >> BR >> >> -sami >> >> >> >> >> >> >> >> 14.9.2018 12.26, "Sami Rajala (FAPPS)" <sami.raj...@fapps.com>: >> >> >> >> >Continue, >> >> > >> >> >I created template from VM-s latest snapshot after 3 days ( one parent >> >>+ >> >> >2 >> >> >child snapshot) - without errors/warning >> >> >CS create VM from this template ok, but Xen could not start VM >> >> > >> >> >so, I return back to situation where every snapshot is full and manual >> >> >removing snapshot from primary >> >> > >> >> >Has anyone any advice? >> >> > >> >> >BR >> >> >-sami >> >> > >> >> > >> >> > >> >> >13.9.2018 7.56, "Sami Rajala (FAPPS)" <sami.raj...@fapps.com>: >> >> > >> >> >>Hello >> >> >> >> >> >>I have had weird issue since I updated CS from 4.9 to 4.10. >> >> >>CS does not delete/remove temporary snapshot from primary SR and I >> >>have >> >> >>had to removed it by manual time to time, before 30 snapshot limit has >> >> >>reached. >> >> >>+ doesn¹t CS not update secondary_storege count on snapshot_count >> >>table >> >> >> >> >> >>It works fine on 4.9, but stop to work on 4.10 and I have wait >> >> >>possibility >> >> >>to update 4.11.1. >> >> >>Update to 4.11.1 has done now and looks like this not work still >> >> >> >> >> >>I removed all snapshot and start over from clean table, snapshot >> >>policy >> >> >>is: DAILY, keep 2, delta = 5 >> >> >>There are now 3 snapshot on primary SR and 3 on secondary SR >> >> >> >> >> >>The Environment is Xen 7 + CS 4.11.1 + NFS storage >> >> >> >> >> >>Is there some other parameter I should look and any other work around >> >>I >> >> >>has to do to get it work ? >> >> >> >> >> >>Any hits? >> >> >> >> >> >>BR >> >> >>-sami >> >> >> >> >> >>VM snapshot Log for last round: >> >> >> >> >> >>2018-09-13 06:02:15,983 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] >> >> >>(API-Job-Executor-83:ctx-cfb1cd95 job-28020) (logid:4c7a1c38) >> >>Executing >> >> >>AsyncJobVO {id:28020, userId: 1, accountId: 7, instanceType: Snapshot, >> >> >>instanceId: 2656, cmd: >> >> >>org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd, >> >> >>cmdInfo: >> >> >> >>>>{"policyid":"17","ctxUserId":"1","volumeid":"238","ctxStartEventId":"1" >> >>>>," >> >> >>i >> >> >>d >> >> >>":"2656","ctxAccountId":"7"}, cmdVersion: 0, status: IN_PROGRESS, >> >> >>processStatus: 0, resultCode: 0, result: null, initMsid: >> >>19873467853209, >> >> >>completeMsid: null, lastUpdated: null, lastPolled: null, created: >> >>null} >> >> >>2018-09-13 06:02:15,991 DEBUG [c.c.u.AccountManagerImpl] >> >> >>(API-Job-Executor-83:ctx-cfb1cd95 job-28020 ctx-5b8156a6) >> >> >>(logid:4c7a1c38) >> >> >>Access to Acct[479c643e-9c84-41fb-9f0a-9bb999893a25-juha] granted to >> >> >>Acct[479c643e-9c84-41fb-9f0a-9bb999893a25-juha] by DomainChecker >> >> >>2018-09-13 06:02:16,045 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] >> >> >>(API-Job-Executor-83:ctx-cfb1cd95 job-28020 ctx-5b8156a6) >> >> >>(logid:4c7a1c38) >> >> >>Sync job-28021 execution on object VmWorkJobQueue.228 >> >> >>2018-09-13 06:02:17,115 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] >> >> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021) >> >>(logid:4c7a1c38) >> >> >>Executing AsyncJobVO {id:28021, userId: 1, accountId: 7, instanceType: >> >> >>null, instanceId: null, cmd: com.cloud.vm.VmWorkTakeVolumeSnapshot, >> >> >>cmdInfo: >> >> >> >>>>rO0ABXNyACVjb20uY2xvdWQudm0uVm1Xb3JrVGFrZVZvbHVtZVNuYXBzaG90BL5gG4Li1c8 >> >>>>CA >> >> >>A >> >> >>Z >> >> >> >>>>aAAthc3luY0JhY2t1cFoACXF1aWVzY2VWbUwADGxvY2F0aW9uVHlwZXQAKUxjb20vY2xvdW >> >>>>Qv >> >> >>c >> >> >>3 >> >> >> >>>>RvcmFnZS9TbmFwc2hvdCRMb2NhdGlvblR5cGU7TAAIcG9saWN5SWR0ABBMamF2YS9sYW5nL >> >>>>0x >> >> >>v >> >> >>b >> >> >> >>>>mc7TAAKc25hcHNob3RJZHEAfgACTAAIdm9sdW1lSWRxAH4AAnhyABNjb20uY2xvdWQudm0u >> >>>>Vm >> >> >>1 >> >> >>X >> >> >> >>>>b3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VySWRKAAR2bUlkTAALaGFuZGxlck5 >> >>>>hb >> >> >>W >> >> >>V >> >> >> >>>>0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAAHAAAAAAAAAAEAAAAAAAAA5HQAFFZvbH >> >>>>Vt >> >> >>Z >> >> >>U >> >> >> >>>>FwaVNlcnZpY2VJbXBsAABwc3IADmphdmEubGFuZy5Mb25nO4vkkMyPI98CAAFKAAV2YWx1Z >> >>>>Xh >> >> >>y >> >> >>A >> >> >> >>>>BBqYXZhLmxhbmcuTnVtYmVyhqyVHQuU4IsCAAB4cAAAAAAAAAARc3EAfgAHAAAAAAAACmBz >> >>>>cQ >> >> >>B >> >> >>- >> >> >>AAcAAAAAAAAA7g, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, >> >> >>resultCode: 0, result: null, initMsid: 19873467853209, completeMsid: >> >> >>null, >> >> >>lastUpdated: null, lastPolled: null, created: Thu Sep 13 06:02:16 EEST >> >> >>2018} >> >> >>2018-09-13 06:02:17,116 DEBUG [c.c.v.VmWorkJobDispatcher] >> >> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021) >> >>(logid:4c7a1c38) >> >> >>Run VM work job: com.cloud.vm.VmWorkTakeVolumeSnapshot for VM 228, job >> >> >>origin: 28020 >> >> >>2018-09-13 06:02:17,117 DEBUG [c.c.v.VmWorkJobHandlerProxy] >> >> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2) >> >> >>(logid:4c7a1c38) Execute VM work job: >> >> >> >>>>com.cloud.vm.VmWorkTakeVolumeSnapshot{"volumeId":238,"policyId":17,"sna >> >>>>ps >> >> >>h >> >> >>o >> >> >> >>>>tId":2656,"quiesceVm":false,"asyncBackup":false,"userId":1,"accountId": >> >>>>7, >> >> >>" >> >> >>v >> >> >>mId":228,"handlerName":"VolumeApiServiceImpl"} >> >> >>2018-09-13 06:02:17,197 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] >> >> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2) >> >> >>(logid:4c7a1c38) getCommandHostDelegation: class >> >> >>org.apache.cloudstack.storage.command.CreateObjectCommand >> >> >>2018-09-13 06:02:17,197 DEBUG [c.c.h.XenServerGuru] >> >> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2) >> >> >>(logid:4c7a1c38) We are returning the default host to execute commands >> >> >>because the command is not of Copy type. >> >> >>2018-09-13 06:02:17,199 DEBUG [c.c.a.t.Request] >> >> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2) >> >> >>(logid:4c7a1c38) Seq 128-4630826316843724436: Sending { Cmd , MgmtId: >> >> >>19873467853209, via: 128(xenserver-15), Ver: v1, Flags: 100011, >> >> >> >>>>[{"org.apache.cloudstack.storage.command.CreateObjectCommand":{"data":{ >> >>>>"o >> >> >>r >> >> >>g >> >> >> >>>>.apache.cloudstack.storage.to.SnapshotObjectTO":{"volume":{"uuid":"969c >> >>>>79 >> >> >>6 >> >> >>d >> >> >> >>>>-072e-4272-adc6-93a492c03ea3","volumeType":"ROOT","dataStore":{"org.apa >> >>>>ch >> >> >>e >> >> >>. >> >> >>cloudstack.storage.to >> >> .PrimaryDataStoreTO":{"uuid":"6568d30f-82b6-35a0-bfe >> >> >>5 >> >> >>- >> >> >> >>>>be17c1398ed8","id":2,"poolType":"NetworkFilesystem","host":"10.10.1.33" >> >>>>," >> >> >>p >> >> >>a >> >> >>th":"/pool3/primary3","port":2049,"url":"NetworkFilesystem:// >> >> 10.10.1.33/p >> >> >>o >> >> >>o >> >> >> >>>>l3/primary3/?ROLE=Primary&STOREUUID=6568d30f-82b6-35a0-bfe5-be17c1398ed >> >>>>8" >> >> >>, >> >> >>" >> >> >> >>>>isManaged":false}},"name":"ROOT-228","size":21474836480,"path":"3d2a971 >> >>>>6- >> >> >>8 >> >> >>f >> >> >> >>>>a1-497e-8753-941117a0ef50","volumeId":238,"vmName":"i-7-228-VM","accoun >> >>>>tI >> >> >>d >> >> >>" >> >> >> >>>>:7,"format":"VHD","provisioningType":"THIN","id":238,"deviceId":0,"hype >> >>>>rv >> >> >>i >> >> >>s >> >> >> >>>>orType":"XenServer"},"parentSnapshotPath":"c7b2824a-7402-4f66-b2e6-06ed >> >>>>45 >> >> >>2 >> >> >>0 >> >> >>72ad","dataStore":{"org.apache.cloudstack.storage.to >> >> .PrimaryDataStoreTO": >> >> >>{ >> >> >>" >> >> >> >>>>uuid":"6568d30f-82b6-35a0-bfe5-be17c1398ed8","id":2,"poolType":"Network >> >>>>Fi >> >> >>l >> >> >>e >> >> >> >>>>system","host":"10.10.1.33","path":"/pool3/primary3","port":2049,"url": >> >>>>"N >> >> >>e >> >> >>t >> >> >>workFilesystem:// >> >> 10.10.1.33/pool3/primary3/?ROLE=Primary&STOREUUID=6568d3 >> >> >>0 >> >> >>f >> >> >> >>>>-82b6-35a0-bfe5-be17c1398ed8","isManaged":false}},"vmName":"i-7-228-VM" >> >>>>," >> >> >>n >> >> >>a >> >> >> >>>>me":"BGI-Maria_ROOT-228_20180913030215","hypervisorType":"XenServer","i >> >>>>d" >> >> >>: >> >> >>2 >> >> >> >>>>656,"quiescevm":false,"parents":["c7b2824a-7402-4f66-b2e6-06ed452072ad" >> >>>>], >> >> >>" >> >> >>p >> >> >>hysicalSize":0}},"wait":0}}] } >> >> >>2018-09-13 06:02:17,199 DEBUG [c.c.a.t.Request] >> >> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2) >> >> >>(logid:4c7a1c38) Seq 128-4630826316843724436: Executing: { Cmd , >> >>MgmtId: >> >> >>19873467853209, via: 128(xenserver-15), Ver: v1, Flags: 100011, >> >> >> >>>>[{"org.apache.cloudstack.storage.command.CreateObjectCommand":{"data":{ >> >>>>"o >> >> >>r >> >> >>g >> >> >> >>>>.apache.cloudstack.storage.to.SnapshotObjectTO":{"volume":{"uuid":"969c >> >>>>79 >> >> >>6 >> >> >>d >> >> >> >>>>-072e-4272-adc6-93a492c03ea3","volumeType":"ROOT","dataStore":{"org.apa >> >>>>ch >> >> >>e >> >> >>. >> >> >>cloudstack.storage.to >> >> .PrimaryDataStoreTO":{"uuid":"6568d30f-82b6-35a0-bfe >> >> >>5 >> >> >>- >> >> >> >>>>be17c1398ed8","id":2,"poolType":"NetworkFilesystem","host":"10.10.1.33" >> >>>>," >> >> >>p >> >> >>a >> >> >>th":"/pool3/primary3","port":2049,"url":"NetworkFilesystem:// >> >> 10.10.1.33/p >> >> >>o >> >> >>o >> >> >> >>>>l3/primary3/?ROLE=Primary&STOREUUID=6568d30f-82b6-35a0-bfe5-be17c1398ed >> >>>>8" >> >> >>, >> >> >>" >> >> >> >>>>isManaged":false}},"name":"ROOT-228","size":21474836480,"path":"3d2a971 >> >>>>6- >> >> >>8 >> >> >>f >> >> >> >>>>a1-497e-8753-941117a0ef50","volumeId":238,"vmName":"i-7-228-VM","accoun >> >>>>tI >> >> >>d >> >> >>" >> >> >> >>>>:7,"format":"VHD","provisioningType":"THIN","id":238,"deviceId":0,"hype >> >>>>rv >> >> >>i >> >> >>s >> >> >> >>>>orType":"XenServer"},"parentSnapshotPath":"c7b2824a-7402-4f66-b2e6-06ed >> >>>>45 >> >> >>2 >> >> >>0 >> >> >>72ad","dataStore":{"org.apache.cloudstack.storage.to >> >> .PrimaryDataStoreTO": >> >> >>{ >> >> >>" >> >> >> >>>>uuid":"6568d30f-82b6-35a0-bfe5-be17c1398ed8","id":2,"poolType":"Network >> >>>>Fi >> >> >>l >> >> >>e >> >> >> >>>>system","host":"10.10.1.33","path":"/pool3/primary3","port":2049,"url": >> >>>>"N >> >> >>e >> >> >>t >> >> >>workFilesystem:// >> >> 10.10.1.33/pool3/primary3/?ROLE=Primary&STOREUUID=6568d3 >> >> >>0 >> >> >>f >> >> >> >>>>-82b6-35a0-bfe5-be17c1398ed8","isManaged":false}},"vmName":"i-7-228-VM" >> >>>>," >> >> >>n >> >> >>a >> >> >> >>>>me":"BGI-Maria_ROOT-228_20180913030215","hypervisorType":"XenServer","i >> >>>>d" >> >> >>: >> >> >>2 >> >> >> >>>>656,"quiescevm":false,"parents":["c7b2824a-7402-4f66-b2e6-06ed452072ad" >> >>>>], >> >> >>" >> >> >>p >> >> >>hysicalSize":0}},"wait":0}}] } >> >> >>2018-09-13 06:02:19,815 DEBUG [c.c.a.m.DirectAgentAttache] >> >> >>(DirectAgent-206:ctx-370f4cd9) (logid:4c7a1c38) Seq >> >> >>128-4630826316843724436: Response Received: >> >> >>2018-09-13 06:02:19,815 DEBUG [c.c.a.t.Request] >> >> >>(DirectAgent-206:ctx-370f4cd9) (logid:4c7a1c38) Seq >> >> >>128-4630826316843724436: Processing: { Ans: , MgmtId: 19873467853209, >> >> >>via: 128(xenserver-15), Ver: v1, Flags: 10, >> >> >> >>>>[{"org.apache.cloudstack.storage.command.CreateObjectAnswer":{"data":{" >> >>>>or >> >> >>g >> >> >>. >> >> >> >>>>apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"e91fb041-779d-4 >> >>>>09 >> >> >>0 >> >> >>- >> >> >> >>>>b14c-4f57f4cda54b","id":0,"quiescevm":false,"physicalSize":0}},"result" >> >>>>:t >> >> >>r >> >> >>u >> >> >>e,"wait":0}}] } >> >> >>2018-09-13 06:02:19,815 DEBUG [c.c.a.t.Request] >> >> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2) >> >> >>(logid:4c7a1c38) Seq 128-4630826316843724436: Received: { Ans: , >> >>MgmtId: >> >> >>19873467853209, via: 128(xenserver-15), Ver: v1, Flags: 10, { >> >> >>CreateObjectAnswer } } >> >> >>2018-09-13 06:02:19,934 DEBUG [o.a.c.s.m.AncientDataMotionStrategy] >> >> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2) >> >> >>(logid:4c7a1c38) copyAsync inspecting src type SNAPSHOT copyAsync >> >> >>inspecting dest type SNAPSHOT >> >> >>2018-09-13 06:02:20,017 ERROR [o.a.c.s.e.DefaultEndPointSelector] >> >> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2) >> >> >>(logid:4c7a1c38) IR24 select BACKUPSNAPSHOT from primary to secondary >> >> >>2656 >> >> >>dest=2656 >> >> >>2018-09-13 06:02:20,023 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] >> >> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2) >> >> >>(logid:4c7a1c38) getCommandHostDelegation: class >> >> >>org.apache.cloudstack.storage.command.CopyCommand >> >> >>2018-09-13 06:02:20,023 DEBUG [c.c.h.XenServerGuru] >> >> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2) >> >> >>(logid:4c7a1c38) We are returning the default host to execute commands >> >> >>because the source and destination objects are not NFS type. >> >> >>2018-09-13 06:02:20,025 DEBUG [c.c.a.t.Request] >> >> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2) >> >> >>(logid:4c7a1c38) Seq 130-7345089517264579763: Sending { Cmd , MgmtId: >> >> >>19873467853209, via: 130(xenserver-81), Ver: v1, Flags: 100111, >> >> >> >>>>[{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.ap >> >>>>ac >> >> >>h >> >> >>e >> >> >>.cloudstack.storage.to >> >> .SnapshotObjectTO":{"path":"e91fb041-779d-4090-b14c >> >> >>- >> >> >>4 >> >> >> >>>>f57f4cda54b","volume":{"uuid":"969c796d-072e-4272-adc6-93a492c03ea3","v >> >>>>ol >> >> >>u >> >> >>m >> >> >> >>>>eType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDat >> >>>>aS >> >> >>t >> >> >>o >> >> >> >>>>reTO":{"uuid":"6568d30f-82b6-35a0-bfe5-be17c1398ed8","id":2,"poolType": >> >>>>"N >> >> >>e >> >> >>t >> >> >> >>>>workFilesystem","host":"10.10.1.33","path":"/pool3/primary3","port":204 >> >>>>9, >> >> >>" >> >> >>u >> >> >>rl":"NetworkFilesystem:// >> >> 10.10.1.33/pool3/primary3/?ROLE=Primary&STOREUUI >> >> >>D >> >> >>= >> >> >> >>>>6568d30f-82b6-35a0-bfe5-be17c1398ed8","isManaged":false}},"name":"ROOT- >> >>>>22 >> >> >>8 >> >> >>" >> >> >> >>>>,"size":21474836480,"path":"3d2a9716-8fa1-497e-8753-941117a0ef50","volu >> >>>>me >> >> >>I >> >> >>d >> >> >> >>>>":238,"vmName":"i-7-228-VM","accountId":7,"format":"VHD","provisioningT >> >>>>yp >> >> >>e >> >> >>" >> >> >> >>>>:"THIN","id":238,"deviceId":0,"hypervisorType":"XenServer"},"parentSnap >> >>>>sh >> >> >>o >> >> >>t >> >> >> >>>>Path":"c7b2824a-7402-4f66-b2e6-06ed452072ad","dataStore":{"org.apache.c >> >>>>lo >> >> >>u >> >> >>d >> >> >>stack.storage.to >> >> .PrimaryDataStoreTO":{"uuid":"6568d30f-82b6-35a0-bfe5-be1 >> >> >>7 >> >> >>c >> >> >> >>>>1398ed8","id":2,"poolType":"NetworkFilesystem","host":"10.10.1.33","pat >> >>>>h" >> >> >>: >> >> >>" >> >> >>/pool3/primary3","port":2049,"url":"NetworkFilesystem:// >> >> 10.10.1.33/pool3/ >> >> >>p >> >> >>r >> >> >> >>>>imary3/?ROLE=Primary&STOREUUID=6568d30f-82b6-35a0-bfe5-be17c1398ed8","i >> >>>>sM >> >> >>a >> >> >>n >> >> >> >>>>aged":false}},"vmName":"i-7-228-VM","name":"BGI-Maria_ROOT-228_20180913 >> >>>>03 >> >> >>0 >> >> >>2 >> >> >> >>>>15","hypervisorType":"XenServer","id":2656,"quiescevm":false,"parents": >> >>>>[" >> >> >>c >> >> >>7 >> >> >> >>>>b2824a-7402-4f66-b2e6-06ed452072ad"],"physicalSize":0}},"destTO":{"org. >> >>>>ap >> >> >>a >> >> >>c >> >> >>he.cloudstack.storage.to >> >> .SnapshotObjectTO":{"path":"snapshots/7/238","vol >> >> >>u >> >> >>m >> >> >> >>>>e":{"uuid":"969c796d-072e-4272-adc6-93a492c03ea3","volumeType":"ROOT"," >> >>>>da >> >> >>t >> >> >>a >> >> >>Store":{"org.apache.cloudstack.storage.to >> >> .PrimaryDataStoreTO":{"uuid":"65 >> >> >>6 >> >> >>8 >> >> >> >>>>d30f-82b6-35a0-bfe5-be17c1398ed8","id":2,"poolType":"NetworkFilesystem" >> >>>>," >> >> >>h >> >> >>o >> >> >> >>>>st":"10.10.1.33","path":"/pool3/primary3","port":2049,"url":"NetworkFil >> >>>>es >> >> >>y >> >> >>s >> >> >>tem:// >> >> 10.10.1.33/pool3/primary3/?ROLE=Primary&STOREUUID=6568d30f-82b6-35a >> >> >>0 >> >> >>- >> >> >> >>>>bfe5-be17c1398ed8","isManaged":false}},"name":"ROOT-228","size":2147483 >> >>>>64 >> >> >>8 >> >> >>0 >> >> >> >>>>,"path":"3d2a9716-8fa1-497e-8753-941117a0ef50","volumeId":238,"vmName": >> >>>>"i >> >> >>- >> >> >>7 >> >> >> >>>>-228-VM","accountId":7,"format":"VHD","provisioningType":"THIN","id":23 >> >>>>8, >> >> >>" >> >> >>d >> >> >> >>>>eviceId":0,"hypervisorType":"XenServer"},"parentSnapshotPath":"snapshot >> >>>>s/ >> >> >>7 >> >> >>/ >> >> >> >>>>238/0c8c6ee3-ce3d-4241-8509-5193d2cff491.vhd","dataStore":{"com.cloud.a >> >>>>ge >> >> >>n >> >> >>t >> >> >>.api.to.NfsTO":{"_url":"nfs://10.10.1.34/pool8/secondary8 >> >> ","_role":"Image >> >> >>" >> >> >>} >> >> >> >>>>},"vmName":"i-7-228-VM","name":"BGI-Maria_ROOT-228_20180913030215","hyp >> >>>>er >> >> >>v >> >> >>i >> >> >> >>>>sorType":"XenServer","id":2656,"quiescevm":false,"parents":["snapshots/ >> >>>>7/ >> >> >>2 >> >> >>3 >> >> >> >>>>8/93a42896-0da1-4b82-8985-d28464f1ac4a.vhd","snapshots/7/238/0c8c6ee3-c >> >>>>e3 >> >> >>d >> >> >>- >> >> >> >>>>4241-8509-5193d2cff491.vhd"],"physicalSize":0}},"executeInSequence":tru >> >>>>e, >> >> >>" >> >> >>o >> >> >>ptions":{"fullSnapshot":"false"},"options2":{},"wait":28800}}] } >> >> >>2018-09-13 06:02:20,025 DEBUG [c.c.a.t.Request] >> >> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2) >> >> >>(logid:4c7a1c38) Seq 130-7345089517264579763: Executing: { Cmd , >> >>MgmtId: >> >> >>19873467853209, via: 130(xenserver-81), Ver: v1, Flags: 100111, >> >> >> >>>>[{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.ap >> >>>>ac >> >> >>h >> >> >>e >> >> >>.cloudstack.storage.to >> >> .SnapshotObjectTO":{"path":"e91fb041-779d-4090-b14c >> >> >>- >> >> >>4 >> >> >> >>>>f57f4cda54b","volume":{"uuid":"969c796d-072e-4272-adc6-93a492c03ea3","v >> >>>>ol >> >> >>u >> >> >>m >> >> >> >>>>eType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDat >> >>>>aS >> >> >>t >> >> >>o >> >> >> >>>>reTO":{"uuid":"6568d30f-82b6-35a0-bfe5-be17c1398ed8","id":2,"poolType": >> >>>>"N >> >> >>e >> >> >>t >> >> >> >>>>workFilesystem","host":"10.10.1.33","path":"/pool3/primary3","port":204 >> >>>>9, >> >> >>" >> >> >>u >> >> >>rl":"NetworkFilesystem:// >> >> 10.10.1.33/pool3/primary3/?ROLE=Primary&STOREUUI >> >> >>D >> >> >>= >> >> >> >>>>6568d30f-82b6-35a0-bfe5-be17c1398ed8","isManaged":false}},"name":"ROOT- >> >>>>22 >> >> >>8 >> >> >>" >> >> >> >>>>,"size":21474836480,"path":"3d2a9716-8fa1-497e-8753-941117a0ef50","volu >> >>>>me >> >> >>I >> >> >>d >> >> >> >>>>":238,"vmName":"i-7-228-VM","accountId":7,"format":"VHD","provisioningT >> >>>>yp >> >> >>e >> >> >>" >> >> >> >>>>:"THIN","id":238,"deviceId":0,"hypervisorType":"XenServer"},"parentSnap >> >>>>sh >> >> >>o >> >> >>t >> >> >> >>>>Path":"c7b2824a-7402-4f66-b2e6-06ed452072ad","dataStore":{"org.apache.c >> >>>>lo >> >> >>u >> >> >>d >> >> >>stack.storage.to >> >> .PrimaryDataStoreTO":{"uuid":"6568d30f-82b6-35a0-bfe5-be1 >> >> >>7 >> >> >>c >> >> >> >>>>1398ed8","id":2,"poolType":"NetworkFilesystem","host":"10.10.1.33","pat >> >>>>h" >> >> >>: >> >> >>" >> >> >>/pool3/primary3","port":2049,"url":"NetworkFilesystem:// >> >> 10.10.1.33/pool3/ >> >> >>p >> >> >>r >> >> >> >>>>imary3/?ROLE=Primary&STOREUUID=6568d30f-82b6-35a0-bfe5-be17c1398ed8","i >> >>>>sM >> >> >>a >> >> >>n >> >> >> >>>>aged":false}},"vmName":"i-7-228-VM","name":"BGI-Maria_ROOT-228_20180913 >> >>>>03 >> >> >>0 >> >> >>2 >> >> >> >>>>15","hypervisorType":"XenServer","id":2656,"quiescevm":false,"parents": >> >>>>[" >> >> >>c >> >> >>7 >> >> >> >>>>b2824a-7402-4f66-b2e6-06ed452072ad"],"physicalSize":0}},"destTO":{"org. >> >>>>ap >> >> >>a >> >> >>c >> >> >>he.cloudstack.storage.to >> >> .SnapshotObjectTO":{"path":"snapshots/7/238","vol >> >> >>u >> >> >>m >> >> >> >>>>e":{"uuid":"969c796d-072e-4272-adc6-93a492c03ea3","volumeType":"ROOT"," >> >>>>da >> >> >>t >> >> >>a >> >> >>Store":{"org.apache.cloudstack.storage.to >> >> .PrimaryDataStoreTO":{"uuid":"65 >> >> >>6 >> >> >>8 >> >> >> >>>>d30f-82b6-35a0-bfe5-be17c1398ed8","id":2,"poolType":"NetworkFilesystem" >> >>>>," >> >> >>h >> >> >>o >> >> >> >>>>st":"10.10.1.33","path":"/pool3/primary3","port":2049,"url":"NetworkFil >> >>>>es >> >> >>y >> >> >>s >> >> >>tem:// >> >> 10.10.1.33/pool3/primary3/?ROLE=Primary&STOREUUID=6568d30f-82b6-35a >> >> >>0 >> >> >>- >> >> >> >>>>bfe5-be17c1398ed8","isManaged":false}},"name":"ROOT-228","size":2147483 >> >>>>64 >> >> >>8 >> >> >>0 >> >> >> >>>>,"path":"3d2a9716-8fa1-497e-8753-941117a0ef50","volumeId":238,"vmName": >> >>>>"i >> >> >>- >> >> >>7 >> >> >> >>>>-228-VM","accountId":7,"format":"VHD","provisioningType":"THIN","id":23 >> >>>>8, >> >> >>" >> >> >>d >> >> >> >>>>eviceId":0,"hypervisorType":"XenServer"},"parentSnapshotPath":"snapshot >> >>>>s/ >> >> >>7 >> >> >>/ >> >> >> >>>>238/0c8c6ee3-ce3d-4241-8509-5193d2cff491.vhd","dataStore":{"com.cloud.a >> >>>>ge >> >> >>n >> >> >>t >> >> >>.api.to.NfsTO":{"_url":"nfs://10.10.1.34/pool8/secondary8 >> >> ","_role":"Image >> >> >>" >> >> >>} >> >> >> >>>>},"vmName":"i-7-228-VM","name":"BGI-Maria_ROOT-228_20180913030215","hyp >> >>>>er >> >> >>v >> >> >>i >> >> >> >>>>sorType":"XenServer","id":2656,"quiescevm":false,"parents":["snapshots/ >> >>>>7/ >> >> >>2 >> >> >>3 >> >> >> >>>>8/93a42896-0da1-4b82-8985-d28464f1ac4a.vhd","snapshots/7/238/0c8c6ee3-c >> >>>>e3 >> >> >>d >> >> >>- >> >> >> >>>>4241-8509-5193d2cff491.vhd"],"physicalSize":0}},"executeInSequence":tru >> >>>>e, >> >> >>" >> >> >>o >> >> >>ptions":{"fullSnapshot":"false"},"options2":{},"wait":28800}}] } >> >> >>2018-09-13 06:04:28,895 DEBUG [c.c.h.x.r.CitrixResourceBase] >> >> >>(DirectAgent-382:ctx-36b817c8) (logid:4c7a1c38) Host 10.10.1.81 >> >> >>OpaqueRef:f83331df-6634-63fd-6b95-44607a6069b8: Removing SR >> >> >>2018-09-13 06:04:28,913 DEBUG [c.c.h.x.r.CitrixResourceBase] >> >> >>(DirectAgent-382:ctx-36b817c8) (logid:4c7a1c38) Host 10.10.1.81 >> >> >>OpaqueRef:05380b03-4cb8-85a7-6a6a-a1595c188f60: Unplugging pbd >> >> >>2018-09-13 06:04:30,881 DEBUG [c.c.h.x.r.CitrixResourceBase] >> >> >>(DirectAgent-382:ctx-36b817c8) (logid:4c7a1c38) Host 10.10.1.81 >> >> >>OpaqueRef:f83331df-6634-63fd-6b95-44607a6069b8: Forgetting >> >> >>2018-09-13 06:04:30,891 DEBUG [c.c.h.x.r.XenServerStorageProcessor] >> >> >>(DirectAgent-382:ctx-36b817c8) (logid:4c7a1c38) Successfully destroyed >> >> >>snapshot on volume: 3d2a9716-8fa1-497e-8753-941117a0ef50 execept this >> >> >>current snapshot e91fb041-779d-4090-b14c-4f57f4cda54b >> >> >>2018-09-13 06:04:30,891 INFO [c.c.h.x.r.XenServerStorageProcessor] >> >> >>(DirectAgent-382:ctx-36b817c8) (logid:4c7a1c38) New snapshot details: >> >> >> >>>>SnapshotTO[datastore=null|volume=null|pathsnapshots/7/238/0560d536-c945 >> >>>>-4 >> >> >>a >> >> >>6 >> >> >>1-8c4c-dbaf29f29993.vhd] >> >> >>2018-09-13 06:04:30,891 INFO [c.c.h.x.r.XenServerStorageProcessor] >> >> >>(DirectAgent-382:ctx-36b817c8) (logid:4c7a1c38) New snapshot physical >> >> >>utilization: 2342932992 >> >> >>2018-09-13 06:04:30,891 DEBUG [c.c.a.m.DirectAgentAttache] >> >> >>(DirectAgent-382:ctx-36b817c8) (logid:4c7a1c38) Seq >> >> >>130-7345089517264579763: Response Received: >> >> >>2018-09-13 06:04:30,891 DEBUG [c.c.a.t.Request] >> >> >>(DirectAgent-382:ctx-36b817c8) (logid:4c7a1c38) Seq >> >> >>130-7345089517264579763: Processing: { Ans: , MgmtId: 19873467853209, >> >> >>via: 130(xenserver-81), Ver: v1, Flags: 110, >> >> >> >>>>[{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{"or >> >>>>g. >> >> >>a >> >> >>p >> >> >>ache.cloudstack.storage.to >> >> .SnapshotObjectTO":{"path":"snapshots/7/238/056 >> >> >>0 >> >> >>d >> >> >> >>>>536-c945-4a61-8c4c-dbaf29f29993.vhd","parentSnapshotPath":"snapshots/7/ >> >>>>23 >> >> >>8 >> >> >>/ >> >> >> >>>>0c8c6ee3-ce3d-4241-8509-5193d2cff491.vhd","id":0,"quiescevm":false,"phy >> >>>>si >> >> >>c >> >> >>a >> >> >>lSize":2342932992}},"result":true,"wait":0}}] } >> >> >>2018-09-13 06:04:30,891 DEBUG [c.c.a.m.AgentAttache] >> >> >>(DirectAgent-382:ctx-36b817c8) (logid:4c7a1c38) Seq >> >> >>130-7345089517264579763: No more commands found >> >> >>2018-09-13 06:04:30,891 DEBUG [c.c.a.t.Request] >> >> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2) >> >> >>(logid:4c7a1c38) Seq 130-7345089517264579763: Received: { Ans: , >> >>MgmtId: >> >> >>19873467853209, via: 130(xenserver-81), Ver: v1, Flags: 110, { >> >> >>CopyCmdAnswer } } >> >> >>2018-09-13 06:04:30,957 DEBUG [c.c.r.ResourceLimitManagerImpl] >> >> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2) >> >> >>(logid:4c7a1c38) Updating resource Type = secondary_storage count for >> >> >>Account = 7 Operation = decreasing Amount = 19131903488 >> >> >>2018-09-13 06:04:30,969 DEBUG [c.c.v.VmWorkJobHandlerProxy] >> >> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2) >> >> >>(logid:4c7a1c38) Done executing VM work job: >> >> >> >>>>com.cloud.vm.VmWorkTakeVolumeSnapshot{"volumeId":238,"policyId":17,"sna >> >>>>ps >> >> >>h >> >> >>o >> >> >> >>>>tId":2656,"quiesceVm":false,"asyncBackup":false,"userId":1,"accountId": >> >>>>7, >> >> >>" >> >> >>v >> >> >>mId":228,"handlerName":"VolumeApiServiceImpl"} >> >> >>2018-09-13 06:04:30,969 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] >> >> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2) >> >> >>(logid:4c7a1c38) Complete async job-28021, jobStatus: SUCCEEDED, >> >> >>resultCode: 0, result: >> >> >> >>>>rO0ABXNyAA5qYXZhLmxhbmcuTG9uZzuL5JDMjyPfAgABSgAFdmFsdWV4cgAQamF2YS5sYW5 >> >>>>nL >> >> >>k >> >> >>5 >> >> >>1bWJlcoaslR0LlOCLAgAAeHAAAAAAAAAKYA >> >> >>2018-09-13 06:04:30,970 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] >> >> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2) >> >> >>(logid:4c7a1c38) Publish async job-28021 complete on message bus >> >> >>2018-09-13 06:04:30,970 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] >> >> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2) >> >> >>(logid:4c7a1c38) Wake up jobs related to job-28021 >> >> >>2018-09-13 06:04:30,970 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] >> >> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2) >> >> >>(logid:4c7a1c38) Update db status for job-28021 >> >> >>2018-09-13 06:04:30,971 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] >> >> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2) >> >> >>(logid:4c7a1c38) Wake up jobs joined with job-28021 and disjoin all >> >> >>subjobs created from job- 28021 >> >> >>2018-09-13 06:04:30,983 DEBUG [c.c.v.VmWorkJobDispatcher] >> >> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021) >> >>(logid:4c7a1c38) >> >> >>Done with run of VM work job: com.cloud.vm.VmWorkTakeVolumeSnapshot >> >>for >> >> >>VM >> >> >>228, job origin: 28020 >> >> >>2018-09-13 06:04:30,983 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] >> >> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021) >> >>(logid:4c7a1c38) >> >> >>Done executing com.cloud.vm.VmWorkTakeVolumeSnapshot for job-28021 >> >> >>2018-09-13 06:04:30,984 INFO [o.a.c.f.j.i.AsyncJobMonitor] >> >> >>(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021) >> >>(logid:4c7a1c38) >> >> >>Remove job-28021 from job monitoring >> >> >>2018-09-13 06:04:31,026 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] >> >> >>(API-Job-Executor-83:ctx-cfb1cd95 job-28020 ctx-5b8156a6) >> >> >>(logid:4c7a1c38) >> >> >>Complete async job-28020, jobStatus: SUCCEEDED, resultCode: 0, result: >> >> >> >>>>org.apache.cloudstack.api.response.SnapshotResponse/snapshot/{"id":"3c8 >> >>>>66 >> >> >>2 >> >> >>b >> >> >> >>>>2-c76f-4e3a-a928-f76dad820122","account":"juha","domainid":"d3c2420f-75 >> >>>>03 >> >> >>- >> >> >>4 >> >> >> >>>>560-a2da-d7e7149ce390","domain":"BGI","snapshottype":"DAILY","volumeid" >> >>>>:" >> >> >>9 >> >> >>6 >> >> >> >>>>9c796d-072e-4272-adc6-93a492c03ea3","volumename":"ROOT-228","volumetype >> >>>>": >> >> >>" >> >> >>R >> >> >> >>>>OOT","created":"2018-09-13T06:02:15+0300","name":"BGI-Maria_ROOT-228_20 >> >>>>18 >> >> >>0 >> >> >>9 >> >> >> >>>>13030215","intervaltype":"DAILY","state":"BackedUp","physicalsize":2342 >> >>>>93 >> >> >>2 >> >> >>9 >> >> >> >>>>92,"zoneid":"533c8f88-2637-4c0e-8210-e0a355b62d82","tags":[],"revertabl >> >>>>e" >> >> >>: >> >> >>f >> >> >> >>>>alse,"ostypeid":"da491b96-3748-11e5-a683-121326fde999","osdisplayname": >> >>>>"C >> >> >>e >> >> >>n >> >> >>tOS 6.5 (64-bit)","virtualsize":21474836480} >> >> >>2018-09-13 06:04:31,027 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] >> >> >>(API-Job-Executor-83:ctx-cfb1cd95 job-28020 ctx-5b8156a6) >> >> >>(logid:4c7a1c38) >> >> >>Publish async job-28020 complete on message bus >> >> >>2018-09-13 06:04:31,027 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] >> >> >>(API-Job-Executor-83:ctx-cfb1cd95 job-28020 ctx-5b8156a6) >> >> >>(logid:4c7a1c38) >> >> >>Wake up jobs related to job-28020 >> >> >>2018-09-13 06:04:31,027 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] >> >> >>(API-Job-Executor-83:ctx-cfb1cd95 job-28020 ctx-5b8156a6) >> >> >>(logid:4c7a1c38) >> >> >>Update db status for job-28020 >> >> >>2018-09-13 06:04:31,028 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] >> >> >>(API-Job-Executor-83:ctx-cfb1cd95 job-28020 ctx-5b8156a6) >> >> >>(logid:4c7a1c38) >> >> >>Wake up jobs joined with job-28020 and disjoin all subjobs created >> >>from >> >> >>job- 28020 >> >> >>2018-09-13 06:04:31,039 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] >> >> >>(API-Job-Executor-83:ctx-cfb1cd95 job-28020) (logid:4c7a1c38) Done >> >> >>executing >> >> >>org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd for >> >> >>job-28020 >> >> >>2018-09-13 06:04:31,039 INFO [o.a.c.f.j.i.AsyncJobMonitor] >> >> >>(API-Job-Executor-83:ctx-cfb1cd95 job-28020) (logid:4c7a1c38) Remove >> >> >>job-28020 from job monitoring >> >> >> >> >> >> >> > >> >-- >> >Rafael Weingärtner >> >> >> > > -- > Rafael Weingärtner