I tested on a fresh environment, acs 4.9.3.0, xenserver 6.5 with all patches and primary storage NFS, we have same problem
- Snapshot files [root@acs01 203]# ls -lh /exports/secondary/snapshots/2/6/ total 2.8G -rw-r--r-- 1 root root 1.7G Oct 31 16:48 cad96f0c-de49-4255-b0a0-9aea5e2297cb.vhd <-- snap-base -rw-r--r-- 1 root root 1.1G Oct 31 16:49 fa4e7e4d-531d-42dd-8a80-df6b274584a8.vhd <-- snap-diff https://i.imgur.com/bWupsAS.png [root@acs01 203]# md5sum /exports/secondary/snapshots/2/6/fa4e7e4d-531d-42dd-8a80-df6b274584a8.vhd 17eec8d4f6d4c128b34b9e2e1876ceb7 /exports/secondary/snapshots/2/6/fa4e7e4d-531d-42dd-8a80-df6b274584a8.vhd - Template files [root@acs01 203]# ls -lh /exports/secondary/template/tmpl/2/203 total 2.0G -rw-r--r-- 1 root root 1.1G Oct 31 16:50 e9def45d-1b9e-4c2e-b843-852fe40f00b2.vhd -rw-r--r-- 1 root root 303 Oct 31 16:50 template.properties [root@acs01 203]# md5sum /exports/secondary/template/tmpl/2/203/e9def45d-1b9e-4c2e-b843-852fe40f00b2.vhd 17eec8d4f6d4c128b34b9e2e1876ceb7 /exports/secondary/template/tmpl/2/203/e9def45d-1b9e-4c2e-b843-852fe40f00b2.vhd It's a huge problem!!! I think issue 9025 is not resolved Regards Marcelo -----Original Message----- From: Rafael Weingärtner [mailto:rafaelweingart...@gmail.com] Sent: terça-feira, 31 de outubro de 2017 16:00 To: us...@cloudstack.apache.org Subject: Re: Strange size of template from snapshot on XenServer Well, reading the description of the issue it seems to be related to the problem you are describing. However, I compared the classes that were changed with PR (https://github.com/apache/cloudstack/pull/1176) and they are the same in ACS 4.9.2.0 and 4.9.3.0. On Tue, Oct 31, 2017 at 6:36 PM, Lotic Lists <li...@lotic.com.br> wrote: > Hi Rafael, thanks for test. > > I found the issue > https://issues.apache.org/jira/browse/CLOUDSTACK-9025 > I think is a same case. I will up a new lab environment now for > simulate the issue > > Regards. > Marcelo > > -----Original Message----- > From: Rafael Weingärtner [mailto:rafaelweingart...@gmail.com] > Sent: segunda-feira, 30 de outubro de 2017 12:08 > To: us...@cloudstack.apache.org > Subject: Re: Strange size of template from snapshot on XenServer > > I just did, and the size of the template is the size of the root disk > + the 1GB file I created in the test vm. > > The system I tested this is an ACS 4.9.2, XenServer 6.5, Primary Stg > ISCSI > > On Mon, Oct 30, 2017 at 10:20 AM, Lotic Lists <li...@lotic.com.br> wrote: > > > Good morning > > > > Guys, who could execute the test below? > > > > 1. Create a manual snapshot of a volume 2. Create a file (1GB) with > > dd on VM 3. Create second snapshot of a volume 4. Convert latest > > snapshot to a template 5. Verify the size of vhd of new template in > > secondary storage. > > > > Here the template have 1GB, exactly the size of latest snapshot. > > > > ACS 4.9.3, XenServer 6.5, Primary Stg ISCSI > > > > > > -----Original Message----- > > From: Lotic Lists [mailto:li...@lotic.com.br] > > Sent: quinta-feira, 26 de outubro de 2017 15:03 > > To: us...@cloudstack.apache.org > > Subject: RE: Strange size of template from snapshot on XenServer > > > > I test the same case on other environment, create template from > > snapshot, acs 4.9.2.0 and XenServer 6.5/iscsi > > > > 1. Was created two snapshots > > > > # ls -lh /var/cloud_mount/e5921d9a-70b0-3501-9fda-6097e721fffb/ > > snapshots/131/617/ > > total 3.0G > > -rw-r--r-- 1 4294967294 root 13G Oct 26 13:46 > > 42c7c51d-4aa7-40d2-b5eb- cb56fc963974.vhd > > -rw-r--r-- 1 4294967294 root 1.2G Oct 26 14:42 > > 53b33a5a-a7d9-4efa-b126- 38eec5482b05.vhd > > > > https://i.imgur.com/cSHZAJ3.png > > > > 2. create a template from latest snapshot > > > > # ls -lh > > total 37M > > -rw-r--r-- 1 4294967294 root 1.2G Oct 26 14:43 > > 637604a6-3085-444e-ba09- b9aab11a2c16.vhd > > -rw-r--r-- 1 4294967294 root 303 Oct 26 14:43 template.properties > > > > # cat template.properties > > #Thu Oct 26 16:43:54 UTC 2017 > > filename=637604a6-3085-444e-ba09-b9aab11a2c16.vhd > > id=1 > > vhd=true > > vhd.filename=637604a6-3085-444e-ba09-b9aab11a2c16.vhd > > public=true > > uniquename=637604a6-3085-444e-ba09-b9aab11a2c16 > > vhd.virtualsize=21474836480 > > virtualsize=21474836480 > > hvm= > > vhd.size=1197752832 > > size=1197752832 > > > > # md5sum /var/cloud_mount/e5921d9a-70b0-3501-9fda-6097e721fffb/ > > snapshots/131/617/53b33a5a-a7d9-4efa-b126-38eec5482b05.vhd > > 9d2078a6f41ca5e1eab824848c06df53 /var/cloud_mount/e5921d9a- > > 70b0-3501-9fda-6097e721fffb/snapshots/131/617/53b33a5a- > > a7d9-4efa-b126-38eec5482b05.vhd > > > > # md5sum /var/cloud_mount/e5921d9a-70b0-3501-9fda-6097e721fffb/ > > template/tmpl/2/294/637604a6-3085-444e-ba09-b9aab11a2c16.vhd > > 9d2078a6f41ca5e1eab824848c06df53 /var/cloud_mount/e5921d9a- > > 70b0-3501-9fda-6097e721fffb/template/tmpl/2/294/637604a6- > > 3085-444e-ba09-b9aab11a2c16.vhd > > > > ACS not merge files of snapshots in a single file for template, I > > think it's a bug. > > > > > > -----Original Message----- > > From: Rafael Weingärtner [mailto:rafaelweingart...@gmail.com] > > Sent: quarta-feira, 25 de outubro de 2017 22:51 > > To: us...@cloudstack.apache.org > > Subject: Re: Strange size of template from snapshot on XenServer > > > > It is easy to test, get one of these files and try to use them in > > Virtualbox. If they run, they do not require anything else. > > > > On Wed, Oct 25, 2017 at 10:49 PM, Lotic Lists <li...@lotic.com.br> > wrote: > > > > > Sure but files on secondary storage don't have parents, I don't > > > know if ACS can connect standalone files. > > > > > > vhd-util scan -f -m'*.vhd' -p > > > vhd=14818597-55bb-49be-9ace-761e9e01c074.vhd capacity=311385128960 > > > size=304124744192 hidden=0 parent=none > > > vhd=bbd9e9b5-53fb-44a2-afbe-f262396a2d84.vhd capacity=311385128960 > > > size=17770869248 hidden=0 parent=none > > > vhd=3b90bbb2-7ce5-41f6-9f7e-fd0ea061cb2d.vhd capacity=311385128960 > > > size=52199817728 hidden=0 parent=none > > > vhd=4b450217-c9ad-45a2-946b-de3cb323469b.vhd capacity=311385128960 > > > size=51615670784 hidden=0 parent=none > > > > > > > > > -----Original Message----- > > > From: Rafael Weingärtner [mailto:rafaelweingart...@gmail.com] > > > Sent: quarta-feira, 25 de outubro de 2017 22:35 > > > To: us...@cloudstack.apache.org > > > Subject: Re: Strange size of template from snapshot on XenServer > > > > > > If it was not executing coalesce you would see the parent > > > reference when you listed the hierarchy, right? > > > > > > On Wed, Oct 25, 2017 at 10:25 PM, Lotic Lists <li...@lotic.com.br> > > wrote: > > > > > > > I think ACS preserve all VHDs for coalesce but coalesce not > > > > occur when create a template or before starting next snapshot scheduled. > > > > If I create a template from any snapshots (2) showing in GUI, > > > > the template have 49GB. 49 GB is the size of the 2 latest files > > > > on secondary > > > storage. > > > > > > > > What do you think? Bug? > > > > > > > > > > > > -----Original Message----- > > > > From: Rafael Weingärtner [mailto:rafaelweingart...@gmail.com] > > > > Sent: quarta-feira, 25 de outubro de 2017 22:07 > > > > To: us...@cloudstack.apache.org > > > > Subject: Re: Strange size of template from snapshot on XenServer > > > > > > > > Aha, The two destroyed entries are showing " removed: NULL". > > > > > > > > In ACS one thing is the "destroying" of a resource, another is > > > > the removal of this resource from the system. I believe there is > > > > something to do with the expunge interval, but I am not sure if > > > > the snapshots complete removal also happens within the expunge > > > > interval of user VMs or if it is configured by something else. > > > > > > > > > > > > On Wed, Oct 25, 2017 at 10:03 PM, Lotic Lists > > > > <li...@lotic.com.br> > > > wrote: > > > > > > > > > I not start manual snapshot, just scheduled with keep=2 > > > > > > > > > > GUI showing 2 snapshots, database showing 4 entries, 2 > > > > > Destroyed and > > > > > 2 BackedUp > > > > > > > > > > *************************** 132. row *************************** > > > > > id: 6380 > > > > > data_center_id: 2 > > > > > account_id: 64 > > > > > domain_id: 49 > > > > > volume_id: 2102 > > > > > disk_offering_id: 230 > > > > > status: Destroyed > > > > > path: NULL > > > > > name: FullCRM_ROOT-542_20171022030205 > > > > > uuid: 1294cdb7-01bf-4e76-b4c0-11b3c6b395e9 > > > > > snapshot_type: 4 > > > > > type_description: DAILY > > > > > size: 311385128960 > > > > > created: 2017-10-22 03:02:05 > > > > > removed: NULL > > > > > backup_snap_id: NULL > > > > > swift_id: NULL > > > > > sechost_id: NULL > > > > > prev_snap_id: NULL > > > > > hypervisor_type: XenServer > > > > > version: 2.2 > > > > > s3_id: NULL > > > > > min_iops: NULL > > > > > max_iops: NULL > > > > > *************************** 133. row *************************** > > > > > id: 6416 > > > > > data_center_id: 2 > > > > > account_id: 64 > > > > > domain_id: 49 > > > > > volume_id: 2102 > > > > > disk_offering_id: 230 > > > > > status: Destroyed > > > > > path: NULL > > > > > name: FullCRM_ROOT-542_20171023030205 > > > > > uuid: 0a8067a3-ad15-40b6-8b16-c1881d7d9f41 > > > > > snapshot_type: 4 > > > > > type_description: DAILY > > > > > size: 311385128960 > > > > > created: 2017-10-23 03:02:05 > > > > > removed: NULL > > > > > backup_snap_id: NULL > > > > > swift_id: NULL > > > > > sechost_id: NULL > > > > > prev_snap_id: NULL > > > > > hypervisor_type: XenServer > > > > > version: 2.2 > > > > > s3_id: NULL > > > > > min_iops: NULL > > > > > max_iops: NULL > > > > > *************************** 134. row *************************** > > > > > id: 6455 > > > > > data_center_id: 2 > > > > > account_id: 64 > > > > > domain_id: 49 > > > > > volume_id: 2102 > > > > > disk_offering_id: 230 > > > > > status: BackedUp > > > > > path: NULL > > > > > name: FullCRM_ROOT-542_20171024030207 > > > > > uuid: 1ca646e5-a155-4a59-aa31-5889eec7536f > > > > > snapshot_type: 4 > > > > > type_description: DAILY > > > > > size: 311385128960 > > > > > created: 2017-10-24 03:02:07 > > > > > removed: NULL > > > > > backup_snap_id: NULL > > > > > swift_id: NULL > > > > > sechost_id: NULL > > > > > prev_snap_id: NULL > > > > > hypervisor_type: XenServer > > > > > version: 2.2 > > > > > s3_id: NULL > > > > > min_iops: NULL > > > > > max_iops: NULL > > > > > *************************** 135. row *************************** > > > > > id: 6497 > > > > > data_center_id: 2 > > > > > account_id: 64 > > > > > domain_id: 49 > > > > > volume_id: 2102 > > > > > disk_offering_id: 230 > > > > > status: BackedUp > > > > > path: NULL > > > > > name: FullCRM_ROOT-542_20171025030208 > > > > > uuid: 759d61dc-e3db-4815-b68b-b90904ccbda5 > > > > > snapshot_type: 4 > > > > > type_description: DAILY > > > > > size: 311385128960 > > > > > created: 2017-10-25 03:02:08 > > > > > removed: NULL > > > > > backup_snap_id: NULL > > > > > swift_id: NULL > > > > > sechost_id: NULL > > > > > prev_snap_id: NULL > > > > > hypervisor_type: XenServer > > > > > version: 2.2 > > > > > s3_id: NULL > > > > > min_iops: NULL > > > > > max_iops: NULL > > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > From: Rafael Weingärtner [mailto:rafaelweingart...@gmail.com] > > > > > Sent: quarta-feira, 25 de outubro de 2017 21:50 > > > > > To: us...@cloudstack.apache.org > > > > > Subject: Re: Strange size of template from snapshot on > > > > > XenServer > > > > > > > > > > The file "14818597-55bb-49be-9ace-761e9e01c074.vhd" seems more > > > > > like a manual snapshot. It was created 2 hours after the "normal" > > > > > time. I would check these snapshots in ACS and see how they > > > > > are > > presented. > > > > > BTW, how many snapshots is ACS showing for this volume? I > > > > > would also check the "snapshots" table for the volume you have > > > > > configured > > them. > > > > > select * from snapshots where volume_id = ? > > > > > > > > > > you may also filter for removed: > > > > > select * from snapshots where volume_id = ? and removed is > > > > > null > > > > > > > > > > On Wed, Oct 25, 2017 at 9:39 PM, Lotic Lists > > > > > <li...@lotic.com.br> > > > wrote: > > > > > > > > > > > Hi Rafael, I check > > > > > > > > > > > > # ls -lrt > > > > > > total 285368382 > > > > > > -rw-r--r-- 1 root root 304124744192 Oct 22 03:28 > > > > > > 14818597-55bb-49be-9ace- 761e9e01c074.vhd > > > > > > -rw-r--r-- 1 root root 17770869248 Oct 23 01:10 > > > > > > bbd9e9b5-53fb-44a2-afbe- f262396a2d84.vhd > > > > > > -rw-r--r-- 1 root root 52199817728 Oct 24 01:30 > > > > > > 3b90bbb2-7ce5-41f6-9f7e- fd0ea061cb2d.vhd > > > > > > -rw-r--r-- 1 root root 51615670784 Oct 25 01:28 > > > > > > 4b450217-c9ad-45a2-946b- de3cb323469b.vhd > > > > > > > > > > > > # vhd-util scan -f -m'*.vhd' -p > > > > > > vhd=14818597-55bb-49be-9ace-761e9e01c074.vhd > > > > > > capacity=311385128960 > > > > > > size=304124744192 hidden=0 parent=none > > > > > > vhd=bbd9e9b5-53fb-44a2-afbe-f262396a2d84.vhd > > > > > > capacity=311385128960 > > > > > > size=17770869248 hidden=0 parent=none > > > > > > vhd=3b90bbb2-7ce5-41f6-9f7e-fd0ea061cb2d.vhd > > > > > > capacity=311385128960 > > > > > > size=52199817728 hidden=0 parent=none > > > > > > vhd=4b450217-c9ad-45a2-946b-de3cb323469b.vhd > > > > > > capacity=311385128960 > > > > > > size=51615670784 hidden=0 parent=none > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > From: Rafael Weingärtner > > > > > > [mailto:rafaelweingart...@gmail.com] > > > > > > Sent: quarta-feira, 25 de outubro de 2017 20:44 > > > > > > To: us...@cloudstack.apache.org > > > > > > Subject: Re: Strange size of template from snapshot on > > > > > > XenServer > > > > > > > > > > > > Did you check the hierarchy of those snapshot files in your > > > > > > secondary storage? > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Oct 25, 2017 at 7:44 PM, Lotic Lists > > > > > > <li...@lotic.com.br> > > > > wrote: > > > > > > > > > > > > > Hi all > > > > > > > > > > > > > > > > > > > > > > > > > > > > Anyone identify problems with templates from snapshots on > > > XenServer? > > > > > > > > > > > > > > I created a recurrent snapshot, first VHD on secondary > > > > > > > storage have a similar size of volume. If I create a > > > > > > > template from latest snapshot, the size of VHD of template > > > > > > > is smaller equal from last snapshot, I think > > > > > > > CloudStack/Xenserver not coalesce VHD files of secondary > > > > > > > storage for create the template > > > > > > > > > > > > > > > > > > > > > > > > > > > > I keep two recurrent snapshots, on secondary storage have > > > > > > > four > > > files. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Environment is ACS 4.9.3.0 and XenServer 6.5, primary > > > > > > > storage is PreSetup with ISCSI. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks > > > > > > > > > > > > > > Marcelo > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Rafael Weingärtner > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > Rafael Weingärtner > > > > > > > > > > > > > > > > > > > > > > -- > > > > Rafael Weingärtner > > > > > > > > > > > > > > > > > -- > > > Rafael Weingärtner > > > > > > > > > > > > -- > > Rafael Weingärtner > > > > > > > > > -- > Rafael Weingärtner > > -- Rafael Weingärtner
