Rafael, you seem to combine two things. Optimization and my fix are two
different commits. I mentioned about optimization (done in commit “8caf52c” not
by me) because that was broken by commit “8a3fd10”. Commit
“2c4ea503f92bcf9c611f409d5cdecb” only fixed the issue for newer versions of
XenServer. Probably you are just going through bug description and missing the
commit “8caf52c”.
Regards,
Anshul
On 10/11/17, 5:25 PM, "Rafael Weingärtner" <rafaelweingart...@gmail.com> wrote:
Anshul,
I will not touch the merit of documentation in code, for now, this is
another discussion. And regarding the “optimization”, you brought it up,
whenever I saw you discussing this issue, you would talk about some sort of
“optimization”… see
https://github.com/apache/cloudstack/pull/1124#issuecomment-326550060. At
least for me, that was quite confusing, and I was not seeing what you were
talking about.
If you diff the files, what broke your “solution” was the removal of the
following lines:
> EndPoint ep = endPointSelector.selectHypervisorHost(new ZoneScope(host.
> getDataCenterId()));
> host = hostDao.findById(ep.getId());
>
The conditional checks were maintained. We only removed something that we
seemed logicless.
With PR https://github.com/apache/cloudstack/pull/2315, I did not add
anything extra. I only re-worked the method to make it clear and more
readable (removed the if-inception). The only point I changed is that,
instead of using “selectOneHypervisorHostByZone”, I create a specific
method for that, which selects a host according to the volume of the
snapshot.
On Fri, Nov 10, 2017 at 8:21 AM, Anshul Gangwar <
anshul.gang...@accelerite.com> wrote:
> Rafael, you are still misunderstanding the fix “
> 2c4ea503f92bcf9c611f409d5cdecb”. That has only fixed the issue due to
> check for specific version i.e. before “2c4ea503f92bcf9c611f409d5cdecb”
> things were fine for xenserver 6.2 but not working for 6.5. Commit doesn’t
> mention anything about optimization. Other than version change it reverts
> the previous commit which is introducing hypervisor specific check in core
> code.
>
> Regarding documentation, “git history”is the best documentation. Code
> documentation may not make sense after some changes as it is often
ignored.
>
> See the PR for more comments.
>
> Regards,
> Anshul
>
> On 10/11/17, 3:14 AM, "Rafael Weingärtner" <rafaelweingart...@gmail.com>
> wrote:
>
> Anshul and Lotic, after debugging and inspecting code for quite a
> while I
> understood the complete picture and created a solution for it. The PR
> with
> the solution is found athttps://github.com/apache/cloudstack/pull/2315
> .
> This was a clear example of what I constantly tell people here, code
> needs
> to be clear, concise, well tested and documented. Here follows some
> explanation regarding the removal of some code introduced with commit
> “2c4ea503f92bcf9c611f409d5cdecb”.
>
> With commit “8caf52c” Daan and I removed a small piece of code
> introduced
> by “2c4ea503f92bcf9c611f409d5cdecb” because it did not seem to make
> much
> sense. We did not remove everything; we only removed the part that was
> looking for a random host in the zone to execute the command. As we
> (Anshul) exchanged some messages in some other PR, the code introduced
> in
> “2c4ea503f92bcf9c611f409d5cdecb” would only work for zones that do
> not have
> other hypervisor types if they are deployed with XenServer clusters.It
> was
> being created a limitation in ACS that should not exist. I stress
again
> that this only happened for the lack of documentation and clear
coding.
> For instance, when I read the commit “2c4ea503f92bcf9c611f409d5cdecb”,
> it
> says it introduced an “optimization”, then I assumed that the process
> executed with our code base before commit “
> 2c4ea503f92bcf9c611f409d5cdecb”
> is working, but not as fast or with as much quality as the code with
> “2c4ea503f92bcf9c611f409d5cdecb”. However, that is not the case; the
> code
> “2c4ea503f92bcf9c611f409d5cdecb” is not optimizing anything; it is, in
> fact, fixing/creating a workflow to create templates from snapshots in
> XenServer deployments.
>
> The first PR(#1176) intended to solve #CLOUDSTACK-9025 was only
> tackling
> the problem for CloudStack deployments that use single hypervisor
types
> (restricted to XenServers host in the same zone; this means, it was
not
> expecting to have multiple hypervisors types in the same zone).
> Additionally, the lack of information regarding that solution
> (documentation, test cases and description in PRs and Jira ticket) led
> the
> code to be removed in #1124 after a long discussion and analysis in
> #1056.
> That piece of code seemed logicless. It would receive a hostId and
> then
> change that hostId for other hostId of the zone without doing any
> check; it
> was not even checking the hypervisor and storage in which the host was
> connected to.
>
> The problem reported in #CLOUDSTACK-9025 is caused by partial
snapshots
> that are taken in XenServer. This means, we do not take a complete
> snapshot, but a partial one that contains only the modified data. This
> requires rebuilding the VHD hierarchy when creating a template out of
> the
> snapshot. The point is that the first hostId received is not a hostId,
> but
> a system VM ID(SSVM). That is why the code in #1176 fixed the problem
> for
> some deployment scenarios, but would cause problems for scenarios
> where we
> have multiple hypervisors in the same zone. We need to execute the
> creation
> of the VHD that represents the template in the hypervisor, so the VHD
> chain
> can be built using the parent links.
>
> The PR #2315 changes the behavior of
> com.cloud.hypervisor.XenServerGuru.getCommandHostDelegation(long,
> Command).
> From now on we replace the hostId that is intended to execute the
“copy
> command” that will create the VHD of the template according to some
> conditions that were already in place. The idea is that starting with
> XenServer 6.2.0 hotFix ESP1004 we need to execute the command in the
> hypervisor host and not from the SSVM. Moreover, the method was
> improved
> making it more readable; it was also created test cases assuring that
> from
> XenServer 6.2.0 hotFix ESP1004 and upward versions we change the
hostId
> that will be used to execute the “copy command”.
>
> Furthermore, we are not selecting a random host from a zone anymore. A
> new
> method was introduced in the HostDao object called
> “findHostConnectedToSnapshotStoragePoolToExecuteCommand”, using this
> method
> we look for a host that is in the cluster that is using the storage
> pool
> where the volume from which the Snapshot is taken of resides. By doing
> this, we guarantee that the host that is connected to the primary
> storage
> where all of the snapshots parent VHDs are stored is used to create
the
> template.
>
> Also, I opened the PR only against master. If someone needs the fix in
> any
> other version we can cherry pick to 4.9.2.0.
>
> On Wed, Nov 8, 2017 at 9:41 AM, Anshul Gangwar <
> anshul.gang...@accelerite.com> wrote:
>
> > Rafael, In ACS 4.5.2 you are facing issue due to XenServer version
> (was
> > only working on xenserver 6.2 with some patches) which got fixed in
> 4.6
> > with commit 2c4ea503f92bcf9c611f409d5cdecb42b0115b69. Now with code
> clean
> > up in commit “8a3fd10” it’s broken. During storage refactoring
> changes in
> > commit “8caf52c” optimization was done to send CopyCommand to
> Hypervisor
> > resource instead of SSVM. Now with commit “8a3fd10” it’s again going
> to
> > SSVM so it’s working fine for first snapshot but broken for linked
> snapshot.
> >
> > Regards,
> > Anshul
> >
> > On 04/11/17, 7:20 PM, "Rafael Weingärtner" <
> rafaelweingart...@gmail.com>
> > wrote:
> >
> > Anshul, I tried with ACS 4.5.2 where commit “8a3fd10” is not
> present.
> > This
> > is the commit where Daan and I removed some code that seemed
> (is? at
> > least
> > for me) logicless (https://github.com/apache/
> cloudstack/pull/1124 and
> > https://github.com/apache/cloudstack/pull/1056).
> >
> > I tested with ACS 4.5.2, which I have in a test environment. I
> have ACS
> > 4.5.2, NFS as the primary storage, and XenServer 6.5 as hosts. I
> > managed to
> > reproduce the problem as Lotic is having.
> >
> > I spin up a VM, then I created a file of 2GB in this VM, and
> later I
> > took a
> > snapshot of the root volume of it (the VM only has the root
> volume).
> > The
> > VHD “687f86f2-1aee-4e90-a64a-3b3c28376fcd.vhd” is the first
> snapshot
> > of the
> > root volume. If I create a template from this snapshot,
> everything
> > works. I
> > created a new VM from the template of the first snapshot, and it
> > worked,
> > and the 2GB file was there. Then, I returned for the first VM
> (from
> > which I
> > took the snapshot), I created another 1GB file. Then I stopped
> the VM
> > again
> > and took the second snapshot (27d37d21-13a6-488d-b2f0-
> > 0ac54236b83a.vhd).
> > The interesting thing here is that the file size reduced!? And I
> also
> > noticed that there is no hierarchy in the VHDs files. I tried to
> start
> > a VM
> > from a template that I created with the second snapshot, and it
> did not
> > work as the problem Lotic is having.
> >
> > [root@xenserver-1 ~]# ls -lah
> > > /var/cloud_mount/963e9072-0fa1-3448-bf48-ffde1f522104/
> snapshots/2/8/
> > > total 6.9G
> > > drwxr-xr-x 2 root root 4.0K Nov 4 10:33 .
> > > drwxr-xr-x 3 root root 4.0K Nov 4 09:49 ..
> > > -rw-r--r-- 1 root root 3.2G Nov 4 10:38
> > > 27d37d21-13a6-488d-b2f0-0ac54236b83a.vhd
> > > -rw-r--r-- 1 root root 3.7G Nov 4 09:55
> > > 687f86f2-1aee-4e90-a64a-3b3c28376fcd.vhd
> > > [root@xenserver-1 ~]# vhd-util scan -f -a -p
> > > /var/cloud_mount/963e9072-0fa1-3448-bf48-ffde1f522104/
> > snapshots/2/8/687f86f2-1aee-4e90-a64a-3b3c28376fcd.vhd
> > > vhd=687f86f2-1aee-4e90-a64a-3b3c28376fcd.vhd
> capacity=21474836480
> > > size=3916767744 hidden=0 parent=none
> > > [root@xenserver-1 ~]# vhd-util scan -f -a -p
> > > /var/cloud_mount/963e9072-0fa1-3448-bf48-ffde1f522104/
> > snapshots/2/8/27d37d21-13a6-488d-b2f0-0ac54236b83a.vhd
> > > vhd=27d37d21-13a6-488d-b2f0-0ac54236b83a.vhd
> capacity=21474836480
> > > size=3420873216 hidden=0 parent=none
> > >
> >
> >
> > Was it not working way before “8a3fd10”?! I think we will need
> help
> > from
> > more folks here to see if this is a problem that only affects
> XenServer
> > deployments. I will also test on my 4.9.2.0 environment again to
> see
> > if I
> > miss anything.
> >
> > On Thu, Nov 2, 2017 at 6:30 AM, Anshul Gangwar <
> > anshul.gang...@accelerite.com> wrote:
> >
> > > Hi Lotic,
> > >
> > > Can you try with any release between 4.6 to 4.8 and see if
> this bug
> > is
> > > there?
> > >
> > > Basically, any release which contains commit
> > 2c4ea503f92bcf9c611f409d5cdecb42b0115b69
> > > and missing commit 8a3fd10.
> > >
> > > On 02/11/17, 1:14 AM, "Lotic Lists" <li...@lotic.com.br>
> wrote:
> > >
> > > Rafael, I know and appreciate your help :)
> > >
> > > I recognize all effort of CloudStack community.
> > > I think all admins use snapshot to backup your virtual
> disks, I
> > think
> > > it is not a isolated problem. Imagine a crash on your storage
> and
> > you need
> > > convert your snapshots to templates.
> > >
> > > Thanks
> > > Marcelo
> > >
> > > -----Original Message-----
> > > From: Rafael Weingärtner [mailto:rafaelweingartner@
> gmail.com]
> > > Sent: quarta-feira, 1 de novembro de 2017 17:17
> > > To: dev@cloudstack.apache.org
> > > Subject: Re: Strange size of template from snapshot on
> XenServer
> > >
> > > Well, I could not reproduce the problem in ACS 4.9.2.0.
So,
> > before
> > > fixing anything, the person that might take on this should
> first
> > find the
> > > root of the problem.
> > >
> > > I understand what you are saying that it is a critical
> problem,
> > but
> > > you have to understand that this is an open-source and free
> software.
> > >
> > > Having said that, ALL of my work here has been pro-bono;
> even
> > though I
> > > am a committer and PMC, I am not willing to dive deeper into
> this
> > issue now
> > > as it does not affect me. It does not mean that I would not
> help
> > people if
> > > I have time, as you can notice whenever I have some free time
> I try
> > to be
> > > around and guide or check things for other people in this
list.
> > >
> > > If this is a huge problem for you right now, I suggest you
> > looking for
> > > companies that provide enterprise support for ACS.
> > >
> > > On Wed, Nov 1, 2017 at 7:35 PM, Lotic Lists <
> li...@lotic.com.br>
> > > wrote:
> > >
> > > > Hi Rafael
> > > >
> > > > Tested on ACS 4.5.2.2, 4.9.2.0, 4.9.3.0 and 4.10. All
> versions
> > with
> > > > same issue
> > > >
> > > > Description and steps to reproduce
> > > > https://issues.apache.org/jira/browse/CLOUDSTACK-10128
> > > >
> > > > Again, it is a critical problem for production
> environments.
> > > >
> > > > Regards
> > > > Marcelo
> > > >
> > > > -----Original Message-----
> > > > From: Rafael Weingärtner [mailto:rafaelweingartner@
> gmail.com]
> > > > Sent: terça-feira, 31 de outubro de 2017 17:10
> > > > To: dev@cloudstack.apache.org
> > > > Subject: Re: Strange size of template from snapshot on
> > XenServer
> > > >
> > > > Yes, it is a big problem. I have not checked the code
> that
> > creates
> > > the
> > > > template, but I am not sure if the problem you are
> > experiencing is
> > > > related to 9025. It might be a problem in some other
> place.
> > > >
> > > > My suggestion is for you to create a Jira ticket
> detailing the
> > > > situation and wait until someone fixes it. Have you
> tested on
> > ACS
> > > > 4.10? If 4.9.2.0 works, it might be something that went
> only
> > into
> > > 4.9.3.0.
> > > >
> > > > On Tue, Oct 31, 2017 at 8:04 PM, Lotic Lists <
> > li...@lotic.com.br>
> > > wrote:
> > > >
> > > > > I tested on a fresh environment, acs 4.9.3.0,
> xenserver 6.5
> > with
> > > all
> > > > > patches and primary storage NFS, we have same problem
> > > > >
> > > > > - Snapshot files
> > > > > [root@acs01 203]# ls -lh /exports/secondary/snapshots/
> 2/6/
> > total
> > > > > 2.8G
> > > > > -rw-r--r-- 1 root root 1.7G Oct 31 16:48
> > > > > cad96f0c-de49-4255-b0a0-9aea5e2297cb.vhd
> > > > > <-- snap-base
> > > > > -rw-r--r-- 1 root root 1.1G Oct 31 16:49
> > > > > fa4e7e4d-531d-42dd-8a80-df6b274584a8.vhd
> > > > > <-- snap-diff
> > > > > https://i.imgur.com/bWupsAS.png
> > > > >
> > > > > [root@acs01 203]# md5sum /exports/secondary/snapshots/
> > > > > 2/6/fa4e7e4d-531d-42dd-8a80-df6b274584a8.vhd
> > > > > 17eec8d4f6d4c128b34b9e2e1876ceb7
> > /exports/secondary/snapshots/
> > > > > 2/6/fa4e7e4d-531d-42dd-8a80-df6b274584a8.vhd
> > > > >
> > > > > - Template files
> > > > > [root@acs01 203]# ls -lh /exports/secondary/template/
> > tmpl/2/203
> > > > > total 2.0G
> > > > > -rw-r--r-- 1 root root 1.1G Oct 31 16:50
> > e9def45d-1b9e-4c2e-b843-
> > > > > 852fe40f00b2.vhd
> > > > > -rw-r--r-- 1 root root 303 Oct 31 16:50
> template.properties
> > > > >
> > > > > [root@acs01 203]# md5sum /exports/secondary/template/
> > > > > tmpl/2/203/e9def45d-1b9e-4c2e-b843-852fe40f00b2.vhd
> > > > > 17eec8d4f6d4c128b34b9e2e1876ceb7
> > /exports/secondary/template/
> > > > > tmpl/2/203/e9def45d-1b9e-4c2e-b843-852fe40f00b2.vhd
> > > > >
> > > > > It's a huge problem!!! I think issue 9025 is not
> resolved
> > > > >
> > > > > Regards
> > > > > Marcelo
> > > > >
> > > > > -----Original Message-----
> > > > > From: Rafael Weingärtner [mailto:rafaelweingartner@
> gmail.com
> > ]
> > > > > Sent: terça-feira, 31 de outubro de 2017 16:00
> > > > > To: us...@cloudstack.apache.org
> > > > > Subject: Re: Strange size of template from snapshot on
> > XenServer
> > > > >
> > > > > Well, reading the description of the issue it seems to
> be
> > related
> > > to
> > > > > the problem you are describing. However, I compared
the
> > classes
> > > that
> > > > > were changed with PR
> > > > > (https://github.com/apache/cloudstack/pull/1176)
> > > > > and they are the same in ACS 4.9.2.0 and 4.9.3.0.
> > > > >
> > > > > On Tue, Oct 31, 2017 at 6:36 PM, Lotic Lists <
> > li...@lotic.com.br>
> > > wrote:
> > > > >
> > > > > > Hi Rafael, thanks for test.
> > > > > >
> > > > > > I found the issue
> > > > > > https://issues.apache.org/
> jira/browse/CLOUDSTACK-9025
> > > > > > I think is a same case. I will up a new lab
> environment
> > now for
> > > > > > simulate the issue
> > > > > >
> > > > > > Regards.
> > > > > > Marcelo
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: Rafael Weingärtner [mailto:rafaelweingartner@
> > gmail.com]
> > > > > > Sent: segunda-feira, 30 de outubro de 2017 12:08
> > > > > > To: us...@cloudstack.apache.org
> > > > > > Subject: Re: Strange size of template from snapshot
> on
> > XenServer
> > > > > >
> > > > > > I just did, and the size of the template is the size
> of
> > the root
> > > > > > disk
> > > > > > + the 1GB file I created in the test vm.
> > > > > >
> > > > > > The system I tested this is an ACS 4.9.2, XenServer
> 6.5,
> > Primary
> > > > > > Stg ISCSI
> > > > > >
> > > > > > On Mon, Oct 30, 2017 at 10:20 AM, Lotic Lists <
> > > li...@lotic.com.br>
> > > > > wrote:
> > > > > >
> > > > > > > Good morning
> > > > > > >
> > > > > > > Guys, who could execute the test below?
> > > > > > >
> > > > > > > 1. Create a manual snapshot of a volume 2. Create
> a file
> > (1GB)
> > > > > > > with dd on VM 3. Create second snapshot of a
> volume 4.
> > Convert
> > > > > > > latest snapshot to a template 5. Verify the size
> of vhd
> > of new
> > > > > > > template in secondary storage.
> > > > > > >
> > > > > > > Here the template have 1GB, exactly the size of
> latest
> > > snapshot.
> > > > > > >
> > > > > > > ACS 4.9.3, XenServer 6.5, Primary Stg ISCSI
> > > > > > >
> > > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: Lotic Lists [mailto:li...@lotic.com.br]
> > > > > > > Sent: quinta-feira, 26 de outubro de 2017 15:03
> > > > > > > To: us...@cloudstack.apache.org
> > > > > > > Subject: RE: Strange size of template from
> snapshot on
> > > XenServer
> > > > > > >
> > > > > > > I test the same case on other environment, create
> > template from
> > > > > > > snapshot, acs 4.9.2.0 and XenServer 6.5/iscsi
> > > > > > >
> > > > > > > 1. Was created two snapshots
> > > > > > >
> > > > > > > # ls -lh /var/cloud_mount/e5921d9a-
> > > 70b0-3501-9fda-6097e721fffb/
> > > > > > > snapshots/131/617/
> > > > > > > total 3.0G
> > > > > > > -rw-r--r-- 1 4294967294 root 13G Oct 26 13:46
> > > > > > > 42c7c51d-4aa7-40d2-b5eb- cb56fc963974.vhd
> > > > > > > -rw-r--r-- 1 4294967294 root 1.2G Oct 26 14:42
> > > > > > > 53b33a5a-a7d9-4efa-b126- 38eec5482b05.vhd
> > > > > > >
> > > > > > > https://i.imgur.com/cSHZAJ3.png
> > > > > > >
> > > > > > > 2. create a template from latest snapshot
> > > > > > >
> > > > > > > # ls -lh
> > > > > > > total 37M
> > > > > > > -rw-r--r-- 1 4294967294 root 1.2G Oct 26 14:43
> > > > > > > 637604a6-3085-444e-ba09- b9aab11a2c16.vhd
> > > > > > > -rw-r--r-- 1 4294967294 root 303 Oct 26 14:43
> > > > > > > template.properties
> > > > > > >
> > > > > > > # cat template.properties
> > > > > > > #Thu Oct 26 16:43:54 UTC 2017
> > > > > > > filename=637604a6-3085-444e-ba09-b9aab11a2c16.vhd
> > > > > > > id=1
> > > > > > > vhd=true
> > > > > > > vhd.filename=637604a6-3085-
> 444e-ba09-b9aab11a2c16.vhd
> > > > > > > public=true
> > > > > > > uniquename=637604a6-3085-444e-ba09-b9aab11a2c16
> > > > > > > vhd.virtualsize=21474836480
> > > > > > > virtualsize=21474836480
> > > > > > > hvm=
> > > > > > > vhd.size=1197752832
> > > > > > > size=1197752832
> > > > > > >
> > > > > > > # md5sum /var/cloud_mount/e5921d9a-
> > > 70b0-3501-9fda-6097e721fffb/
> > > > > > > snapshots/131/617/53b33a5a-
> a7d9-4efa-b126-38eec5482b05.
> > vhd
> > > > > > > 9d2078a6f41ca5e1eab824848c06df53
> > /var/cloud_mount/e5921d9a-
> > > > > > > 70b0-3501-9fda-6097e721fffb/
> snapshots/131/617/53b33a5a-
> > > > > > > a7d9-4efa-b126-38eec5482b05.vhd
> > > > > > >
> > > > > > > # md5sum /var/cloud_mount/e5921d9a-
> > > 70b0-3501-9fda-6097e721fffb/
> > > > > > > template/tmpl/2/294/637604a6-
> > 3085-444e-ba09-b9aab11a2c16.vhd
> > > > > > > 9d2078a6f41ca5e1eab824848c06df53
> > /var/cloud_mount/e5921d9a-
> > > > > > > 70b0-3501-9fda-6097e721fffb/
> > template/tmpl/2/294/637604a6-
> > > > > > > 3085-444e-ba09-b9aab11a2c16.vhd
> > > > > > >
> > > > > > > ACS not merge files of snapshots in a single file
> for
> > template,
> > > > > > > I think it's a bug.
> > > > > > >
> > > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: Rafael Weingärtner [mailto:
> rafaelweingartner@
> > gmail.com]
> > > > > > > Sent: quarta-feira, 25 de outubro de 2017 22:51
> > > > > > > To: us...@cloudstack.apache.org
> > > > > > > Subject: Re: Strange size of template from
> snapshot on
> > > XenServer
> > > > > > >
> > > > > > > It is easy to test, get one of these files and try
> to
> > use them
> > > > > > > in Virtualbox. If they run, they do not require
> anything
> > else.
> > > > > > >
> > > > > > > On Wed, Oct 25, 2017 at 10:49 PM, Lotic Lists
> > > > > > > <li...@lotic.com.br>
> > > > > > wrote:
> > > > > > >
> > > > > > > > Sure but files on secondary storage don't have
> > parents, I
> > > > > > > > don't know if ACS can connect standalone files.
> > > > > > > >
> > > > > > > > vhd-util scan -f -m'*.vhd' -p
> > > > > > > > vhd=14818597-55bb-49be-9ace-761e9e01c074.vhd
> > > > > > > > capacity=311385128960
> > > > > > > > size=304124744192 hidden=0 parent=none
> > > > > > > > vhd=bbd9e9b5-53fb-44a2-afbe-f262396a2d84.vhd
> > > > > > > > capacity=311385128960
> > > > > > > > size=17770869248 hidden=0 parent=none
> > > > > > > > vhd=3b90bbb2-7ce5-41f6-9f7e-fd0ea061cb2d.vhd
> > > > > > > > capacity=311385128960
> > > > > > > > size=52199817728 hidden=0 parent=none
> > > > > > > > vhd=4b450217-c9ad-45a2-946b-de3cb323469b.vhd
> > > > > > > > capacity=311385128960
> > > > > > > > size=51615670784 hidden=0 parent=none
> > > > > > > >
> > > > > > > >
> > > > > > > > -----Original Message-----
> > > > > > > > From: Rafael Weingärtner [mailto:
> rafaelweingartner@
> > gmail.com
> > > ]
> > > > > > > > Sent: quarta-feira, 25 de outubro de 2017 22:35
> > > > > > > > To: us...@cloudstack.apache.org
> > > > > > > > Subject: Re: Strange size of template from
> snapshot on
> > > > > > > > XenServer
> > > > > > > >
> > > > > > > > If it was not executing coalesce you would see
> the
> > parent
> > > > > > > > reference when you listed the hierarchy, right?
> > > > > > > >
> > > > > > > > On Wed, Oct 25, 2017 at 10:25 PM, Lotic Lists
> > > > > > > > <li...@lotic.com.br>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > I think ACS preserve all VHDs for coalesce but
> > coalesce not
> > > > > > > > > occur when create a template or before
> starting next
> > > > > > > > > snapshot
> > > > > scheduled.
> > > > > > > > > If I create a template from any snapshots (2)
> > showing in
> > > > > > > > > GUI, the template have 49GB. 49 GB is the size
> of
> > the 2
> > > > > > > > > latest files on secondary
> > > > > > > > storage.
> > > > > > > > >
> > > > > > > > > What do you think? Bug?
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > -----Original Message-----
> > > > > > > > > From: Rafael Weingärtner
> > > > > > > > > [mailto:rafaelweingart...@gmail.com]
> > > > > > > > > Sent: quarta-feira, 25 de outubro de 2017
22:07
> > > > > > > > > To: us...@cloudstack.apache.org
> > > > > > > > > Subject: Re: Strange size of template from
> snapshot
> > on
> > > > > > > > > XenServer
> > > > > > > > >
> > > > > > > > > Aha, The two destroyed entries are showing "
> removed:
> > > NULL".
> > > > > > > > >
> > > > > > > > > In ACS one thing is the "destroying" of a
> resource,
> > another
> > > > > > > > > is the removal of this resource from the
> system. I
> > believe
> > > > > > > > > there is something to do with the expunge
> interval,
> > but I
> > > am
> > > > > > > > > not sure if the snapshots complete removal
also
> > happens
> > > > > > > > > within the expunge interval of user VMs or if
> it is
> > > > > > > > > configured by something
> > > > else.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Wed, Oct 25, 2017 at 10:03 PM, Lotic Lists
> > > > > > > > > <li...@lotic.com.br>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > I not start manual snapshot, just scheduled
> with
> > keep=2
> > > > > > > > > >
> > > > > > > > > > GUI showing 2 snapshots, database showing 4
> > entries, 2
> > > > > > > > > > Destroyed and
> > > > > > > > > > 2 BackedUp
> > > > > > > > > >
> > > > > > > > > > *************************** 132. row
> > > > ***************************
> > > > > > > > > > id: 6380
> > > > > > > > > > data_center_id: 2
> > > > > > > > > > account_id: 64
> > > > > > > > > > domain_id: 49
> > > > > > > > > > volume_id: 2102
> > > > > > > > > > disk_offering_id: 230
> > > > > > > > > > status: Destroyed
> > > > > > > > > > path: NULL
> > > > > > > > > > name: FullCRM_ROOT-542_
> 20171022030205
> > > > > > > > > > uuid: 1294cdb7-01bf-4e76-b4c0-
> > 11b3c6b395e9
> > > > > > > > > > snapshot_type: 4
> > > > > > > > > > type_description: DAILY
> > > > > > > > > > size: 311385128960
> > > > > > > > > > created: 2017-10-22 03:02:05
> > > > > > > > > > removed: NULL
> > > > > > > > > > backup_snap_id: NULL
> > > > > > > > > > swift_id: NULL
> > > > > > > > > > sechost_id: NULL
> > > > > > > > > > prev_snap_id: NULL
> > > > > > > > > > hypervisor_type: XenServer
> > > > > > > > > > version: 2.2
> > > > > > > > > > s3_id: NULL
> > > > > > > > > > min_iops: NULL
> > > > > > > > > > max_iops: NULL
> > > > > > > > > > *************************** 133. row
> > > > ***************************
> > > > > > > > > > id: 6416
> > > > > > > > > > data_center_id: 2
> > > > > > > > > > account_id: 64
> > > > > > > > > > domain_id: 49
> > > > > > > > > > volume_id: 2102
> > > > > > > > > > disk_offering_id: 230
> > > > > > > > > > status: Destroyed
> > > > > > > > > > path: NULL
> > > > > > > > > > name: FullCRM_ROOT-542_
> 20171023030205
> > > > > > > > > > uuid: 0a8067a3-ad15-40b6-8b16-
> > c1881d7d9f41
> > > > > > > > > > snapshot_type: 4
> > > > > > > > > > type_description: DAILY
> > > > > > > > > > size: 311385128960
> > > > > > > > > > created: 2017-10-23 03:02:05
> > > > > > > > > > removed: NULL
> > > > > > > > > > backup_snap_id: NULL
> > > > > > > > > > swift_id: NULL
> > > > > > > > > > sechost_id: NULL
> > > > > > > > > > prev_snap_id: NULL
> > > > > > > > > > hypervisor_type: XenServer
> > > > > > > > > > version: 2.2
> > > > > > > > > > s3_id: NULL
> > > > > > > > > > min_iops: NULL
> > > > > > > > > > max_iops: NULL
> > > > > > > > > > *************************** 134. row
> > > > ***************************
> > > > > > > > > > id: 6455
> > > > > > > > > > data_center_id: 2
> > > > > > > > > > account_id: 64
> > > > > > > > > > domain_id: 49
> > > > > > > > > > volume_id: 2102
> > > > > > > > > > disk_offering_id: 230
> > > > > > > > > > status: BackedUp
> > > > > > > > > > path: NULL
> > > > > > > > > > name: FullCRM_ROOT-542_
> 20171024030207
> > > > > > > > > > uuid: 1ca646e5-a155-4a59-aa31-
> > 5889eec7536f
> > > > > > > > > > snapshot_type: 4
> > > > > > > > > > type_description: DAILY
> > > > > > > > > > size: 311385128960
> > > > > > > > > > created: 2017-10-24 03:02:07
> > > > > > > > > > removed: NULL
> > > > > > > > > > backup_snap_id: NULL
> > > > > > > > > > swift_id: NULL
> > > > > > > > > > sechost_id: NULL
> > > > > > > > > > prev_snap_id: NULL
> > > > > > > > > > hypervisor_type: XenServer
> > > > > > > > > > version: 2.2
> > > > > > > > > > s3_id: NULL
> > > > > > > > > > min_iops: NULL
> > > > > > > > > > max_iops: NULL
> > > > > > > > > > *************************** 135. row
> > > > ***************************
> > > > > > > > > > id: 6497
> > > > > > > > > > data_center_id: 2
> > > > > > > > > > account_id: 64
> > > > > > > > > > domain_id: 49
> > > > > > > > > > volume_id: 2102
> > > > > > > > > > disk_offering_id: 230
> > > > > > > > > > status: BackedUp
> > > > > > > > > > path: NULL
> > > > > > > > > > name: FullCRM_ROOT-542_
> 20171025030208
> > > > > > > > > > uuid: 759d61dc-e3db-4815-b68b-
> > b90904ccbda5
> > > > > > > > > > snapshot_type: 4
> > > > > > > > > > type_description: DAILY
> > > > > > > > > > size: 311385128960
> > > > > > > > > > created: 2017-10-25 03:02:08
> > > > > > > > > > removed: NULL
> > > > > > > > > > backup_snap_id: NULL
> > > > > > > > > > swift_id: NULL
> > > > > > > > > > sechost_id: NULL
> > > > > > > > > > prev_snap_id: NULL
> > > > > > > > > > hypervisor_type: XenServer
> > > > > > > > > > version: 2.2
> > > > > > > > > > s3_id: NULL
> > > > > > > > > > min_iops: NULL
> > > > > > > > > > max_iops: NULL
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > -----Original Message-----
> > > > > > > > > > From: Rafael Weingärtner
> > > > > > > > > > [mailto:rafaelweingart...@gmail.com]
> > > > > > > > > > Sent: quarta-feira, 25 de outubro de 2017
> 21:50
> > > > > > > > > > To: us...@cloudstack.apache.org
> > > > > > > > > > Subject: Re: Strange size of template from
> > snapshot on
> > > > > > > > > > XenServer
> > > > > > > > > >
> > > > > > > > > > The file "14818597-55bb-49be-9ace-
> > 761e9e01c074.vhd"
> > > seems
> > > > > > > > > > more like a manual snapshot. It was created
2
> > hours after
> > > > > > > > > > the
> > > > "normal"
> > > > > > > > > > time. I would check these snapshots in ACS
> and see
> > how
> > > > > > > > > > they are
> > > > > > > presented.
> > > > > > > > > > BTW, how many snapshots is ACS showing for
> this
> > volume? I
> > > > > > > > > > would also check the "snapshots" table for
> the
> > volume you
> > > > > > > > > > have configured
> > > > > > > them.
> > > > > > > > > > select * from snapshots where volume_id = ?
> > > > > > > > > >
> > > > > > > > > > you may also filter for removed:
> > > > > > > > > > select * from snapshots where volume_id = ?
> and
> > removed
> > > is
> > > > > > > > > > null
> > > > > > > > > >
> > > > > > > > > > On Wed, Oct 25, 2017 at 9:39 PM, Lotic Lists
> > > > > > > > > > <li...@lotic.com.br>
> > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Rafael, I check
> > > > > > > > > > >
> > > > > > > > > > > # ls -lrt
> > > > > > > > > > > total 285368382
> > > > > > > > > > > -rw-r--r-- 1 root root 304124744192 Oct 22
> 03:28
> > > > > > > > > > > 14818597-55bb-49be-9ace- 761e9e01c074.vhd
> > > > > > > > > > > -rw-r--r-- 1 root root 17770869248 Oct 23
> 01:10
> > > > > > > > > > > bbd9e9b5-53fb-44a2-afbe- f262396a2d84.vhd
> > > > > > > > > > > -rw-r--r-- 1 root root 52199817728 Oct 24
> 01:30
> > > > > > > > > > > 3b90bbb2-7ce5-41f6-9f7e- fd0ea061cb2d.vhd
> > > > > > > > > > > -rw-r--r-- 1 root root 51615670784 Oct 25
> 01:28
> > > > > > > > > > > 4b450217-c9ad-45a2-946b- de3cb323469b.vhd
> > > > > > > > > > >
> > > > > > > > > > > # vhd-util scan -f -m'*.vhd' -p
> > > > > > > > > > > vhd=14818597-55bb-49be-9ace-
> 761e9e01c074.vhd
> > > > > > > > > > > capacity=311385128960
> > > > > > > > > > > size=304124744192 hidden=0 parent=none
> > > > > > > > > > > vhd=bbd9e9b5-53fb-44a2-afbe-
> f262396a2d84.vhd
> > > > > > > > > > > capacity=311385128960
> > > > > > > > > > > size=17770869248 hidden=0 parent=none
> > > > > > > > > > > vhd=3b90bbb2-7ce5-41f6-9f7e-
> fd0ea061cb2d.vhd
> > > > > > > > > > > capacity=311385128960
> > > > > > > > > > > size=52199817728 hidden=0 parent=none
> > > > > > > > > > > vhd=4b450217-c9ad-45a2-946b-
> de3cb323469b.vhd
> > > > > > > > > > > capacity=311385128960
> > > > > > > > > > > size=51615670784 hidden=0 parent=none
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > -----Original Message-----
> > > > > > > > > > > From: Rafael Weingärtner
> > > > > > > > > > > [mailto:rafaelweingart...@gmail.com]
> > > > > > > > > > > Sent: quarta-feira, 25 de outubro de 2017
> 20:44
> > > > > > > > > > > To: us...@cloudstack.apache.org
> > > > > > > > > > > Subject: Re: Strange size of template from
> > snapshot on
> > > > > > > > > > > XenServer
> > > > > > > > > > >
> > > > > > > > > > > Did you check the hierarchy of those
> snapshot
> > files in
> > > > > > > > > > > your secondary storage?
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Wed, Oct 25, 2017 at 7:44 PM, Lotic
> Lists
> > > > > > > > > > > <li...@lotic.com.br>
> > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi all
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Anyone identify problems with templates
> from
> > > snapshots
> > > > > > > > > > > > on
> > > > > > > > XenServer?
> > > > > > > > > > > >
> > > > > > > > > > > > I created a recurrent snapshot, first
> VHD on
> > > secondary
> > > > > > > > > > > > storage have a similar size of volume.
> If I
> > create a
> > > > > > > > > > > > template from latest snapshot, the size
> of VHD
> > of
> > > > > > > > > > > > template is smaller equal from last
> snapshot,
> > I think
> > > > > > > > > > > > CloudStack/Xenserver not coalesce VHD
> files of
> > > > > > > > > > > > secondary storage for create the
template
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > I keep two recurrent snapshots, on
> secondary
> > storage
> > > > > > > > > > > > have four
> > > > > > > > files.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Environment is ACS 4.9.3.0 and XenServer
> 6.5,
> > primary
> > > > > > > > > > > > storage is PreSetup with ISCSI.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks
> > > > > > > > > > > >
> > > > > > > > > > > > Marcelo
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > > Rafael Weingärtner
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > Rafael Weingärtner
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Rafael Weingärtner
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Rafael Weingärtner
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Rafael Weingärtner
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Rafael Weingärtner
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Rafael Weingärtner
> > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > Rafael Weingärtner
> > > >
> > > >
> > >
> > >
> > > --
> > > Rafael Weingärtner
> > >
> > >
> > >
> > > DISCLAIMER
> > > ==========
> > > This e-mail may contain privileged and confidential
information
> > which is
> > > the property of Accelerite, a Persistent Systems business. It
> is
> > intended
> > > only for the use of the individual or entity to which it is
> > addressed. If
> > > you are not the intended recipient, you are not authorized to
> read,
> > retain,
> > > copy, print, distribute or use this message. If you have
> received
> > this
> > > communication in error, please notify the sender and delete
all
> > copies of
> > > this message. Accelerite, a Persistent Systems business does
> not
> > accept any
> > > liability for virus infected mails.
> > >
> >
> >
> >
> > --
> > Rafael Weingärtner
> >
> >
> > DISCLAIMER
> > ==========
> > This e-mail may contain privileged and confidential information
> which is
> > the property of Accelerite, a Persistent Systems business. It is
> intended
> > only for the use of the individual or entity to which it is
> addressed. If
> > you are not the intended recipient, you are not authorized to read,
> retain,
> > copy, print, distribute or use this message. If you have received
> this
> > communication in error, please notify the sender and delete all
> copies of
> > this message. Accelerite, a Persistent Systems business does not
> accept any
> > liability for virus infected mails.
> >
>
>
>
> --
> Rafael Weingärtner
>
>
> DISCLAIMER
> ==========
> This e-mail may contain privileged and confidential information which is
> the property of Accelerite, a Persistent Systems business. It is intended
> only for the use of the individual or entity to which it is addressed. If
> you are not the intended recipient, you are not authorized to read,
retain,
> copy, print, distribute or use this message. If you have received this
> communication in error, please notify the sender and delete all copies of
> this message. Accelerite, a Persistent Systems business does not accept
any
> liability for virus infected mails.
>
--
Rafael Weingärtner
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the
property of Accelerite, a Persistent Systems business. It is intended only for
the use of the individual or entity to which it is addressed. If you are not
the intended recipient, you are not authorized to read, retain, copy, print,
distribute or use this message. If you have received this communication in
error, please notify the sender and delete all copies of this message.
Accelerite, a Persistent Systems business does not accept any liability for
virus infected mails.
