The underlying issue has existed forever AFAIK, but the issue has been masked since Infernalis due to the deep-flatten changes. If you were to use krbd or an older librbd client to write to the image, the same issue would appear (until the PR is merged and backported).
On Nov 30, 2016, at 1:00 AM, Zhongyan Gu <zhongyan...@gmail.com> wrote: Thanks for the quick confirm and fix. I reviewed the pull request. Seems that this fix is going into Jewel and Hammer. So does this mean this issue also exists in Jewel? why I test Jewel and cannot reproduce this issue?? Zhongyan On Wed, Nov 30, 2016 at 12:32 PM, Venky Shankar <vshan...@redhat.com> wrote: > On 16-11-30 12:07:33, Zhongyan Gu wrote: > > Jason, > > I test Jewel and confirmed Jewel has no such issue. > > Could you tell me what is the specific pull that can be backported to > > hammer to fix this issue?? > > PR: https://github.com/ceph/ceph/pull/12218 > > > > > Zhongyan > > > > On Tue, Nov 29, 2016 at 10:52 PM, Jason Dillaman <jdill...@redhat.com> > > wrote: > > > > > You are correct that there is an issue with the Hammer version when > > > calculating diffs. If the clone has an object that obscures an extent > > > within the parent image but didn't exist within the first snapshot of > > > the clone, the diff results from the parent image won't be included in > > > the result. I'll open a ticket against this issue, but in the meantime > > > I'd suggest trying a Jewel-based librbd client instead of Hammer. > > > > > > On Mon, Nov 28, 2016 at 10:18 PM, Zhongyan Gu <zhongyan...@gmail.com> > > > wrote: > > > > Thank you Jason. > > > > We are designing a backup system for production cluster based on > ceph's > > > > export /import diff feature. > > > > We found this issue and hopefully it can be confirmed and then fixed > > > soon. > > > > If you need any more information for debugging, Please just let me > know. > > > > > > > > Thanks, > > > > Zhongyan > > > > > > > > On Mon, Nov 28, 2016 at 10:05 PM, Jason Dillaman < > jdill...@redhat.com> > > > > wrote: > > > >> > > > >> OK, in that case, none of my previous explanation is relevant. I'll > > > >> spin up a hammer cluster and try to reproduce. > > > >> > > > >> On Wed, Nov 23, 2016 at 9:13 PM, Zhongyan Gu <zhongyan...@gmail.com > > > > > >> wrote: > > > >> > BTW, I used Hammer 0.94.5 to do the test. > > > >> > > > > >> > Zhongyan > > > >> > > > > >> > On Thu, Nov 24, 2016 at 10:07 AM, Zhongyan Gu < > zhongyan...@gmail.com> > > > >> > wrote: > > > >> >> > > > >> >> Thank you Jason. My test shows in the following case, image B > will be > > > >> >> exactly same: > > > >> >> 1. clone image A from parent: > > > >> >> #rbd clone 1124-parent@snap1 A > > > >> >> > > > >> >> 2. create snap for A > > > >> >> #rbd snap create A@snap1 > > > >> >> > > > >> >> 3. create empty image B > > > >> >> #rbd create B -s 1 > > > >> >> > > > >> >> 4. export-diff A then impor-diff B: > > > >> >> #rbd export-diff A@snap1 -|./rbd import-diff - B > > > >> >> > > > >> >> 5. check A@snap1 equals B@snap1 > > > >> >> #rbd export A@snap1 -|md5sum > > > >> >> Exporting image: 100% complete...done. > > > >> >> 880709d7352b6c9926beb1d829673366 - > > > >> >> #rbd export B@snap1 -|md5sum > > > >> >> Exporting image: 100% complete...done. > > > >> >> 880709d7352b6c9926beb1d829673366 - > > > >> >> output shows A@snap1 equals B@snap1 > > > >> >> > > > >> >> However, in the following case, image B will not be exactly same: > > > >> >> 1. clone image A from parent: > > > >> >> #rbd clone 1124-parent@snap1 A > > > >> >> > > > >> >> 2. create snap for A > > > >> >> #rbd snap create A@snap1 > > > >> >> > > > >> >> 3. use fio make some change to A > > > >> >> > > > >> >> 4. create empty image B > > > >> >> #rbd create B -s 1 > > > >> >> > > > >> >> 4. export-diff A then impor-diff B: > > > >> >> #rbd export-diff A@snap1 -|./rbd import-diff - B > > > >> >> > > > >> >> 5. check A@snap1 equals B@snap1 > > > >> >> #rbd export A@snap1 -|md5sum > > > >> >> Exporting image: 100% complete...done. > > > >> >> 880709d7352b6c9926beb1d829673366 - > > > >> >> #rbd export B@snap1 -|md5sum > > > >> >> Exporting image: 100% complete...done. > > > >> >> bbf7cf69a84f3978c66f5eb082fb91ec - > > > >> >> output shows A@snap1 DOES NOT equal B@snap1 > > > >> >> > > > >> >> The second case can always be reproduced. What is wrong with the > > > second > > > >> >> case? > > > >> >> > > > >> >> Thanks, > > > >> >> Zhongyan > > > >> >> > > > >> >> > > > >> >> On Wed, Nov 23, 2016 at 10:11 PM, Jason Dillaman < > > > jdill...@redhat.com> > > > >> >> wrote: > > > >> >>> > > > >> >>> What you are seeing sounds like a side-effect of deep-flatten > > > support. > > > >> >>> If you write to an unallocated extent within a cloned image, the > > > >> >>> associated object extent must be read from the parent image, > > > modified, > > > >> >>> and written to the clone image. > > > >> >>> > > > >> >>> Since the Infernalis release, this process has been tweaked if > the > > > >> >>> cloned image has a snapshot. In that case, the associated object > > > >> >>> extent is still read from the parent, but instead of being > modified > > > >> >>> and written to the HEAD revision, it is left unmodified and is > > > written > > > >> >>> to "pre" snapshot history followed by writing the original > > > >> >>> modification (w/o the parent's object extent data) to the HEAD > > > >> >>> revision. > > > >> >>> > > > >> >>> This change to the IO path was made to support flattening > clones and > > > >> >>> dissociating them from their parents even if the clone had > > > snapshots. > > > >> >>> > > > >> >>> Therefore, what you are seeing with export-diff is actually the > > > >> >>> backing object extent of data from the parent image written to > the > > > >> >>> clone's "pre" snapshot history. If you had two snapshots and > your > > > >> >>> export-diff'ed from the first to second snapshot, you wouldn't > see > > > >> >>> this extra data. > > > >> >>> > > > >> >>> To your question about how to prepare image B to make sure it > will > > > be > > > >> >>> exactly the same, the answer is that you don't need to do > anything. > > > In > > > >> >>> your example above, I am assuming you are manually creating an > empty > > > >> >>> Image B and using "import-diff" to populate it. The difference > in > > > the > > > >> >>> export-diff is most likely related to fact that the clone lost > its > > > >> >>> sparseness on any backing object that was written (e.g. instead > of a > > > >> >>> one or more 512 byte diffs within a backing object extent, you > will > > > >> >>> see a single, full-object extent with zeroes where the parent > image > > > >> >>> had no data). > > > >> >>> > > > >> >>> > > > >> >>> On Wed, Nov 23, 2016 at 5:06 AM, Zhongyan Gu < > zhongyan...@gmail.com > > > > > > > >> >>> wrote: > > > >> >>> > Let me make the issue more clear. > > > >> >>> > Suppose I cloned image A from a parent image and create snap1 > for > > > >> >>> > image > > > >> >>> > A > > > >> >>> > and then make some change of image A. > > > >> >>> > If I did the rbd export-diff @snap1. how should I prepare the > > > >> >>> > existing > > > >> >>> > image > > > >> >>> > B to make sure it will be exactly same with image A@snap1 > after > > > >> >>> > import-diff > > > >> >>> > against this image B. > > > >> >>> > > > > >> >>> > Thanks, > > > >> >>> > Zhongyan > > > >> >>> > > > > >> >>> > > > > >> >>> > On Wed, Nov 23, 2016 at 11:34 AM, Zhongyan Gu > > > >> >>> > <zhongyan...@gmail.com> > > > >> >>> > wrote: > > > >> >>> >> > > > >> >>> >> Thanks Jason, very clear explanation. > > > >> >>> >> However, I found some strange behavior when export-diff on a > > > cloned > > > >> >>> >> image, > > > >> >>> >> not sure it is a bug on calc_snap_set_diff(). > > > >> >>> >> The test is, > > > >> >>> >> Image A is cloned from a parent image. then create snap1 for > > > image > > > >> >>> >> A. > > > >> >>> >> The content of export-diff A@snap1 will be changed when > update > > > >> >>> >> image > > > >> >>> >> A. > > > >> >>> >> Only after image A has no overlap with parent, the content of > > > >> >>> >> export-diff > > > >> >>> >> A@snap1 is stabled, which is almost zero. > > > >> >>> >> I don't think it is a designed behavior. export-diff A@snap1 > > > should > > > >> >>> >> always > > > >> >>> >> get a stable output no matter image A is cloned or not. > > > >> >>> >> > > > >> >>> >> Please correct me if anything wrong. > > > >> >>> >> > > > >> >>> >> Thanks, > > > >> >>> >> Zhongyan > > > >> >>> >> > > > >> >>> >> > > > >> >>> >> > > > >> >>> >> > > > >> >>> >> On Tue, Nov 22, 2016 at 10:31 PM, Jason Dillaman > > > >> >>> >> <jdill...@redhat.com> > > > >> >>> >> wrote: > > > >> >>> >>> > > > >> >>> >>> On Tue, Nov 22, 2016 at 5:31 AM, Zhongyan Gu > > > >> >>> >>> <zhongyan...@gmail.com> > > > >> >>> >>> wrote: > > > >> >>> >>> > So if initial snapshot is NOT specified, then: > > > >> >>> >>> > rbd export-diff image@snap1 will diff all data to snap1. > this > > > >> >>> >>> > cmd > > > >> >>> >>> > equals to > > > >> >>> >>> > : > > > >> >>> >>> > rbd export image@snap1. Is my understand right or not?? > > > >> >>> >>> > > > >> >>> >>> > > > >> >>> >>> While they will both export all data associated w/ > image@snap1, > > > >> >>> >>> the > > > >> >>> >>> "export" command will generate a raw, non-sparse dump of the > > > full > > > >> >>> >>> image whereas "export-diff" will export only sections of the > > > image > > > >> >>> >>> that contain data. The file generated from "export" can be > used > > > >> >>> >>> with > > > >> >>> >>> the "import" command to create a new image, whereas the file > > > >> >>> >>> generated > > > >> >>> >>> from "export-diff" can only be used with "import-diff" > against > > > an > > > >> >>> >>> existing image. > > > >> >>> >>> > > > >> >>> >>> -- > > > >> >>> >>> Jason > > > >> >>> >> > > > >> >>> >> > > > >> >>> > > > > >> >>> > > > >> >>> > > > >> >>> > > > >> >>> -- > > > >> >>> Jason > > > >> >> > > > >> >> > > > >> > > > > >> > > > >> > > > >> > > > >> -- > > > >> Jason > > > > > > > > > > > > > > > > > > > > -- > > > Jason > > > > > > _______________________________________________ > > ceph-users mailing list > > ceph-users@lists.ceph.com > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > >
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
