On Thu, Jul 19, 2018 at 2:29 PM, Lian, George (NSB - CN/Hangzhou) < george.l...@nokia-sbell.com> wrote:
> Hi, Gluster Experts, > > > > In glusterfs version 3.12.3, There seems a “fstat” issue for ctime after > we use fsync, > > We have a demo execute binary which write some data and then do fsync for > this file, it named as “tt”, > > Then run tar command right after “tt” command, it will always error with > “tar: /mnt/test/file1.txt: file changed as we read it” > > > > The command output is list as the below, the source code and volume info > configuration attached FYI, > > This issue will be 100% reproducible! (/mnt/test is mountpoint of > glusterfs volume “test” , which the volume info is attached in mail) > > ------------------------------------------------------------------ > > ./tt;tar -czvf /tmp/abc.gz /mnt/test/file1.txt > > mtime:1531247107.272000000 > > ctime:1531247107.272000000 > > tar: Removing leading `/' from member names > > /mnt/test/file1.txt > > tar: /mnt/test/file1.txt: file changed as we read it > > ---------------------------------------------------------------------- > > > > After my investigation, the xattrop for changelog is later than the fsync > response , this is mean: > > In function “afr_fsync_cbk” will call afr_delayed_changelog_wake_resume > (this, local->fd, stub); > > > > In our case, it always a pending changelog , so glusterfs save the > metadata information to stub, and handle pending changelog first, > > But the changelog will also change the ctime, from the packet captured by > tcpdump, the response packet of xattrop will not include the metadata > information, and the wake_resume also not handle this metadata changed > case. > > > > So in this case, the metadata in mdc_cache is not right, and when cache is > valid, the application will get WRONG metadata! > > > > For verify my guess, if I change the configuration for this volume > > “gluster v set test md-cache-timeout 0” or > > “gluster v set export stat-prefetch off” > > This issue will be GONE! > We recently identified an issue with stat-prefetch. Fix can be found at: https://review.gluster.org/#/c/20410/11 Can you let us know whether this helps? > > > > And I restore the configuration to default, which mean stat-prefetch is on > and md-cache-timeout is 1 second, > > I try invalidate the md-cache in source code as the below in function > mdc_fync_cbk on md-cache.c > > The issue also will be GONE! > > > > So GLusterFS Experts, > > Could you please verify this issue, and share your comments on my > investigation? > > And your finally solutions is highly appreciated! > > Does the following fix you've posted solves the problem? > > ------------------------------------changes in function “mdc_fsync_cbk” > > int > > mdc_fsync_cbk (call_frame_t *frame, void *cookie, xlator_t *this, > > int32_t op_ret, int32_t op_errno, > > struct iatt *prebuf, struct iatt *postbuf, dict_t *xdata) > > { > > mdc_local_t *local = NULL; > > > > local = frame->local; > > > > if (op_ret != 0) > > goto out; > > > > if (!local) > > goto out; > > > > mdc_inode_iatt_set_validate(this, local->fd->inode, prebuf, > postbuf, > > _gf_true); > > /* new added for ctime issue*/ > > mdc_inode_iatt_invalidate(this, local->fd->inode); > > > /* new added end*/ > > out: > > MDC_STACK_UNWIND (fsync, frame, op_ret, op_errno, prebuf, postbuf, > > xdata); > > > > return 0; > > } > > ------------------------------------------------------------ > --------------------- > > Best Regards, > > George >
_______________________________________________ Gluster-devel mailing list Gluster-devel@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-devel