Hi, Gluster Experts,
In glusterfs version 3.12.3, There seems a “fstat” issue for ctime after we use
fsync,
We have a demo execute binary which write some data and then do fsync for this
file, it named as “tt”,
Then run tar command right after “tt” command, it will always error with “tar:
/mnt/test/file1.txt: file changed as we read it”
The command output is list as the below, the source code and volume info
configuration attached FYI,
This issue will be 100% reproducible! (/mnt/test is mountpoint of glusterfs
volume “test” , which the volume info is attached in mail)
------------------------------------------------------------------
./tt;tar -czvf /tmp/abc.gz /mnt/test/file1.txt
mtime:1531247107.272000000
ctime:1531247107.272000000
tar: Removing leading `/' from member names
/mnt/test/file1.txt
tar: /mnt/test/file1.txt: file changed as we read it
----------------------------------------------------------------------
After my investigation, the xattrop for changelog is later than the fsync
response , this is mean:
In function “afr_fsync_cbk” will call afr_delayed_changelog_wake_resume (this,
local->fd, stub);
In our case, it always a pending changelog , so glusterfs save the metadata
information to stub, and handle pending changelog first,
But the changelog will also change the ctime, from the packet captured by
tcpdump, the response packet of xattrop will not include the metadata
information, and the wake_resume also not handle this metadata changed case.
So in this case, the metadata in mdc_cache is not right, and when cache is
valid, the application will get WRONG metadata!
For verify my guess, if I change the configuration for this volume
“gluster v set test md-cache-timeout 0” or
“gluster v set export stat-prefetch off”
This issue will be GONE!
And I restore the configuration to default, which mean stat-prefetch is on and
md-cache-timeout is 1 second,
I try invalidate the md-cache in source code as the below in function
mdc_fync_cbk on md-cache.c
The issue also will be GONE!
So GLusterFS Experts,
Could you please verify this issue, and share your comments on my investigation?
And your finally solutions is highly appreciated!
------------------------------------changes in function “mdc_fsync_cbk”
int
mdc_fsync_cbk (call_frame_t *frame, void *cookie, xlator_t *this,
int32_t op_ret, int32_t op_errno,
struct iatt *prebuf, struct iatt *postbuf, dict_t *xdata)
{
mdc_local_t *local = NULL;
local = frame->local;
if (op_ret != 0)
goto out;
if (!local)
goto out;
mdc_inode_iatt_set_validate(this, local->fd->inode, prebuf, postbuf,
_gf_true);
/* new added for ctime issue*/
mdc_inode_iatt_invalidate(this, local->fd->inode);
/* new added end*/
out:
MDC_STACK_UNWIND (fsync, frame, op_ret, op_errno, prebuf, postbuf,
xdata);
return 0;
}
---------------------------------------------------------------------------------
Best Regards,
George
#include <stdio.h>
#include <unistd.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <time.h>
#include <sys/time.h>
void main() {
char* fileName = "/mnt/test/file1.txt";
char buf[128];
struct stat st;
struct timeval tv_begin, tv_end;
// create and write a file, then fflush and fsync
FILE* stream = fopen(fileName,"w");
fwrite("0123456789", sizeof(char), 10, stream);
fflush(stream);
fsync(fileno(stream));
//fsync(stream);
fclose(stream);
// last file status change timestamp
stat(fileName, &st);
printf("mtime:%06d.%06d\n", st.st_mtim.tv_sec, st.st_mtim.tv_nsec);
printf("ctime:%06d.%06d\n", st.st_ctim.tv_sec, st.st_ctim.tv_nsec);
}
bash-4.4# gluster v info test
Volume Name: test
Type: Replicate
Volume ID: 9373eba9-eb84-4618-a54c-f2837345daec
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x (2 + 1) = 3
Transport-type: tcp
Bricks:
Brick1: rcp:/trunk/brick/test1/sn0
Brick2: rcp:/trunk/brick/test1/sn1
Brick3: rcp:/trunk/brick/test1/sn2 (arbiter)
Options Reconfigured:
performance.client-io-threads: off
nfs.disable: on
transport.address-family: inet
cluster.quorum-type: none
cluster.quorum-reads: no
cluster.favorite-child-policy: mtime
diagnostics.client-log-level: INFO
_______________________________________________
Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel
