Re: [Gluster-devel] The ctime of fstat is not correct which lead to "tar" utility error
On Mon, Jul 23, 2018 at 8:40 AM, Lian, George (NSB - CN/Hangzhou) < george.l...@nokia-sbell.com> wrote: > Hi, > > > > I tested both patchset1 and patchset2 of https://review.gluster.org/20549, > the ctime issue seems both be there. > > And I use my test c program and “dd” program, the issue both be there. > Strange. Tests on my laptop succeeded. > > But when use the patch of https://review.gluster.org/#/c/20410/11, > > My test C program and “dd” to an exist file will pass, > > ONLY “dd” to new file will be failed. > Thanks. I'll think more about this. > > Best Regards, > > George > > > > > > > > *From:* gluster-devel-boun...@gluster.org [mailto:gluster-devel-bounces@ > gluster.org] *On Behalf Of *Raghavendra Gowdappa > *Sent:* Monday, July 23, 2018 10:37 AM > > *To:* Lian, George (NSB - CN/Hangzhou) > *Cc:* Zhang, Bingxuan (NSB - CN/Hangzhou) ; > Raghavendra G ; Gluster-devel@gluster.org > *Subject:* Re: [Gluster-devel] The ctime of fstat is not correct which > lead to "tar" utility error > > > > > > > > On Sun, Jul 22, 2018 at 1:41 PM, Raghavendra Gowdappa > wrote: > > George, > > Sorry. I sent you a version of the fix which was stale. Can you try with: > https://review.gluster.org/20549 > > This patch passes the test case you've given. > > > > Patchset 1 solves this problem. However, it ran into dbench failures as > md-cache was slow to update its cache. Once I fixed it, I am seeing > failures again. with performance.stat-prefetch off, the error goes away. > But, I can see only ctime changes. Wondering whether this is related to > ctime translator or an issue in md-cache. Note that md-cache caches stats > from codepaths which don't result in stat updation in kernel too. So, it > could be either, > > * a bug in md-cache > > * or a bug where in those codepaths wrong/changed stat was sent. > > > > I'll probe the first hypothesis. @Pranith/@Ravi, > > > > What do you think about second hypothesis? > > > > regards, > > Raghavendra > > > > regards, > > Raghavendra > > > > On Fri, Jul 20, 2018 at 2:59 PM, Lian, George (NSB - CN/Hangzhou) < > george.l...@nokia-sbell.com> wrote: > > Hi, > > > > Sorry, there seems still have issue. > > > > We use “dd” application of linux tools instead of my demo program, and if > the file is not exist before dd, the issue still be there. > > > > The test command is > > rm -rf /mnt/test/file.txt ; dd if=/dev/zero of=/mnt/test/file.txt bs=512 > count=1 oflag=sync;stat /mnt/test/file.txt;tar -czvf /tmp/abc.gz > > 1) If we set md-cache-timeout to 0, the issue will not happen > > 2) If we set md-cache-timeout to 1, the issue will 100% reproduced! > (with new patch you mentioned in the mail) > > > > Please see detail test result as the below: > > > > bash-4.4# gluster v set export md-cache-timeout 0 > > volume set: failed: Volume export does not exist > > bash-4.4# gluster v set test md-cache-timeout 0 > > volume set: success > > bash-4.4# dd if=/dev/zero of=/mnt/test/file.txt bs=512 count=1 > oflag=sync;stat /mnt/test/file.txt;tar -czvf /tmp/abc.gz > /mnt/test/file.txt;stat /mnt/test/file.txt^C > > bash-4.4# rm /mnt/test/file.txt > > bash-4.4# dd if=/dev/zero of=/mnt/test/file.txt bs=512 count=1 > oflag=sync;stat /mnt/test/file.txt;tar -czvf /tmp/abc.gz > /mnt/test/file.txt;stat /mnt/test/file.txt > > 1+0 records in > > 1+0 records out > > 512 bytes copied, 0.00932571 s, 54.9 kB/s > > File: /mnt/test/file.txt > > Size: 512 Blocks: 1 IO Block: 131072 regular file > > Device: 33h/51d Inode: 9949244856126716752 Links: 1 > > Access: (0644/-rw-r--r--) Uid: (0/root) Gid: (0/root) > > Access: 2018-07-13 17:55:02.75600 + > > Modify: 2018-07-13 17:55:02.76400 + > > Change: 2018-07-13 17:55:02.76800 + > > Birth: - > > tar: Removing leading `/' from member names > > /mnt/test/file.txt > > File: /mnt/test/file.txt > > Size: 512 Blocks: 1 IO Block: 131072 regular file > > Device: 33h/51d Inode: 9949244856126716752 Links: 1 > > Access: (0644/-rw-r--r--) Uid: (0/root) Gid: (0/root) > > Access: 2018-07-13 17:55:02.77600 + > > Modify: 2018-07-13 17:55:02.76400 + > > Change: 2018-07-13 17:55:02.76800 + > > Birth: - > > bash-4.4# gluster v set test md-cache-timeout 1 > > volume set: success > > bash-4.4# rm /mnt/test/file.txt > > bash-4.4# dd if=/dev/zero of=/mnt/test/file.txt bs=512 count=1 > oflag=sync;stat /mnt/test/file.txt;tar -czvf /tmp/abc.gz > /mnt/test/file.txt;stat /mnt/test/file.txt > > 1+0 records in > > 1+0 records out > > 512 bytes copied, 0.0107589 s, 47.6 kB/s > > File: /mnt/test/file.txt > > Size: 512 Blocks: 1 IO Block: 131072 regular file > > Device: 33h/51d Inode: 13569976446871695205 Links: 1 > > Access: (0644/-rw-r--r--) Uid: (0/root) Gid: (0/root) > > Access: 2018-07-13 17:55:11.54800 + > > Modify: 2018-07-13 17:55:11.56000 + > > Change: 2018-07-13 17:55:11.56000
Re: [Gluster-devel] The ctime of fstat is not correct which lead to "tar" utility error
On Sun, Jul 22, 2018 at 1:41 PM, Raghavendra Gowdappa wrote: > George, > > Sorry. I sent you a version of the fix which was stale. Can you try with: > https://review.gluster.org/20549 > > This patch passes the test case you've given. > Patchset 1 solves this problem. However, it ran into dbench failures as md-cache was slow to update its cache. Once I fixed it, I am seeing failures again. with performance.stat-prefetch off, the error goes away. But, I can see only ctime changes. Wondering whether this is related to ctime translator or an issue in md-cache. Note that md-cache caches stats from codepaths which don't result in stat updation in kernel too. So, it could be either, * a bug in md-cache * or a bug where in those codepaths wrong/changed stat was sent. I'll probe the first hypothesis. @Pranith/@Ravi, What do you think about second hypothesis? regards, Raghavendra > > regards, > Raghavendra > > On Fri, Jul 20, 2018 at 2:59 PM, Lian, George (NSB - CN/Hangzhou) < > george.l...@nokia-sbell.com> wrote: > >> Hi, >> >> >> >> Sorry, there seems still have issue. >> >> >> >> We use “dd” application of linux tools instead of my demo program, and >> if the file is not exist before dd, the issue still be there. >> >> >> >> The test command is >> >> rm -rf /mnt/test/file.txt ; dd if=/dev/zero of=/mnt/test/file.txt bs=512 >> count=1 oflag=sync;stat /mnt/test/file.txt;tar -czvf /tmp/abc.gz >> >> 1) If we set md-cache-timeout to 0, the issue will not happen >> >> 2) If we set md-cache-timeout to 1, the issue will 100% reproduced! >> (with new patch you mentioned in the mail) >> >> >> >> Please see detail test result as the below: >> >> >> >> bash-4.4# gluster v set export md-cache-timeout 0 >> >> volume set: failed: Volume export does not exist >> >> bash-4.4# gluster v set test md-cache-timeout 0 >> >> volume set: success >> >> bash-4.4# dd if=/dev/zero of=/mnt/test/file.txt bs=512 count=1 >> oflag=sync;stat /mnt/test/file.txt;tar -czvf /tmp/abc.gz >> /mnt/test/file.txt;stat /mnt/test/file.txt^C >> >> bash-4.4# rm /mnt/test/file.txt >> >> bash-4.4# dd if=/dev/zero of=/mnt/test/file.txt bs=512 count=1 >> oflag=sync;stat /mnt/test/file.txt;tar -czvf /tmp/abc.gz >> /mnt/test/file.txt;stat /mnt/test/file.txt >> >> 1+0 records in >> >> 1+0 records out >> >> 512 bytes copied, 0.00932571 s, 54.9 kB/s >> >> File: /mnt/test/file.txt >> >> Size: 512 Blocks: 1 IO Block: 131072 regular file >> >> Device: 33h/51d Inode: 9949244856126716752 Links: 1 >> >> Access: (0644/-rw-r--r--) Uid: (0/root) Gid: (0/root) >> >> Access: 2018-07-13 17:55:02.75600 + >> >> Modify: 2018-07-13 17:55:02.76400 + >> >> Change: 2018-07-13 17:55:02.76800 + >> >> Birth: - >> >> tar: Removing leading `/' from member names >> >> /mnt/test/file.txt >> >> File: /mnt/test/file.txt >> >> Size: 512 Blocks: 1 IO Block: 131072 regular file >> >> Device: 33h/51d Inode: 9949244856126716752 Links: 1 >> >> Access: (0644/-rw-r--r--) Uid: (0/root) Gid: (0/root) >> >> Access: 2018-07-13 17:55:02.77600 + >> >> Modify: 2018-07-13 17:55:02.76400 + >> >> Change: 2018-07-13 17:55:02.76800 + >> >> Birth: - >> >> bash-4.4# gluster v set test md-cache-timeout 1 >> >> volume set: success >> >> bash-4.4# rm /mnt/test/file.txt >> >> bash-4.4# dd if=/dev/zero of=/mnt/test/file.txt bs=512 count=1 >> oflag=sync;stat /mnt/test/file.txt;tar -czvf /tmp/abc.gz >> /mnt/test/file.txt;stat /mnt/test/file.txt >> >> 1+0 records in >> >> 1+0 records out >> >> 512 bytes copied, 0.0107589 s, 47.6 kB/s >> >> File: /mnt/test/file.txt >> >> Size: 512 Blocks: 1 IO Block: 131072 regular file >> >> Device: 33h/51d Inode: 13569976446871695205 Links: 1 >> >> Access: (0644/-rw-r--r--) Uid: (0/root) Gid: (0/root) >> >> Access: 2018-07-13 17:55:11.54800 + >> >> Modify: 2018-07-13 17:55:11.56000 + >> >> Change: 2018-07-13 17:55:11.56000 + >> >> Birth: - >> >> tar: Removing leading `/' from member names >> >> /mnt/test/file.txt >> >> tar: /mnt/test/file.txt: file changed as we read it >> >> File: /mnt/test/file.txt >> >> Size: 512 Blocks: 1 IO Block: 131072 regular file >> >> Device: 33h/51d Inode: 13569976446871695205 Links: 1 >> >> Access: (0644/-rw-r--r--) Uid: (0/root) Gid: (0/root) >> >> Access: 2018-07-13 17:55:11.58000 + >> >> Modify: 2018-07-13 17:55:11.56000 + >> >> Change: 2018-07-13 17:55:11.56400 + >> >> Birth: - >> >> >> >> >> >> Best Regards, >> >> George >> >> *From:* gluster-devel-boun...@gluster.org [mailto:gluster-devel-bounces@ >> gluster.org] *On Behalf Of *Raghavendra Gowdappa >> *Sent:* Friday, July 20, 2018 4:01 PM >> *To:* Lian, George (NSB - CN/Hangzhou) >> *Cc:* Zhang, Bingxuan (NSB - CN/Hangzhou) ; >> Raghavendra G ; Gluster-devel@gluster.org >> >> *Subject:* Re: [Gluster-devel] The ctime of
[Gluster-devel] Weekly Untriaged Bugs
[...truncated 6 lines...] https://bugzilla.redhat.com/1599203 / build: Compilation failure due to python-devel dependency than configure time failure https://bugzilla.redhat.com/1599219 / build: configure fails complaining absence of libxml2-devel https://bugzilla.redhat.com/1598900 / build: tmpfiles snippet tries to create folder owned by nonexistent user https://bugzilla.redhat.com/1600349 / core: call stack group list leaks https://bugzilla.redhat.com/1603220 / core: glusterfs-server depends on deprecated liblvm2app https://bugzilla.redhat.com/1596746 / core: High Memory Utilization Glusterfs https://bugzilla.redhat.com/1605066 / disperse: RFE: Need to optimize on time taken by heal info to display o/p when large number of entries exist https://bugzilla.redhat.com/1596686 / distribute: key = trusted.glusterfs.protect.writes [Invalid argument]; key = glusterfs.avoid.overwrite [Invalid argument] https://bugzilla.redhat.com/1597798 / encryption-xlator: 'mv' of directory on encrypted volume fails https://bugzilla.redhat.com/1603576 / fuse: glusterfs dying with SIGSEGV https://bugzilla.redhat.com/1595916 / geo-replication: geo-replication faies with xsync https://bugzilla.redhat.com/1605077 / glusterd: If a node disconnects during volume delete, it assumes deleted volume as a freshly created volume when it is back online https://bugzilla.redhat.com/1601356 / glusterd: Problem with SSL/TLS encryption on Gluster 4.0 & 4.1 https://bugzilla.redhat.com/1603063 / glusterd: ./tests/bugs/glusterd/validating-server-quorum.t is generated core https://bugzilla.redhat.com/1602824 / libgfapi: SMBD crashes when streams_attr VFS is used with Gluster VFS https://bugzilla.redhat.com/1596904 / object-storage: remove-brick is failing to remove hidden dot files https://bugzilla.redhat.com/1601532 / packaging: debian packages miss /etc/glusterfs/gsyncd.conf / breaks geo-replication / sate_file entry missing / unable to get statefile's name https://bugzilla.redhat.com/1599275 / posix-acl: Default ACL cannot be removed https://bugzilla.redhat.com/1594857 / project-infrastructure: Make smoke runs detect test cases added to patch https://bugzilla.redhat.com/1597731 / project-infrastructure: need 'shellcheck' in smoke. https://bugzilla.redhat.com/1598326 / project-infrastructure: Setup CI for gluster-block https://bugzilla.redhat.com/1597351 / project-infrastructure: Smoke should fail on commits against closed github issues https://bugzilla.redhat.com/1598328 / project-infrastructure: Unable to subscribe to automated-testing mailing list https://bugzilla.redhat.com/1596787 / quota: glusterfs rpc-clnt.c: error returned while attempting to connect to host: (null), port 0 https://bugzilla.redhat.com/1598769 / rdma: Brick is not operational when using RDMA transport https://bugzilla.redhat.com/1595088 / read-ahead: Gluster clients seem to be reading blocks from server multiple times https://bugzilla.redhat.com/1596188 / rpc: XDR files contain illegal "long" integer declarations. https://bugzilla.redhat.com/1594819 / snapshot: Failed to take snapshot (occasionally) https://bugzilla.redhat.com/1602282 / tests: tests/bugs/bug-1371806_acl.t fails for distributed regression framework https://bugzilla.redhat.com/1602262 / tests: tests/bugs/posix/bug-990028.t fails for distributed regression framework [...truncated 2 lines...] build.log Description: Binary data ___ Gluster-devel mailing list Gluster-devel@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] The ctime of fstat is not correct which lead to "tar" utility error
George, Sorry. I sent you a version of the fix which was stale. Can you try with: https://review.gluster.org/20549 This patch passes the test case you've given. regards, Raghavendra On Fri, Jul 20, 2018 at 2:59 PM, Lian, George (NSB - CN/Hangzhou) < george.l...@nokia-sbell.com> wrote: > Hi, > > > > Sorry, there seems still have issue. > > > > We use “dd” application of linux tools instead of my demo program, and if > the file is not exist before dd, the issue still be there. > > > > The test command is > > rm -rf /mnt/test/file.txt ; dd if=/dev/zero of=/mnt/test/file.txt bs=512 > count=1 oflag=sync;stat /mnt/test/file.txt;tar -czvf /tmp/abc.gz > > 1) If we set md-cache-timeout to 0, the issue will not happen > > 2) If we set md-cache-timeout to 1, the issue will 100% reproduced! > (with new patch you mentioned in the mail) > > > > Please see detail test result as the below: > > > > bash-4.4# gluster v set export md-cache-timeout 0 > > volume set: failed: Volume export does not exist > > bash-4.4# gluster v set test md-cache-timeout 0 > > volume set: success > > bash-4.4# dd if=/dev/zero of=/mnt/test/file.txt bs=512 count=1 > oflag=sync;stat /mnt/test/file.txt;tar -czvf /tmp/abc.gz > /mnt/test/file.txt;stat /mnt/test/file.txt^C > > bash-4.4# rm /mnt/test/file.txt > > bash-4.4# dd if=/dev/zero of=/mnt/test/file.txt bs=512 count=1 > oflag=sync;stat /mnt/test/file.txt;tar -czvf /tmp/abc.gz > /mnt/test/file.txt;stat /mnt/test/file.txt > > 1+0 records in > > 1+0 records out > > 512 bytes copied, 0.00932571 s, 54.9 kB/s > > File: /mnt/test/file.txt > > Size: 512 Blocks: 1 IO Block: 131072 regular file > > Device: 33h/51d Inode: 9949244856126716752 Links: 1 > > Access: (0644/-rw-r--r--) Uid: (0/root) Gid: (0/root) > > Access: 2018-07-13 17:55:02.75600 + > > Modify: 2018-07-13 17:55:02.76400 + > > Change: 2018-07-13 17:55:02.76800 + > > Birth: - > > tar: Removing leading `/' from member names > > /mnt/test/file.txt > > File: /mnt/test/file.txt > > Size: 512 Blocks: 1 IO Block: 131072 regular file > > Device: 33h/51d Inode: 9949244856126716752 Links: 1 > > Access: (0644/-rw-r--r--) Uid: (0/root) Gid: (0/root) > > Access: 2018-07-13 17:55:02.77600 + > > Modify: 2018-07-13 17:55:02.76400 + > > Change: 2018-07-13 17:55:02.76800 + > > Birth: - > > bash-4.4# gluster v set test md-cache-timeout 1 > > volume set: success > > bash-4.4# rm /mnt/test/file.txt > > bash-4.4# dd if=/dev/zero of=/mnt/test/file.txt bs=512 count=1 > oflag=sync;stat /mnt/test/file.txt;tar -czvf /tmp/abc.gz > /mnt/test/file.txt;stat /mnt/test/file.txt > > 1+0 records in > > 1+0 records out > > 512 bytes copied, 0.0107589 s, 47.6 kB/s > > File: /mnt/test/file.txt > > Size: 512 Blocks: 1 IO Block: 131072 regular file > > Device: 33h/51d Inode: 13569976446871695205 Links: 1 > > Access: (0644/-rw-r--r--) Uid: (0/root) Gid: (0/root) > > Access: 2018-07-13 17:55:11.54800 + > > Modify: 2018-07-13 17:55:11.56000 + > > Change: 2018-07-13 17:55:11.56000 + > > Birth: - > > tar: Removing leading `/' from member names > > /mnt/test/file.txt > > tar: /mnt/test/file.txt: file changed as we read it > > File: /mnt/test/file.txt > > Size: 512 Blocks: 1 IO Block: 131072 regular file > > Device: 33h/51d Inode: 13569976446871695205 Links: 1 > > Access: (0644/-rw-r--r--) Uid: (0/root) Gid: (0/root) > > Access: 2018-07-13 17:55:11.58000 + > > Modify: 2018-07-13 17:55:11.56000 + > > Change: 2018-07-13 17:55:11.56400 + > > Birth: - > > > > > > Best Regards, > > George > > *From:* gluster-devel-boun...@gluster.org [mailto:gluster-devel-bounces@ > gluster.org] *On Behalf Of *Raghavendra Gowdappa > *Sent:* Friday, July 20, 2018 4:01 PM > *To:* Lian, George (NSB - CN/Hangzhou) > *Cc:* Zhang, Bingxuan (NSB - CN/Hangzhou) ; > Raghavendra G ; Gluster-devel@gluster.org > > *Subject:* Re: [Gluster-devel] The ctime of fstat is not correct which > lead to "tar" utility error > > > > > > > > On Fri, Jul 20, 2018 at 1:22 PM, Lian, George (NSB - CN/Hangzhou) < > george.l...@nokia-sbell.com> wrote: > > >>>We recently identified an issue with stat-prefetch. Fix can be found at: > > >>>https://review.gluster.org/#/c/20410/11 > > > > >>>Can you let us know whether this helps? > > > > > > The patch can resolve this issue, I have verified in Gluster 4.2(master > trunk branch) and Gluster 3.12.3! > > > > Thanks we'll merge it. > > > > > > Thanks & Best Regards, > > George > > > > *From:* gluster-devel-boun...@gluster.org [mailto:gluster-devel-bounces@ > gluster.org] *On Behalf Of *Raghavendra Gowdappa > *Sent:* Thursday, July 19, 2018 5:06 PM > *To:* Lian, George (NSB - CN/Hangzhou) > *Cc:* Zhang, Bingxuan (NSB - CN/Hangzhou) ; > Gluster-devel@gluster.org; Raghavendra G > *Subject:* Re: [Gluster-devel] The
