Sorry for the late reply. On 4/16/2017 22:31, Vijay Bellur wrote: > > > On Wed, Apr 12, 2017 at 10:37 PM, Kinglong Mee <mijinl...@open-fs.com > <mailto:mijinl...@open-fs.com>> wrote: > > Yes, this one is silly rename, > > >>> rm: cannot remove > ‘/mnt/nfs/ltp-JEYAuky2dz/.nfsaa46457a6a72f8ea000014f5’: Device or resource > busy > > But, the other one isn't client's silly rename, > > >>>> rm: cannot remove ‘/mnt/nfs/ltp-JEYAuky2dz/rmderQsjV’: Directory > not empty > > Also, the second one often appears when testing ganesha nfs by ltp. > > I try to get an tcpdump, I found, when ls under the nfs client directory, > > the glusterfs client doesn't send readdir to glusterfs server, so that, > the nfs client gets an empty directory information, but it's empty > underlying. > > ls at some other directory, > the glusterfs client send readdir to glusterfs server, nfs client gets > right dirctory > information as the underlying. > > So, I guess maybe there are some problems in MDCHANE or glusterfs client? > > ]# cat /var/lib/glusterd/vols/gvtest/trusted-gvtest.tcp-fuse.vol > volume gvtest-client-0 > type protocol/client > option send-gids true > option password d80765c8-95ae-46ed-912a-0c98fdd1e7cd > option username 7bc6276f-245c-477a-99d0-7ead4fcb2968 > option transport.address-family inet > option transport-type tcp > option remote-subvolume /gluster-test/gvtest > option remote-host 192.168.9.111 > option ping-timeout 42 > end-volume > > volume gvtest-client-1 > type protocol/client > option send-gids true > option password d80765c8-95ae-46ed-912a-0c98fdd1e7cd > option username 7bc6276f-245c-477a-99d0-7ead4fcb2968 > option transport.address-family inet > option transport-type tcp > option remote-subvolume /gluster-test/gvtest > option remote-host 192.168.9.111 > option ping-timeout 42 > end-volume > > volume gvtest-dht > type cluster/distribute > option lock-migration off > subvolumes gvtest-client-0 gvtest-client-1 > end-volume > > volume gvtest > type debug/io-stats > option count-fop-hits off > option latency-measurement off > option log-level INFO > subvolumes gvtest-dht > end-volume > > > > > Have you checked the gluster export directories after receiving ENOTEMPTY? If > you observe any files there, can you please check if they are 0-byte files > with access mode 600 and sticky bit set? >
No. I have check those files at the latest test. rm: cannot remove ‘/mnt/nfs/ltp-fpeUthdR0W/rmdVGrwdj’: Directory not empty The back-end directory, # ll /gvtest/ltp-fpeUthdR0W/ total 264 drwxrwxrwx 2 root root 253952 Apr 20 15:53 fann7LHNa drwxrwxrwx 2 root root 4096 Apr 20 15:54 inop4q8Wi drwxrwxrwx 3 root root 4096 Apr 20 15:58 linu6YVST drwxrwxrwx 3 root root 4096 Apr 20 15:59 rmdVGrwdj # ll /gvtest/ltp-fpeUthdR0W/rmdVGrwdj/testdir/ total 0 The nfs client directory mounted through ganesha.nfsd, # ll /mnt/nfs/ltp-fpeUthdR0W/ total 12 drwxrwxrwx 2 root root 4096 Apr 20 15:53 fann7LHNa drwxrwxrwx 2 root root 4096 Apr 20 15:54 inop4q8Wi drwxrwxrwx 3 root root 4096 Apr 20 15:58 linu6YVST # ll /mnt/nfs/ltp-fpeUthdR0W/rmdVGrwdj/ total 4 drwxrwxrwx 2 root root 4096 Apr 20 15:59 testdir [root@C0N1 ~]# ll /mnt/nfs/ltp-fpeUthdR0W/rmdVGrwdj/testdir/ total 0 -rwxrwxrwx 1 root root 0 Apr 20 15:59 testfile 1. nfs client can't get "rmdVGrwdj" when ll the upper directory, 2. but nfs client can access it by name directly, 3. A "testfile" under nfs client's "rmdVGrwdj/testdir", but it's empty in the back-end gluster filesystem. 4. after ltp's rm finish, a duplicate rm can remove "rmdVGrwdj". So, I guess there are some problem of the ganesha.nfsd cache when deleting files. thanks, Kinglong Mee > When a file is being renamed, if the hash value of its new name happens to > fall in a range that is different than that of the old name, gluster creates > one such file. In the past we have seen instances of these files not being > cleaned up properly in some cases and that causes a ENOTEMPTY when rmdir > lands on the parent directory. Trying to see if we are hitting the same > problem here. > > Regards, > Vijay ------------------------------------------------------------------------------ Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot _______________________________________________ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
