>> Also can u please mention which version of ganesha and details of >> ganesha.conf . Latest stable release for I am using ganesha.nfsd Release = V2.3.2
ganesha.conf is the standard glusterfs FSAL. >> I am wondering why it is sending gettattr call on "security.selinux". Fyi, selinux is disabled both on server & client. >> Check /var/log/ganesha.log and /var/log/ganesha-gfapi.log for more clues They are kosher. I replied to my earlier post. I guess the delay/pause in the ls l or rm -rf command is something todo with my VM setup/network as I see the pause/delay in the kernel.NFS also. -- Deepak -----Original Message----- From: Jiffin Tony Thottan [mailto:jthot...@redhat.com] Sent: Thursday, August 11, 2016 10:36 PM To: Deepak Naidu; Vijay Bellur; gluster-users@gluster.org Subject: Re: [Gluster-users] Linux (ls -l) command pauses/slow on GlusterFS mounts On 12/08/16 07:23, Deepak Naidu wrote: > I tried more things to figure out the issue. Like upgrading NFS-ganesha to > the latest version(as the earlier version had some bug regarding crashing), > that helped a bit. > > But still again the ls -ls or rm -rf files were hanging but not much as > earlier. So upgrade of NFS ganesha to stable version did help help a bit. > > I did strace again, looks like its pausing/hanging at "lstat" I had to > [crtl+c] to get the exact hang/pausing line. > > lgetxattr("/mnt/gluster/rand.26.0", "security.selinux", 0x1990a00, > 255) = -1 ENODATA (No data available) lstat("/mnt/gluster/rand.25.0", > {st_mode=S_IFREG|0644, st_size=2147483648, ...}) = 0 > lgetxattr("/mnt/gluster/rand.25.0", "security.selinux", 0x1990a20, > 255) = -1 ENODATA (No data available) lstat("/mnt/gluster/rand.24.0", > ^C > > > NOTE: I am running fio to generate some write operation & hangs are seen when > issuing ls during write operation. > > Next thing, I might try is to use NFS mount rather than Glustefs fuse to see > if its related to fuse client. > > ====strace of ls -l /mnt/gluster/====== > > munmap(0x7efebec71000, 4096) = 0 > openat(AT_FDCWD, "/mnt/gluster/", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) > = 3 > getdents(3, /* 14 entries */, 32768) = 464 > lstat("/mnt/gluster/9e50d562-5846-4a60-ad75-e95dcbe0e38a.vhd", > {st_mode=S_IFREG|0644, st_size=19474461184, ...}) = 0 > lgetxattr("/mnt/gluster/9e50d562-5846-4a60-ad75-e95dcbe0e38a.vhd", > "security.selinux", 0x1990900, 255) = -1 ENODATA (No data available) > lstat("/mnt/gluster/file1", {st_mode=S_IFREG|0644, > st_size=19474461184, ...}) = 0 lgetxattr("/mnt/gluster/file1", > "security.selinux", 0x1990940, 255) = -1 ENODATA (No data available) > lstat("/mnt/gluster/rand.0.0", {st_mode=S_IFREG|0644, > st_size=2147483648, ...}) = 0 lgetxattr("/mnt/gluster/rand.0.0", > "security.selinux", 0x1990940, 255) = -1 ENODATA (No data available) > lstat("/mnt/gluster/rand.31.0", {st_mode=S_IFREG|0644, > st_size=2147483648, ...}) = 0 lgetxattr("/mnt/gluster/rand.31.0", > "security.selinux", 0x1990960, 255) = -1 ENODATA (No data available) > lstat("/mnt/gluster/rand.30.0", {st_mode=S_IFREG|0644, > st_size=2147483648, ...}) = 0 lgetxattr("/mnt/gluster/rand.30.0", > "security.selinux", 0x1990980, 255) = -1 ENODATA (No data available) > lstat("/mnt/gluster/rand.29.0", {st_mode=S_IFREG|0644, > st_size=2147483648, ...}) = 0 lgetxattr("/mnt/gluster/rand.29.0", > "security.selinux", 0x19909a0, 255) = -1 ENODATA (No data available) > lstat("/mnt/gluster/rand.28.0", {st_mode=S_IFREG|0644, > st_size=2147483648, ...}) = 0 lgetxattr("/mnt/gluster/rand.28.0", > "security.selinux", 0x19909c0, 255) = -1 ENODATA (No data available) > lstat("/mnt/gluster/rand.27.0", {st_mode=S_IFREG|0644, > st_size=2147483648, ...}) = 0 lgetxattr("/mnt/gluster/rand.27.0", > "security.selinux", 0x19909e0, 255) = -1 ENODATA (No data available) > lstat("/mnt/gluster/rand.26.0", {st_mode=S_IFREG|0644, > st_size=2147483648, ...}) = 0 lgetxattr("/mnt/gluster/rand.26.0", > "security.selinux", 0x1990a00, 255) = -1 ENODATA (No data available) > lstat("/mnt/gluster/rand.25.0", {st_mode=S_IFREG|0644, > st_size=2147483648, ...}) = 0 lgetxattr("/mnt/gluster/rand.25.0", > "security.selinux", 0x1990a20, 255) = -1 ENODATA (No data available) > lstat("/mnt/gluster/rand.24.0", ^C > > ====strace of end - ls -l /mnt/gluster/====== I am wondering why it is sending gettattr call on "security.selinux". Also can u please mention which version of ganesha and details of ganesha.conf . Latest stable release for ganesha(2.3.3) is pretty close. Check /var/log/ganesha.log and /var/log/ganesha-gfapi.log for more clues -- Jiffin > -----Original Message----- > From: Deepak Naidu > Sent: Wednesday, August 10, 2016 2:25 PM > To: Vijay Bellur > Cc: gluster-users@gluster.org > Subject: RE: [Gluster-users] Linux (ls -l) command pauses/slow on > GlusterFS mounts > > To be more precious the hang is clearly seen when there is some IO(write) to > the mount point. Even rm -rf takes time to clear the files. > > Below, time command showing the delay. Typically it should take less then a > second, but glusterfs take more than 5seconds just to list 32x 2GB files. > > [root@client-host ~]# time ls -l /mnt/gluster/ total 34575680 -rw-r--r--. 1 > root root 2147483648 Aug 10 12:23 rand.0.0 -rw-r--r--. 1 root root 2147483648 > Aug 10 12:23 rand.1.0 -rw-r--r--. 1 root root 2147454976 Aug 10 12:23 > rand.10.0 -rw-r--r--. 1 root root 2147463168 Aug 10 12:23 rand.11.0 > -rw-r--r--. 1 root root 2147467264 Aug 10 12:23 rand.12.0 -rw-r--r--. 1 root > root 2147475456 Aug 10 12:23 rand.13.0 -rw-r--r--. 1 root root 2147479552 Aug > 10 12:23 rand.14.0 -rw-r--r--. 1 root root 2147479552 Aug 10 12:23 rand.15.0 > -rw-r--r--. 1 root root 2147483648 Aug 10 12:23 rand.16.0 -rw-r--r--. 1 root > root 2147479552 Aug 10 12:23 rand.17.0 -rw-r--r--. 1 root root 2147483648 Aug > 10 12:23 rand.18.0 -rw-r--r--. 1 root root 2147467264 Aug 10 12:23 rand.19.0 > -rw-r--r--. 1 root root 2147483648 Aug 10 12:23 rand.2.0 -rw-r--r--. 1 root > root 2147475456 Aug 10 12:23 rand.20.0 -rw-r--r--. 1 root root 2147479552 Aug > 10 12:23 rand.21.0 -rw-r--r--. 1 root root 2147483648 Aug 10 12:23 rand.22.0 - rw > -r--r--. 1 root root 2147459072 Aug 10 12:23 rand.23.0 -rw-r--r--. 1 > root root 2147483648 Aug 10 12:23 rand.24.0 -rw-r--r--. 1 root root > 2147471360 Aug 10 12:23 rand.25.0 -rw-r--r--. 1 root root 2147483648 > Aug 10 12:23 rand.26.0 -rw-r--r--. 1 root root 2147483648 Aug 10 12:23 > rand.27.0 -rw-r--r--. 1 root root 2147479552 Aug 10 12:23 rand.28.0 > -rw-r--r--. 1 root root 2147483648 Aug 10 12:23 rand.29.0 -rw-r--r--. > 1 root root 2147483648 Aug 10 12:23 rand.3.0 -rw-r--r--. 1 root root > 2147442688 Aug 10 12:23 rand.30.0 -rw-r--r--. 1 root root 2147483648 > Aug 10 12:23 rand.31.0 -rw-r--r--. 1 root root 2147483648 Aug 10 12:23 > rand.4.0 -rw-r--r--. 1 root root 2147483648 Aug 10 12:23 rand.5.0 > -rw-r--r--. 1 root root 2147483648 Aug 10 12:23 rand.6.0 -rw-r--r--. 1 > root root 2147483648 Aug 10 12:23 rand.7.0 -rw-r--r--. 1 root root > 2147483648 Aug 10 12:23 rand.8.0 -rw-r--r--. 1 root root 2147483648 > Aug 10 12:23 rand.9.0 > > real 0m7.478s > user 0m0.001s > sys 0m0.005s > [root@client-host ~]# > > -- > Deepak > > -----Original Message----- > From: gluster-users-boun...@gluster.org > [mailto:gluster-users-boun...@gluster.org] On Behalf Of Deepak Naidu > Sent: Wednesday, August 10, 2016 2:18 PM > To: Vijay Bellur > Cc: gluster-users@gluster.org > Subject: Re: [Gluster-users] Linux (ls -l) command pauses/slow on > GlusterFS mounts > > * PGP Signed: 08/10/2016 at 02:18:22 PM, Decrypted > > I did strace & its waiting on IO. > > -- > Deepak > > -----Original Message----- > From: Vijay Bellur [mailto:vbel...@redhat.com] > Sent: Wednesday, August 10, 2016 2:17 PM > To: Deepak Naidu > Cc: gluster-users@gluster.org > Subject: Re: [Gluster-users] Linux (ls -l) command pauses/slow on > GlusterFS mounts > > On 08/10/2016 05:12 PM, Deepak Naidu wrote: >> Before we can try physical we wanted POC on VM. >> >> Just a note the VMs are decently powerful 18cpus, 10gig NIC, 45GB Ram 1TB >> SSD drives. This is per node spec. >> >> I don't see the ls -l command hanging when I try to list the files from the >> gluster-node VMs itself So the question. > The reason I alluded to a physical setup was to remove the variables that can > affect performance in a virtual setup. The behavior is not usual for the > scale of deployment that you mention. You could use strace in conjunction > with gluster volume profile to determine where the latency is stemming from. > > Regards, > Vijay > >> -- >> Deepak >> >>> On Aug 10, 2016, at 2:01 PM, Vijay Bellur <vbel...@redhat.com> wrote: >>> >>>> On 08/10/2016 04:54 PM, Deepak Naidu wrote: >>>> Anyone who has seen the issue in their env ? >>> >>>> -- >>>> Deepak >>>> >>>> -----Original Message----- >>>> From: gluster-users-boun...@gluster.org >>>> [mailto:gluster-users-boun...@gluster.org] On Behalf Of Deepak >>>> Naidu >>>> Sent: Tuesday, August 09, 2016 9:14 PM >>>> To: gluster-users@gluster.org >>>> Subject: [Gluster-users] Linux (ls -l) command pauses/slow on >>>> GlusterFS mounts >>>> >>>> Greetings, >>>> >>>> I have 3node GlusterFS on VM for POC each node has 2x bricks of 200GB. >>>> Regardless of what type of volume I create when listing files under >>>> directory using ls command the GlusterFS mount hangs pauses for few >>>> seconds. This is same if there're 2-5 19gb file each or 2gb file each. >>>> There are less than 10 files under the GlusterFS mount. >>>> >>>> I am using NFS-Ganesha for NFS server with GlusterFS and the Linux client >>>> is mounted using GlusterFS fuse mount with direct-io enabled. >>>> >>>> GlusterFS version 3.8(latest) >>>> >>>> >>>> Any insight is appreciated. >>> This does not seem usual for the deployment that you describe. Can you try >>> on a physical setup to see if the same behavior is observed? >>> >>> -Vijay >>> >>> >> --------------------------------------------------------------------- >> - >> ------------- This email message is for the sole use of the intended >> recipient(s) and may contain confidential information. Any >> unauthorized review, use, disclosure or distribution is prohibited. >> If you are not the intended recipient, please contact the sender by >> reply email and destroy all copies of the original message. >> --------------------------------------------------------------------- >> - >> ------------- >> > _______________________________________________ > Gluster-users mailing list > Gluster-users@gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users > > * Deepak Naidu <dna...@nvidia.com> > * 0x15098040 > _______________________________________________ > Gluster-users mailing list > Gluster-users@gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users _______________________________________________ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
