Re: [Gluster-users] Setting up a new gluster volume using a block device
Your gluster brick must be a directory, not a block device. The filesystem that directory is located on must supported xattr. David On 7/19/12 1:16 PM, Sallee, Stephen (Jake) wrote: I am new to gluster so please be a bit patient with me. I am trying to setup a gluster volume with the bricks being /dev/sdb1 (or /dev/sdb, I have tried both) ... etc. however I get an error message and the operation fails. I do not have the error message (sorry!) the only brick I have available for testing is tied up at the moment so I cant rerun the test to get the exact error. If I can I will post it later. However if I mount the partition I am able to create the volume no problem. I have read the admin guide several times but cannot find any reference to using a block device only a mounted device. Is it even possible, or am I crazy for even trying? Jake Sallee Godfather of Bandwidth System Engineer University of Mary Hardin-Baylor 900 College St. Belton TX. 76513 Fone: 254-295-4658 Phax: 254-295-4221 HTTP://WWW.UMHB.EDU ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
[Gluster-users] Setting up a new gluster volume using a block device
I am new to gluster so please be a bit patient with me. I am trying to setup a gluster volume with the bricks being /dev/sdb1 (or /dev/sdb, I have tried both) ... etc. however I get an error message and the operation fails. I do not have the error message (sorry!) the only brick I have available for testing is tied up at the moment so I cant rerun the test to get the exact error. If I can I will post it later. However if I mount the partition I am able to create the volume no problem. I have read the admin guide several times but cannot find any reference to using a block device only a mounted device. Is it even possible, or am I crazy for even trying? Jake Sallee Godfather of Bandwidth System Engineer University of Mary Hardin-Baylor 900 College St. Belton TX. 76513 Fone: 254-295-4658 Phax: 254-295-4221 HTTP://WWW.UMHB.EDU ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] glusterfs process eats memory until OOM kills it
On 07/19/2012 08:57 AM, Vijay Bellur wrote: > On 07/19/2012 01:46 AM, Andreas Kurz wrote: >> Hi, >> >> I'm running GlusterFS 3.2.6 in AWS on CentOS 6.2, running a >> distributed/replicated setup. >> >> Type: Distributed-Replicate >> Status: Started >> Number of Bricks: 4 x 2 = 8 >> >> Whenever geo-replication is activated, the corresponding glusterfs >> process on the server starts eating memory (and using lot of cpu) until >> oom killer strikes back. >> >> This happens once user starting to change files via glusterfs mount and >> gsyncd starts crawling through the directory tree looking for changes. >> Network traffic between the servers is quite high, typically 10Mbit/s >> ... the vast majority are lookups and getxattr request from the server >> running geo-replication. >> >> I also created a state dump (5MB bzip2 archive) of this glusterfs >> process when eating about 9GB if that is needed for debugging I can >> upload it somewhere (Bugzilla?). Dropping dentries and inodes reclaims >> about 1GB. >> >> Any ideas? A bug? Any recommended tunings, maybe a gsyncd option? I >> changed these values: >> >> performance.stat-prefetch: off >> performance.quick-read: off >> performance.cache-refresh-timeout: 1 >> performance.read-ahead: off >> geo-replication.indexing: on >> nfs.disable: on >> network.ping-timeout: 10 >> performance.cache-size: 1073741824 >> > > > Can you also try with performance.io-cache being set to off? If that > doesn't show any improvement, please raise a bug and attach the > statedump to it. Done ... https://bugzilla.redhat.com/show_bug.cgi?id=841617 Thanks! Regards, Andreas > > Thanks, > Vijay > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users signature.asc Description: OpenPGP digital signature ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] "Granular locking" - does this need to be enabled in 3.3.0 ?
On Thu, Jul 19, 2012 at 2:14 AM, Jake Grimmett wrote: > Dear Pranith /Anand , > > Update on our progress with using KVM & Gluster: > > We built a two server (Dell R710) cluster, each box has... > 5 x 500 GB SATA RAID5 array (software raid) > an Intel 10GB ethernet HBA. > One box has 8GB RAM, the other 48GB > both have 2 x E5520 Xeon > Centos 6.3 installed > Gluster 3.3 installed from the rpm files on the gluster site > > > 1) create a replicated gluster volume (on top of xfs) > 2) setup qemu/kvm with a gluster volume (mounts localhost:/gluster-vol) > 3) sanlock configured (this is evil!) > 4) build a virtual machines with 30GB qcow2 image, 1GB RAM > 5) clone this VM into 4 machines > 6) check that live migration works (OK) > > Start basic test cycle: > a) migrate all machines to host #1, then reboot host #2 > b) watch logs for self-heal to complete > c) migrate VM's to host #2, reboot host #1 > d) check logs for self heal > > The above cycle can be repeated numerous times, and completes without > error, provided that no (or little) load is on the VM. > > > If I give the VM's a work load, such by running "bonnie++" on each VM, > things start to break. > 1) it becomes almost impossible to log in to each VM > 2) the kernel on each VM starts giving timeout errors > i.e. "echo 0 > /proc/sys/kernel/hung_task_**timeout_secs" > 3) top / uptime on the hosts shows load average of up to 24 > 4) dd write speed (block size 1K) to gluster is around 3MB/s on the host > > > While I agree that running bonnie++ on four VM's is possibly unfair, there > are load spikes on quiet machines (yum updates etc). I suspect that the I/O > of one VM starts blocking that of another VM, and the pressure builds up > rapidly on gluster - which does not seem to cope well under pressure. > Possibly this is the access pattern / block size of qcow2 disks? > > I'm (slightly) disappointed. > > Though it doesn't corrupt data, the I/O performance is < 1% of my > hardwares capability. Hopefully work on buffering and other tuning will fix > this ? Or maybe the work mentioned getting qemu talking directly to gluster > will fix this? > > Do you mean that the I/O is bad when you are performing the migration? Or bad in general? If it is bad in general the qemu driver should help. Also try presenting each VM a FUSE mount point of its own (we have seen that help improve the overall system IOPs) If it is slow performance only during failover/failback, we probably need to do some more internal QoS tuning to de-prioritize self-heal traffic from preempting VM traffic for resources. Avati ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] NFS mounts with glusterd on localhost - reliable or not?
On Thu, Jul 19, 2012 at 05:16:24AM -0400, Krishna Srinivas wrote: > It was pretty confusing to read this thread. Hope I can clarify the > questions here. Thanks. I was confused. > The other discussion in this thread was related to NLM which has been > implemented in 3.3.0. This is to support locking calls from the NFS > clients to support fcntl() locking for the applications running on nfs > client. NLM server is implemented in glusterfs as well as kernel. NLM > server implemented in kernel is used by kernel-nfsd as well as > kernel-nfs-client. Hence if you have an nfs mount point, the > kernel-nfs-client automatically starts kernel NLM server. So if > glusterfs-nfs process is already running on a system (and hence it > also runs its own NLM server) and if you try to do "mount -t nfs > someserver:/export /mnt/nfs" on the same system it fails as > kernel-nfs-client won't be able to start kernel-NLM-server (because > glusterfs NLM server would have already registered with portmapper for > NLM service and hence kernel-NLM-server registration with portmapper > fails). Workaround is "mount -t nfs -o nolock someserver:/export > /mnt/nfs" if you really want to have an nfs mount on the same machine > where glusterfs-nfs process is running. So, if you want to run both at once, only one can lock. Is the architecture of NLM such that there could never be a single NLM server for both Gluster and kernel (whether that single server be Gluster's or kernel's)? Thanks, Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] NFS mounts with glusterd on localhost - reliable or not?
On Wed, Jul 18, 2012 at 01:56:04AM -0400, Rajesh Amaravathi wrote: > As to whether we can disable parts of kernel NFS (I'm assuming e.g NLM), I > think > its not really necessary since we can mount other exports with nolock option. > If we take out NLM or disable NLM at the kernel level, then every time we need > NLM from kernel, we need to recompile the kernel/have a secondary kernel with > NLM > and reboot, much tedious than simply killing Gluster/fuse NFS and after > kernel NLM's > work is done, restart Gluster/fuse NFS. My $0.02 :) Since there's good reason to want locking with Gluster NFS, wouldn't the answer be to add code to kernel that would allow the kernel's locking to be turned off and on in the standard way - a file called something like kernel_nfs_locking that would hold either a 0 or 1? Obviously the kernel's NFS support was built on the assumption that no one with kernel NFS available would want to run userland NFS. Gluster shows that assumption is wrong. So wouldn't it be sensible for someone on the Gluster team to be submitting kernel patches to fix this oversight? Best, Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] NFS subdirectory Solaris client
Hi Rajesh, Thank you for the workaround, it works fine. Can we expect a patch in the future release to avoid this procedure ? Regards, Anthony > Date: Thu, 19 Jul 2012 05:30:24 -0400 > From: raj...@redhat.com > To: sokar6...@hotmail.com > CC: gluster-users@gluster.org; sgo...@redhat.com > Subject: Re: [Gluster-users] NFS subdirectory Solaris client > > fwding from krishna's mail, need to update the docs with this info: > > Here are the steps: > on linux client machine: > * make sure you have the directory you want to export is already created, if > not create it. > if /mnt/nfs is the mount point do this: > mkdir /mnt/nfs/subdir > * on storage node do: > gluster volume set nfs.export-dir /subdir > gluster volume set nfs.mount-udp on > * do a "showmount -e " to see that subdir is exported too. > * on a LINUX client do this: > mount -o proto=tcp ://subdir /mnt/nfs > i.e you are mounting the subdir. > * now on SOLARIS client do this: > mount nfs://://subdir /mnt/nfs > you should be able to access the exported subdir on Solaris machine. note > that you have to mount the subdir on a linux machine first with proto=tcp > before trying to mount on solaris machine. > > Regards, > Rajesh Amaravathi, > Software Engineer, GlusterFS > RedHat Inc. > > - Original Message - > From: "anthony garnier" > To: raj...@redhat.com > Cc: gluster-users@gluster.org, sgo...@redhat.com > Sent: Thursday, July 19, 2012 2:19:24 PM > Subject: RE: [Gluster-users] NFS subdirectory Solaris client > > > Hi Rajesh , > My mistake, I didn't specify that I was in 3.3. > I didn't see any particular comment in the Admin doc about subdirectory > export. > Are you referring to this option: nfs.export-dir, but by default every > subdirectory are exported. > > Regards, > > Anthony > > > > > > Date: Thu, 19 Jul 2012 04:42:20 -0400 > > From: raj...@redhat.com > > To: sokar6...@hotmail.com > > CC: gluster-users@gluster.org; sgo...@redhat.com > > Subject: Re: [Gluster-users] NFS subdirectory Solaris client > > > > Hi Anthony, > > sub directory mount is not possible with GlusterNfs in 3.2.x version. > > you can get the 3.3 gNfs to work for subdir mount on solaris, though it > > requires some oblique steps to get it working. The steps are provided in > > the documentation (Admin guide i think) for 3.3. > > > > Regards, > > Rajesh Amaravathi, > > Software Engineer, GlusterFS > > RedHat Inc. > > > > - Original Message - > > From: "anthony garnier" > > To: sgo...@redhat.com > > Cc: gluster-users@gluster.org > > Sent: Thursday, July 19, 2012 12:50:41 PM > > Subject: Re: [Gluster-users] NFS subdirectory Solaris client > > > > > > > > Hi Shishir, > > > > I have reconfigured the port to 2049 so there is no need to specify it. > > Moreover the mount of the volume works fine, it's only the mount of the > > subdirectory that doesn't work. > > > > Thx and Regards, > > > > Anthony > > > > > > > > > > > Date: Wed, 18 Jul 2012 22:54:28 -0400 > > > From: sgo...@redhat.com > > > To: sokar6...@hotmail.com > > > CC: gluster-users@gluster.org > > > Subject: Re: [Gluster-users] NFS subdirectory Solaris client > > > > > > Hi Anthony, > > > > > > Please also specify this option port=38467, and try mounting it. > > > > > > With regards, > > > Shishir > > > > > > - Original Message - > > > From: "anthony garnier" > > > To: gluster-users@gluster.org > > > Sent: Wednesday, July 18, 2012 3:21:36 PM > > > Subject: [Gluster-users] NFS subdirectory Solaris client > > > > > > > > > > > > Hi everyone, > > > > > > I still have problem to mount subdirectory in NFS on Solaris client : > > > > > > # mount -o proto=tcp,vers=3 nfs://yval1010:/test/test2 > > > /users/glusterfs_mnt > > > nfs mount: yval1010: : RPC: Program not registered > > > nfs mount: retrying: /users/glusterfs_mnt > > > > > > > > > [2012-07-18 11:43:43.484994] E [nfs3.c:305:__nfs3_get_volume_id] > > > (-->/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_lookup+0x125) > > > [0x7f5418ea4e15] > > > (-->/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_lookup_reply+0x48) > > > [0x7f5418e9b908] > > > (-->/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_request_xlator_deviceid+0x4c) > > > [0x7f5418e9a54c]))) 0-nfs-nfsv3: invalid argument: xl > > > [2012-07-18 11:43:43.491088] E [nfs3.c:305:__nfs3_get_volume_id] > > > (-->/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_lookup+0x125) > > > [0x7f5418ea4e15] > > > (-->/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_lookup_reply+0x48) > > > [0x7f5418e9b908] > > > (-->/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_request_xlator_deviceid+0x4c) > > > [0x7f5418e9a54c]))) 0-nfs-nfsv3: invalid argument: xl > > > [2012-07-18 11:43:48.494268] E [nfs3.c:305:__nfs3_get_volume_id] > > > (-->/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_lookup+0x125) > > > [0x7f5418ea4e15
Re: [Gluster-users] NFS subdirectory Solaris client
fwding from krishna's mail, need to update the docs with this info: Here are the steps: on linux client machine: * make sure you have the directory you want to export is already created, if not create it. if /mnt/nfs is the mount point do this: mkdir /mnt/nfs/subdir * on storage node do: gluster volume set nfs.export-dir /subdir gluster volume set nfs.mount-udp on * do a "showmount -e " to see that subdir is exported too. * on a LINUX client do this: mount -o proto=tcp ://subdir /mnt/nfs i.e you are mounting the subdir. * now on SOLARIS client do this: mount nfs://://subdir /mnt/nfs you should be able to access the exported subdir on Solaris machine. note that you have to mount the subdir on a linux machine first with proto=tcp before trying to mount on solaris machine. Regards, Rajesh Amaravathi, Software Engineer, GlusterFS RedHat Inc. - Original Message - From: "anthony garnier" To: raj...@redhat.com Cc: gluster-users@gluster.org, sgo...@redhat.com Sent: Thursday, July 19, 2012 2:19:24 PM Subject: RE: [Gluster-users] NFS subdirectory Solaris client Hi Rajesh , My mistake, I didn't specify that I was in 3.3. I didn't see any particular comment in the Admin doc about subdirectory export. Are you referring to this option: nfs.export-dir, but by default every subdirectory are exported. Regards, Anthony > Date: Thu, 19 Jul 2012 04:42:20 -0400 > From: raj...@redhat.com > To: sokar6...@hotmail.com > CC: gluster-users@gluster.org; sgo...@redhat.com > Subject: Re: [Gluster-users] NFS subdirectory Solaris client > > Hi Anthony, > sub directory mount is not possible with GlusterNfs in 3.2.x version. > you can get the 3.3 gNfs to work for subdir mount on solaris, though it > requires some oblique steps to get it working. The steps are provided in > the documentation (Admin guide i think) for 3.3. > > Regards, > Rajesh Amaravathi, > Software Engineer, GlusterFS > RedHat Inc. > > - Original Message - > From: "anthony garnier" > To: sgo...@redhat.com > Cc: gluster-users@gluster.org > Sent: Thursday, July 19, 2012 12:50:41 PM > Subject: Re: [Gluster-users] NFS subdirectory Solaris client > > > > Hi Shishir, > > I have reconfigured the port to 2049 so there is no need to specify it. > Moreover the mount of the volume works fine, it's only the mount of the > subdirectory that doesn't work. > > Thx and Regards, > > Anthony > > > > > > Date: Wed, 18 Jul 2012 22:54:28 -0400 > > From: sgo...@redhat.com > > To: sokar6...@hotmail.com > > CC: gluster-users@gluster.org > > Subject: Re: [Gluster-users] NFS subdirectory Solaris client > > > > Hi Anthony, > > > > Please also specify this option port=38467, and try mounting it. > > > > With regards, > > Shishir > > > > - Original Message - > > From: "anthony garnier" > > To: gluster-users@gluster.org > > Sent: Wednesday, July 18, 2012 3:21:36 PM > > Subject: [Gluster-users] NFS subdirectory Solaris client > > > > > > > > Hi everyone, > > > > I still have problem to mount subdirectory in NFS on Solaris client : > > > > # mount -o proto=tcp,vers=3 nfs://yval1010:/test/test2 /users/glusterfs_mnt > > nfs mount: yval1010: : RPC: Program not registered > > nfs mount: retrying: /users/glusterfs_mnt > > > > > > [2012-07-18 11:43:43.484994] E [nfs3.c:305:__nfs3_get_volume_id] > > (-->/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_lookup+0x125) > > [0x7f5418ea4e15] > > (-->/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_lookup_reply+0x48) > > [0x7f5418e9b908] > > (-->/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_request_xlator_deviceid+0x4c) > > [0x7f5418e9a54c]))) 0-nfs-nfsv3: invalid argument: xl > > [2012-07-18 11:43:43.491088] E [nfs3.c:305:__nfs3_get_volume_id] > > (-->/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_lookup+0x125) > > [0x7f5418ea4e15] > > (-->/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_lookup_reply+0x48) > > [0x7f5418e9b908] > > (-->/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_request_xlator_deviceid+0x4c) > > [0x7f5418e9a54c]))) 0-nfs-nfsv3: invalid argument: xl > > [2012-07-18 11:43:48.494268] E [nfs3.c:305:__nfs3_get_volume_id] > > (-->/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_lookup+0x125) > > [0x7f5418ea4e15] > > (-->/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_lookup_reply+0x48) > > [0x7f5418e9b908] > > (-->/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_request_xlator_deviceid+0x4c) > > [0x7f5418e9a54c]))) 0-nfs-nfsv3: invalid argument: xl > > [2012-07-18 11:43:48.992370] W [socket.c:195:__socket_rwv] > > 0-socket.nfs-server: readv failed (Connection reset by peer) > > [2012-07-18 11:43:57.422070] W [socket.c:195:__socket_rwv] > > 0-socket.nfs-server: readv failed (Connection reset by peer) > > [2012-07-18 11:43:58.498666] E [nfs3.c:305:__nfs3_get_volume_id] > > (-->/usr/local/lib//glusterfs/3.3.
Re: [Gluster-users] NFS mounts with glusterd on localhost - reliable or not?
It was pretty confusing to read this thread. Hope I can clarify the questions here. The original question by Tomasz was whether the behavior seen in https://bugzilla.redhat.com/show_bug.cgi?id=GLUSTER-2320 is still seen in 3.3.0 - yes - the deadlock can not be avoided and still seen when the machine is running low on memory as a write call by gluster-nfs process triggers for an nfs-client cache flush in kernel which in turn tries to write the cached data to an already blocked glusterfs-nfs. Hence avoid this kind of setup. The other discussion in this thread was related to NLM which has been implemented in 3.3.0. This is to support locking calls from the NFS clients to support fcntl() locking for the applications running on nfs client. NLM server is implemented in glusterfs as well as kernel. NLM server implemented in kernel is used by kernel-nfsd as well as kernel-nfs-client. Hence if you have an nfs mount point, the kernel-nfs-client automatically starts kernel NLM server. So if glusterfs-nfs process is already running on a system (and hence it also runs its own NLM server) and if you try to do "mount -t nfs someserver:/export /mnt/nfs" on the same system it fails as kernel-nfs-client won't be able to start kernel-NLM-server (because glusterfs NLM server would have already registered with portmapper for NLM service and hence kernel-NLM-server registration with portmapper fails). Workaround is "mount -t nfs -o nolock someserver:/export /mnt/nfs" if you really want to have an nfs mount on the same machine where glusterfs-nfs process is running. ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] "Granular locking" - does this need to be enabled in 3.3.0 ?
Dear Pranith /Anand , Update on our progress with using KVM & Gluster: We built a two server (Dell R710) cluster, each box has... 5 x 500 GB SATA RAID5 array (software raid) an Intel 10GB ethernet HBA. One box has 8GB RAM, the other 48GB both have 2 x E5520 Xeon Centos 6.3 installed Gluster 3.3 installed from the rpm files on the gluster site 1) create a replicated gluster volume (on top of xfs) 2) setup qemu/kvm with a gluster volume (mounts localhost:/gluster-vol) 3) sanlock configured (this is evil!) 4) build a virtual machines with 30GB qcow2 image, 1GB RAM 5) clone this VM into 4 machines 6) check that live migration works (OK) Start basic test cycle: a) migrate all machines to host #1, then reboot host #2 b) watch logs for self-heal to complete c) migrate VM's to host #2, reboot host #1 d) check logs for self heal The above cycle can be repeated numerous times, and completes without error, provided that no (or little) load is on the VM. If I give the VM's a work load, such by running "bonnie++" on each VM, things start to break. 1) it becomes almost impossible to log in to each VM 2) the kernel on each VM starts giving timeout errors i.e. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 3) top / uptime on the hosts shows load average of up to 24 4) dd write speed (block size 1K) to gluster is around 3MB/s on the host While I agree that running bonnie++ on four VM's is possibly unfair, there are load spikes on quiet machines (yum updates etc). I suspect that the I/O of one VM starts blocking that of another VM, and the pressure builds up rapidly on gluster - which does not seem to cope well under pressure. Possibly this is the access pattern / block size of qcow2 disks? I'm (slightly) disappointed. Though it doesn't corrupt data, the I/O performance is < 1% of my hardwares capability. Hopefully work on buffering and other tuning will fix this ? Or maybe the work mentioned getting qemu talking directly to gluster will fix this? best wishes Jake -- Dr Jake Grimmett Head Of Scientific Computing MRC Laboratory of Molecular Biology Hills Road, Cambridge, CB2 0QH, UK. Phone 01223 402219 Mobile 0776 9886539 ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] NFS subdirectory Solaris client
Hi Rajesh , My mistake, I didn't specify that I was in 3.3. I didn't see any particular comment in the Admin doc about subdirectory export. Are you referring to this option: nfs.export-dir, but by default every subdirectory are exported. Regards, Anthony > Date: Thu, 19 Jul 2012 04:42:20 -0400 > From: raj...@redhat.com > To: sokar6...@hotmail.com > CC: gluster-users@gluster.org; sgo...@redhat.com > Subject: Re: [Gluster-users] NFS subdirectory Solaris client > > Hi Anthony, >sub directory mount is not possible with GlusterNfs in 3.2.x version. >you can get the 3.3 gNfs to work for subdir mount on solaris, though it >requires some oblique steps to get it working. The steps are provided in >the documentation (Admin guide i think) for 3.3. > > Regards, > Rajesh Amaravathi, > Software Engineer, GlusterFS > RedHat Inc. > > - Original Message - > From: "anthony garnier" > To: sgo...@redhat.com > Cc: gluster-users@gluster.org > Sent: Thursday, July 19, 2012 12:50:41 PM > Subject: Re: [Gluster-users] NFS subdirectory Solaris client > > > > Hi Shishir, > > I have reconfigured the port to 2049 so there is no need to specify it. > Moreover the mount of the volume works fine, it's only the mount of the > subdirectory that doesn't work. > > Thx and Regards, > > Anthony > > > > > > Date: Wed, 18 Jul 2012 22:54:28 -0400 > > From: sgo...@redhat.com > > To: sokar6...@hotmail.com > > CC: gluster-users@gluster.org > > Subject: Re: [Gluster-users] NFS subdirectory Solaris client > > > > Hi Anthony, > > > > Please also specify this option port=38467, and try mounting it. > > > > With regards, > > Shishir > > > > - Original Message - > > From: "anthony garnier" > > To: gluster-users@gluster.org > > Sent: Wednesday, July 18, 2012 3:21:36 PM > > Subject: [Gluster-users] NFS subdirectory Solaris client > > > > > > > > Hi everyone, > > > > I still have problem to mount subdirectory in NFS on Solaris client : > > > > # mount -o proto=tcp,vers=3 nfs://yval1010:/test/test2 /users/glusterfs_mnt > > nfs mount: yval1010: : RPC: Program not registered > > nfs mount: retrying: /users/glusterfs_mnt > > > > > > [2012-07-18 11:43:43.484994] E [nfs3.c:305:__nfs3_get_volume_id] > > (-->/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_lookup+0x125) > > [0x7f5418ea4e15] > > (-->/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_lookup_reply+0x48) > > [0x7f5418e9b908] > > (-->/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_request_xlator_deviceid+0x4c) > > [0x7f5418e9a54c]))) 0-nfs-nfsv3: invalid argument: xl > > [2012-07-18 11:43:43.491088] E [nfs3.c:305:__nfs3_get_volume_id] > > (-->/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_lookup+0x125) > > [0x7f5418ea4e15] > > (-->/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_lookup_reply+0x48) > > [0x7f5418e9b908] > > (-->/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_request_xlator_deviceid+0x4c) > > [0x7f5418e9a54c]))) 0-nfs-nfsv3: invalid argument: xl > > [2012-07-18 11:43:48.494268] E [nfs3.c:305:__nfs3_get_volume_id] > > (-->/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_lookup+0x125) > > [0x7f5418ea4e15] > > (-->/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_lookup_reply+0x48) > > [0x7f5418e9b908] > > (-->/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_request_xlator_deviceid+0x4c) > > [0x7f5418e9a54c]))) 0-nfs-nfsv3: invalid argument: xl > > [2012-07-18 11:43:48.992370] W [socket.c:195:__socket_rwv] > > 0-socket.nfs-server: readv failed (Connection reset by peer) > > [2012-07-18 11:43:57.422070] W [socket.c:195:__socket_rwv] > > 0-socket.nfs-server: readv failed (Connection reset by peer) > > [2012-07-18 11:43:58.498666] E [nfs3.c:305:__nfs3_get_volume_id] > > (-->/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_lookup+0x125) > > [0x7f5418ea4e15] > > (-->/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_lookup_reply+0x48) > > [0x7f5418e9b908] > > (-->/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_request_xlator_deviceid+0x4c) > > [0x7f5418e9a54c]))) 0-nfs-nfsv3: invalid argument: xl > > > > > > Can someone confirm that ? > > > > Regards, > > > > Anthony > > > > ___ > > Gluster-users mailing list > > Gluster-users@gluster.org > > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users > > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] NFS subdirectory Solaris client
Hi Anthony, sub directory mount is not possible with GlusterNfs in 3.2.x version. you can get the 3.3 gNfs to work for subdir mount on solaris, though it requires some oblique steps to get it working. The steps are provided in the documentation (Admin guide i think) for 3.3. Regards, Rajesh Amaravathi, Software Engineer, GlusterFS RedHat Inc. - Original Message - From: "anthony garnier" To: sgo...@redhat.com Cc: gluster-users@gluster.org Sent: Thursday, July 19, 2012 12:50:41 PM Subject: Re: [Gluster-users] NFS subdirectory Solaris client Hi Shishir, I have reconfigured the port to 2049 so there is no need to specify it. Moreover the mount of the volume works fine, it's only the mount of the subdirectory that doesn't work. Thx and Regards, Anthony > Date: Wed, 18 Jul 2012 22:54:28 -0400 > From: sgo...@redhat.com > To: sokar6...@hotmail.com > CC: gluster-users@gluster.org > Subject: Re: [Gluster-users] NFS subdirectory Solaris client > > Hi Anthony, > > Please also specify this option port=38467, and try mounting it. > > With regards, > Shishir > > - Original Message - > From: "anthony garnier" > To: gluster-users@gluster.org > Sent: Wednesday, July 18, 2012 3:21:36 PM > Subject: [Gluster-users] NFS subdirectory Solaris client > > > > Hi everyone, > > I still have problem to mount subdirectory in NFS on Solaris client : > > # mount -o proto=tcp,vers=3 nfs://yval1010:/test/test2 /users/glusterfs_mnt > nfs mount: yval1010: : RPC: Program not registered > nfs mount: retrying: /users/glusterfs_mnt > > > [2012-07-18 11:43:43.484994] E [nfs3.c:305:__nfs3_get_volume_id] > (-->/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_lookup+0x125) > [0x7f5418ea4e15] > (-->/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_lookup_reply+0x48) > [0x7f5418e9b908] > (-->/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_request_xlator_deviceid+0x4c) > [0x7f5418e9a54c]))) 0-nfs-nfsv3: invalid argument: xl > [2012-07-18 11:43:43.491088] E [nfs3.c:305:__nfs3_get_volume_id] > (-->/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_lookup+0x125) > [0x7f5418ea4e15] > (-->/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_lookup_reply+0x48) > [0x7f5418e9b908] > (-->/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_request_xlator_deviceid+0x4c) > [0x7f5418e9a54c]))) 0-nfs-nfsv3: invalid argument: xl > [2012-07-18 11:43:48.494268] E [nfs3.c:305:__nfs3_get_volume_id] > (-->/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_lookup+0x125) > [0x7f5418ea4e15] > (-->/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_lookup_reply+0x48) > [0x7f5418e9b908] > (-->/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_request_xlator_deviceid+0x4c) > [0x7f5418e9a54c]))) 0-nfs-nfsv3: invalid argument: xl > [2012-07-18 11:43:48.992370] W [socket.c:195:__socket_rwv] > 0-socket.nfs-server: readv failed (Connection reset by peer) > [2012-07-18 11:43:57.422070] W [socket.c:195:__socket_rwv] > 0-socket.nfs-server: readv failed (Connection reset by peer) > [2012-07-18 11:43:58.498666] E [nfs3.c:305:__nfs3_get_volume_id] > (-->/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_lookup+0x125) > [0x7f5418ea4e15] > (-->/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_lookup_reply+0x48) > [0x7f5418e9b908] > (-->/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_request_xlator_deviceid+0x4c) > [0x7f5418e9a54c]))) 0-nfs-nfsv3: invalid argument: xl > > > Can someone confirm that ? > > Regards, > > Anthony > > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] NFS performance degradation in 3.3
I've just make more tests and without any error log, the NFS glusterfs server raised up to 6.00 load (in a 4 core server) and in the 2 bricks where the "real" files where stored, reached loads of 10. No error message in log files (nfs, bricks, gluster). Will deactivate NLM improve performance? Any other options? Thanks in advance for any hint, Samuel. On 19 July 2012 08:44, samuel wrote: > This are the parameters that are set: > > 59: volume nfs-server > 60: type nfs/server > 61: option nfs.dynamic-volumes on > 62: option nfs.nlm on > 63: option rpc-auth.addr.cloud.allow * > 64: option nfs3.cloud.volume-id 84fcec8c-d11a-43b6-9689-3f39700732b3 > 65: option nfs.enable-ino32 off > 66: option nfs3.cloud.volume-access read-write > 67: option nfs.cloud.disable off > 68: subvolumes cloud > 69: end-volume > > And some errors are: > [2012-07-18 17:57:00.391104] W [socket.c:195:__socket_rwv] > 0-socket.nfs-server: readv failed (Connection reset by peer) > [2012-07-18 17:57:29.805684] W [socket.c:195:__socket_rwv] > 0-socket.nfs-server: readv failed (Connection reset by peer) > [2012-07-18 18:04:08.603822] W [nfs3.c:3525:nfs3svc_rmdir_cbk] 0-nfs: > d037df6: /one/var/datastores/0/99/disk.0 => -1 (Directory not empty) > [2012-07-18 18:04:08.625753] W [nfs3.c:3525:nfs3svc_rmdir_cbk] 0-nfs: > d037dfe: /one/var/datastores/0/99 => -1 (Directory not empty) > > The directory not empty is just an attempt to delete a directory with > files inside but I guess that it should not increase the CPU load. > > Above case is just one of the many times that the NFS daemon started using > CPU but it's not the only scenario (deleting not empyt directory) that > causes the degradation. Sometimes it has happened wihout any concrete error > on the log files. I'll try to make more tests and offer more debug > information. > > Thanks for your answer so far, > Samuel. > > > On 18 July 2012 21:54, Anand Avati wrote: > >> Is there anything in the nfs logs? >> >> Avati >> >> On Wed, Jul 18, 2012 at 9:44 AM, samuel wrote: >> >>> Hi all, >>> >>> We're experiencing with a 4 nodes distributed-replicated environment >>> (replica 2). We were using gluster native client to access the volumes, but >>> we were asked to add NFS accessibility to the volume. We then started the >>> NFS daemon on the bricks. Everything went ok but we started experiencing >>> some performance degradation accessing the volume. >>> We debugged the problem and found out that quite often the NFS glusterfs >>> process (NOT the glusterfsd) eats up all the CPU and the server where the >>> NFS is being exported starts offering really bad performance. >>> >>> Is there any issue with 3.3 and NFS performance? Are there any NFS >>> parameters to play with that can mitigate this degradation (standard R/W >>> values drops to a quarter of standard values)? >>> >>> Thanks in advance for any help, >>> >>> Samuel. >>> >>> ___ >>> Gluster-users mailing list >>> Gluster-users@gluster.org >>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users >>> >>> >> > ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] NFS subdirectory Solaris client
Hi Shishir, I have reconfigured the port to 2049 so there is no need to specify it. Moreover the mount of the volume works fine, it's only the mount of the subdirectory that doesn't work. Thx and Regards, Anthony > Date: Wed, 18 Jul 2012 22:54:28 -0400 > From: sgo...@redhat.com > To: sokar6...@hotmail.com > CC: gluster-users@gluster.org > Subject: Re: [Gluster-users] NFS subdirectory Solaris client > > Hi Anthony, > > Please also specify this option port=38467, and try mounting it. > > With regards, > Shishir > > - Original Message - > From: "anthony garnier" > To: gluster-users@gluster.org > Sent: Wednesday, July 18, 2012 3:21:36 PM > Subject: [Gluster-users] NFS subdirectory Solaris client > > > > Hi everyone, > > I still have problem to mount subdirectory in NFS on Solaris client : > > # mount -o proto=tcp,vers=3 nfs://yval1010:/test/test2 /users/glusterfs_mnt > nfs mount: yval1010: : RPC: Program not registered > nfs mount: retrying: /users/glusterfs_mnt > > > [2012-07-18 11:43:43.484994] E [nfs3.c:305:__nfs3_get_volume_id] > (-->/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_lookup+0x125) > [0x7f5418ea4e15] > (-->/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_lookup_reply+0x48) > [0x7f5418e9b908] > (-->/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_request_xlator_deviceid+0x4c) > [0x7f5418e9a54c]))) 0-nfs-nfsv3: invalid argument: xl > [2012-07-18 11:43:43.491088] E [nfs3.c:305:__nfs3_get_volume_id] > (-->/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_lookup+0x125) > [0x7f5418ea4e15] > (-->/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_lookup_reply+0x48) > [0x7f5418e9b908] > (-->/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_request_xlator_deviceid+0x4c) > [0x7f5418e9a54c]))) 0-nfs-nfsv3: invalid argument: xl > [2012-07-18 11:43:48.494268] E [nfs3.c:305:__nfs3_get_volume_id] > (-->/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_lookup+0x125) > [0x7f5418ea4e15] > (-->/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_lookup_reply+0x48) > [0x7f5418e9b908] > (-->/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_request_xlator_deviceid+0x4c) > [0x7f5418e9a54c]))) 0-nfs-nfsv3: invalid argument: xl > [2012-07-18 11:43:48.992370] W [socket.c:195:__socket_rwv] > 0-socket.nfs-server: readv failed (Connection reset by peer) > [2012-07-18 11:43:57.422070] W [socket.c:195:__socket_rwv] > 0-socket.nfs-server: readv failed (Connection reset by peer) > [2012-07-18 11:43:58.498666] E [nfs3.c:305:__nfs3_get_volume_id] > (-->/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_lookup+0x125) > [0x7f5418ea4e15] > (-->/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_lookup_reply+0x48) > [0x7f5418e9b908] > (-->/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_request_xlator_deviceid+0x4c) > [0x7f5418e9a54c]))) 0-nfs-nfsv3: invalid argument: xl > > > Can someone confirm that ? > > Regards, > > Anthony > > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
