Re: [Gluster-users] glusterfs process eats memory until OOM kills it
On 07/19/2012 01:46 AM, Andreas Kurz wrote: Hi, I'm running GlusterFS 3.2.6 in AWS on CentOS 6.2, running a distributed/replicated setup. Type: Distributed-Replicate Status: Started Number of Bricks: 4 x 2 = 8 Whenever geo-replication is activated, the corresponding glusterfs process on the server starts eating memory (and using lot of cpu) until oom killer strikes back. This happens once user starting to change files via glusterfs mount and gsyncd starts crawling through the directory tree looking for changes. Network traffic between the servers is quite high, typically 10Mbit/s ... the vast majority are lookups and getxattr request from the server running geo-replication. I also created a state dump (5MB bzip2 archive) of this glusterfs process when eating about 9GB if that is needed for debugging I can upload it somewhere (Bugzilla?). Dropping dentries and inodes reclaims about 1GB. Any ideas? A bug? Any recommended tunings, maybe a gsyncd option? I changed these values: performance.stat-prefetch: off performance.quick-read: off performance.cache-refresh-timeout: 1 performance.read-ahead: off geo-replication.indexing: on nfs.disable: on network.ping-timeout: 10 performance.cache-size: 1073741824 Can you also try with performance.io-cache being set to off? If that doesn't show any improvement, please raise a bug and attach the statedump to it. Thanks, Vijay ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] NFS subdirectory Solaris client
Hi Shishir, I have reconfigured the port to 2049 so there is no need to specify it. Moreover the mount of the volume works fine, it's only the mount of the subdirectory that doesn't work. Thx and Regards, Anthony Date: Wed, 18 Jul 2012 22:54:28 -0400 From: sgo...@redhat.com To: sokar6...@hotmail.com CC: gluster-users@gluster.org Subject: Re: [Gluster-users] NFS subdirectory Solaris client Hi Anthony, Please also specify this option port=38467, and try mounting it. With regards, Shishir - Original Message - From: anthony garnier sokar6...@hotmail.com To: gluster-users@gluster.org Sent: Wednesday, July 18, 2012 3:21:36 PM Subject: [Gluster-users] NFS subdirectory Solaris client Hi everyone, I still have problem to mount subdirectory in NFS on Solaris client : # mount -o proto=tcp,vers=3 nfs://yval1010:/test/test2 /users/glusterfs_mnt nfs mount: yval1010: : RPC: Program not registered nfs mount: retrying: /users/glusterfs_mnt [2012-07-18 11:43:43.484994] E [nfs3.c:305:__nfs3_get_volume_id] (--/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_lookup+0x125) [0x7f5418ea4e15] (--/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_lookup_reply+0x48) [0x7f5418e9b908] (--/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_request_xlator_deviceid+0x4c) [0x7f5418e9a54c]))) 0-nfs-nfsv3: invalid argument: xl [2012-07-18 11:43:43.491088] E [nfs3.c:305:__nfs3_get_volume_id] (--/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_lookup+0x125) [0x7f5418ea4e15] (--/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_lookup_reply+0x48) [0x7f5418e9b908] (--/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_request_xlator_deviceid+0x4c) [0x7f5418e9a54c]))) 0-nfs-nfsv3: invalid argument: xl [2012-07-18 11:43:48.494268] E [nfs3.c:305:__nfs3_get_volume_id] (--/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_lookup+0x125) [0x7f5418ea4e15] (--/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_lookup_reply+0x48) [0x7f5418e9b908] (--/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_request_xlator_deviceid+0x4c) [0x7f5418e9a54c]))) 0-nfs-nfsv3: invalid argument: xl [2012-07-18 11:43:48.992370] W [socket.c:195:__socket_rwv] 0-socket.nfs-server: readv failed (Connection reset by peer) [2012-07-18 11:43:57.422070] W [socket.c:195:__socket_rwv] 0-socket.nfs-server: readv failed (Connection reset by peer) [2012-07-18 11:43:58.498666] E [nfs3.c:305:__nfs3_get_volume_id] (--/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_lookup+0x125) [0x7f5418ea4e15] (--/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_lookup_reply+0x48) [0x7f5418e9b908] (--/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_request_xlator_deviceid+0x4c) [0x7f5418e9a54c]))) 0-nfs-nfsv3: invalid argument: xl Can someone confirm that ? Regards, Anthony ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] NFS performance degradation in 3.3
I've just make more tests and without any error log, the NFS glusterfs server raised up to 6.00 load (in a 4 core server) and in the 2 bricks where the real files where stored, reached loads of 10. No error message in log files (nfs, bricks, gluster). Will deactivate NLM improve performance? Any other options? Thanks in advance for any hint, Samuel. On 19 July 2012 08:44, samuel sam...@gmail.com wrote: This are the parameters that are set: 59: volume nfs-server 60: type nfs/server 61: option nfs.dynamic-volumes on 62: option nfs.nlm on 63: option rpc-auth.addr.cloud.allow * 64: option nfs3.cloud.volume-id 84fcec8c-d11a-43b6-9689-3f39700732b3 65: option nfs.enable-ino32 off 66: option nfs3.cloud.volume-access read-write 67: option nfs.cloud.disable off 68: subvolumes cloud 69: end-volume And some errors are: [2012-07-18 17:57:00.391104] W [socket.c:195:__socket_rwv] 0-socket.nfs-server: readv failed (Connection reset by peer) [2012-07-18 17:57:29.805684] W [socket.c:195:__socket_rwv] 0-socket.nfs-server: readv failed (Connection reset by peer) [2012-07-18 18:04:08.603822] W [nfs3.c:3525:nfs3svc_rmdir_cbk] 0-nfs: d037df6: /one/var/datastores/0/99/disk.0 = -1 (Directory not empty) [2012-07-18 18:04:08.625753] W [nfs3.c:3525:nfs3svc_rmdir_cbk] 0-nfs: d037dfe: /one/var/datastores/0/99 = -1 (Directory not empty) The directory not empty is just an attempt to delete a directory with files inside but I guess that it should not increase the CPU load. Above case is just one of the many times that the NFS daemon started using CPU but it's not the only scenario (deleting not empyt directory) that causes the degradation. Sometimes it has happened wihout any concrete error on the log files. I'll try to make more tests and offer more debug information. Thanks for your answer so far, Samuel. On 18 July 2012 21:54, Anand Avati anand.av...@gmail.com wrote: Is there anything in the nfs logs? Avati On Wed, Jul 18, 2012 at 9:44 AM, samuel sam...@gmail.com wrote: Hi all, We're experiencing with a 4 nodes distributed-replicated environment (replica 2). We were using gluster native client to access the volumes, but we were asked to add NFS accessibility to the volume. We then started the NFS daemon on the bricks. Everything went ok but we started experiencing some performance degradation accessing the volume. We debugged the problem and found out that quite often the NFS glusterfs process (NOT the glusterfsd) eats up all the CPU and the server where the NFS is being exported starts offering really bad performance. Is there any issue with 3.3 and NFS performance? Are there any NFS parameters to play with that can mitigate this degradation (standard R/W values drops to a quarter of standard values)? Thanks in advance for any help, Samuel. ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] NFS subdirectory Solaris client
Hi Anthony, sub directory mount is not possible with GlusterNfs in 3.2.x version. you can get the 3.3 gNfs to work for subdir mount on solaris, though it requires some oblique steps to get it working. The steps are provided in the documentation (Admin guide i think) for 3.3. Regards, Rajesh Amaravathi, Software Engineer, GlusterFS RedHat Inc. - Original Message - From: anthony garnier sokar6...@hotmail.com To: sgo...@redhat.com Cc: gluster-users@gluster.org Sent: Thursday, July 19, 2012 12:50:41 PM Subject: Re: [Gluster-users] NFS subdirectory Solaris client Hi Shishir, I have reconfigured the port to 2049 so there is no need to specify it. Moreover the mount of the volume works fine, it's only the mount of the subdirectory that doesn't work. Thx and Regards, Anthony Date: Wed, 18 Jul 2012 22:54:28 -0400 From: sgo...@redhat.com To: sokar6...@hotmail.com CC: gluster-users@gluster.org Subject: Re: [Gluster-users] NFS subdirectory Solaris client Hi Anthony, Please also specify this option port=38467, and try mounting it. With regards, Shishir - Original Message - From: anthony garnier sokar6...@hotmail.com To: gluster-users@gluster.org Sent: Wednesday, July 18, 2012 3:21:36 PM Subject: [Gluster-users] NFS subdirectory Solaris client Hi everyone, I still have problem to mount subdirectory in NFS on Solaris client : # mount -o proto=tcp,vers=3 nfs://yval1010:/test/test2 /users/glusterfs_mnt nfs mount: yval1010: : RPC: Program not registered nfs mount: retrying: /users/glusterfs_mnt [2012-07-18 11:43:43.484994] E [nfs3.c:305:__nfs3_get_volume_id] (--/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_lookup+0x125) [0x7f5418ea4e15] (--/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_lookup_reply+0x48) [0x7f5418e9b908] (--/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_request_xlator_deviceid+0x4c) [0x7f5418e9a54c]))) 0-nfs-nfsv3: invalid argument: xl [2012-07-18 11:43:43.491088] E [nfs3.c:305:__nfs3_get_volume_id] (--/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_lookup+0x125) [0x7f5418ea4e15] (--/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_lookup_reply+0x48) [0x7f5418e9b908] (--/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_request_xlator_deviceid+0x4c) [0x7f5418e9a54c]))) 0-nfs-nfsv3: invalid argument: xl [2012-07-18 11:43:48.494268] E [nfs3.c:305:__nfs3_get_volume_id] (--/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_lookup+0x125) [0x7f5418ea4e15] (--/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_lookup_reply+0x48) [0x7f5418e9b908] (--/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_request_xlator_deviceid+0x4c) [0x7f5418e9a54c]))) 0-nfs-nfsv3: invalid argument: xl [2012-07-18 11:43:48.992370] W [socket.c:195:__socket_rwv] 0-socket.nfs-server: readv failed (Connection reset by peer) [2012-07-18 11:43:57.422070] W [socket.c:195:__socket_rwv] 0-socket.nfs-server: readv failed (Connection reset by peer) [2012-07-18 11:43:58.498666] E [nfs3.c:305:__nfs3_get_volume_id] (--/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_lookup+0x125) [0x7f5418ea4e15] (--/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_lookup_reply+0x48) [0x7f5418e9b908] (--/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_request_xlator_deviceid+0x4c) [0x7f5418e9a54c]))) 0-nfs-nfsv3: invalid argument: xl Can someone confirm that ? Regards, Anthony ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] NFS subdirectory Solaris client
Hi Rajesh , My mistake, I didn't specify that I was in 3.3. I didn't see any particular comment in the Admin doc about subdirectory export. Are you referring to this option: nfs.export-dir, but by default every subdirectory are exported. Regards, Anthony Date: Thu, 19 Jul 2012 04:42:20 -0400 From: raj...@redhat.com To: sokar6...@hotmail.com CC: gluster-users@gluster.org; sgo...@redhat.com Subject: Re: [Gluster-users] NFS subdirectory Solaris client Hi Anthony, sub directory mount is not possible with GlusterNfs in 3.2.x version. you can get the 3.3 gNfs to work for subdir mount on solaris, though it requires some oblique steps to get it working. The steps are provided in the documentation (Admin guide i think) for 3.3. Regards, Rajesh Amaravathi, Software Engineer, GlusterFS RedHat Inc. - Original Message - From: anthony garnier sokar6...@hotmail.com To: sgo...@redhat.com Cc: gluster-users@gluster.org Sent: Thursday, July 19, 2012 12:50:41 PM Subject: Re: [Gluster-users] NFS subdirectory Solaris client Hi Shishir, I have reconfigured the port to 2049 so there is no need to specify it. Moreover the mount of the volume works fine, it's only the mount of the subdirectory that doesn't work. Thx and Regards, Anthony Date: Wed, 18 Jul 2012 22:54:28 -0400 From: sgo...@redhat.com To: sokar6...@hotmail.com CC: gluster-users@gluster.org Subject: Re: [Gluster-users] NFS subdirectory Solaris client Hi Anthony, Please also specify this option port=38467, and try mounting it. With regards, Shishir - Original Message - From: anthony garnier sokar6...@hotmail.com To: gluster-users@gluster.org Sent: Wednesday, July 18, 2012 3:21:36 PM Subject: [Gluster-users] NFS subdirectory Solaris client Hi everyone, I still have problem to mount subdirectory in NFS on Solaris client : # mount -o proto=tcp,vers=3 nfs://yval1010:/test/test2 /users/glusterfs_mnt nfs mount: yval1010: : RPC: Program not registered nfs mount: retrying: /users/glusterfs_mnt [2012-07-18 11:43:43.484994] E [nfs3.c:305:__nfs3_get_volume_id] (--/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_lookup+0x125) [0x7f5418ea4e15] (--/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_lookup_reply+0x48) [0x7f5418e9b908] (--/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_request_xlator_deviceid+0x4c) [0x7f5418e9a54c]))) 0-nfs-nfsv3: invalid argument: xl [2012-07-18 11:43:43.491088] E [nfs3.c:305:__nfs3_get_volume_id] (--/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_lookup+0x125) [0x7f5418ea4e15] (--/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_lookup_reply+0x48) [0x7f5418e9b908] (--/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_request_xlator_deviceid+0x4c) [0x7f5418e9a54c]))) 0-nfs-nfsv3: invalid argument: xl [2012-07-18 11:43:48.494268] E [nfs3.c:305:__nfs3_get_volume_id] (--/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_lookup+0x125) [0x7f5418ea4e15] (--/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_lookup_reply+0x48) [0x7f5418e9b908] (--/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_request_xlator_deviceid+0x4c) [0x7f5418e9a54c]))) 0-nfs-nfsv3: invalid argument: xl [2012-07-18 11:43:48.992370] W [socket.c:195:__socket_rwv] 0-socket.nfs-server: readv failed (Connection reset by peer) [2012-07-18 11:43:57.422070] W [socket.c:195:__socket_rwv] 0-socket.nfs-server: readv failed (Connection reset by peer) [2012-07-18 11:43:58.498666] E [nfs3.c:305:__nfs3_get_volume_id] (--/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_lookup+0x125) [0x7f5418ea4e15] (--/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_lookup_reply+0x48) [0x7f5418e9b908] (--/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_request_xlator_deviceid+0x4c) [0x7f5418e9a54c]))) 0-nfs-nfsv3: invalid argument: xl Can someone confirm that ? Regards, Anthony ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Granular locking - does this need to be enabled in 3.3.0 ?
Dear Pranith /Anand , Update on our progress with using KVM Gluster: We built a two server (Dell R710) cluster, each box has... 5 x 500 GB SATA RAID5 array (software raid) an Intel 10GB ethernet HBA. One box has 8GB RAM, the other 48GB both have 2 x E5520 Xeon Centos 6.3 installed Gluster 3.3 installed from the rpm files on the gluster site 1) create a replicated gluster volume (on top of xfs) 2) setup qemu/kvm with a gluster volume (mounts localhost:/gluster-vol) 3) sanlock configured (this is evil!) 4) build a virtual machines with 30GB qcow2 image, 1GB RAM 5) clone this VM into 4 machines 6) check that live migration works (OK) Start basic test cycle: a) migrate all machines to host #1, then reboot host #2 b) watch logs for self-heal to complete c) migrate VM's to host #2, reboot host #1 d) check logs for self heal The above cycle can be repeated numerous times, and completes without error, provided that no (or little) load is on the VM. If I give the VM's a work load, such by running bonnie++ on each VM, things start to break. 1) it becomes almost impossible to log in to each VM 2) the kernel on each VM starts giving timeout errors i.e. echo 0 /proc/sys/kernel/hung_task_timeout_secs 3) top / uptime on the hosts shows load average of up to 24 4) dd write speed (block size 1K) to gluster is around 3MB/s on the host While I agree that running bonnie++ on four VM's is possibly unfair, there are load spikes on quiet machines (yum updates etc). I suspect that the I/O of one VM starts blocking that of another VM, and the pressure builds up rapidly on gluster - which does not seem to cope well under pressure. Possibly this is the access pattern / block size of qcow2 disks? I'm (slightly) disappointed. Though it doesn't corrupt data, the I/O performance is 1% of my hardwares capability. Hopefully work on buffering and other tuning will fix this ? Or maybe the work mentioned getting qemu talking directly to gluster will fix this? best wishes Jake -- Dr Jake Grimmett Head Of Scientific Computing MRC Laboratory of Molecular Biology Hills Road, Cambridge, CB2 0QH, UK. Phone 01223 402219 Mobile 0776 9886539 ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] NFS subdirectory Solaris client
Hi Rajesh, Thank you for the workaround, it works fine. Can we expect a patch in the future release to avoid this procedure ? Regards, Anthony Date: Thu, 19 Jul 2012 05:30:24 -0400 From: raj...@redhat.com To: sokar6...@hotmail.com CC: gluster-users@gluster.org; sgo...@redhat.com Subject: Re: [Gluster-users] NFS subdirectory Solaris client fwding from krishna's mail, need to update the docs with this info: Here are the steps: on linux client machine: * make sure you have the directory you want to export is already created, if not create it. if /mnt/nfs is the mount point do this: mkdir /mnt/nfs/subdir * on storage node do: gluster volume set volname nfs.export-dir /subdir gluster volume set volname nfs.mount-udp on * do a showmount -e storage-node-ip to see that subdir is exported too. * on a LINUX client do this: mount -o proto=tcp storage-node-ip:/volname/subdir /mnt/nfs i.e you are mounting the subdir. * now on SOLARIS client do this: mount nfs://storage-node-ip:/volname/subdir /mnt/nfs you should be able to access the exported subdir on Solaris machine. note that you have to mount the subdir on a linux machine first with proto=tcp before trying to mount on solaris machine. Regards, Rajesh Amaravathi, Software Engineer, GlusterFS RedHat Inc. - Original Message - From: anthony garnier sokar6...@hotmail.com To: raj...@redhat.com Cc: gluster-users@gluster.org, sgo...@redhat.com Sent: Thursday, July 19, 2012 2:19:24 PM Subject: RE: [Gluster-users] NFS subdirectory Solaris client Hi Rajesh , My mistake, I didn't specify that I was in 3.3. I didn't see any particular comment in the Admin doc about subdirectory export. Are you referring to this option: nfs.export-dir, but by default every subdirectory are exported. Regards, Anthony Date: Thu, 19 Jul 2012 04:42:20 -0400 From: raj...@redhat.com To: sokar6...@hotmail.com CC: gluster-users@gluster.org; sgo...@redhat.com Subject: Re: [Gluster-users] NFS subdirectory Solaris client Hi Anthony, sub directory mount is not possible with GlusterNfs in 3.2.x version. you can get the 3.3 gNfs to work for subdir mount on solaris, though it requires some oblique steps to get it working. The steps are provided in the documentation (Admin guide i think) for 3.3. Regards, Rajesh Amaravathi, Software Engineer, GlusterFS RedHat Inc. - Original Message - From: anthony garnier sokar6...@hotmail.com To: sgo...@redhat.com Cc: gluster-users@gluster.org Sent: Thursday, July 19, 2012 12:50:41 PM Subject: Re: [Gluster-users] NFS subdirectory Solaris client Hi Shishir, I have reconfigured the port to 2049 so there is no need to specify it. Moreover the mount of the volume works fine, it's only the mount of the subdirectory that doesn't work. Thx and Regards, Anthony Date: Wed, 18 Jul 2012 22:54:28 -0400 From: sgo...@redhat.com To: sokar6...@hotmail.com CC: gluster-users@gluster.org Subject: Re: [Gluster-users] NFS subdirectory Solaris client Hi Anthony, Please also specify this option port=38467, and try mounting it. With regards, Shishir - Original Message - From: anthony garnier sokar6...@hotmail.com To: gluster-users@gluster.org Sent: Wednesday, July 18, 2012 3:21:36 PM Subject: [Gluster-users] NFS subdirectory Solaris client Hi everyone, I still have problem to mount subdirectory in NFS on Solaris client : # mount -o proto=tcp,vers=3 nfs://yval1010:/test/test2 /users/glusterfs_mnt nfs mount: yval1010: : RPC: Program not registered nfs mount: retrying: /users/glusterfs_mnt [2012-07-18 11:43:43.484994] E [nfs3.c:305:__nfs3_get_volume_id] (--/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_lookup+0x125) [0x7f5418ea4e15] (--/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_lookup_reply+0x48) [0x7f5418e9b908] (--/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_request_xlator_deviceid+0x4c) [0x7f5418e9a54c]))) 0-nfs-nfsv3: invalid argument: xl [2012-07-18 11:43:43.491088] E [nfs3.c:305:__nfs3_get_volume_id] (--/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_lookup+0x125) [0x7f5418ea4e15] (--/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_lookup_reply+0x48) [0x7f5418e9b908] (--/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_request_xlator_deviceid+0x4c) [0x7f5418e9a54c]))) 0-nfs-nfsv3: invalid argument: xl [2012-07-18 11:43:48.494268] E [nfs3.c:305:__nfs3_get_volume_id] (--/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_lookup+0x125) [0x7f5418ea4e15] (--/usr/local/lib//glusterfs/3.3.0/xlator/nfs/server.so(nfs3_lookup_reply+0x48) [0x7f5418e9b908]
Re: [Gluster-users] NFS mounts with glusterd on localhost - reliable or not?
On Wed, Jul 18, 2012 at 01:56:04AM -0400, Rajesh Amaravathi wrote: As to whether we can disable parts of kernel NFS (I'm assuming e.g NLM), I think its not really necessary since we can mount other exports with nolock option. If we take out NLM or disable NLM at the kernel level, then every time we need NLM from kernel, we need to recompile the kernel/have a secondary kernel with NLM and reboot, much tedious than simply killing Gluster/fuse NFS and after kernel NLM's work is done, restart Gluster/fuse NFS. My $0.02 :) Since there's good reason to want locking with Gluster NFS, wouldn't the answer be to add code to kernel that would allow the kernel's locking to be turned off and on in the standard way - a file called something like kernel_nfs_locking that would hold either a 0 or 1? Obviously the kernel's NFS support was built on the assumption that no one with kernel NFS available would want to run userland NFS. Gluster shows that assumption is wrong. So wouldn't it be sensible for someone on the Gluster team to be submitting kernel patches to fix this oversight? Best, Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] NFS mounts with glusterd on localhost - reliable or not?
On Thu, Jul 19, 2012 at 05:16:24AM -0400, Krishna Srinivas wrote: It was pretty confusing to read this thread. Hope I can clarify the questions here. Thanks. I was confused. The other discussion in this thread was related to NLM which has been implemented in 3.3.0. This is to support locking calls from the NFS clients to support fcntl() locking for the applications running on nfs client. NLM server is implemented in glusterfs as well as kernel. NLM server implemented in kernel is used by kernel-nfsd as well as kernel-nfs-client. Hence if you have an nfs mount point, the kernel-nfs-client automatically starts kernel NLM server. So if glusterfs-nfs process is already running on a system (and hence it also runs its own NLM server) and if you try to do mount -t nfs someserver:/export /mnt/nfs on the same system it fails as kernel-nfs-client won't be able to start kernel-NLM-server (because glusterfs NLM server would have already registered with portmapper for NLM service and hence kernel-NLM-server registration with portmapper fails). Workaround is mount -t nfs -o nolock someserver:/export /mnt/nfs if you really want to have an nfs mount on the same machine where glusterfs-nfs process is running. So, if you want to run both at once, only one can lock. Is the architecture of NLM such that there could never be a single NLM server for both Gluster and kernel (whether that single server be Gluster's or kernel's)? Thanks, Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Granular locking - does this need to be enabled in 3.3.0 ?
On Thu, Jul 19, 2012 at 2:14 AM, Jake Grimmett j...@mrc-lmb.cam.ac.ukwrote: Dear Pranith /Anand , Update on our progress with using KVM Gluster: We built a two server (Dell R710) cluster, each box has... 5 x 500 GB SATA RAID5 array (software raid) an Intel 10GB ethernet HBA. One box has 8GB RAM, the other 48GB both have 2 x E5520 Xeon Centos 6.3 installed Gluster 3.3 installed from the rpm files on the gluster site 1) create a replicated gluster volume (on top of xfs) 2) setup qemu/kvm with a gluster volume (mounts localhost:/gluster-vol) 3) sanlock configured (this is evil!) 4) build a virtual machines with 30GB qcow2 image, 1GB RAM 5) clone this VM into 4 machines 6) check that live migration works (OK) Start basic test cycle: a) migrate all machines to host #1, then reboot host #2 b) watch logs for self-heal to complete c) migrate VM's to host #2, reboot host #1 d) check logs for self heal The above cycle can be repeated numerous times, and completes without error, provided that no (or little) load is on the VM. If I give the VM's a work load, such by running bonnie++ on each VM, things start to break. 1) it becomes almost impossible to log in to each VM 2) the kernel on each VM starts giving timeout errors i.e. echo 0 /proc/sys/kernel/hung_task_**timeout_secs 3) top / uptime on the hosts shows load average of up to 24 4) dd write speed (block size 1K) to gluster is around 3MB/s on the host While I agree that running bonnie++ on four VM's is possibly unfair, there are load spikes on quiet machines (yum updates etc). I suspect that the I/O of one VM starts blocking that of another VM, and the pressure builds up rapidly on gluster - which does not seem to cope well under pressure. Possibly this is the access pattern / block size of qcow2 disks? I'm (slightly) disappointed. Though it doesn't corrupt data, the I/O performance is 1% of my hardwares capability. Hopefully work on buffering and other tuning will fix this ? Or maybe the work mentioned getting qemu talking directly to gluster will fix this? Do you mean that the I/O is bad when you are performing the migration? Or bad in general? If it is bad in general the qemu driver should help. Also try presenting each VM a FUSE mount point of its own (we have seen that help improve the overall system IOPs) If it is slow performance only during failover/failback, we probably need to do some more internal QoS tuning to de-prioritize self-heal traffic from preempting VM traffic for resources. Avati ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] glusterfs process eats memory until OOM kills it
On 07/19/2012 08:57 AM, Vijay Bellur wrote: On 07/19/2012 01:46 AM, Andreas Kurz wrote: Hi, I'm running GlusterFS 3.2.6 in AWS on CentOS 6.2, running a distributed/replicated setup. Type: Distributed-Replicate Status: Started Number of Bricks: 4 x 2 = 8 Whenever geo-replication is activated, the corresponding glusterfs process on the server starts eating memory (and using lot of cpu) until oom killer strikes back. This happens once user starting to change files via glusterfs mount and gsyncd starts crawling through the directory tree looking for changes. Network traffic between the servers is quite high, typically 10Mbit/s ... the vast majority are lookups and getxattr request from the server running geo-replication. I also created a state dump (5MB bzip2 archive) of this glusterfs process when eating about 9GB if that is needed for debugging I can upload it somewhere (Bugzilla?). Dropping dentries and inodes reclaims about 1GB. Any ideas? A bug? Any recommended tunings, maybe a gsyncd option? I changed these values: performance.stat-prefetch: off performance.quick-read: off performance.cache-refresh-timeout: 1 performance.read-ahead: off geo-replication.indexing: on nfs.disable: on network.ping-timeout: 10 performance.cache-size: 1073741824 Can you also try with performance.io-cache being set to off? If that doesn't show any improvement, please raise a bug and attach the statedump to it. Done ... https://bugzilla.redhat.com/show_bug.cgi?id=841617 Thanks! Regards, Andreas Thanks, Vijay ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users signature.asc Description: OpenPGP digital signature ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
[Gluster-users] Setting up a new gluster volume using a block device
I am new to gluster so please be a bit patient with me. I am trying to setup a gluster volume with the bricks being /dev/sdb1 (or /dev/sdb, I have tried both) ... etc. however I get an error message and the operation fails. I do not have the error message (sorry!) the only brick I have available for testing is tied up at the moment so I cant rerun the test to get the exact error. If I can I will post it later. However if I mount the partition I am able to create the volume no problem. I have read the admin guide several times but cannot find any reference to using a block device only a mounted device. Is it even possible, or am I crazy for even trying? Jake Sallee Godfather of Bandwidth System Engineer University of Mary Hardin-Baylor 900 College St. Belton TX. 76513 Fone: 254-295-4658 Phax: 254-295-4221 HTTP://WWW.UMHB.EDU ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Setting up a new gluster volume using a block device
Your gluster brick must be a directory, not a block device. The filesystem that directory is located on must supported xattr. David On 7/19/12 1:16 PM, Sallee, Stephen (Jake) wrote: I am new to gluster so please be a bit patient with me. I am trying to setup a gluster volume with the bricks being /dev/sdb1 (or /dev/sdb, I have tried both) ... etc. however I get an error message and the operation fails. I do not have the error message (sorry!) the only brick I have available for testing is tied up at the moment so I cant rerun the test to get the exact error. If I can I will post it later. However if I mount the partition I am able to create the volume no problem. I have read the admin guide several times but cannot find any reference to using a block device only a mounted device. Is it even possible, or am I crazy for even trying? Jake Sallee Godfather of Bandwidth System Engineer University of Mary Hardin-Baylor 900 College St. Belton TX. 76513 Fone: 254-295-4658 Phax: 254-295-4221 HTTP://WWW.UMHB.EDU ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
