Re: [Gluster-users] Invalid argument on delete with GlusterFS 3.1.1 client
Hello All, I have seen what appears to be the same problem when a user tried to change the ownership of some files. The client log file entries were like this: [2011-01-05 17:10:02.93] W [fuse-bridge.c:648:fuse_setattr_cbk] glusterfs-fuse: 16210806: SETATTR() /users/rle/INTERIM/INTERIM_2003_VOR850.nc = -1 (Invalid argument) The error reported on the command line was permission denied I am told, not invalid argument as with the file deletion problem. Unfortunately I didn't get a chance set the log level to TRACE because the user in question went to an NFS client to change the ownership before telling me there had been a problem. However this might make it easier to reproduce the problem, now that we know that it isn't restricted to file deletion. -Dan. - On 01/06/2011 09:07 AM, Thai. Ngo Bao wrote: Dear All, I'd like to confirm that I am having the same problem. I am using glusterfs 3.1.1. I did set volume diagnostics.client-log-level to TRACE, please see the attached file for further information. Thanks for your support. ~Thai -Original Message- From: gluster-users-boun...@gluster.org [mailto:gluster-users-boun...@gluster.org] On Behalf Of Vijay Bellur Sent: Wednesday, January 05, 2011 1:30 AM To: Dan Bretherton Cc: gluster-users Subject: Re: [Gluster-users] Invalid argument on delete with GlusterFS 3.1.1 client On Tuesday 04 January 2011 11:21 PM, Dan Bretherton wrote: Hello Vijay, There is nothing except routine accepted client messages in the brick log files on the servers, and I can't see anything relevant in the /var/log/messages files. On the client the only messages relating to this problem were included in my original mailing list message. Are there some other log files I should be looking at? I don't know how to recreate this problem I'm afraid - it's just a case of waiting until it crops up again. When it does I will try to persuade the users to leave the files in place while I investigate further. When the time comes what should I look for? When it happens again, please set the glusterfs client log level to TRACE through #gluster volume setvolname diagnostics.client-log-level TRACE, perform the delete operation on such files and send across the client log file to us. You can revert back the log level to INFO after you are done with this. Thanks, Vijay ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Invalid argument on delete with GlusterFS 3.1.1 client
No I don't think it is load dependent. The user reported the problem again during the Christmas holiday when very few other people (if any) were using the clients or the servers. -Dan, On 03/01/2011 17:28, Lana Deere wrote: I have seen this same problem but have not been able to find a workaround other than to delete the file from the server directly. I was not able to figure out a way to reproduce the symptom reliably, but in my case I suspect it was related to heavy concurrent access. Does that seem plausible in light of your access patterns? .. Lana (lana.de...@gmail.com) On Wed, Dec 29, 2010 at 3:30 PM, Dan Bretherton d.a.brether...@reading.ac.uk wrote: We have an occasional problem that prevents deletion of certain GlusterFS mounted files. See the following, for example, with corresponding log file message. ke...@cd /glusterfs/atmos/users/kih/ECHAM5/TS4-TEMP ke...@rm HYBRID_TEMP_207212 rm: cannot remove `HYBRID_TEMP_207212': Invalid argument [2010-12-28 00:59:04.298331] W [fuse-bridge.c:888:fuse_unlink_cbk] glusterfs-fuse: 3997: UNLINK() /users/kih/ECHAM5/TS4-TEMP/HYBRID_TEMP_207212 = -1 (Invalid argument) The file was deleted without error on a machine where the volume was mounted via NFS. I have four compute servers that are using the GlusterFS client for performance reasons. Operating system and GlusterFS package details are as follows. [r...@nemo1 TS4-TEMP]# cat /etc/redhat-release CentOS release 5.5 (Final) [r...@nemo1 TS4-TEMP]# uname -a Linux nemo1.nerc-essc.ac.uk 2.6.18-194.el5 #1 SMP Fri Apr 2 14:58:14 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux [r...@nemo1 TS4-TEMP]# rpm -qa | grep -i gluster glusterfs-fuse-3.1.1-1 glusterfs-core-3.1.1-1 Is there anything I can do to stop this from happening, other than using NFS instead of GlusterFS client? -Dan. ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Invalid argument on delete with GlusterFS 3.1.1 client
Hi there, I just want to add that we have exactly the same problem, with many many files on our infrastructure. If I want to delete a file, DHT returned Invalid argument. And in the log file: [2011-01-04 15:20:43.641438] I [dht-common.c:369:dht_revalidate_cbk] dns-dht: subvolume dns-replicate-1 returned -1 (Invalid argument) [2011-01-04 15:20:52.510538] I [dht-common.c:369:dht_revalidate_cbk] dns-dht: subvolume dns-replicate-0 returned -1 (Invalid argument) (and all over again...). Regards. Sam -Message d'origine- De : gluster-users-boun...@gluster.org [mailto:gluster-users-boun...@gluster.org] De la part de Dan Bretherton Envoyé : mardi 4 janvier 2011 14:29 À : Lana Deere Cc : gluster-users Objet : Re: [Gluster-users] Invalid argument on delete with GlusterFS 3.1.1 client No I don't think it is load dependent. The user reported the problem again during the Christmas holiday when very few other people (if any) were using the clients or the servers. -Dan, On 03/01/2011 17:28, Lana Deere wrote: I have seen this same problem but have not been able to find a workaround other than to delete the file from the server directly. I was not able to figure out a way to reproduce the symptom reliably, but in my case I suspect it was related to heavy concurrent access. Does that seem plausible in light of your access patterns? .. Lana (lana.de...@gmail.com) On Wed, Dec 29, 2010 at 3:30 PM, Dan Bretherton d.a.brether...@reading.ac.uk wrote: We have an occasional problem that prevents deletion of certain GlusterFS mounted files. See the following, for example, with corresponding log file message. ke...@cd /glusterfs/atmos/users/kih/ECHAM5/TS4-TEMP ke...@rm HYBRID_TEMP_207212 rm: cannot remove `HYBRID_TEMP_207212': Invalid argument [2010-12-28 00:59:04.298331] W [fuse-bridge.c:888:fuse_unlink_cbk] glusterfs-fuse: 3997: UNLINK() /users/kih/ECHAM5/TS4-TEMP/HYBRID_TEMP_207212 = -1 (Invalid argument) The file was deleted without error on a machine where the volume was mounted via NFS. I have four compute servers that are using the GlusterFS client for performance reasons. Operating system and GlusterFS package details are as follows. [r...@nemo1 TS4-TEMP]# cat /etc/redhat-release CentOS release 5.5 (Final) [r...@nemo1 TS4-TEMP]# uname -a Linux nemo1.nerc-essc.ac.uk 2.6.18-194.el5 #1 SMP Fri Apr 2 14:58:14 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux [r...@nemo1 TS4-TEMP]# rpm -qa | grep -i gluster glusterfs-fuse-3.1.1-1 glusterfs-core-3.1.1-1 Is there anything I can do to stop this from happening, other than using NFS instead of GlusterFS client? -Dan. ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Invalid argument on delete with GlusterFS 3.1.1 client
On 01/03/2011 12:28 PM, Lana Deere wrote: I have seen this same problem but have not been able to find a workaround other than to delete the file from the server directly. I was not able to figure out a way to reproduce the symptom reliably, but in my case I suspect it was related to heavy concurrent access. Does that seem plausible in light of your access patterns? Lana (et al) Check your system times. Make sure all the clocks are sync'ed. A quick pdsh date (assuming you have pdsh installed/configured across your storage nodes) will tell you. We've encountered some odd problems with files disappearing or similar due (in part) to this. This said, there is definitely still a lurking bug in DHT that the time issue won't address, that is similar to this (has to do with strange permissions). The other thing we've tried (see an email back in Dec 2010 time frame) is to turn off some of the stat caching and other bits. Try this and see if it helps: [r...@manager ~]# gluster volume set nfs performance.cache-refresh-timeout 0 Set volume successful [r...@manager ~]# gluster volume set nfs performance.stat-prefetch 0 Set volume successful In one customer case, they have decided to cease using the NFS interface and use the native gluster interface, as this bug was not as visible or less impactful with that. We have some support tickets open on this (though we saw one closed yesterday that wasn't resolved, so we've got to re-open it). Regards, Joe -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics, Inc. email: land...@scalableinformatics.com web : http://scalableinformatics.com http://scalableinformatics.com/sicluster phone: +1 734 786 8423 x121 fax : +1 866 888 3112 cell : +1 734 612 4615 ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Invalid argument on delete with GlusterFS 3.1.1 client
Hello Vijay, There is nothing except routine accepted client messages in the brick log files on the servers, and I can't see anything relevant in the /var/log/messages files. On the client the only messages relating to this problem were included in my original mailing list message. Are there some other log files I should be looking at? I don't know how to recreate this problem I'm afraid - it's just a case of waiting until it crops up again. When it does I will try to persuade the users to leave the files in place while I investigate further. When the time comes what should I look for? -Dan. On 03/01/2011 17:36, Vijay Bellur wrote: On Thursday 30 December 2010 02:00 AM, Dan Bretherton wrote: We have an occasional problem that prevents deletion of certain GlusterFS mounted files. See the following, for example, with corresponding log file message. ke...@cd /glusterfs/atmos/users/kih/ECHAM5/TS4-TEMP ke...@rm HYBRID_TEMP_207212 rm: cannot remove `HYBRID_TEMP_207212': Invalid argument [2010-12-28 00:59:04.298331] W [fuse-bridge.c:888:fuse_unlink_cbk] glusterfs-fuse: 3997: UNLINK() /users/kih/ECHAM5/TS4-TEMP/HYBRID_TEMP_207212 = -1 (Invalid argument) Can you please provide log files from the client and servers when this problem is seen? Thanks, Vijay ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Invalid argument on delete with GlusterFS 3.1.1 client
I have seen this same problem but have not been able to find a workaround other than to delete the file from the server directly. I was not able to figure out a way to reproduce the symptom reliably, but in my case I suspect it was related to heavy concurrent access. Does that seem plausible in light of your access patterns? .. Lana (lana.de...@gmail.com) On Wed, Dec 29, 2010 at 3:30 PM, Dan Bretherton d.a.brether...@reading.ac.uk wrote: We have an occasional problem that prevents deletion of certain GlusterFS mounted files. See the following, for example, with corresponding log file message. ke...@cd /glusterfs/atmos/users/kih/ECHAM5/TS4-TEMP ke...@rm HYBRID_TEMP_207212 rm: cannot remove `HYBRID_TEMP_207212': Invalid argument [2010-12-28 00:59:04.298331] W [fuse-bridge.c:888:fuse_unlink_cbk] glusterfs-fuse: 3997: UNLINK() /users/kih/ECHAM5/TS4-TEMP/HYBRID_TEMP_207212 = -1 (Invalid argument) The file was deleted without error on a machine where the volume was mounted via NFS. I have four compute servers that are using the GlusterFS client for performance reasons. Operating system and GlusterFS package details are as follows. [r...@nemo1 TS4-TEMP]# cat /etc/redhat-release CentOS release 5.5 (Final) [r...@nemo1 TS4-TEMP]# uname -a Linux nemo1.nerc-essc.ac.uk 2.6.18-194.el5 #1 SMP Fri Apr 2 14:58:14 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux [r...@nemo1 TS4-TEMP]# rpm -qa | grep -i gluster glusterfs-fuse-3.1.1-1 glusterfs-core-3.1.1-1 Is there anything I can do to stop this from happening, other than using NFS instead of GlusterFS client? -Dan. ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Invalid argument on delete with GlusterFS 3.1.1 client
On Thursday 30 December 2010 02:00 AM, Dan Bretherton wrote: We have an occasional problem that prevents deletion of certain GlusterFS mounted files. See the following, for example, with corresponding log file message. ke...@cd /glusterfs/atmos/users/kih/ECHAM5/TS4-TEMP ke...@rm HYBRID_TEMP_207212 rm: cannot remove `HYBRID_TEMP_207212': Invalid argument [2010-12-28 00:59:04.298331] W [fuse-bridge.c:888:fuse_unlink_cbk] glusterfs-fuse: 3997: UNLINK() /users/kih/ECHAM5/TS4-TEMP/HYBRID_TEMP_207212 = -1 (Invalid argument) Can you please provide log files from the client and servers when this problem is seen? Thanks, Vijay ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
[Gluster-users] Invalid argument on delete with GlusterFS 3.1.1 client
We have an occasional problem that prevents deletion of certain GlusterFS mounted files. See the following, for example, with corresponding log file message. ke...@cd /glusterfs/atmos/users/kih/ECHAM5/TS4-TEMP ke...@rm HYBRID_TEMP_207212 rm: cannot remove `HYBRID_TEMP_207212': Invalid argument [2010-12-28 00:59:04.298331] W [fuse-bridge.c:888:fuse_unlink_cbk] glusterfs-fuse: 3997: UNLINK() /users/kih/ECHAM5/TS4-TEMP/HYBRID_TEMP_207212 = -1 (Invalid argument) The file was deleted without error on a machine where the volume was mounted via NFS. I have four compute servers that are using the GlusterFS client for performance reasons. Operating system and GlusterFS package details are as follows. [r...@nemo1 TS4-TEMP]# cat /etc/redhat-release CentOS release 5.5 (Final) [r...@nemo1 TS4-TEMP]# uname -a Linux nemo1.nerc-essc.ac.uk 2.6.18-194.el5 #1 SMP Fri Apr 2 14:58:14 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux [r...@nemo1 TS4-TEMP]# rpm -qa | grep -i gluster glusterfs-fuse-3.1.1-1 glusterfs-core-3.1.1-1 Is there anything I can do to stop this from happening, other than using NFS instead of GlusterFS client? -Dan. ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
