Re: [Gluster-users] nfs acces denied
Just another thing i noticed: when i mount the volume as nfs on linux and i do a: find . i get -no- errors at all! it seems to me the issue i'm expericencing on linux and windows is related to the permissions of the files. Whenever nfs tries to read the permissions of a file it fails a lot of the times. So what could be causing this? How does the nfs part of gluster read and transfer the file permissions to the client? (again, when mounted as gluster volume, everything is fine) Van: Carlos Capriotti [capriotti.car...@gmail.com] Verzonden: donderdag 3 april 2014 15:37 To: VAN CAUSBROECK Wannes Cc: gluster-users@gluster.org Onderwerp: Re: [Gluster-users] nfs acces denied I have a bad feeling about this. Sounds like you may have two problems: name resolution and some corrupt or conflicting info on gluster's config files. Those that are under /var. I have NO idea how gluster's NFS server does name resolution, but it is clear is is wrong, and not working wright. There is an option that disables nfs name resolution on volumes, regarding peers, which I use to try speeding up NFS access: nfs.addr-namelookup off You might want to give this a try. Now, it does NOT help with smb name resolution. If you try, from Windows, or even linux, to access the smb share via IP, does it get the right volume ? On Thu, Apr 3, 2014 at 3:13 PM, VAN CAUSBROECK Wannes wannes.vancausbro...@onprvp.fgov.bemailto:wannes.vancausbro...@onprvp.fgov.be wrote: Now it gets even better : If I try to open \\lpr-nas01\caviar_data11 on windows, it automatically opens caviar_data1… which is a completely different volume! Same goes for caviar_data12. Those are the two volumes I deleted and recreated on another disk. Again, mounting as a gluster filesystem works fine, and when mounting on linux as nfs it behaves strangely. What’s also weird in linux as an nfs mount: if I do an ls, it works fine, but the more directories I ‘ls’, or when I do a lot of recursive ‘ls’s, the listing of the directory contents slows down and even freezes all the time. Could this be related to some kind of timeout or buffer or whatever that gets full on the server side? From: Carlos Capriotti [mailto:capriotti.car...@gmail.commailto:capriotti.car...@gmail.com] Sent: donderdag 3 april 2014 13:43 To: VAN CAUSBROECK Wannes Subject: Re: [Gluster-users] nfs acces denied Wannes: It is funny the way life keeps playing with us. I used to live in Belgium until about 6 months ago, working for Kodak. Anyway, back to your problem, I think you have already destroyed and re-created those volumes, if memory serves me well, but I could be wrong. The fact is that gluster creates several configuration files under /var (and I have their guts for this), and they are somewhat complex. In the past I had to change the IP of gluster nodes, on multihomed servers, and I was not able to make gluster work anymore. Because it was a test structure I just scrapped the entire volume and started over. I would try to (re) create the volume if possible. If not, use the same physical mountpoint, create a new folder below that, and create a new volume with that, but this time, use the Ip addresses of the nodes instead of the names. I am a bit paranoid about name resolution, so I tend to hard code IPs everywhere. Your case DOES ring a bell. You might want to search the list in the last couple of weeks. Not farther than 6 weeks, which is more or less the time I am part of it. Now, if you get the chance to update your Os to 6.5, that might be beneficial also. Cheers. On Thu, Apr 3, 2014 at 1:22 PM, VAN CAUSBROECK Wannes wannes.vancausbro...@onprvp.fgov.bemailto:wannes.vancausbro...@onprvp.fgov.be wrote: Hi Carlos, Belgian government, indeed ☺ (pension fund to be more precise) Did: mount -t nfs -o mountproto=tcp,vers=3 localhost:/caviar_data11 /media when I do an ls of the directory, I still get the same weird kind of directory listing. In the logs: [2014-04-03 11:16:58.005527] E [nfs3-helpers.c:3595:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: Lookup failed: gfid:7553c77d-884b-4e28-a3ae-330b3a24b055: Invalid argument [2014-04-03 11:16:58.005559] E [acl3.c:334:acl3_getacl_resume] 0-nfs-ACL: Unable to resolve FH: (127.0.0.1:762http://127.0.0.1:762) caviar_data11 : 7553c77d-884b-4e28-a3ae-330b3a24b055 [2014-04-03 11:16:58.005577] E [acl3.c:342:acl3_getacl_resume] 0-nfs-ACL: unable to open_and_resume [2014-04-03 11:16:58.005814] E [dht-helper.c:429:dht_subvol_get_hashed] (--/usr/lib64/glusterfs/3.4.2/xlator/debug/io-stats.so(io_stats_lookup+0x157) [0x7f4465bf52e7] (--/usr/lib64/libglusterfs.so.0(default_lookup+0x6d) [0x3dfe01c03d] (--/usr/lib64/glusterfs/3.4.2/xlator/cluster/distribute.so(dht_lookup+0xa7e) [0x7f4466037f2e]))) 0-caviar_data11-dht: invalid argument: loc-parent [2014-04-03 11:16:58.006087] W [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-caviar_data11-client-0: remote
Re: [Gluster-users] nfs acces denied
Hello Carlos, This is just one of the volumes that has the problem, but there are several : [root@lpr-nas01 ~]# gluster vol info caviar_data11 Volume Name: caviar_data11 Type: Distribute Volume ID: 4440ef2e-5d60-41a9-bac8-cf4751fc9be2 Status: Started Number of Bricks: 1 Transport-type: tcp Bricks: Brick1: lpr-nas01:/brick-xiv/caviar_data11 [root@lpr-nas01 ~]# gluster vol status caviar_data11 detail Status of volume: caviar_data11 -- Brick: Brick lpr-nas01:/brick-xiv/caviar_data11 Port : 49186 Online : Y Pid : 4592 File System : ext4 Device : /dev/sdb Mount Options: rw Inode Size : 256 Disk Space Free : 7.2TB Total Disk Space : 22.7TB Inode Count : 381786112 Free Inodes : 115286880 Mounting with glusterfs works fine and I have no problems whatsoever. Nfs service is switched off No auto-heal is running Name resolution shouldn't be a problem as the bricks are mounted on the same host running the gluster daemon. I also test the mount from that server. [root@lpr-nas01 ~]# ping lpr-nas01 PING lpr-nas01.onprvp.fgov.be (192.168.151.21) 56(84) bytes of data. 64 bytes from lpr-nas01.onprvp.fgov.be (192.168.151.21): icmp_seq=1 ttl=64 time=0.023 ms [root@lpr-nas01 ~]# nslookup 192.168.151.21 Server: 192.168.147.31 Address: 192.168.147.31#53 21.151.168.192.in-addr.arpa name = lpr-nas01.onprvp.fgov.be. The files were created on a linux machine with those uid/gid, the windows machine need only read acces, so it always falls into the others category. Selinux/iptables are switched off. Thanks for helping out! From: Carlos Capriotti [mailto:capriotti.car...@gmail.com] Sent: maandag 31 maart 2014 18:03 To: VAN CAUSBROECK Wannes Cc: gluster-users@gluster.org1) Subject: Re: [Gluster-users] nfs acces denied maybe it would be nice to see your volume info for affected volumes. Also, on the server side, what happens if you mount the share using glusterfs instead of nfs ? any change the native nfs server is running on your server ? Are there any auto-heal processes running ? There are a few name resolution messages on your logs, that seem to refer to the nodes themselves. Any DNS conflicts ? Maybe add the names of servers to the hosts file ? You MS client seems to be having issues with user/group translation. It seems to create files with gid 1003. (I could be wrong). Again, is SElinux/ACLs/iptables disabled ? All is very inconclusive os far. On Mon, Mar 31, 2014 at 5:26 PM, VAN CAUSBROECK Wannes wannes.vancausbro...@onprvp.fgov.bemailto:wannes.vancausbro...@onprvp.fgov.be wrote: Well, with 'client' i do actually mean the server itself. i've tried forcing linux and windows to nfs V3 and tcp, and on windows i played around with the uid and gid, but the result is always the same On 31 Mar 2014, at 17:22, Carlos Capriotti capriotti.car...@gmail.commailto:capriotti.car...@gmail.com wrote: Well, saying your client-side is linux does not help much. Distro, flavor, etc helps a lot, but I'll take a wild guess here. First, force your NFS mount (client) to use nfs version 3. The same for Microsoft. (It is fair to say I have no idea if the MS client supports v4 or not). Additionally, check that firewalls are disabled on both sides, just for testing. The same goes for SElinux. Windows and ACL, and user mapping is something that might be in your way too. There is a Technet document that describes how to handle this mapping if I am not wrong. Just for testing, mount your nfs share you your own server, using localhost:/nfs_share and see how it goes. It is a good start. Kr, Carlos On Mon, Mar 31, 2014 at 3:58 PM, VAN CAUSBROECK Wannes wannes.vancausbro...@onprvp.fgov.bemailto:wannes.vancausbro...@onprvp.fgov.be wrote: Hello all, I've already tried to post this, but i'm unsure it arrived to the mailing list. I have some issues regarding my nfs mounts. My setup is as follows: Rhel 6.4, gluster 3.4.2-1 running on a vm (4 cores, 8GB ram) attached to a san. I have one disk on which are all the bricks (formatted ext4 in 64 bit mode) of 25TB. On the gluster side of things, everything works without issues. The trouble starts when I mount a volume as an nfs mount. Lots of volumes work without issues, but others behave strangely. The volumes that act weird generally contain many files (can be accidental?). The volumes in question mount without issues, but when I try to go into any subdirectory sometimes it works, sometimes I get errors. On windows with nfs client: access denied In nfslog: [2014-03-31 13:57:58.771241] I [dht-layout.c:638:dht_layout_normalize] 0-caviar_data11-dht: found anomalies in gfid:c8d94120-6851-46ea-9f28-c629a44b1015. holes=1 overlaps=0 [2014-03-31 13:57:58.771348] E [nfs3-helpers.c:3595:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3
Re: [Gluster-users] nfs acces denied
[2014-03-31 12:44:18.941083] I [dht-layout.c:638:dht_layout_normalize] 0-caviar_data11-dht: found anomalies in /2011/201107/20110716/55. holes=1 overlaps=0 This tells us that there is a chunk of hash map missing. My guess would be that the server you're nfs mounting from cannot connect to one of the bricks. On March 31, 2014 6:58:53 AM PDT, VAN CAUSBROECK Wannes wannes.vancausbro...@onprvp.fgov.be wrote: Hello all, I've already tried to post this, but i'm unsure it arrived to the mailing list. I have some issues regarding my nfs mounts. My setup is as follows: Rhel 6.4, gluster 3.4.2-1 running on a vm (4 cores, 8GB ram) attached to a san. I have one disk on which are all the bricks (formatted ext4 in 64 bit mode) of 25TB. On the gluster side of things, everything works without issues. The trouble starts when I mount a volume as an nfs mount. Lots of volumes work without issues, but others behave strangely. The volumes that act weird generally contain many files (can be accidental?). The volumes in question mount without issues, but when I try to go into any subdirectory sometimes it works, sometimes I get errors. On windows with nfs client: access denied In nfslog: [2014-03-31 13:57:58.771241] I [dht-layout.c:638:dht_layout_normalize] 0-caviar_data11-dht: found anomalies in gfid:c8d94120-6851-46ea-9f28-c629a44b1015. holes=1 overlaps=0 [2014-03-31 13:57:58.771348] E [nfs3-helpers.c:3595:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: Lookup failed: gfid:c8d94120-6851-46ea-9f28-c629a44b1015: Invalid argument [2014-03-31 13:57:58.771380] E [nfs3.c:1380:nfs3_lookup_resume] 0-nfs-nfsv3: Unable to resolve FH: (192.168.148.46:984) caviar_data11 : c8d94120-6851-46ea-9f28-c629a44b1015 [2014-03-31 13:57:58.771819] W [nfs3-helpers.c:3380:nfs3_log_common_res] 0-nfs-nfsv3: XID: 1ec28530, LOOKUP: NFS: 22(Invalid argument for operation), POSIX: 14(Bad address) [2014-03-31 13:57:58.798967] I [dht-layout.c:638:dht_layout_normalize] 0-caviar_data11-dht: found anomalies in gfid:14972193-1039-4d7a-aed5-0d7e7eccf57b. holes=1 overlaps=0 [2014-03-31 13:57:58.799039] E [nfs3-helpers.c:3595:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: Lookup failed: gfid:14972193-1039-4d7a-aed5-0d7e7eccf57b: Invalid argument [2014-03-31 13:57:58.799056] E [nfs3.c:1380:nfs3_lookup_resume] 0-nfs-nfsv3: Unable to resolve FH: (192.168.148.46:984) caviar_data11 : 14972193-1039-4d7a-aed5-0d7e7eccf57b [2014-03-31 13:57:58.799088] W [nfs3-helpers.c:3380:nfs3_log_common_res] 0-nfs-nfsv3: XID: 1ec28531, LOOKUP: NFS: 22(Invalid argument for operation), POSIX: 14(Bad address) On linux: [root@lpr-nas01 brick-xiv2]# ll /media/2011/201105/20110530/ ls: /media/2011/201105/20110530/37: No such file or directory total 332 ... drwxrwsr-x 2 nfsnobody 1003 4096 Jun 6 2011 32 drwxrwsr-x 2 nfsnobody 1003 4096 Jun 6 2011 34 drwxrwsr-x 2 nfsnobody 1003 4096 Jun 6 2011 35 drwxrwsr-x 2 nfsnobody 1003 4096 Jun 6 2011 36 drwxrwsr-x 2 nfsnobody 1003 4096 Jun 6 2011 37 ... [root@lpr-nas01 brick-xiv2]# ll /media/2011/201105/20110530/37 ls: /media/2011/201105/20110530/37/NN.073824357.1.tif: No such file or directory ls: /media/2011/201105/20110530/37/NN.073824357.3.tif: No such file or directory total 54 -rwxrwxr-x 0 nfsnobody 1003 9340 Jun 6 2011 NN.073824357.1.tif -rwxrwxr-x 1 nfsnobody 1003 35312 Jun 6 2011 NN.073824357.2.tif -rwxrwxr-x 0 nfsnobody 1003 9340 Jun 6 2011 NN.073824357.3.tif I see in the nfslog: ... [2014-03-31 12:44:18.941083] I [dht-layout.c:638:dht_layout_normalize] 0-caviar_data11-dht: found anomalies in /2011/201107/20110716/55. holes=1 overlaps=0 [2014-03-31 12:44:18.958078] I [dht-layout.c:638:dht_layout_normalize] 0-caviar_data11-dht: found anomalies in /2011/201107/20110716/30. holes=1 overlaps=0 [2014-03-31 12:44:18.959980] I [dht-layout.c:638:dht_layout_normalize] 0-caviar_data11-dht: found anomalies in /2011/201107/20110716/90. holes=1 overlaps=0 [2014-03-31 12:44:18.961094] E [dht-helper.c:429:dht_subvol_get_hashed] (--/usr/lib64/glusterfs/3.4.2/xlator/debug/io-stats.so(io_stats_lookup+0x157) [0x7fd6a61282e7] (--/usr/lib64/libglusterfs.so.0(default_lookup+0x6d) [0x3dfe01c03d] (--/usr/lib64/glusterfs/3.4.2/xlator/cluster/distribute.so(dht_lookup+0xa7e) [0x7fd6a656af2e]))) 0-caviar_data11-dht: invalid argument: loc-parent [2014-03-31 12:44:18.961283] W [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-caviar_data11-client-0: remote operation failed: Invalid argument. Path: gfid:---- (----) [2014-03-31 12:44:18.961319] E [acl3.c:334:acl3_getacl_resume] 0-nfs-ACL: Unable to resolve FH: (192.168.151.21:740) caviar_data11 : ---- [2014-03-31 12:44:18.961338] E [acl3.c:342:acl3_getacl_resume] 0-nfs-ACL: unable to open_and_resume ... The weirdest thing is it changes from time to time which files and directories work and which don't Any ideas? Thanks!
Re: [Gluster-users] nfs acces denied
Hello Joe, Could it be that gluster has issues with the 64 bit ext4 filesystem? As a test I’m moving one of the volumes to an xfs volume… I hope I’ll be able to answer that question myself soon. From: Joe Julian [mailto:j...@julianfamily.org] Sent: dinsdag 1 april 2014 16:14 To: VAN CAUSBROECK Wannes; 'gluster-users@gluster.org' Subject: Re: [Gluster-users] nfs acces denied [2014-03-31 12:44:18.941083] I [dht-layout.c:638:dht_layout_normalize] 0-caviar_data11-dht: found anomalies in /2011/201107/20110716/55. holes=1 overlaps=0 This tells us that there is a chunk of hash map missing. My guess would be that the server you're nfs mounting from cannot connect to one of the bricks. On March 31, 2014 6:58:53 AM PDT, VAN CAUSBROECK Wannes wannes.vancausbro...@onprvp.fgov.be wrote: Hello all, I’ve already tried to post this, but i’m unsure it arrived to the mailing list. I have some issues regarding my nfs mounts. My setup is as follows: Rhel 6.4, gluster 3.4.2-1 running on a vm (4 cores, 8GB ram) attached to a san. I have one disk on which are all the bricks (formatted ext4 in 64 bit mode) of 25TB. On the gluster side of things, everything works without issues. The trouble starts when I mount a volume as an nfs mount. Lots of volumes work without issues, but others behave strangely. The volumes that act weird generally contain many files (can be accidental?). The volumes in question mount without issues, but when I try to go into any subdirectory sometimes it works, sometimes I get errors. On windows with nfs client: access denied In nfslog: [2014-03-31 13:57:58.771241] I [dht-layout.c:638:dht_layout_normalize] 0-caviar_data11-dht: found anomalies in gfid:c8d94120-6851-46ea-9f28-c629a44b1015. holes=1 overlaps=0 [2014-03-31 13:57:58.771348] E [nfs3-helpers.c:3595:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: Lookup failed: gfid:c8d94120-6851-46ea-9f28-c629a44b1015: Invalid argument [2014-03-31 13:57:58.771380] E [nfs3.c:1380:nfs3_lookup_resume] 0-nfs-nfsv3: Unable to resolve FH: (192.168.148.46:984) caviar_data11 : c8d94120-6851-46ea-9f28-c629a44b1015 [2014-03-31 13:57:58.771819] W [nfs3-helpers.c:3380:nfs3_log_common_res] 0-nfs-nfsv3: XID: 1ec28530, LOOKUP: NFS: 22(Invalid argument for operation), POSIX: 14(Bad address) [2014-03-31 13:57:58.798967] I [dht-layout.c:638:dht_layout_normalize] 0-caviar_data11-dht: found anomalies in gfid:14972193-1039-4d7a-aed5-0d7e7eccf57b. holes=1 overlaps=0 [2014-03-31 13:57:58.799039] E [nfs3-helpers.c:3595:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: Lookup failed: gfid:14972193-1039-4d7a-aed5-0d7e7eccf57b: Invalid argument [2014-03-31 13:57:58.799056] E [nfs3.c:1380:nfs3_lookup_resume] 0-nfs-nfsv3: Unable to resolve FH: (192.168.148.46:984) caviar_data11 : 14972193-1039-4d7a-aed5-0d7e7eccf57b [2014-03-31 13:57:58.799088] W [nfs3-helpers.c:3380:nfs3_log_common_res] 0-nfs-nfsv3: XID: 1ec28531, LOOKUP: NFS: 22(Invalid argument for operation), POSIX: 14(Bad address) …. On linux: [root@lpr-nas01 brick-xiv2]# ll /media/2011/201105/20110530/ ls: /media/2011/201105/20110530/37: No such file or directory total 332 … drwxrwsr-x 2 nfsnobody 1003 4096 Jun 6 2011 32 drwxrwsr-x 2 nfsnobody 1003 4096 Jun 6 2011 34 drwxrwsr-x 2 nfsnobody 1003 4096 Jun 6 2011 35 drwxrwsr-x 2 nfsnobody 1003 4096 Jun 6 2011 36 drwxrwsr-x 2 nfsnobody 1003 4096 Jun 6 2011 37 … [root@lpr-nas01 brick-xiv2]# ll /media/2011/201105/20110530/37 ls: /media/2011/201105/20110530/37/NN.073824357.1.tif: No such file or directory ls: /media/2011/201105/20110530/37/NN.073824357.3.tif: No such file or directory total 54 -rwxrwxr-x 0 nfsnobody 1003 9340 Jun 6 2011 NN.073824357.1.tif -rwxrwxr-x 1 nfsnobody 1003 35312 Jun 6 2011 NN.073824357.2.tif -rwxrwxr-x 0 nfsnobody 1003 9340 Jun 6 2011 NN.073824357.3.tif I see in the nfslog: … [2014-03-31 12:44:18.941083] I [dht-layout.c:638:dht_layout_normalize] 0-caviar_data11-dht: found anomalies in /2011/201107/20110716/55. holes=1 overlaps=0 [2014-03-31 12:44:18.958078] I [dht-layout.c:638:dht_layout_normalize] 0-caviar_data11-dht: found anomalies in /2011/201107/20110716/30. holes=1 overlaps=0 [2014-03-31 12:44:18.959980] I [dht-layout.c:638:dht_layout_normalize] 0-caviar_data11-dht: found anomalies in /2011/201107/20110716/90. holes=1 overlaps=0 [2014-03-31 12:44:18.961094] E [dht-helper.c:429:dht_subvol_get_hashed] (--/usr/lib64/glusterfs/3.4.2/xlator/debug/io-stats.so(io_stats_lookup+0x157) [0x7fd6a61282e7] (--/usr/lib64/libglusterfs.so.0(default_lookup+0x6d) [0x3dfe01c03d] (--/usr/lib64/glusterfs/3.4.2/xlator/cluster/distribute.so(dht_lookup+0xa7e) [0x7fd6a656af2e]))) 0-caviar_data11-dht: invalid argument: loc-parent [2014-03-31 12:44:18.961283] W [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-caviar_data11-client-0: remote operation failed: Invalid argument. Path: gfid:---- (----) [2014-03-31
Re: [Gluster-users] nfs acces denied
Wannes: There is a current issue with EXT4, the Linux Kernel and Gluster. It is a long story, but, in the end there is a bug there that won't be addressed any time soon. If at all possible, like gluster's/ RH's documentation recommends, use XFS and make sure you prepare the partition with -i size=512. At least then you are fully compliant. On Tue, Apr 1, 2014 at 4:21 PM, VAN CAUSBROECK Wannes wannes.vancausbro...@onprvp.fgov.be wrote: Hello Joe, Could it be that gluster has issues with the 64 bit ext4 filesystem? As a test I'm moving one of the volumes to an xfs volume... I hope I'll be able to answer that question myself soon. *From:* Joe Julian [mailto:j...@julianfamily.org] *Sent:* dinsdag 1 april 2014 16:14 *To:* VAN CAUSBROECK Wannes; 'gluster-users@gluster.org' *Subject:* Re: [Gluster-users] nfs acces denied [2014-03-31 12:44:18.941083] I [dht-layout.c:638:dht_layout_normalize] 0-caviar_data11-dht: found anomalies in /2011/201107/20110716/55. holes=1 overlaps=0 This tells us that there is a chunk of hash map missing. My guess would be that the server you're nfs mounting from cannot connect to one of the bricks. On March 31, 2014 6:58:53 AM PDT, VAN CAUSBROECK Wannes wannes.vancausbro...@onprvp.fgov.be wrote: Hello all, I've already tried to post this, but i'm unsure it arrived to the mailing list. I have some issues regarding my nfs mounts. My setup is as follows: Rhel 6.4, gluster 3.4.2-1 running on a vm (4 cores, 8GB ram) attached to a san. I have one disk on which are all the bricks (formatted ext4 in 64 bit mode) of 25TB. On the gluster side of things, everything works without issues. The trouble starts when I mount a volume as an nfs mount. Lots of volumes work without issues, but others behave strangely. The volumes that act weird generally contain many files (can be accidental?). The volumes in question mount without issues, but when I try to go into any subdirectory sometimes it works, sometimes I get errors. On windows with nfs client: access denied In nfslog: [2014-03-31 13:57:58.771241] I [dht-layout.c:638:dht_layout_normalize] 0-caviar_data11-dht: found anomalies in gfid:c8d94120-6851-46ea-9f28-c629a44b1015. holes=1 overlaps=0 [2014-03-31 13:57:58.771348] E [nfs3-helpers.c:3595:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: Lookup failed: gfid:c8d94120-6851-46ea-9f28-c629a44b1015: Invalid argument [2014-03-31 13:57:58.771380] E [nfs3.c:1380:nfs3_lookup_resume] 0-nfs-nfsv3: Unable to resolve FH: (192.168.148.46:984) caviar_data11 : c8d94120-6851-46ea-9f28-c629a44b1015 [2014-03-31 13:57:58.771819] W [nfs3-helpers.c:3380:nfs3_log_common_res] 0-nfs-nfsv3: XID: 1ec28530, LOOKUP: NFS: 22(Invalid argument for operation), POSIX: 14(Bad address) [2014-03-31 13:57:58.798967] I [dht-layout.c:638:dht_layout_normalize] 0-caviar_data11-dht: found anomalies in gfid:14972193-1039-4d7a-aed5-0d7e7eccf57b. holes=1 overlaps=0 [2014-03-31 13:57:58.799039] E [nfs3-helpers.c:3595:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: Lookup failed: gfid:14972193-1039-4d7a-aed5-0d7e7eccf57b: Invalid argument [2014-03-31 13:57:58.799056] E [nfs3.c:1380:nfs3_lookup_resume] 0-nfs-nfsv3: Unable to resolve FH: (192.168.148.46:984) caviar_data11 : 14972193-1039-4d7a-aed5-0d7e7eccf57b [2014-03-31 13:57:58.799088] W [nfs3-helpers.c:3380:nfs3_log_common_res] 0-nfs-nfsv3: XID: 1ec28531, LOOKUP: NFS: 22(Invalid argument for operation), POSIX: 14(Bad address) On linux: [root@lpr-nas01 brick-xiv2]# ll /media/2011/201105/20110530/ ls: /media/2011/201105/20110530/37: No such file or directory total 332 ... drwxrwsr-x 2 nfsnobody 1003 4096 Jun 6 2011 32 drwxrwsr-x 2 nfsnobody 1003 4096 Jun 6 2011 34 drwxrwsr-x 2 nfsnobody 1003 4096 Jun 6 2011 35 drwxrwsr-x 2 nfsnobody 1003 4096 Jun 6 2011 36 drwxrwsr-x 2 nfsnobody 1003 4096 Jun 6 2011 37 ... [root@lpr-nas01 brick-xiv2]# ll /media/2011/201105/20110530/37 ls: /media/2011/201105/20110530/37/NN.073824357.1.tif: No such file or directory ls: /media/2011/201105/20110530/37/NN.073824357.3.tif: No such file or directory total 54 -rwxrwxr-x 0 nfsnobody 1003 9340 Jun 6 2011 NN.073824357.1.tif -rwxrwxr-x 1 nfsnobody 1003 35312 Jun 6 2011 NN.073824357.2.tif -rwxrwxr-x 0 nfsnobody 1003 9340 Jun 6 2011 NN.073824357.3.tif I see in the nfslog: ... [2014-03-31 12:44:18.941083] I [dht-layout.c:638:dht_layout_normalize] 0-caviar_data11-dht: found anomalies in /2011/201107/20110716/55. holes=1 overlaps=0 [2014-03-31 12:44:18.958078] I [dht-layout.c:638:dht_layout_normalize] 0-caviar_data11-dht: found anomalies in /2011/201107/20110716/30. holes=1 overlaps=0 [2014-03-31 12:44:18.959980] I [dht-layout.c:638:dht_layout_normalize] 0-caviar_data11-dht: found anomalies in /2011/201107/20110716/90. holes=1 overlaps=0 [2014-03-31 12:44:18.961094] E [dht-helper.c:429
Re: [Gluster-users] nfs acces denied
thanks Carlos, this confirms my suspicions... xfs it is then! On 01 Apr 2014, at 17:11, Carlos Capriotti capriotti.car...@gmail.commailto:capriotti.car...@gmail.com wrote: Wannes: There is a current issue with EXT4, the Linux Kernel and Gluster. It is a long story, but, in the end there is a bug there that won't be addressed any time soon. If at all possible, like gluster's/ RH's documentation recommends, use XFS and make sure you prepare the partition with -i size=512. At least then you are fully compliant. On Tue, Apr 1, 2014 at 4:21 PM, VAN CAUSBROECK Wannes wannes.vancausbro...@onprvp.fgov.bemailto:wannes.vancausbro...@onprvp.fgov.be wrote: Hello Joe, Could it be that gluster has issues with the 64 bit ext4 filesystem? As a test I’m moving one of the volumes to an xfs volume… I hope I’ll be able to answer that question myself soon. From: Joe Julian [mailto:j...@julianfamily.orgmailto:j...@julianfamily.org] Sent: dinsdag 1 april 2014 16:14 To: VAN CAUSBROECK Wannes; 'gluster-users@gluster.orgmailto:gluster-users@gluster.org' Subject: Re: [Gluster-users] nfs acces denied [2014-03-31 12:44:18.941083] I [dht-layout.c:638:dht_layout_normalize] 0-caviar_data11-dht: found anomalies in /2011/201107/20110716/55. holes=1 overlaps=0 This tells us that there is a chunk of hash map missing. My guess would be that the server you're nfs mounting from cannot connect to one of the bricks. On March 31, 2014 6:58:53 AM PDT, VAN CAUSBROECK Wannes wannes.vancausbro...@onprvp.fgov.bemailto:wannes.vancausbro...@onprvp.fgov.be wrote: Hello all, I’ve already tried to post this, but i’m unsure it arrived to the mailing list. I have some issues regarding my nfs mounts. My setup is as follows: Rhel 6.4, gluster 3.4.2-1 running on a vm (4 cores, 8GB ram) attached to a san. I have one disk on which are all the bricks (formatted ext4 in 64 bit mode) of 25TB. On the gluster side of things, everything works without issues. The trouble starts when I mount a volume as an nfs mount. Lots of volumes work without issues, but others behave strangely. The volumes that act weird generally contain many files (can be accidental?). The volumes in question mount without issues, but when I try to go into any subdirectory sometimes it works, sometimes I get errors. On windows with nfs client: access denied In nfslog: [2014-03-31 13:57:58.771241] I [dht-layout.c:638:dht_layout_normalize] 0-caviar_data11-dht: found anomalies in gfid:c8d94120-6851-46ea-9f28-c629a44b1015. holes=1 overlaps=0 [2014-03-31 13:57:58.771348] E [nfs3-helpers.c:3595:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: Lookup failed: gfid:c8d94120-6851-46ea-9f28-c629a44b1015: Invalid argument [2014-03-31 13:57:58.771380] E [nfs3.c:1380:nfs3_lookup_resume] 0-nfs-nfsv3: Unable to resolve FH: (192.168.148.46:984http://192.168.148.46:984) caviar_data11 : c8d94120-6851-46ea-9f28-c629a44b1015 [2014-03-31 13:57:58.771819] W [nfs3-helpers.c:3380:nfs3_log_common_res] 0-nfs-nfsv3: XID: 1ec28530, LOOKUP: NFS: 22(Invalid argument for operation), POSIX: 14(Bad address) [2014-03-31 13:57:58.798967] I [dht-layout.c:638:dht_layout_normalize] 0-caviar_data11-dht: found anomalies in gfid:14972193-1039-4d7a-aed5-0d7e7eccf57b. holes=1 overlaps=0 [2014-03-31 13:57:58.799039] E [nfs3-helpers.c:3595:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: Lookup failed: gfid:14972193-1039-4d7a-aed5-0d7e7eccf57b: Invalid argument [2014-03-31 13:57:58.799056] E [nfs3.c:1380:nfs3_lookup_resume] 0-nfs-nfsv3: Unable to resolve FH: (192.168.148.46:984http://192.168.148.46:984) caviar_data11 : 14972193-1039-4d7a-aed5-0d7e7eccf57b [2014-03-31 13:57:58.799088] W [nfs3-helpers.c:3380:nfs3_log_common_res] 0-nfs-nfsv3: XID: 1ec28531, LOOKUP: NFS: 22(Invalid argument for operation), POSIX: 14(Bad address) …. On linux: [root@lpr-nas01 brick-xiv2]# ll /media/2011/201105/20110530/ ls: /media/2011/201105/20110530/37: No such file or directory total 332 … drwxrwsr-x 2 nfsnobody 1003 4096 Jun 6 2011 32 drwxrwsr-x 2 nfsnobody 1003 4096 Jun 6 2011 34 drwxrwsr-x 2 nfsnobody 1003 4096 Jun 6 2011 35 drwxrwsr-x 2 nfsnobody 1003 4096 Jun 6 2011 36 drwxrwsr-x 2 nfsnobody 1003 4096 Jun 6 2011 37 … [root@lpr-nas01 brick-xiv2]# ll /media/2011/201105/20110530/37 ls: /media/2011/201105/20110530/37/NN.073824357.1.tif: No such file or directory ls: /media/2011/201105/20110530/37/NN.073824357.3.tif: No such file or directory total 54 -rwxrwxr-x 0 nfsnobody 1003 9340 Jun 6 2011 NN.073824357.1.tif -rwxrwxr-x 1 nfsnobody 1003 35312 Jun 6 2011 NN.073824357.2.tif -rwxrwxr-x 0 nfsnobody 1003 9340 Jun 6 2011 NN.073824357.3.tif I see in the nfslog: … [2014-03-31 12:44:18.941083] I [dht-layout.c:638:dht_layout_normalize] 0-caviar_data11-dht: found anomalies in /2011/201107/20110716/55. holes=1 overlaps=0 [2014-03-31 12:44:18.958078] I [dht-layout.c:638:dht_layout_normalize] 0-caviar_data11-dht: found anomalies in /2011/201107
Re: [Gluster-users] nfs acces denied
Well, saying your client-side is linux does not help much. Distro, flavor, etc helps a lot, but I'll take a wild guess here. First, force your NFS mount (client) to use nfs version 3. The same for Microsoft. (It is fair to say I have no idea if the MS client supports v4 or not). Additionally, check that firewalls are disabled on both sides, just for testing. The same goes for SElinux. Windows and ACL, and user mapping is something that might be in your way too. There is a Technet document that describes how to handle this mapping if I am not wrong. Just for testing, mount your nfs share you your own server, using localhost:/nfs_share and see how it goes. It is a good start. Kr, Carlos On Mon, Mar 31, 2014 at 3:58 PM, VAN CAUSBROECK Wannes wannes.vancausbro...@onprvp.fgov.be wrote: Hello all, I've already tried to post this, but i'm unsure it arrived to the mailing list. I have some issues regarding my nfs mounts. My setup is as follows: Rhel 6.4, gluster 3.4.2-1 running on a vm (4 cores, 8GB ram) attached to a san. I have one disk on which are all the bricks (formatted ext4 in 64 bit mode) of 25TB. On the gluster side of things, everything works without issues. The trouble starts when I mount a volume as an nfs mount. Lots of volumes work without issues, but others behave strangely. The volumes that act weird generally contain many files (can be accidental?). The volumes in question mount without issues, but when I try to go into any subdirectory sometimes it works, sometimes I get errors. On windows with nfs client: access denied In nfslog: [2014-03-31 13:57:58.771241] I [dht-layout.c:638:dht_layout_normalize] 0-caviar_data11-dht: found anomalies in gfid:c8d94120-6851-46ea-9f28-c629a44b1015. holes=1 overlaps=0 [2014-03-31 13:57:58.771348] E [nfs3-helpers.c:3595:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: Lookup failed: gfid:c8d94120-6851-46ea-9f28-c629a44b1015: Invalid argument [2014-03-31 13:57:58.771380] E [nfs3.c:1380:nfs3_lookup_resume] 0-nfs-nfsv3: Unable to resolve FH: (192.168.148.46:984) caviar_data11 : c8d94120-6851-46ea-9f28-c629a44b1015 [2014-03-31 13:57:58.771819] W [nfs3-helpers.c:3380:nfs3_log_common_res] 0-nfs-nfsv3: XID: 1ec28530, LOOKUP: NFS: 22(Invalid argument for operation), POSIX: 14(Bad address) [2014-03-31 13:57:58.798967] I [dht-layout.c:638:dht_layout_normalize] 0-caviar_data11-dht: found anomalies in gfid:14972193-1039-4d7a-aed5-0d7e7eccf57b. holes=1 overlaps=0 [2014-03-31 13:57:58.799039] E [nfs3-helpers.c:3595:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: Lookup failed: gfid:14972193-1039-4d7a-aed5-0d7e7eccf57b: Invalid argument [2014-03-31 13:57:58.799056] E [nfs3.c:1380:nfs3_lookup_resume] 0-nfs-nfsv3: Unable to resolve FH: (192.168.148.46:984) caviar_data11 : 14972193-1039-4d7a-aed5-0d7e7eccf57b [2014-03-31 13:57:58.799088] W [nfs3-helpers.c:3380:nfs3_log_common_res] 0-nfs-nfsv3: XID: 1ec28531, LOOKUP: NFS: 22(Invalid argument for operation), POSIX: 14(Bad address) On linux: [root@lpr-nas01 brick-xiv2]# ll /media/2011/201105/20110530/ ls: /media/2011/201105/20110530/37: No such file or directory total 332 ... drwxrwsr-x 2 nfsnobody 1003 4096 Jun 6 2011 32 drwxrwsr-x 2 nfsnobody 1003 4096 Jun 6 2011 34 drwxrwsr-x 2 nfsnobody 1003 4096 Jun 6 2011 35 drwxrwsr-x 2 nfsnobody 1003 4096 Jun 6 2011 36 drwxrwsr-x 2 nfsnobody 1003 4096 Jun 6 2011 37 ... [root@lpr-nas01 brick-xiv2]# ll /media/2011/201105/20110530/37 ls: /media/2011/201105/20110530/37/NN.073824357.1.tif: No such file or directory ls: /media/2011/201105/20110530/37/NN.073824357.3.tif: No such file or directory total 54 -rwxrwxr-x 0 nfsnobody 1003 9340 Jun 6 2011 NN.073824357.1.tif -rwxrwxr-x 1 nfsnobody 1003 35312 Jun 6 2011 NN.073824357.2.tif -rwxrwxr-x 0 nfsnobody 1003 9340 Jun 6 2011 NN.073824357.3.tif I see in the nfslog: ... [2014-03-31 12:44:18.941083] I [dht-layout.c:638:dht_layout_normalize] 0-caviar_data11-dht: found anomalies in /2011/201107/20110716/55. holes=1 overlaps=0 [2014-03-31 12:44:18.958078] I [dht-layout.c:638:dht_layout_normalize] 0-caviar_data11-dht: found anomalies in /2011/201107/20110716/30. holes=1 overlaps=0 [2014-03-31 12:44:18.959980] I [dht-layout.c:638:dht_layout_normalize] 0-caviar_data11-dht: found anomalies in /2011/201107/20110716/90. holes=1 overlaps=0 [2014-03-31 12:44:18.961094] E [dht-helper.c:429:dht_subvol_get_hashed] (--/usr/lib64/glusterfs/3.4.2/xlator/debug/io-stats.so(io_stats_lookup+0x157) [0x7fd6a61282e7] (--/usr/lib64/libglusterfs.so.0(default_lookup+0x6d) [0x3dfe01c03d] (--/usr/lib64/glusterfs/3.4.2/xlator/cluster/distribute.so(dht_lookup+0xa7e) [0x7fd6a656af2e]))) 0-caviar_data11-dht: invalid argument: loc-parent [2014-03-31 12:44:18.961283] W [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-caviar_data11-client-0: remote operation failed: Invalid
Re: [Gluster-users] nfs acces denied
Well, with 'client' i do actually mean the server itself. i've tried forcing linux and windows to nfs V3 and tcp, and on windows i played around with the uid and gid, but the result is always the same On 31 Mar 2014, at 17:22, Carlos Capriotti capriotti.car...@gmail.commailto:capriotti.car...@gmail.com wrote: Well, saying your client-side is linux does not help much. Distro, flavor, etc helps a lot, but I'll take a wild guess here. First, force your NFS mount (client) to use nfs version 3. The same for Microsoft. (It is fair to say I have no idea if the MS client supports v4 or not). Additionally, check that firewalls are disabled on both sides, just for testing. The same goes for SElinux. Windows and ACL, and user mapping is something that might be in your way too. There is a Technet document that describes how to handle this mapping if I am not wrong. Just for testing, mount your nfs share you your own server, using localhost:/nfs_share and see how it goes. It is a good start. Kr, Carlos On Mon, Mar 31, 2014 at 3:58 PM, VAN CAUSBROECK Wannes wannes.vancausbro...@onprvp.fgov.bemailto:wannes.vancausbro...@onprvp.fgov.be wrote: Hello all, I’ve already tried to post this, but i’m unsure it arrived to the mailing list. I have some issues regarding my nfs mounts. My setup is as follows: Rhel 6.4, gluster 3.4.2-1 running on a vm (4 cores, 8GB ram) attached to a san. I have one disk on which are all the bricks (formatted ext4 in 64 bit mode) of 25TB. On the gluster side of things, everything works without issues. The trouble starts when I mount a volume as an nfs mount. Lots of volumes work without issues, but others behave strangely. The volumes that act weird generally contain many files (can be accidental?). The volumes in question mount without issues, but when I try to go into any subdirectory sometimes it works, sometimes I get errors. On windows with nfs client: access denied In nfslog: [2014-03-31 13:57:58.771241] I [dht-layout.c:638:dht_layout_normalize] 0-caviar_data11-dht: found anomalies in gfid:c8d94120-6851-46ea-9f28-c629a44b1015. holes=1 overlaps=0 [2014-03-31 13:57:58.771348] E [nfs3-helpers.c:3595:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: Lookup failed: gfid:c8d94120-6851-46ea-9f28-c629a44b1015: Invalid argument [2014-03-31 13:57:58.771380] E [nfs3.c:1380:nfs3_lookup_resume] 0-nfs-nfsv3: Unable to resolve FH: (192.168.148.46:984http://192.168.148.46:984) caviar_data11 : c8d94120-6851-46ea-9f28-c629a44b1015 [2014-03-31 13:57:58.771819] W [nfs3-helpers.c:3380:nfs3_log_common_res] 0-nfs-nfsv3: XID: 1ec28530, LOOKUP: NFS: 22(Invalid argument for operation), POSIX: 14(Bad address) [2014-03-31 13:57:58.798967] I [dht-layout.c:638:dht_layout_normalize] 0-caviar_data11-dht: found anomalies in gfid:14972193-1039-4d7a-aed5-0d7e7eccf57b. holes=1 overlaps=0 [2014-03-31 13:57:58.799039] E [nfs3-helpers.c:3595:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: Lookup failed: gfid:14972193-1039-4d7a-aed5-0d7e7eccf57b: Invalid argument [2014-03-31 13:57:58.799056] E [nfs3.c:1380:nfs3_lookup_resume] 0-nfs-nfsv3: Unable to resolve FH: (192.168.148.46:984http://192.168.148.46:984) caviar_data11 : 14972193-1039-4d7a-aed5-0d7e7eccf57b [2014-03-31 13:57:58.799088] W [nfs3-helpers.c:3380:nfs3_log_common_res] 0-nfs-nfsv3: XID: 1ec28531, LOOKUP: NFS: 22(Invalid argument for operation), POSIX: 14(Bad address) …. On linux: [root@lpr-nas01 brick-xiv2]# ll /media/2011/201105/20110530/ ls: /media/2011/201105/20110530/37: No such file or directory total 332 … drwxrwsr-x 2 nfsnobody 1003 4096 Jun 6 2011 32 drwxrwsr-x 2 nfsnobody 1003 4096 Jun 6 2011 34 drwxrwsr-x 2 nfsnobody 1003 4096 Jun 6 2011 35 drwxrwsr-x 2 nfsnobody 1003 4096 Jun 6 2011 36 drwxrwsr-x 2 nfsnobody 1003 4096 Jun 6 2011 37 … [root@lpr-nas01 brick-xiv2]# ll /media/2011/201105/20110530/37 ls: /media/2011/201105/20110530/37/NN.073824357.1.tif: No such file or directory ls: /media/2011/201105/20110530/37/NN.073824357.3.tif: No such file or directory total 54 -rwxrwxr-x 0 nfsnobody 1003 9340 Jun 6 2011 NN.073824357.1.tif -rwxrwxr-x 1 nfsnobody 1003 35312 Jun 6 2011 NN.073824357.2.tif -rwxrwxr-x 0 nfsnobody 1003 9340 Jun 6 2011 NN.073824357.3.tif I see in the nfslog: … [2014-03-31 12:44:18.941083] I [dht-layout.c:638:dht_layout_normalize] 0-caviar_data11-dht: found anomalies in /2011/201107/20110716/55. holes=1 overlaps=0 [2014-03-31 12:44:18.958078] I [dht-layout.c:638:dht_layout_normalize] 0-caviar_data11-dht: found anomalies in /2011/201107/20110716/30. holes=1 overlaps=0 [2014-03-31 12:44:18.959980] I [dht-layout.c:638:dht_layout_normalize] 0-caviar_data11-dht: found anomalies in /2011/201107/20110716/90. holes=1 overlaps=0 [2014-03-31 12:44:18.961094] E [dht-helper.c:429:dht_subvol_get_hashed] (--/usr/lib64/glusterfs/3.4.2/xlator/debug/io-stats.so(io_stats_lookup+0x157) [0x7fd6a61282e7] (--/usr/lib64/libglusterfs.so.0(default_lookup+0x6d)
Re: [Gluster-users] nfs acces denied
maybe it would be nice to see your volume info for affected volumes. Also, on the server side, what happens if you mount the share using glusterfs instead of nfs ? any change the native nfs server is running on your server ? Are there any auto-heal processes running ? There are a few name resolution messages on your logs, that seem to refer to the nodes themselves. Any DNS conflicts ? Maybe add the names of servers to the hosts file ? You MS client seems to be having issues with user/group translation. It seems to create files with gid 1003. (I could be wrong). Again, is SElinux/ACLs/iptables disabled ? All is very inconclusive os far. On Mon, Mar 31, 2014 at 5:26 PM, VAN CAUSBROECK Wannes wannes.vancausbro...@onprvp.fgov.be wrote: Well, with 'client' i do actually mean the server itself. i've tried forcing linux and windows to nfs V3 and tcp, and on windows i played around with the uid and gid, but the result is always the same On 31 Mar 2014, at 17:22, Carlos Capriotti capriotti.car...@gmail.com wrote: Well, saying your client-side is linux does not help much. Distro, flavor, etc helps a lot, but I'll take a wild guess here. First, force your NFS mount (client) to use nfs version 3. The same for Microsoft. (It is fair to say I have no idea if the MS client supports v4 or not). Additionally, check that firewalls are disabled on both sides, just for testing. The same goes for SElinux. Windows and ACL, and user mapping is something that might be in your way too. There is a Technet document that describes how to handle this mapping if I am not wrong. Just for testing, mount your nfs share you your own server, using localhost:/nfs_share and see how it goes. It is a good start. Kr, Carlos On Mon, Mar 31, 2014 at 3:58 PM, VAN CAUSBROECK Wannes wannes.vancausbro...@onprvp.fgov.be wrote: Hello all, I've already tried to post this, but i'm unsure it arrived to the mailing list. I have some issues regarding my nfs mounts. My setup is as follows: Rhel 6.4, gluster 3.4.2-1 running on a vm (4 cores, 8GB ram) attached to a san. I have one disk on which are all the bricks (formatted ext4 in 64 bit mode) of 25TB. On the gluster side of things, everything works without issues. The trouble starts when I mount a volume as an nfs mount. Lots of volumes work without issues, but others behave strangely. The volumes that act weird generally contain many files (can be accidental?). The volumes in question mount without issues, but when I try to go into any subdirectory sometimes it works, sometimes I get errors. On windows with nfs client: access denied In nfslog: [2014-03-31 13:57:58.771241] I [dht-layout.c:638:dht_layout_normalize] 0-caviar_data11-dht: found anomalies in gfid:c8d94120-6851-46ea-9f28-c629a44b1015. holes=1 overlaps=0 [2014-03-31 13:57:58.771348] E [nfs3-helpers.c:3595:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: Lookup failed: gfid:c8d94120-6851-46ea-9f28-c629a44b1015: Invalid argument [2014-03-31 13:57:58.771380] E [nfs3.c:1380:nfs3_lookup_resume] 0-nfs-nfsv3: Unable to resolve FH: (192.168.148.46:984) caviar_data11 : c8d94120-6851-46ea-9f28-c629a44b1015 [2014-03-31 13:57:58.771819] W [nfs3-helpers.c:3380:nfs3_log_common_res] 0-nfs-nfsv3: XID: 1ec28530, LOOKUP: NFS: 22(Invalid argument for operation), POSIX: 14(Bad address) [2014-03-31 13:57:58.798967] I [dht-layout.c:638:dht_layout_normalize] 0-caviar_data11-dht: found anomalies in gfid:14972193-1039-4d7a-aed5-0d7e7eccf57b. holes=1 overlaps=0 [2014-03-31 13:57:58.799039] E [nfs3-helpers.c:3595:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: Lookup failed: gfid:14972193-1039-4d7a-aed5-0d7e7eccf57b: Invalid argument [2014-03-31 13:57:58.799056] E [nfs3.c:1380:nfs3_lookup_resume] 0-nfs-nfsv3: Unable to resolve FH: (192.168.148.46:984) caviar_data11 : 14972193-1039-4d7a-aed5-0d7e7eccf57b [2014-03-31 13:57:58.799088] W [nfs3-helpers.c:3380:nfs3_log_common_res] 0-nfs-nfsv3: XID: 1ec28531, LOOKUP: NFS: 22(Invalid argument for operation), POSIX: 14(Bad address) On linux: [root@lpr-nas01 brick-xiv2]# ll /media/2011/201105/20110530/ ls: /media/2011/201105/20110530/37: No such file or directory total 332 ... drwxrwsr-x 2 nfsnobody 1003 4096 Jun 6 2011 32 drwxrwsr-x 2 nfsnobody 1003 4096 Jun 6 2011 34 drwxrwsr-x 2 nfsnobody 1003 4096 Jun 6 2011 35 drwxrwsr-x 2 nfsnobody 1003 4096 Jun 6 2011 36 drwxrwsr-x 2 nfsnobody 1003 4096 Jun 6 2011 37 ... [root@lpr-nas01 brick-xiv2]# ll /media/2011/201105/20110530/37 ls: /media/2011/201105/20110530/37/NN.073824357.1.tif: No such file or directory ls: /media/2011/201105/20110530/37/NN.073824357.3.tif: No such file or directory total 54 -rwxrwxr-x 0 nfsnobody 1003 9340 Jun 6 2011 NN.073824357.1.tif -rwxrwxr-x 1 nfsnobody 1003 35312 Jun 6 2011 NN.073824357.2.tif -rwxrwxr-x 0 nfsnobody 1003 9340 Jun 6 2011
