Re: [Gluster-users] Renaming permission denied
On 12/15/2011 03:35 AM, Brian Rosner wrote: On Dec 13, 2011, at 7:54 PM, Pranith Kumar K wrote: On 12/13/2011 10:56 PM, Brian Rosner wrote: On Dec 12, 2011, at 1:41 AM, Pranith Kumar K wrote: Seems like the issue with that specific file your application is trying to rename. Could you check if that file has correct permissions on the backends?. It is difficult for me to test that due to the file being created and unlinked during the reindexing process. The fact it has permission to unlink and not rename seems very odd to me. Brian Rosner Brian, Could you give the permissions/ownership of the directory the file is being created, permissions/ownership of the file that is being created. Is the group of the file secondary group? It is very difficult for us to find the root cause without a good test case. Ok, I have a bit more info. I was able to reproduce it with a tool that I can easily modify. Here is some info from this new case: log: [2011-12-14 21:53:15.373241] W [fuse-bridge.c:1348:fuse_rename_cbk] 0-glusterfs-fuse: 265: /i130/whoosh_index/_MAIN_1.toc.1323899595.36 - /i130/whoosh_index/_MAIN_1.toc = -1 (Permission denied) % sudo ls -l /mnt/i130/whoosh_index/_MAIN_1.toc.1323899595.36 -rw-r- 1 1630 nogroup 1522 2011-12-14 21:52 /mnt/i130/whoosh_index/_MAIN_1.toc.1323899595.36 % sudo ls -l /mnt/i130/ drwxr-x--- 2 1630 nogroup 8192 2011-12-14 21:52 whoosh_index % sudo ls -l /mnt | grep i130 drwxr-x--x 4 1630 anduin 8192 2011-12-14 21:52 i130 I've confirmed the user and group of the process are 1630 and nogroup. Brian Rosner Brian, Could you give the output of the same files/dirs on the backends as well. Pranith ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Renaming permission denied
On 12/15/2011 05:23 PM, Pranith Kumar K wrote: On 12/15/2011 03:35 AM, Brian Rosner wrote: On Dec 13, 2011, at 7:54 PM, Pranith Kumar K wrote: On 12/13/2011 10:56 PM, Brian Rosner wrote: On Dec 12, 2011, at 1:41 AM, Pranith Kumar K wrote: Seems like the issue with that specific file your application is trying to rename. Could you check if that file has correct permissions on the backends?. It is difficult for me to test that due to the file being created and unlinked during the reindexing process. The fact it has permission to unlink and not rename seems very odd to me. Brian Rosner Brian, Could you give the permissions/ownership of the directory the file is being created, permissions/ownership of the file that is being created. Is the group of the file secondary group? It is very difficult for us to find the root cause without a good test case. Ok, I have a bit more info. I was able to reproduce it with a tool that I can easily modify. Here is some info from this new case: log: [2011-12-14 21:53:15.373241] W [fuse-bridge.c:1348:fuse_rename_cbk] 0-glusterfs-fuse: 265: /i130/whoosh_index/_MAIN_1.toc.1323899595.36 - /i130/whoosh_index/_MAIN_1.toc = -1 (Permission denied) % sudo ls -l /mnt/i130/whoosh_index/_MAIN_1.toc.1323899595.36 -rw-r- 1 1630 nogroup 1522 2011-12-14 21:52 /mnt/i130/whoosh_index/_MAIN_1.toc.1323899595.36 % sudo ls -l /mnt/i130/ drwxr-x--- 2 1630 nogroup 8192 2011-12-14 21:52 whoosh_index % sudo ls -l /mnt | grep i130 drwxr-x--x 4 1630 anduin 8192 2011-12-14 21:52 i130 I've confirmed the user and group of the process are 1630 and nogroup. Brian Rosner Brian, Could you give the output of the same files/dirs on the backends as well. Pranith ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users Brian, Could you also give the output of the 'id 1630' on the backend bricks. Pranith ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Renaming permission denied
On 12/15/2011 05:31 PM, Pranith Kumar K wrote: On 12/15/2011 05:23 PM, Pranith Kumar K wrote: On 12/15/2011 03:35 AM, Brian Rosner wrote: On Dec 13, 2011, at 7:54 PM, Pranith Kumar K wrote: On 12/13/2011 10:56 PM, Brian Rosner wrote: On Dec 12, 2011, at 1:41 AM, Pranith Kumar K wrote: Seems like the issue with that specific file your application is trying to rename. Could you check if that file has correct permissions on the backends?. It is difficult for me to test that due to the file being created and unlinked during the reindexing process. The fact it has permission to unlink and not rename seems very odd to me. Brian Rosner Brian, Could you give the permissions/ownership of the directory the file is being created, permissions/ownership of the file that is being created. Is the group of the file secondary group? It is very difficult for us to find the root cause without a good test case. Ok, I have a bit more info. I was able to reproduce it with a tool that I can easily modify. Here is some info from this new case: log: [2011-12-14 21:53:15.373241] W [fuse-bridge.c:1348:fuse_rename_cbk] 0-glusterfs-fuse: 265: /i130/whoosh_index/_MAIN_1.toc.1323899595.36 - /i130/whoosh_index/_MAIN_1.toc = -1 (Permission denied) % sudo ls -l /mnt/i130/whoosh_index/_MAIN_1.toc.1323899595.36 -rw-r- 1 1630 nogroup 1522 2011-12-14 21:52 /mnt/i130/whoosh_index/_MAIN_1.toc.1323899595.36 % sudo ls -l /mnt/i130/ drwxr-x--- 2 1630 nogroup 8192 2011-12-14 21:52 whoosh_index % sudo ls -l /mnt | grep i130 drwxr-x--x 4 1630 anduin 8192 2011-12-14 21:52 i130 I've confirmed the user and group of the process are 1630 and nogroup. Brian Rosner Brian, Could you give the output of the same files/dirs on the backends as well. Pranith ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users Brian, Could you also give the output of the 'id 1630' on the backend bricks. Pranith ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users Brian, Sorry, we need the 'id 1630' output on the client machine. Pranith ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Renaming permission denied
On Dec 15, 2011, at 5:05 AM, Pranith Kumar K wrote: Brian, Sorry, we need the 'id 1630' output on the client machine. I should have clarified that 1630 is the UID of user i130. I ran 'id i130' on the client machine: uid=1630(i130) gid=65534(nogroup) groups=65534(nogroup) User i130 does not exist on the backend systems. Brian Rosner ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Renaming permission denied
On Dec 15, 2011, at 4:53 AM, Pranith Kumar K wrote: Brian, Could you give the output of the same files/dirs on the backends as well. I had to recreate the situation again so here is the new log entry to show the new file it is renaming: [2011-12-15 13:47:14.513882] W [fuse-bridge.c:1348:fuse_rename_cbk] 0-glusterfs-fuse: 395: /i130/whoosh_index/_MAIN_0.toc.1323956834.5 - /i130/whoosh_index/_MAIN_0.toc = -1 (Permission denied) Backends (file only exists on brick 3 and 4): Brick1: 10.x.x.106: % sudo ls -l /var/g/data-instances/ | grep i130 drwxr-x--x 4 1630 anduin 4096 2011-12-15 13:46 i130 % sudo ls -l /var/g/data-instances/i130 | grep whoosh_index drwxr-x--- 2 1630 nogroup 4096 2011-12-15 13:46 whoosh_index % sudo ls -l /var/g/data-instances/i130/whoosh_index total 0 Brick2: 10.x.x.131: % sudo ls -l /var/g/data-instances/ | grep i130 drwxr-x--x 4 1630 anduin 4096 2011-12-15 13:46 i130 % sudo ls -l /var/g/data-instances/i130 | grep whoosh_index drwxr-x--- 2 1630 nogroup 4096 2011-12-15 13:46 whoosh_index % sudo ls -l /var/g/data-instances/i130/whoosh_index total 8 -rw-r- 1 1630 nogroup 1267 2011-12-15 13:46 _MAIN_0.toc.1323956834.5 Brick3: 10.x.x.0: % sudo ls -l /var/g/data-instances/ | grep i130 drwxr-x--x 4 1630 anduin 4096 2011-12-15 13:46 i130 % sudo ls -l /var/g/data-instances/i130 | grep whoosh_index drwxr-x--- 2 1630 nogroup 4096 2011-12-15 13:46 whoosh_index % sudo ls -l /var/g/data-instances/i130/whoosh_index total 0 Brick4: 10.x.x.201: % sudo ls -l /var/g/data-instances/ | grep i130 drwxr-x--x 4 1630 anduin 4096 2011-12-15 13:46 i130 % sudo ls -l /var/g/data-instances/i130 | grep whoosh_index drwxr-x--- 2 1630 nogroup 4096 2011-12-15 13:46 whoosh_index % sudo ls -l /var/g/data-instances/i130/whoosh_index total 8 -rw-r- 1 1630 nogroup 1267 2011-12-15 13:46 _MAIN_0.toc.1323956834.5 Brian Rosner ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Renaming permission denied
On Dec 15, 2011, at 7:03 AM, Brian Rosner wrote: Backends (file only exists on brick 3 and 4): Slight correction: I meant to say bricks 2 and 4 Brian Rosner ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Renaming permission denied
On Dec 13, 2011, at 7:54 PM, Pranith Kumar K wrote: On 12/13/2011 10:56 PM, Brian Rosner wrote: On Dec 12, 2011, at 1:41 AM, Pranith Kumar K wrote: Seems like the issue with that specific file your application is trying to rename. Could you check if that file has correct permissions on the backends?. It is difficult for me to test that due to the file being created and unlinked during the reindexing process. The fact it has permission to unlink and not rename seems very odd to me. Brian Rosner Brian, Could you give the permissions/ownership of the directory the file is being created, permissions/ownership of the file that is being created. Is the group of the file secondary group? It is very difficult for us to find the root cause without a good test case. Ok, I have a bit more info. I was able to reproduce it with a tool that I can easily modify. Here is some info from this new case: log: [2011-12-14 21:53:15.373241] W [fuse-bridge.c:1348:fuse_rename_cbk] 0-glusterfs-fuse: 265: /i130/whoosh_index/_MAIN_1.toc.1323899595.36 - /i130/whoosh_index/_MAIN_1.toc = -1 (Permission denied) % sudo ls -l /mnt/i130/whoosh_index/_MAIN_1.toc.1323899595.36 -rw-r- 1 1630 nogroup 1522 2011-12-14 21:52 /mnt/i130/whoosh_index/_MAIN_1.toc.1323899595.36 % sudo ls -l /mnt/i130/ drwxr-x--- 2 1630 nogroup 8192 2011-12-14 21:52 whoosh_index % sudo ls -l /mnt | grep i130 drwxr-x--x 4 1630 anduin 8192 2011-12-14 21:52 i130 I've confirmed the user and group of the process are 1630 and nogroup. Brian Rosner ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Renaming permission denied
On Dec 12, 2011, at 1:41 AM, Pranith Kumar K wrote: Seems like the issue with that specific file your application is trying to rename. Could you check if that file has correct permissions on the backends?. It is difficult for me to test that due to the file being created and unlinked during the reindexing process. The fact it has permission to unlink and not rename seems very odd to me. Brian Rosner ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Renaming permission denied
On 12/13/2011 10:56 PM, Brian Rosner wrote: On Dec 12, 2011, at 1:41 AM, Pranith Kumar K wrote: Seems like the issue with that specific file your application is trying to rename. Could you check if that file has correct permissions on the backends?. It is difficult for me to test that due to the file being created and unlinked during the reindexing process. The fact it has permission to unlink and not rename seems very odd to me. Brian Rosner Brian, Could you give the permissions/ownership of the directory the file is being created, permissions/ownership of the file that is being created. Is the group of the file secondary group? It is very difficult for us to find the root cause without a good test case. Pranith ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Renaming permission denied
On 12/11/2011 12:25 AM, Brian Rosner wrote: Here is my setup: Volume Name: g-data-instances Type: Distributed-Replicate Status: Started Number of Bricks: 2 x 2 = 4 Transport-type: tcp Bricks: Brick1: 10.x.x.106:/var/g/data-instances Brick2: 10.x.x.131:/var/g/data-instances Brick3: 10.x.x.0:/var/g/data-instances Brick4: 10.x.x.201:/var/g/data-instances running on 3.2.5. The client is mounted: mount -t glusterfs 10.x.x.106:/g-data-instances /mnt I am seeing: [2011-12-10 18:45:24.792339] I [client3_1-fops.c:818:client3_1_setxattr_cbk] 0-g-data-instances-client-0: remote operation failed: Permission denied [2011-12-10 18:45:24.792502] I [client3_1-fops.c:818:client3_1_setxattr_cbk] 0-g-data-instances-client-1: remote operation failed: Permission denied [2011-12-10 18:45:24.795528] W [fuse-bridge.c:1348:fuse_rename_cbk] 0-glusterfs-fuse: 484: /i130/xapian/position.tmp - /i130/xapian/position.baseB = -1 (Permission denied) I have tried to produce this in other ways such as: sudo -u i130 bash -c umask 027 ; python open(/mnt/i130/xapian/a, wb).write(hello) import os os.rename(/mnt/i130/xapian/a, /mnt/i130/xapian/b) however, this example works flawlessly. I've tracked down the line of code in xapian which is seeing the error and it looks to be a straightforward rename system call. Does anyone have ideas on what I can try to resolve this issue? Seems like the issue with that specific file your application is trying to rename. Could you check if that file has correct permissions on the backends?. Pranith ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
[Gluster-users] Renaming permission denied
Here is my setup: Volume Name: g-data-instances Type: Distributed-Replicate Status: Started Number of Bricks: 2 x 2 = 4 Transport-type: tcp Bricks: Brick1: 10.x.x.106:/var/g/data-instances Brick2: 10.x.x.131:/var/g/data-instances Brick3: 10.x.x.0:/var/g/data-instances Brick4: 10.x.x.201:/var/g/data-instances running on 3.2.5. The client is mounted: mount -t glusterfs 10.x.x.106:/g-data-instances /mnt I am seeing: [2011-12-10 18:45:24.792339] I [client3_1-fops.c:818:client3_1_setxattr_cbk] 0-g-data-instances-client-0: remote operation failed: Permission denied [2011-12-10 18:45:24.792502] I [client3_1-fops.c:818:client3_1_setxattr_cbk] 0-g-data-instances-client-1: remote operation failed: Permission denied [2011-12-10 18:45:24.795528] W [fuse-bridge.c:1348:fuse_rename_cbk] 0-glusterfs-fuse: 484: /i130/xapian/position.tmp - /i130/xapian/position.baseB = -1 (Permission denied) I have tried to produce this in other ways such as: sudo -u i130 bash -c umask 027 ; python open(/mnt/i130/xapian/a, wb).write(hello) import os os.rename(/mnt/i130/xapian/a, /mnt/i130/xapian/b) however, this example works flawlessly. I've tracked down the line of code in xapian which is seeing the error and it looks to be a straightforward rename system call. Does anyone have ideas on what I can try to resolve this issue? -- Brian Rosner http://twitter.com/brosner ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
