Re: [Gluster-users] ACL
Hi Anush, Thanks for your quick reply. There was no error when set ACL. Below is what I has done at Client Side: 1. Under root user: mount -t glusterfs IP_server:/volume_name -o acl /mnt 2. Under root user at client, create a folder named thainb: mkdir -p /mnt/thainb 3. Under root user: chown -R thainb:thainb /mnt/thainb 4. Under root user: set -m u:thainb:rw /mnt/thainb 5. su thainb 6. create some folder and file under thainb folder 7. su peter 8. remove file and folder 9. Permission denied log from glusterfs: [2011-10-20 08:58:17.603705] W [fuse-bridge.c:847:fuse_err_cbk] 0-glusterfs-fuse: 2646: ACCESS() /thainb/readme = -1 (Permission denied) [2011-10-20 08:58:19.7369] W [fuse-bridge.c:908:fuse_unlink_cbk] 0-glusterfs-fuse: 2648: UNLINK() /thainb/readme = -1 (Permission denied) 10. Exit su and under root user, remove file and folder 11. No log from gluster ---snip- [root@GSO_DB_Local4 thainb]# su peter [peter@GSO_DB_Local4 thainb]$ ls -al total 56 drwxrwxr-x+ 3 thainb thainb 8192 Oct 20 08:56 . drwxr-xr-x 26 root root 8192 Oct 20 08:34 .. -rw-rw-r-- 1 thainb thainb6 Oct 20 08:56 readme drwxrwxr-x 2 thainb thainb 8192 Oct 20 08:40 test [peter@GSO_DB_Local4 thainb]$ rm readme rm: remove write-protected regular file `readme'? y rm: cannot remove `readme': Permission denied [peter@GSO_DB_Local4 thainb]$ exit exit [root@GSO_DB_Local4 thainb]# ls readme test [root@GSO_DB_Local4 thainb]# rm readme rm: remove regular file `readme'? y [root@GSO_DB_Local4 thainb]# rm -f test rm: cannot remove `test': Is a directory [root@GSO_DB_Local4 thainb]# rm -rf test [root@GSO_DB_Local4 thainb]# ls [root@GSO_DB_Local4 thainb]# getfacl . # file: . # owner: thainb # group: thainb user::rwx user:thainb:rw- group::r-x mask::rwx other::r-x [root@GSO_DB_Local4 thainb]# pwd /mnt/thainb [root@GSO_DB_Local4 thainb]# df -h FilesystemSize Used Avail Use% Mounted on /dev/sda3 74G 7.1G 63G 11% / /dev/sda1 99M 12M 82M 13% /boot tmpfs 2.0G 0 2.0G 0% /dev/shm /dev/sda3 74G 7.1G 63G 11% /data glusterfs#lab3:/farm53 148G 9.5G 131G 7% /mnt -end-- What do you suggest? Thanks, ~Thai From: Anush Shetty [mailto:an...@gluster.com] Sent: Wednesday, October 19, 2011 5:58 PM To: Thai. Ngo Bao; gluster-users@gluster.org Subject: RE: [Gluster-users] ACL Hi, Do you see any error messages when you try to set ACL? Can you also paste the logs here? - Anush From: gluster-users-boun...@gluster.org [gluster-users-boun...@gluster.org] on behalf of Thai. Ngo Bao [tha...@vng.com.vn] Sent: 19 October 2011 16:16:36 To: gluster-users@gluster.org Subject: [Gluster-users] ACL Hi, I am testing gluster 3.2.4 with ACL on small linux cluster: 1. All exports (bricks) mounted with -o acl option 2. All the glusterfs clients mounted with -o acl option Acl works perfectly when clients mount under normal users but it will not work if clients mount under root user. What am I missing? How can we force clients to mount under normal user rather than root user? Thanks, ~Thai ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
[Gluster-users] Notify when reaching quota limit
Hi guys, What do you suggest if I would like to give our users some notification (via email) about their reaching quota limits? My current approach is to get periodically the output of gluster volume quota vol_name list and send to users but it is not efficient and causes some impact on performance of our system. Any insight into this is much appreciated. Thanks, ~Thai ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] NFS expose subfolders only
Anush, thanks for the quick reply. Below is the output of showmount at server side: [root@GS_BackupFarm_Cluster01 ~]# showmount -e localhost Export list for localhost: /bkfarm/00 * Output from netstat: [root@GS_BackupFarm_Cluster01 ~]# netstat -vtlpn Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp0 0 0.0.0.0:38465 0.0.0.0:* LISTEN 1531/glusterfs tcp0 0 0.0.0.0:38466 0.0.0.0:* LISTEN 1531/glusterfs tcp0 0 0.0.0.0:38467 0.0.0.0:* LISTEN 1531/glusterfs tcp0 0 0.0.0.0:805 0.0.0.0:* LISTEN 4015/rpc.statd tcp0 0 0.0.0.0:111 0.0.0.0:* LISTEN 3934/portmap tcp0 0 :::24007:::* LISTEN 22996/glusterd tcp0 0 :::24009:::* LISTEN 8653/glusterfsd -client side --- [root@GSO_DB_Local1 ~]# showmount -e localhost mount clntudp_create: RPC: Program not registered [root@GSO_DB_Local1 ~]# mount -o vers=3 bkf3:/00 /bkfarm/ mount: bkf3:/00 failed, reason given by server: No such file or directory Any ideas? Thanks, ~Thai From: Anush Shetty [mailto:an...@gluster.com] Sent: Monday, November 14, 2011 5:40 PM To: Thai. Ngo Bao; gluster-users@gluster.org Subject: RE: NFS expose subfolders only Hi, Please make sure that nfs-kernel-server isn't running. Can you pase your showmount -e output? The right way to mount Gluster NFS is, mount -o vers=3 bkf3:/00 /bkfarm/ - Anush From: Thai. Ngo Bao [tha...@vng.com.vn] Sent: 14 November 2011 16:07:05 To: Anush Shetty; gluster-users@gluster.org Subject: RE: NFS expose subfolders only Hi, I have tried the trick several times but had no success so far. Below is the info of my testing environment [root@GS_BackupFarm_Cluster01 ~]# gluster volume set bkfarm nfs.export-dir /00 Set volume successful [root@GS_BackupFarm_Cluster01 ~]# gluster volume set bkfarm nfs.export-volumes off Set volume successful [root@GS_BackupFarm_Cluster01 ~]# gluster volume info Volume Name: bkfarm Type: Distributed-Replicate Status: Started Number of Bricks: 4 x 2 = 8 Transport-type: tcp Bricks: Brick1: bkf1:/sfarm Brick2: bkf2:/sfarm Brick3: bkf3:/sfarm Brick4: bkf4:/sfarm Brick5: bkf5:/sfarm Brick6: bkf6:/sfarm Brick7: bkf7:/sfarm Brick8: bkf8:/sfarm Options Reconfigured: nfs.disable: Off nfs.export-dir: /00 nfs.export-volumes: off diagnostics.client-log-level: ERROR diagnostics.brick-log-level: ERROR cluster.min-free-disk: 10% performance.cache-size: 2048MB --client side - [root@GSO_DB_Local1 ~]# mount -t nfs bkf3:/00 /bkfarm/ mount: bkf3:/00 failed, reason given by server: No such file or directory What am I missing? Any thought? Thanks, ~Thai From: Anush Shetty [mailto:an...@gluster.com] Sent: Monday, November 14, 2011 5:21 PM To: Thai. Ngo Bao; gluster-users@gluster.org Subject: RE: NFS expose subfolders only Hi, These volume set options should do the trick gluster volume set xfsvol nfs.export-dir /subdir gluster volume set xfsvol nfs.export-volumes off Let me know. - Anush From: gluster-users-boun...@gluster.org [gluster-users-boun...@gluster.org] on behalf of Thai. Ngo Bao [tha...@vng.com.vn] Sent: 14 November 2011 15:39:46 To: gluster-users@gluster.org Subject: [Gluster-users] NFS expose subfolders only Hi, AFAIK one can expose subfolders thru gluster NFS for a client if he/she has option nfs.export-volumes ON and nfs.export-dir Enable. There will be a case (for security purpose) when people just want to expose only subfolders for clients thru NFS. I wonder if it is possible to expose only subfolders thru gluster NFS protocol for a client? Any insight into this is much appreciated. Thanks, ~Thai No virus found in this message. Checked by AVG - www.avg.comhttp://www.avg.com Version: 2012.0.1869 / Virus Database: 2092/4615 - Release Date: 11/13/11 No virus found in this message. Checked by AVG - www.avg.comhttp://www.avg.com Version: 2012.0.1869 / Virus Database: 2092/4615 - Release Date: 11/13/11 ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] NFS expose subfolders only
Bingo. It works. Shehjar, thanks for your hint. ~Thai -Original Message- From: Shehjar Tikoo [mailto:shehj...@gluster.com] Sent: Monday, November 14, 2011 5:55 PM To: Thai. Ngo Bao Cc: Anush Shetty; gluster-users@gluster.org Subject: Re: [Gluster-users] NFS expose subfolders only Directory exports are enabled by default. You just need to mount using /bkfarm/00 as the export dir, not /00. -Shehjar Thai. Ngo Bao wrote: Anush, thanks for the quick reply. Below is the output of showmount at server side: [root@GS_BackupFarm_Cluster01 ~]# showmount -e localhost Export list for localhost: /bkfarm/00 * Output from netstat: [root@GS_BackupFarm_Cluster01 ~]# netstat -vtlpn Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp0 0 0.0.0.0:38465 0.0.0.0:* LISTEN 1531/glusterfs tcp0 0 0.0.0.0:38466 0.0.0.0:* LISTEN 1531/glusterfs tcp0 0 0.0.0.0:38467 0.0.0.0:* LISTEN 1531/glusterfs tcp0 0 0.0.0.0:805 0.0.0.0:* LISTEN 4015/rpc.statd tcp0 0 0.0.0.0:111 0.0.0.0:* LISTEN 3934/portmap tcp0 0 :::24007 :::*LISTEN 22996/glusterd tcp0 0 :::24009 :::*LISTEN 8653/glusterfsd -client side --- [root@GSO_DB_Local1 ~]# showmount -e localhost mount clntudp_create: RPC: Program not registered [root@GSO_DB_Local1 ~]# mount -o vers=3 bkf3:/00 /bkfarm/ mount: bkf3:/00 failed, reason given by server: No such file or directory Any ideas? Thanks, ~Thai *From:* Anush Shetty [mailto:an...@gluster.com] *Sent:* Monday, November 14, 2011 5:40 PM *To:* Thai. Ngo Bao; gluster-users@gluster.org *Subject:* RE: NFS expose subfolders only Hi, Please make sure that nfs-kernel-server isn't running. Can you pase your showmount -e output? The right way to mount Gluster NFS is, mount -o vers=3 bkf3:/00 /bkfarm/ - Anush *From:* Thai. Ngo Bao [tha...@vng.com.vn] *Sent:* 14 November 2011 16:07:05 *To:* Anush Shetty; gluster-users@gluster.org *Subject:* RE: NFS expose subfolders only Hi, I have tried the trick several times but had no success so far. Below is the info of my testing environment [root@GS_BackupFarm_Cluster01 ~]# gluster volume set bkfarm nfs.export-dir /00 Set volume successful [root@GS_BackupFarm_Cluster01 ~]# gluster volume set bkfarm nfs.export-volumes off Set volume successful [root@GS_BackupFarm_Cluster01 ~]# gluster volume info Volume Name: bkfarm Type: Distributed-Replicate Status: Started Number of Bricks: 4 x 2 = 8 Transport-type: tcp Bricks: Brick1: bkf1:/sfarm Brick2: bkf2:/sfarm Brick3: bkf3:/sfarm Brick4: bkf4:/sfarm Brick5: bkf5:/sfarm Brick6: bkf6:/sfarm Brick7: bkf7:/sfarm Brick8: bkf8:/sfarm Options Reconfigured: nfs.disable: Off nfs.export-dir: /00 - No virus found in this message. Checked by AVG - www.avg.com Version: 2012.0.1869 / Virus Database: 2092/4615 - Release Date: 11/13/11 ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] NFS expose subfolders only
Hi, Is it possible to expose multiple sub-folders using nfs.export-dir? I do not have an access to my test environment right now, so I guess something similar should be done: gluster volume set volume_name nfs.export-dir /sub-folder1 /sub-folder2 ... ? Any insight into this is much appreciated, and I will definitely figure out what should be done tomorrow for the case. Thanks, ~Thai From: gluster-users-boun...@gluster.org [gluster-users-boun...@gluster.org] On Behalf Of Thai. Ngo Bao [tha...@vng.com.vn] Sent: Monday, November 14, 2011 6:02 PM To: Shehjar Tikoo Cc: gluster-users@gluster.org Subject: Re: [Gluster-users] NFS expose subfolders only Bingo. It works. Shehjar, thanks for your hint. ~Thai -Original Message- From: Shehjar Tikoo [mailto:shehj...@gluster.com] Sent: Monday, November 14, 2011 5:55 PM To: Thai. Ngo Bao Cc: Anush Shetty; gluster-users@gluster.org Subject: Re: [Gluster-users] NFS expose subfolders only Directory exports are enabled by default. You just need to mount using /bkfarm/00 as the export dir, not /00. -Shehjar Thai. Ngo Bao wrote: Anush, thanks for the quick reply. Below is the output of showmount at server side: [root@GS_BackupFarm_Cluster01 ~]# showmount -e localhost Export list for localhost: /bkfarm/00 * Output from netstat: [root@GS_BackupFarm_Cluster01 ~]# netstat -vtlpn Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp0 0 0.0.0.0:38465 0.0.0.0:* LISTEN 1531/glusterfs tcp0 0 0.0.0.0:38466 0.0.0.0:* LISTEN 1531/glusterfs tcp0 0 0.0.0.0:38467 0.0.0.0:* LISTEN 1531/glusterfs tcp0 0 0.0.0.0:805 0.0.0.0:* LISTEN 4015/rpc.statd tcp0 0 0.0.0.0:111 0.0.0.0:* LISTEN 3934/portmap tcp0 0 :::24007 :::*LISTEN 22996/glusterd tcp0 0 :::24009 :::*LISTEN 8653/glusterfsd -client side --- [root@GSO_DB_Local1 ~]# showmount -e localhost mount clntudp_create: RPC: Program not registered [root@GSO_DB_Local1 ~]# mount -o vers=3 bkf3:/00 /bkfarm/ mount: bkf3:/00 failed, reason given by server: No such file or directory Any ideas? Thanks, ~Thai *From:* Anush Shetty [mailto:an...@gluster.com] *Sent:* Monday, November 14, 2011 5:40 PM *To:* Thai. Ngo Bao; gluster-users@gluster.org *Subject:* RE: NFS expose subfolders only Hi, Please make sure that nfs-kernel-server isn't running. Can you pase your showmount -e output? The right way to mount Gluster NFS is, mount -o vers=3 bkf3:/00 /bkfarm/ - Anush *From:* Thai. Ngo Bao [tha...@vng.com.vn] *Sent:* 14 November 2011 16:07:05 *To:* Anush Shetty; gluster-users@gluster.org *Subject:* RE: NFS expose subfolders only Hi, I have tried the trick several times but had no success so far. Below is the info of my testing environment [root@GS_BackupFarm_Cluster01 ~]# gluster volume set bkfarm nfs.export-dir /00 Set volume successful [root@GS_BackupFarm_Cluster01 ~]# gluster volume set bkfarm nfs.export-volumes off Set volume successful [root@GS_BackupFarm_Cluster01 ~]# gluster volume info Volume Name: bkfarm Type: Distributed-Replicate Status: Started Number of Bricks: 4 x 2 = 8 Transport-type: tcp Bricks: Brick1: bkf1:/sfarm Brick2: bkf2:/sfarm Brick3: bkf3:/sfarm Brick4: bkf4:/sfarm Brick5: bkf5:/sfarm Brick6: bkf6:/sfarm Brick7: bkf7:/sfarm Brick8: bkf8:/sfarm Options Reconfigured: nfs.disable: Off nfs.export-dir: /00 - No virus found in this message. Checked by AVG - www.avg.com Version: 2012.0.1869 / Virus Database: 2092/4615 - Release Date: 11/13/11 ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users - No virus found in this message. Checked by AVG - www.avg.com Version: 2012.0.1869 / Virus Database: 2092/4615 - Release Date: 11/13/11 ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] ACL
Hi Anush, Well, I was aware of this feature of glusterfs several months ago and one will lose the advantage of glusterfs (such as high performance, fail-over, etc) if he/she uses gluster nfs instead of gluster native protocol. I believe using gluster nfs will solve the problem I am concerning with (export some subdirectory per client). Could you please point me out what one will lose when using gluster nsf instead of native protocol in terms of performance? Thanks, ~Thai From: Anush Shetty [mailto:an...@gluster.com] Sent: Friday, October 21, 2011 12:07 PM To: Thai. Ngo Bao; gluster-users@gluster.org Subject: RE: [Gluster-users] ACL Hi Thai, Yes, we do. http://community.gluster.org/q/export-subdirectories-of-volumes-with-nfs/ - Anush From: Thai. Ngo Bao [tha...@vng.com.vn] Sent: 21 October 2011 08:21:32 To: Anush Shetty; gluster-users@gluster.org Subject: RE: [Gluster-users] ACL Hi Anush, Thanks for the info. Also, I wonder if gluster concerns with some feature allowing glusterfs to export subdirectories for clients by gluster native protocol like NFS? Thanks, ~Thai From: Anush Shetty [mailto:an...@gluster.com] Sent: Thursday, October 20, 2011 1:37 PM To: Thai. Ngo Bao; gluster-users@gluster.org Subject: RE: [Gluster-users] ACL Hi Thai, As of now, there are no provisions to mount GlusterFS client as a normal user. - Anush From: gluster-users-boun...@gluster.org [gluster-users-boun...@gluster.org] on behalf of Thai. Ngo Bao [tha...@vng.com.vn] Sent: 20 October 2011 12:00:44 To: Anush Shetty; gluster-users@gluster.org Subject: Re: [Gluster-users] ACL Hi Anush, Thanks for your response. I do know that there is almost no difference in behavior of these 2 cases (POSIX ACL) except the thing I concerned with by the test. Please correct me if I am wrong. It turns out that glusterfs ACL has no effect if client mounts gluster volume under root user. That's why I asked in previous email if we can anyhow force (from glusterfs servers) clients to mount under a normal user instead of root. What do you think? ~Thai From: Anush Shetty [mailto:an...@gluster.com] Sent: Thursday, October 20, 2011 1:03 PM To: Thai. Ngo Bao; gluster-users@gluster.org Subject: RE: [Gluster-users] ACL Hi Thai, Have you tried these steps without glusterfs in the picture? I see no difference in behaviour when tried directly on the backend filesystem. - Anush From: Thai. Ngo Bao [tha...@vng.com.vn] Sent: 20 October 2011 08:04:31 To: Anush Shetty; gluster-users@gluster.org Subject: RE: [Gluster-users] ACL Hi Anush, Thanks for your quick reply. There was no error when set ACL. Below is what I has done at Client Side: 1. Under root user: mount -t glusterfs IP_server:/volume_name -o acl /mnt 2. Under root user at client, create a folder named thainb: mkdir -p /mnt/thainb 3. Under root user: chown -R thainb:thainb /mnt/thainb 4. Under root user: set -m u:thainb:rw /mnt/thainb 5. su thainb 6. create some folder and file under thainb folder 7. su peter 8. remove file and folder 9. Permission denied log from glusterfs: [2011-10-20 08:58:17.603705] W [fuse-bridge.c:847:fuse_err_cbk] 0-glusterfs-fuse: 2646: ACCESS() /thainb/readme = -1 (Permission denied) [2011-10-20 08:58:19.7369] W [fuse-bridge.c:908:fuse_unlink_cbk] 0-glusterfs-fuse: 2648: UNLINK() /thainb/readme = -1 (Permission denied) 10. Exit su and under root user, remove file and folder 11. No log from gluster ---snip- [root@GSO_DB_Local4 thainb]# su peter [peter@GSO_DB_Local4 thainb]$ ls -al total 56 drwxrwxr-x+ 3 thainb thainb 8192 Oct 20 08:56 . drwxr-xr-x 26 root root 8192 Oct 20 08:34 .. -rw-rw-r-- 1 thainb thainb6 Oct 20 08:56 readme drwxrwxr-x 2 thainb thainb 8192 Oct 20 08:40 test [peter@GSO_DB_Local4 thainb]$ rm readme rm: remove write-protected regular file `readme'? y rm: cannot remove `readme': Permission denied [peter@GSO_DB_Local4 thainb]$ exit exit [root@GSO_DB_Local4 thainb]# ls readme test [root@GSO_DB_Local4 thainb]# rm readme rm: remove regular file `readme'? y [root@GSO_DB_Local4 thainb]# rm -f test rm: cannot remove `test': Is a directory [root@GSO_DB_Local4 thainb]# rm -rf test [root@GSO_DB_Local4 thainb]# ls [root@GSO_DB_Local4 thainb]# getfacl . # file: . # owner: thainb # group: thainb user::rwx user:thainb:rw- group::r-x mask::rwx other::r-x [root@GSO_DB_Local4 thainb]# pwd /mnt/thainb [root@GSO_DB_Local4 thainb]# df -h FilesystemSize Used Avail Use% Mounted on /dev/sda3 74G 7.1G 63G 11% / /dev/sda1 99M 12M 82M 13% /boot tmpfs 2.0G 0 2.0G 0% /dev/shm /dev/sda3 74G 7.1G 63G 11% /data glusterfs#lab3:/farm53 148G 9.5G 131G 7% /mnt -end-- What do you suggest? Thanks, ~Thai From: Anush Shetty
Re: [Gluster-users] ACL
Hi Anush, Thanks for your response. I do know that there is almost no difference in behavior of these 2 cases (POSIX ACL) except the thing I concerned with by the test. Please correct me if I am wrong. It turns out that glusterfs ACL has no effect if client mounts gluster volume under root user. That's why I asked in previous email if we can anyhow force (from glusterfs servers) clients to mount under a normal user instead of root. What do you think? ~Thai From: Anush Shetty [mailto:an...@gluster.com] Sent: Thursday, October 20, 2011 1:03 PM To: Thai. Ngo Bao; gluster-users@gluster.org Subject: RE: [Gluster-users] ACL Hi Thai, Have you tried these steps without glusterfs in the picture? I see no difference in behaviour when tried directly on the backend filesystem. - Anush From: Thai. Ngo Bao [tha...@vng.com.vn] Sent: 20 October 2011 08:04:31 To: Anush Shetty; gluster-users@gluster.org Subject: RE: [Gluster-users] ACL Hi Anush, Thanks for your quick reply. There was no error when set ACL. Below is what I has done at Client Side: 1. Under root user: mount -t glusterfs IP_server:/volume_name -o acl /mnt 2. Under root user at client, create a folder named thainb: mkdir -p /mnt/thainb 3. Under root user: chown -R thainb:thainb /mnt/thainb 4. Under root user: set -m u:thainb:rw /mnt/thainb 5. su thainb 6. create some folder and file under thainb folder 7. su peter 8. remove file and folder 9. Permission denied log from glusterfs: [2011-10-20 08:58:17.603705] W [fuse-bridge.c:847:fuse_err_cbk] 0-glusterfs-fuse: 2646: ACCESS() /thainb/readme = -1 (Permission denied) [2011-10-20 08:58:19.7369] W [fuse-bridge.c:908:fuse_unlink_cbk] 0-glusterfs-fuse: 2648: UNLINK() /thainb/readme = -1 (Permission denied) 10. Exit su and under root user, remove file and folder 11. No log from gluster ---snip- [root@GSO_DB_Local4 thainb]# su peter [peter@GSO_DB_Local4 thainb]$ ls -al total 56 drwxrwxr-x+ 3 thainb thainb 8192 Oct 20 08:56 . drwxr-xr-x 26 root root 8192 Oct 20 08:34 .. -rw-rw-r-- 1 thainb thainb6 Oct 20 08:56 readme drwxrwxr-x 2 thainb thainb 8192 Oct 20 08:40 test [peter@GSO_DB_Local4 thainb]$ rm readme rm: remove write-protected regular file `readme'? y rm: cannot remove `readme': Permission denied [peter@GSO_DB_Local4 thainb]$ exit exit [root@GSO_DB_Local4 thainb]# ls readme test [root@GSO_DB_Local4 thainb]# rm readme rm: remove regular file `readme'? y [root@GSO_DB_Local4 thainb]# rm -f test rm: cannot remove `test': Is a directory [root@GSO_DB_Local4 thainb]# rm -rf test [root@GSO_DB_Local4 thainb]# ls [root@GSO_DB_Local4 thainb]# getfacl . # file: . # owner: thainb # group: thainb user::rwx user:thainb:rw- group::r-x mask::rwx other::r-x [root@GSO_DB_Local4 thainb]# pwd /mnt/thainb [root@GSO_DB_Local4 thainb]# df -h FilesystemSize Used Avail Use% Mounted on /dev/sda3 74G 7.1G 63G 11% / /dev/sda1 99M 12M 82M 13% /boot tmpfs 2.0G 0 2.0G 0% /dev/shm /dev/sda3 74G 7.1G 63G 11% /data glusterfs#lab3:/farm53 148G 9.5G 131G 7% /mnt -end-- What do you suggest? Thanks, ~Thai From: Anush Shetty [mailto:an...@gluster.com] Sent: Wednesday, October 19, 2011 5:58 PM To: Thai. Ngo Bao; gluster-users@gluster.org Subject: RE: [Gluster-users] ACL Hi, Do you see any error messages when you try to set ACL? Can you also paste the logs here? - Anush From: gluster-users-boun...@gluster.org [gluster-users-boun...@gluster.org] on behalf of Thai. Ngo Bao [tha...@vng.com.vn] Sent: 19 October 2011 16:16:36 To: gluster-users@gluster.org Subject: [Gluster-users] ACL Hi, I am testing gluster 3.2.4 with ACL on small linux cluster: 1. All exports (bricks) mounted with -o acl option 2. All the glusterfs clients mounted with -o acl option Acl works perfectly when clients mount under normal users but it will not work if clients mount under root user. What am I missing? How can we force clients to mount under normal user rather than root user? Thanks, ~Thai No virus found in this message. Checked by AVG - www.avg.comhttp://www.avg.com Version: 2012.0.1831 / Virus Database: 2092/4562 - Release Date: 10/19/11 ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
[Gluster-users] ACL
Hi, I am testing gluster 3.2.4 with ACL on small linux cluster: 1. All exports (bricks) mounted with –o acl option 2. All the glusterfs clients mounted with –o acl option Acl works perfectly when clients mount under normal users but it will not work if clients mount under root user. What am I missing? How can we force clients to mount under normal user rather than root user? Thanks, ~Thai From: gluster-users-boun...@gluster.org [mailto:gluster-users-boun...@gluster.org] On Behalf Of Amar Tumballi Sent: Tuesday, August 02, 2011 2:57 AM To: Papp Tamas Cc: gluster-users@gluster.org Subject: Re: [Gluster-users] xfs+acl To use 'acl' below thing should be minimum setup: * glusterfs version 3.2.2 or higher * all exports (ie bricks) mounted with '-oacl' option * all the glusterfs clients mounted with '-oacl' option Regards, Amar On Tue, Aug 2, 2011 at 1:06 AM, Papp Tamas tom...@martos.bme.humailto:tom...@martos.bme.hu wrote: On 2011-08-01 21:32, Liam Slusser wrote: We use XFS with Gluster with great success. We also do. I'm asking about acl. Thank you, tamas ___ Gluster-users mailing list Gluster-users@gluster.orgmailto:Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users No virus found in this message. Checked by AVG - www.avg.comhttp://www.avg.com Version: 2012.0.1831 / Virus Database: 2092/4560 - Release Date: 10/18/11 ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
[Gluster-users] where are files physically stored in recent version (3.1.x)
Hello list, I have been developing a mapreduce - application running on GlusterFS. Therefore files’ extended attributes are pretty much critical for the application. I am curious to know what is the official way of finding out where files are physically stored. Should you have any suggestions or experience you want to share, please let me know. Thanks, Thai Ngo ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
[Gluster-users] glusterfs integration with Hadoop
Hello, Is there any one trying to integrate glusterfs with hadoop? Would you please give me some advice on it? Thank you in advance, ~Thai ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] No space left on device
Hi, Are you doing rebalance? If not, can you try: 1. umount the mountpoint 2. unload the FUSE module 3. re-load FUSE module 4. re-mount the mountpoint Lets us know if it is fixed. Also, what does a output of: getfattr -d -m '\.' Filename look like? (in your gluster system) Good luck, ~Thai -Original Message- From: gluster-users-boun...@gluster.org [mailto:gluster-users-boun...@gluster.org] On Behalf Of Daniel Zander Sent: Wednesday, January 19, 2011 7:47 PM Cc: gluster-users@gluster.org Subject: Re: [Gluster-users] No space left on device Hi, yes, we already checked that FilesystemInodes IUsed IFree IUse% Mounted on /dev/sda13055616 51143 30044732% / none 2057204 720 20564841% /dev none 2058311 1 20583101% /dev/shm none 2058311 22 20582891% /var/run none 2058311 2 20583091% /var/lock none 2058311 3 20583081% /lib/init/rw /dev/sdb 4818066560 4277250 48137893101% /storage/6 /dev/sda612206082232 12183761% /var /dev/sda72445984 12 24459721% /tmp We managed to free some more space (now ~1TB free) and creating files seem to work again. However, during the migration of a user, some files don't show up in his directory, although we are able to find them via ls filename. We decided to reverse the migration and try again, but then I was not able to delete his directory: rm: FATAL: directory `delete_me/rootfiles/dplusdminus' changed dev/ino Also, when we tried chown -R user:group user/ the following error popped up. chown: fts_read failed: No such file or directory Additionally, some directories show up in ls -l like that ?- ? ???? b_dd This all is really confusing to us. Can someone shed some light on this mess? Regards, Daniel ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] permission denied and no such file for real files
Dear All, I am having a problem, and I think it is similar to Lana's case. The problem is glusterfs prevents clients to create/delete files and folders. This problem appears so regularly. I attached the log file of our client. Please also keep informed that I'm running glusterfs 3.1.1 with 4 bricks, volumes distributed, tcp transport, fuse clients. The servers and clients are running CentOS 5.4. Would you have any suggestions, please let me know. Thanks. ~Thai -Original Message- From: gluster-users-boun...@gluster.org [mailto:gluster-users-boun...@gluster.org] On Behalf Of Lana Deere Sent: Tuesday, December 21, 2010 6:27 AM To: gluster-users@gluster.org Subject: [Gluster-users] permission denied and no such file for real files I'm running glusterfs 3.1.1, 4 volumes distributed, tcp transport, fuse clients. The servers are running CentOS 5.5, the clients CentOS 5.4. The symptom I'm having which reproduces the most frequently goes like this. There are a bunch of jobs (several hundred) running on various client nodes (~50), but for no obvious reason one or several of them will get a failure to open a file (errno 13, permission denied). Subsequently it will not be possible to delete that directory because it is not empty. Attempts to delete the files which make it non-empty fail, no such file or directory. I attached the program and driver script which I use to reproduce this, but not some log files (I ran them with trace and even compressed they are 150M). It is not perfect, it only reproduces maybe one-third of the runs. Here's a typical output: $ ATTACK-QRSH Quantity of jobs: 1000 Mon Dec 20 14:59:52 EST 2010: rm . Mon Dec 20 14:59:52 EST 2010: create data . Mon Dec 20 14:59:54 EST 2010: wait for creates. Couldn't create data/051/307-this!:is:!thefilenameanditisverystrange.tmp: errno 13 (Permission denied) Couldn't create data/051/563-this!:is:!thefilenameanditisverystrange.tmp: errno 13 (Permission denied) Mon Dec 20 15:04:45 EST 2010: check data . Mon Dec 20 15:04:47 EST 2010: wait for checks. Unable to reopen for read data/051/563-this!:is:!thefilenameanditisverystrange, errno 2 (No such file or directory) Unable to reopen for read data/051/307-this!:is:!thefilenameanditisverystrange, errno 2 (No such file or directory) Mon Dec 20 15:08:53 EST 2010: rm again . rm: cannot remove directory `data/051': Directory not empty Mon Dec 20 15:09:16 EST 2010: done $ cd data/051 $ ls 179-this!:is:!thefilenameanditisverystrange 435-this!:is:!thefilenameanditisverystrange $ rm * rm: cannot remove `179-this!:is:!thefilenameanditisverystrange': No such file or directory rm: cannot remove `435-this!:is:!thefilenameanditisverystrange': No such file or directory $ cd .. $ rm -r 051 rm: cannot remove `051/179-this!:is:!thefilenameanditisverystrange': No such file or directory rm: cannot remove `051/435-this!:is:!thefilenameanditisverystrange': No such file or directory Note that the files which won't delete are in the same directory as had the permission error earlier. .. Lana (lana.de...@gmail.com) ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
[Gluster-users] How does gluster identify which brick is the closest
Hello all, First of all, I would like to thank you for the great FS. I have some basic questions regarding Elastic Hash Algorithm (EHA). I understand that EHA decides data placement at a host level and then places complete files on its selected storage. EHA also takes one step further and optimizes selection based not just on availability, but performance as well. Thus, if there is a copy of the file that is closer to the user, the closest copy is used. So my questions are: How does gluster identify where the closest copy is? How does gluster identify which brick storing the copy closest to user? Which bricks are in a same or different subnets? Which value(s) does gluster take to compare for that purpose? Are they dynamic values, the results of EHA? Also, please point me out where is the code segments/modules of gluster doing these jobs. I appreciate your time, and thank you in advance. Regards, Thai Ngo ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
