Re: [Gluster-users] Sorry for x-post: RDMA Mounts drop with transport endpoints not connected
I've got my OFED (client server) versions in-sync with the stock version that are part of the latest Scientific Linux 6.1 packages (removing QLogic's OFED distribution). I'm going to see if the stability issues ensue. SL 6.1 Kernel: 2.6.32-220.2.1.el6.x86_64 HCA: InfiniBand: QLogic Corp. IBA7322 QDR InfiniBand HCA (rev 02) Switches: roots: 1 x QLogic 12300 w/ SM, 2 x QLogic 12200 edge: 5 x QLogic 12200 ibv_devinfo -v hca_id:qib0 transport:InfiniBand (0) fw_ver:0.0.0 node_guid:0011:7500:0078:b690 sys_image_guid:0011:7500:0078:b690 vendor_id:0x1175 vendor_part_id:29474 hw_ver:0x2 board_id:InfiniPath_QLE7340 phys_port_cnt:1 max_mr_size:0x page_size_cap:0x1000 max_qp:16384 max_qp_wr:16383 device_cap_flags:0x3d06 max_sge:96 max_sge_rd:0 max_cq:131071 max_cqe:196607 max_mr:65536 max_pd:65535 max_qp_rd_atom:16 max_ee_rd_atom:0 max_res_rd_atom:0 max_qp_init_rd_atom:255 max_ee_init_rd_atom:0 atomic_cap:ATOMIC_GLOB (2) max_ee:0 max_rdd:0 max_mw:0 max_raw_ipv6_qp:0 max_raw_ethy_qp:0 max_mcast_grp:16384 max_mcast_qp_attach:16 max_total_mcast_qp_attach:262144 max_ah:65535 max_fmr:65536 max_map_per_fmr:32767 max_srq:1024 max_srq_wr:131071 max_srq_sge:128 max_pkeys:4 local_ca_ack_delay:0 port:1 state:PORT_ACTIVE (4) max_mtu:2048 (4) active_mtu:2048 (4) sm_lid:97 port_lid:51 port_lmc:0x00 max_msg_sz:0x8000 port_cap_flags:0x07610868 max_vl_num:2 (2) bad_pkey_cntr:0x0 qkey_viol_cntr:0x0 sm_sl:0 pkey_tbl_len:4 gid_tbl_len:5 subnet_timeout:17 init_type_reply:0 active_width:4X (2) active_speed:10.0 Gbps (4) phys_state:LINK_UP (5) GID[ 0]:fe80::::0011:7500:0078:b690 -Brian On 02/01/2012 04:51 PM, Joe Landman wrote: On 02/01/2012 04:49 PM, Brian Smith wrote: Having serious issues w/ glusterfs 3.2.5 over rdma. Clients are periodically dropping off with transport endpoint not connected. Any help would be appreciated. Environment is HPC. GlusterFS is being used as a shared /work|/scratch directory. Standard distributed volume configuration. Nothing fancy. Pastie log snippet is here: http://pastie.org/3291330 Any help would be appreciated! What OS, kernel rev, OFED, etc. What HCAs, switch, etc. What does ibv_devinfo report for nodes experiencing the transport endpoint issue? ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
[Gluster-users] Sorry for x-post: RDMA Mounts drop with transport endpoints not connected
Having serious issues w/ glusterfs 3.2.5 over rdma. Clients are periodically dropping off with transport endpoint not connected. Any help would be appreciated. Environment is HPC. GlusterFS is being used as a shared /work|/scratch directory. Standard distributed volume configuration. Nothing fancy. Pastie log snippet is here: http://pastie.org/3291330 Any help would be appreciated! -- Brian Smith Senior Systems Administrator IT Research Computing, University of South Florida 4202 E. Fowler Ave. ENB308 Office Phone: +1 813 974-1467 Organization URL: http://rc.usf.edu ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
[Gluster-users] Adding bricks to enable distribute-replicate
Hi, all, I've searched through the archives and saw that this has come up before, but it was not clear what the best way was to accomplish this. I'm currently on 3.2.3 and would like to convert a two-brick distribute volume to a four-brick disribute-replicate volume. Would editing the vol file directly by encapsulating the bricks I want to replicate in the replicate translator, then encapsulating that in the distribute translator be the way to go, then ls -RA to trigger a self-heal ? Is there a way to do this from the gluster console command? Thanks, -Brian Brian Smith Senior Systems Administrator IT Research Computing, University of South Florida 4202 E. Fowler Ave. ENB308 Office Phone: +1 813 974-1467 Organization URL: http://rc.usf.edu On 10/27/2011 02:00 AM, Mohammed Junaid wrote: Hi Daniel, However, I've been trying to setup an extra swift proxy to offload some of the CPU usage I've been experiencing in our tests and I can't really find a way to do so. What's the correct way to proceed, with gluster's object-storage? Anyone else has done it? To increase the proxy server count, edit the file /etc/gluster-object/proxy-server.conf and add the line workers =count Following Swift's documentation I get stuck at copying the ring files (do they exist in gluster's object-storage?). Then I've thought I could simply setup gluster object-storage in a proxy machine and somehow redirect the requests from the proxy to the storage machine, but I'm failing miserably that way as well. For using gluster object-storage, there is no need to generate the ring files. Please follow the gluster object-storage documentation to setup the machines. Thanks, Junaid ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] xfs+acl
What version of glusterfs are you on? AFAIK, only =3.2.2 has ACL support enabled out-of-the box. Also, you must use -o acl on both the backing-store XFS mounts AND the glusterfs client mounts in order for it to work. -Brian Brian Smith Senior Systems Administrator IT Research Computing, University of South Florida 4202 E. Fowler Ave. ENB308 Office Phone: +1 813 974-1467 Organization URL: http://rc.usf.edu On 08/01/2011 03:22 PM, Papp Tamas wrote: hi! Is this possible? Mounting with -o acl doesn't change mount options: gl0:/w-vol on /W/Projects type fuse.glusterfs (rw,allow_other,default_permissions,max_read=131072) Of course setfacl is not working. On server side -o acl is not possible for xfs filesystems. Thank you, tamas ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Issue with Gluster Quota
Updated to the 3.2.2 release and found that Quotas do not work when using the POSIX-ACL translator. In fact, after I disabled the ACLs, I had to remove the quota'd directory (presumably, to remove some attributes) and start over in order to get them to work. Once I disabled ACLs and re-created my directory, quotas worked as expected. Is this a known limitation of using POSIX ACLs? I happen to need both features, so that could pose an issue :) -Brian Brian Smith Senior Systems Administrator IT Research Computing, University of South Florida 4202 E. Fowler Ave. ENB308 Office Phone: +1 813 974-1467 Organization URL: http://rc.usf.edu On 07/11/2011 12:48 PM, Brian Smith wrote: According to the logs, the last commit was: commit 5c20eb3bbf870edadd22d06babb5d38dad222533 Author: shishir gowda shishi...@gluster.com Date: Tue Jul 5 03:41:51 2011 + [root@gluster1 glusterfs-3.2git]# gluster volume quota home list path limit_set size -- /brs 10485760 81965056 [root@gluster1 glusterfs-3.2git]# gluster volume info Volume Name: home Type: Distribute Status: Started Number of Bricks: 2 Transport-type: tcp,rdma Bricks: Brick1: gluster1:/glusterfs/home Brick2: gluster2:/glusterfs/home Options Reconfigured: features.limit-usage: /brs:10MB features.quota: on -Brian Brian Smith Senior Systems Administrator IT Research Computing, University of South Florida 4202 E. Fowler Ave. ENB308 Office Phone: +1 813 974-1467 Organization URL: http://rc.usf.edu On 07/11/2011 08:29 AM, Saurabh Jain wrote: Hello Brian, I synced my gluster repository back to July 5th and tried quota on a certain dir of a distribute and the quota was implemeted properly on that, here are the logs, [root@centos-qa-client-3 glusterfs]# /root/july6git/inst/sbin/gluster volume quota dist list path limit_set size -- /dir 10485760 10485760 [root@centos-qa-client-3 glusterfs]# /root/july6git/inst/sbin/gluster volume info Volume Name: dist Type: Distribute Status: Started Number of Bricks: 2 Transport-type: tcp,rdma Bricks: Brick1: 10.1.12.134:/mnt/dist Brick2: 10.1.12.135:/mnt/dist Options Reconfigured: features.limit-usage: /dir:10MB features.quota: on [root@centos-qa-client-3 glusterfs]# requesting you to please inform us about the commit id to which your workspace is synced. Thanks, Saurabh From: gluster-users-boun...@gluster.org [gluster-users-boun...@gluster.org] on behalf of gluster-users-requ...@gluster.org [gluster-users-requ...@gluster.org] Sent: Friday, July 08, 2011 12:30 AM To: gluster-users@gluster.org Subject: Gluster-users Digest, Vol 39, Issue 13 Send Gluster-users mailing list submissions to gluster-users@gluster.org To subscribe or unsubscribe via the World Wide Web, visit http://gluster.org/cgi-bin/mailman/listinfo/gluster-users or, via email, send a message with subject or body 'help' to gluster-users-requ...@gluster.org You can reach the person managing the list at gluster-users-ow...@gluster.org When replying, please edit your Subject line so it is more specific than Re: Contents of Gluster-users digest... Today's Topics: 1. Re: Issue with Gluster Quota (Brian Smith) 2. Re: Issues with geo-rep (Carl Chenet) -- Message: 1 Date: Thu, 07 Jul 2011 13:10:06 -0400 From: Brian Smith b...@usf.edu Subject: Re: [Gluster-users] Issue with Gluster Quota To: gluster-users@gluster.org Message-ID: 4e15e86e.6030...@usf.edu Content-Type: text/plain; charset=ISO-8859-1 Sorry about that. I re-populated with an 82MB dump from dd: [root@gluster1 ~]# gluster volume quota home list path limit_set size -- /brs 10485760 81965056 [root@gluster1 ~]# getfattr -m . -d -e hex /glusterfs/home/brs getfattr: Removing leading '/' from absolute path names # file: glusterfs/home/brs security.selinux=0x726f6f743a6f626a6563745f723a66696c655f743a733000 trusted.gfid=0x1bbcb9a08bf64406b440f3bb3ad334ed trusted.glusterfs.dht=0x00017fff trusted.glusterfs.quota.----0001.contri=0x6000 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.size=0x6000 [root@gluster2 ~]# getfattr -m . -d -e hex /glusterfs/home/brs getfattr: Removing leading '/' from absolute path names # file: glusterfs/home/brs security.selinux=0x726f6f743a6f626a6563745f723a66696c655f743a733000 trusted.gfid
Re: [Gluster-users] Issue with Gluster Quota
After further tests, it appears both now work after the update. Seems the attributes set while using the git source build on the 'brs' directory were gummed up. When I recreated the directory and remounted with -o acl, ACLs worked and so did quota enforcement. I'll keep testing and post if anything else comes up. So far, so good. -Brian Brian Smith Senior Systems Administrator IT Research Computing, University of South Florida 4202 E. Fowler Ave. ENB308 Office Phone: +1 813 974-1467 Organization URL: http://rc.usf.edu On 07/18/2011 11:57 AM, Brian Smith wrote: Updated to the 3.2.2 release and found that Quotas do not work when using the POSIX-ACL translator. In fact, after I disabled the ACLs, I had to remove the quota'd directory (presumably, to remove some attributes) and start over in order to get them to work. Once I disabled ACLs and re-created my directory, quotas worked as expected. Is this a known limitation of using POSIX ACLs? I happen to need both features, so that could pose an issue :) -Brian Brian Smith Senior Systems Administrator IT Research Computing, University of South Florida 4202 E. Fowler Ave. ENB308 Office Phone: +1 813 974-1467 Organization URL: http://rc.usf.edu On 07/11/2011 12:48 PM, Brian Smith wrote: According to the logs, the last commit was: commit 5c20eb3bbf870edadd22d06babb5d38dad222533 Author: shishir gowda shishi...@gluster.com Date: Tue Jul 5 03:41:51 2011 + [root@gluster1 glusterfs-3.2git]# gluster volume quota home list path limit_set size -- /brs 10485760 81965056 [root@gluster1 glusterfs-3.2git]# gluster volume info Volume Name: home Type: Distribute Status: Started Number of Bricks: 2 Transport-type: tcp,rdma Bricks: Brick1: gluster1:/glusterfs/home Brick2: gluster2:/glusterfs/home Options Reconfigured: features.limit-usage: /brs:10MB features.quota: on -Brian Brian Smith Senior Systems Administrator IT Research Computing, University of South Florida 4202 E. Fowler Ave. ENB308 Office Phone: +1 813 974-1467 Organization URL: http://rc.usf.edu On 07/11/2011 08:29 AM, Saurabh Jain wrote: Hello Brian, I synced my gluster repository back to July 5th and tried quota on a certain dir of a distribute and the quota was implemeted properly on that, here are the logs, [root@centos-qa-client-3 glusterfs]# /root/july6git/inst/sbin/gluster volume quota dist list path limit_set size -- /dir 10485760 10485760 [root@centos-qa-client-3 glusterfs]# /root/july6git/inst/sbin/gluster volume info Volume Name: dist Type: Distribute Status: Started Number of Bricks: 2 Transport-type: tcp,rdma Bricks: Brick1: 10.1.12.134:/mnt/dist Brick2: 10.1.12.135:/mnt/dist Options Reconfigured: features.limit-usage: /dir:10MB features.quota: on [root@centos-qa-client-3 glusterfs]# requesting you to please inform us about the commit id to which your workspace is synced. Thanks, Saurabh From: gluster-users-boun...@gluster.org [gluster-users-boun...@gluster.org] on behalf of gluster-users-requ...@gluster.org [gluster-users-requ...@gluster.org] Sent: Friday, July 08, 2011 12:30 AM To: gluster-users@gluster.org Subject: Gluster-users Digest, Vol 39, Issue 13 Send Gluster-users mailing list submissions to gluster-users@gluster.org To subscribe or unsubscribe via the World Wide Web, visit http://gluster.org/cgi-bin/mailman/listinfo/gluster-users or, via email, send a message with subject or body 'help' to gluster-users-requ...@gluster.org You can reach the person managing the list at gluster-users-ow...@gluster.org When replying, please edit your Subject line so it is more specific than Re: Contents of Gluster-users digest... Today's Topics: 1. Re: Issue with Gluster Quota (Brian Smith) 2. Re: Issues with geo-rep (Carl Chenet) -- Message: 1 Date: Thu, 07 Jul 2011 13:10:06 -0400 From: Brian Smith b...@usf.edu Subject: Re: [Gluster-users] Issue with Gluster Quota To: gluster-users@gluster.org Message-ID: 4e15e86e.6030...@usf.edu Content-Type: text/plain; charset=ISO-8859-1 Sorry about that. I re-populated with an 82MB dump from dd: [root@gluster1 ~]# gluster volume quota home list path limit_set size -- /brs 10485760 81965056 [root@gluster1 ~]# getfattr -m . -d -e hex /glusterfs/home/brs getfattr: Removing leading '/' from absolute path names # file: glusterfs/home/brs security.selinux
[Gluster-users] Questions on Replication and Design
Hi, all, We're looking to replace our DRBD/Ext4/NFS file storage configuration, using RHEL cluster w/ GlusterFS 3.2.2 (or greater, depending on the timeline). Currently, our configuration includes 2 four node clusters at different sites with 1. 8 @ 1.5TB LVM LVs on top of an HP-P2000 storage array 2. Each LV is mirrored to an identical LV on a cluster at a remote site using DRBD 3. Each LV/DRBD is an Ext4 volume and an NFS mount-point 4. DRBD is started as primary in one site and secondary in the remote site and can be switched easily in the event of a failure. 5. RHEL cluster w/ some patches runs DRBD, floating IP, ext4, NFS as a service for each of the 8 mounts. We went this route because it was the only way to get POSIX-ACL support and a working Quota implementation w/ replication of a large-ish volume without spending incredible amounts of money. With GlusterFS 3.2.2, these are both supported features. My proposed layout for the new configuration would look like so: 1. 8 @ 1.5TB LVM LVs on top of an HP-P2000 storage array 2. Each LV is an Ext4 FS 3. RHEL cluster runs a glusterd instance, floating IP and, ext4 mount for each of the 8 LVs. 4. Each of the 8 LVs is configured with a replicated pair in our remote site while they distribute across the local site. For instance: site1-node1: site2-node1: gluster1:/glusterfs -- gluster9:/glusterfs gluster2:/glusterfs -- gluster10:/glusterfs site1-node2: gluster3:/glusterfs -- gluster11:/glusterfs gluster4:/glusterfs -- gluster12:/glusterfs site1-node3: gluster5:/glusterfs -- gluster13:/glusterfs gluster6:/glusterfs -- gluster14:/glusterfs site1-node4: gluster7:/glusterfs -- gluster15:/glusterfs gluster8:/glusterfs -- gluster16:/glusterfs Distributed -- | | | | client client client client We will still use RHEL cluster to facilitate HA and failover of the glusterd/ip/fs instances on each cluster site. What say the experts about this approach and what caveats/issues should I be looking out for? I'll be building a test environment, but was wondering, before I start, whether this is a supportable configuration in the event we decide to get support, etc. Many thanks in advance! -Brian -- Brian Smith Senior Systems Administrator IT Research Computing, University of South Florida 4202 E. Fowler Ave. ENB308 Office Phone: +1 813 974-1467 Organization URL: http://rc.usf.edu ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Issue with Gluster Quota
According to the logs, the last commit was: commit 5c20eb3bbf870edadd22d06babb5d38dad222533 Author: shishir gowda shishi...@gluster.com Date: Tue Jul 5 03:41:51 2011 + [root@gluster1 glusterfs-3.2git]# gluster volume quota home list path limit_set size -- /brs 10485760 81965056 [root@gluster1 glusterfs-3.2git]# gluster volume info Volume Name: home Type: Distribute Status: Started Number of Bricks: 2 Transport-type: tcp,rdma Bricks: Brick1: gluster1:/glusterfs/home Brick2: gluster2:/glusterfs/home Options Reconfigured: features.limit-usage: /brs:10MB features.quota: on -Brian Brian Smith Senior Systems Administrator IT Research Computing, University of South Florida 4202 E. Fowler Ave. ENB308 Office Phone: +1 813 974-1467 Organization URL: http://rc.usf.edu On 07/11/2011 08:29 AM, Saurabh Jain wrote: Hello Brian, I synced my gluster repository back to July 5th and tried quota on a certain dir of a distribute and the quota was implemeted properly on that, here are the logs, [root@centos-qa-client-3 glusterfs]# /root/july6git/inst/sbin/gluster volume quota dist list path limit_set size -- /dir 10485760 10485760 [root@centos-qa-client-3 glusterfs]# /root/july6git/inst/sbin/gluster volume info Volume Name: dist Type: Distribute Status: Started Number of Bricks: 2 Transport-type: tcp,rdma Bricks: Brick1: 10.1.12.134:/mnt/dist Brick2: 10.1.12.135:/mnt/dist Options Reconfigured: features.limit-usage: /dir:10MB features.quota: on [root@centos-qa-client-3 glusterfs]# requesting you to please inform us about the commit id to which your workspace is synced. Thanks, Saurabh From: gluster-users-boun...@gluster.org [gluster-users-boun...@gluster.org] on behalf of gluster-users-requ...@gluster.org [gluster-users-requ...@gluster.org] Sent: Friday, July 08, 2011 12:30 AM To: gluster-users@gluster.org Subject: Gluster-users Digest, Vol 39, Issue 13 Send Gluster-users mailing list submissions to gluster-users@gluster.org To subscribe or unsubscribe via the World Wide Web, visit http://gluster.org/cgi-bin/mailman/listinfo/gluster-users or, via email, send a message with subject or body 'help' to gluster-users-requ...@gluster.org You can reach the person managing the list at gluster-users-ow...@gluster.org When replying, please edit your Subject line so it is more specific than Re: Contents of Gluster-users digest... Today's Topics: 1. Re: Issue with Gluster Quota (Brian Smith) 2. Re: Issues with geo-rep (Carl Chenet) -- Message: 1 Date: Thu, 07 Jul 2011 13:10:06 -0400 From: Brian Smith b...@usf.edu Subject: Re: [Gluster-users] Issue with Gluster Quota To: gluster-users@gluster.org Message-ID: 4e15e86e.6030...@usf.edu Content-Type: text/plain; charset=ISO-8859-1 Sorry about that. I re-populated with an 82MB dump from dd: [root@gluster1 ~]# gluster volume quota home list path limit_set size -- /brs 10485760 81965056 [root@gluster1 ~]# getfattr -m . -d -e hex /glusterfs/home/brs getfattr: Removing leading '/' from absolute path names # file: glusterfs/home/brs security.selinux=0x726f6f743a6f626a6563745f723a66696c655f743a733000 trusted.gfid=0x1bbcb9a08bf64406b440f3bb3ad334ed trusted.glusterfs.dht=0x00017fff trusted.glusterfs.quota.----0001.contri=0x6000 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.size=0x6000 [root@gluster2 ~]# getfattr -m . -d -e hex /glusterfs/home/brs getfattr: Removing leading '/' from absolute path names # file: glusterfs/home/brs security.selinux=0x726f6f743a6f626a6563745f723a66696c655f743a733000 trusted.gfid=0x1bbcb9a08bf64406b440f3bb3ad334ed trusted.glusterfs.dht=0x00017ffe trusted.glusterfs.quota.----0001.contri=0x04e25000 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.size=0x04e25000 Brian Smith Senior Systems Administrator IT Research Computing, University of South Florida 4202 E. Fowler Ave. ENB308 Office Phone: +1 813 974-1467 Organization URL: http://rc.usf.edu On 07/07/2011 04:50 AM, Mohammed Junaid wrote: [root@gluster1 ~]# getfattr -m . -d -e hex /glusterfs/home/brs getfattr: Removing leading '/' from absolute path names # file: glusterfs/home/brs security.selinux
Re: [Gluster-users] Issue with Gluster Quota
Sorry about that. I re-populated with an 82MB dump from dd: [root@gluster1 ~]# gluster volume quota home list path limit_set size -- /brs 10485760 81965056 [root@gluster1 ~]# getfattr -m . -d -e hex /glusterfs/home/brs getfattr: Removing leading '/' from absolute path names # file: glusterfs/home/brs security.selinux=0x726f6f743a6f626a6563745f723a66696c655f743a733000 trusted.gfid=0x1bbcb9a08bf64406b440f3bb3ad334ed trusted.glusterfs.dht=0x00017fff trusted.glusterfs.quota.----0001.contri=0x6000 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.size=0x6000 [root@gluster2 ~]# getfattr -m . -d -e hex /glusterfs/home/brs getfattr: Removing leading '/' from absolute path names # file: glusterfs/home/brs security.selinux=0x726f6f743a6f626a6563745f723a66696c655f743a733000 trusted.gfid=0x1bbcb9a08bf64406b440f3bb3ad334ed trusted.glusterfs.dht=0x00017ffe trusted.glusterfs.quota.----0001.contri=0x04e25000 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.size=0x04e25000 Brian Smith Senior Systems Administrator IT Research Computing, University of South Florida 4202 E. Fowler Ave. ENB308 Office Phone: +1 813 974-1467 Organization URL: http://rc.usf.edu On 07/07/2011 04:50 AM, Mohammed Junaid wrote: [root@gluster1 ~]# getfattr -m . -d -e hex /glusterfs/home/brs getfattr: Removing leading '/' from absolute path names # file: glusterfs/home/brs security.selinux=0x726f6f743a6f626a6563745f723a66696c655f743a733000 trusted.gfid=0x1bbcb9a08bf64406b440f3bb3ad334ed trusted.glusterfs.dht=0x00017fff trusted.glusterfs.quota.----0001.contri=0x6000 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.size=0x6000 and [root@gluster2 ~]# getfattr -m . -d -e hex /glusterfs/home/brs getfattr: Removing leading '/' from absolute path names # file: glusterfs/home/brs security.selinux=0x726f6f743a6f626a6563745f723a66696c655f743a733000 trusted.gfid=0x1bbcb9a08bf64406b440f3bb3ad334ed trusted.glusterfs.dht=0x00017ffe trusted.glusterfs.quota.----0001.contri=0x2000 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.size=0x2000 trusted.glusterfs.quota.size=0x6000 trusted.glusterfs.quota.size=0x2000 So, quota adds these values to calculate the size of the directory. They are in hex so when I add them the value comes upto 32kb. So I suspect that you have deleted some data from your volume. These values wont be helpful now. Can you please re-run the same test and report the values when such a problem occurs again. ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Issue with Gluster Quota
I'm running into the same issue. But be aware, the documentation says that you should use an absolute path _relative_ to the volume. So if your volume is, for example: volume name: home and a top-level directory in home is test, you would simply do volume quota home limit-usage /test 10MB So, in your case: volume quota crlgfs1 limit-usage /dheeraj 10MB At least, that's how I read the documentation. This is why you don't even see any usage numbers on the 'size' column. I get the usage numbers in my configuration, but the quota is not enforced. this is my config: gluster volume info Volume Name: home Type: Distribute Status: Started Number of Bricks: 2 Transport-type: tcp,rdma Bricks: Brick1: gluster1:/glusterfs/home Brick2: gluster2:/glusterfs/home Options Reconfigured: features.limit-usage: /brs:10MB features.quota: on Here's my quota information: gluster volume quota home list path limit_set size -- /brs 10485760 16424960 FYI, I am running a check-out of the latest git release-3.2 branch, so that I can test POSIX ACLs (which seem to work nicely, BTW). -Brian Brian Smith Senior Systems Administrator IT Research Computing, University of South Florida 4202 E. Fowler Ave. ENB308 Office Phone: +1 813 974-1467 Organization URL: http://rc.usf.edu On 06/28/2011 07:30 AM, Dheeraj KV wrote: Hi I have installed GlusterFS-3.2 with two servers and two bricks. I am able to mount it on a client node also. I need to enable quota on the directories of glusterfs. *# gluster volume info* *Volume Name: crlgfs1* *Type: Distribute* *Status: Started* *Number of Bricks: 2* *Transport-type: rdma* *Bricks:* *Brick1: glus01-ib:/data/gluster/brick-1* *Brick2: glus02-ib:/data/gluster/brick-2* *Options Reconfigured:* *diagnostics.latency-measurement: on* *diagnostics.count-fop-hits: on* *features.quota: on* *features.limit-usage: /data/gluster/brick-2/dheeraj:10MB,/data/gluster/brick-1/dheeraj:10MB* Please check the above command to check the configuration. When I give the below given command from the glusterfs server, it will me the following output. ** *# gluster volume quota crlgfs1 list /data/gluster/brick-1/dheeraj* *path limit_set size* *--* */data/gluster/brick-1/dheeraj 10MB* I have created quota from one glusterfs server only The result is that I m able to create files of size more than 10 MB . Even 8 GB file I created even though the quota is for 10 MB. Please put some light on it. Any help is much appreciated. Thanks Regards Dheeraj K V ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Gluster-users Digest, Vol 27, Issue 1
Each of the two bricks has just one XFS file system w/ inode64 enabled on a 10TB LVM LV. Each of the volumes is less than 40% full and inode counts look reasonable. I'm working to get a test environment going so I can reproduce this off-production and add the trace translator. I've sent in a bunch of trace data and opened a bug: http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=1040 -Brian -- Brian Smith Senior Systems Administrator IT Research Computing, University of South Florida 4202 E. Fowler Ave. ENB204 Office Phone: +1 813 974-1467 Organization URL: http://rc.usf.edu Date: Wed, 30 Jun 2010 15:33:24 -0400 From: Brian Smith b...@usf.edu Subject: Re: [Gluster-users] Revisit: FORTRAN Codes and File I/O To: Harshavardhana har...@gluster.com Cc: Gluster General Discussion List gluster-users@gluster.org Message-ID: 1277926404.2687.60.ca...@localhost.localdomain Content-Type: text/plain; charset=UTF-8 Spoke too soon. Same problem occurs minus all performance translators. Debug logs on the server show [2010-06-30 15:30:54] D [server-protocol.c:2104:server_create_cbk] server-tcp: create(/b/brs/Si/CHGCAR) inode (ptr=0x2aaab00e05b0, ino=2159011921, gen=5488651098262601749) found conflict (ptr=0x2aaab40cca00, ino=2159011921, gen=5488651098262601749) [2010-06-30 15:30:54] D [server-resolve.c:386:resolve_entry_simple] server-tcp: inode (pointer: 0x2aaab40cca00 ino:2159011921) found for path (/b/brs/Si/CHGCAR) while type is RESOLVE_NOT [2010-06-30 15:30:54] D [server-protocol.c:2132:server_create_cbk] server-tcp: 72: CREATE (null) (0) == -1 (File exists) -Brian -- Brian Smith Senior Systems Administrator IT Research Computing, University of South Florida 4202 E. Fowler Ave. ENB204 Office Phone: +1 813 974-1467 Organization URL: http://rc.usf.edu On Wed, 2010-06-30 at 13:06 -0400, Brian Smith wrote: I received these in my debug output during a run that failed: [2010-06-30 12:34:25] D [read-ahead.c:468:ra_readv] readahead: unexpected offset (8192 != 1062) resetting [2010-06-30 12:34:25] D [read-ahead.c:468:ra_readv] readahead: unexpected offset (8192 != 1062) resetting [2010-06-30 12:34:25] D [read-ahead.c:468:ra_readv] readahead: unexpected offset (8192 != 1062) resetting [2010-06-30 12:34:25] D [read-ahead.c:468:ra_readv] readahead: unexpected offset (8192 != 1062) resetting I disabled the read-ahead translator as well as the three other performance translators commented out in my vol file (I'm on GigE; the docs say I can still reach link max anyway) and my processes appear to be running smoothly. I'll go ahead and submit the bug report with tracing enabled as well. -Brian Date: Wed, 30 Jun 2010 21:17:59 -0400 From: Jeff Darcy jda...@redhat.com Subject: Re: [Gluster-users] Revisit: FORTRAN Codes and File I/O To: gluster-users@gluster.org Message-ID: 4c2becc7.5010...@redhat.com Content-Type: text/plain; charset=iso-8859-1; Format=flowed On 06/30/2010 03:33 PM, Brian Smith wrote: Spoke too soon. Same problem occurs minus all performance translators. Debug logs on the server show [2010-06-30 15:30:54] D [server-protocol.c:2104:server_create_cbk] server-tcp: create(/b/brs/Si/CHGCAR) inode (ptr=0x2aaab00e05b0, ino=2159011921, gen=5488651098262601749) found conflict (ptr=0x2aaab40cca00, ino=2159011921, gen=5488651098262601749) [2010-06-30 15:30:54] D [server-resolve.c:386:resolve_entry_simple] server-tcp: inode (pointer: 0x2aaab40cca00 ino:2159011921) found for path (/b/brs/Si/CHGCAR) while type is RESOLVE_NOT [2010-06-30 15:30:54] D [server-protocol.c:2132:server_create_cbk] server-tcp: 72: CREATE (null) (0) == -1 (File exists) The first line almost looks like a create attempt for a file that already exists at the server. The second and third lines look like *yet another* create attempt, failing this time before the request is even passed to the next translator. This might be a good time to drag out the debug/trace translator, and sit it on top of brick1 to watch the create calls. That will help nail down the exact sequence of events as the server sees them, so we don't go looking in the wrong places. It might even be useful to do the same on the client side, but perhaps not yet. Instructions are here: http://www.gluster.com/community/documentation/index.php/Translators/debug/trace In the mean time, to further identity which code paths are most likely to be relevant, it would be helpful to know a couple more things. (1) Is each storage/posix volume using just one local filesystem, or is it possible that the underlying directory tree spans more than one? This could lead to inode-number duplication, which requires extra handling. (2) Are either of the server-side volumes close to being full? This could result in creating an extra linkfile on the subvolume/server where we'd normally create the file, pointing to where we really
Re: [Gluster-users] Revisit: FORTRAN Codes and File I/O
Spoke too soon. Same problem occurs minus all performance translators. Debug logs on the server show [2010-06-30 15:30:54] D [server-protocol.c:2104:server_create_cbk] server-tcp: create(/b/brs/Si/CHGCAR) inode (ptr=0x2aaab00e05b0, ino=2159011921, gen=5488651098262601749) found conflict (ptr=0x2aaab40cca00, ino=2159011921, gen=5488651098262601749) [2010-06-30 15:30:54] D [server-resolve.c:386:resolve_entry_simple] server-tcp: inode (pointer: 0x2aaab40cca00 ino:2159011921) found for path (/b/brs/Si/CHGCAR) while type is RESOLVE_NOT [2010-06-30 15:30:54] D [server-protocol.c:2132:server_create_cbk] server-tcp: 72: CREATE (null) (0) == -1 (File exists) -Brian -- Brian Smith Senior Systems Administrator IT Research Computing, University of South Florida 4202 E. Fowler Ave. ENB204 Office Phone: +1 813 974-1467 Organization URL: http://rc.usf.edu On Wed, 2010-06-30 at 13:06 -0400, Brian Smith wrote: I received these in my debug output during a run that failed: [2010-06-30 12:34:25] D [read-ahead.c:468:ra_readv] readahead: unexpected offset (8192 != 1062) resetting [2010-06-30 12:34:25] D [read-ahead.c:468:ra_readv] readahead: unexpected offset (8192 != 1062) resetting [2010-06-30 12:34:25] D [read-ahead.c:468:ra_readv] readahead: unexpected offset (8192 != 1062) resetting [2010-06-30 12:34:25] D [read-ahead.c:468:ra_readv] readahead: unexpected offset (8192 != 1062) resetting I disabled the read-ahead translator as well as the three other performance translators commented out in my vol file (I'm on GigE; the docs say I can still reach link max anyway) and my processes appear to be running smoothly. I'll go ahead and submit the bug report with tracing enabled as well. -Brian ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
[Gluster-users] Revisit: FORTRAN Codes and File I/O
It's obviously been a while since I brought this issue up, but it has cropped up again for us. We're now on 3.0.3 and I've included my glusterfs*.vol files below. We end up with file i/o errors like the ones below: forrtl: File exists forrtl: severe (10): cannot overwrite existing file, unit 18, file /work/b/brs/vdWSi/CHGCAR Even if the file existed, it shouldn't really be a problem. Other file systems work just fine. I'll get some more verbose logging going and share my output. glusterfsd.vol is the same in the referenced e-mails below. Thanks in advance, -Brian ## file auto generated by /usr/bin/glusterfs-volgen (mount.vol) # Cmd line: # $ /usr/bin/glusterfs-volgen --name work pvfs0:/pvfs/glusterfs pvfs1:/pvfs/glusterfs # TRANSPORT-TYPE tcp volume pvfs0-1 type protocol/client option transport-type tcp option remote-host pvfs0 option remote-port 6996 option transport.socket.nodelay on option remote-subvolume brick1 end-volume volume pvfs1-1 type protocol/client option transport-type tcp option remote-host pvfs1 option remote-port 6996 option transport.socket.nodelay on option remote-subvolume brick1 end-volume volume distribute type cluster/distribute subvolumes pvfs0-1 pvfs1-1 end-volume volume writebehind type performance/write-behind option cache-size 4MB subvolumes distribute end-volume volume readahead type performance/read-ahead option page-count 4 subvolumes writebehind end-volume #volume iocache #type performance/io-cache #option cache-size 1GB #option cache-timeout 1 #subvolumes readahead #end-volume #volume quickread #type performance/quick-read #option cache-timeout 1 #option max-file-size 64kB #subvolumes iocache #end-volume #volume statprefetch #type performance/stat-prefetch #subvolumes quickread #end-volume == * replies inline * On Fri, Feb 12, 2010 at 9:52 PM, Brian Smith brs at usf.edu wrote: Hi, Harshavardhana, Thanks for the reply. My volume files are below. Unfortunately, there is no helpful information in the logs as it seems the log verbosity is set too low. I'll update that and hopefully get some more information. I'm using distribute and not NUFA. Perfect. ## file auto generated by /usr/bin/glusterfs-volgen (export.vol) # Cmd line: # $ /usr/bin/glusterfs-volgen --name work pvfs0:/pvfs/glusterfs pvfs1:/pvfs/glusterfs volume posix1 type storage/posix option directory /pvfs/glusterfs end-volume volume locks1 type features/locks subvolumes posix1 option mandatory-locks on end-volume volume brick1 type performance/io-threads option thread-count 8 subvolumes locks1 end-volume volume server-tcp type protocol/server option transport-type tcp option auth.addr.brick1.allow * option transport.socket.listen-port 6996 option transport.socket.nodelay on subvolumes brick1 end-volume ## file auto generated by /usr/bin/glusterfs-volgen (mount.vol) # Cmd line: # $ /usr/bin/glusterfs-volgen --name work pvfs0:/pvfs/glusterfs pvfs1:/pvfs/glusterfs # TRANSPORT-TYPE tcp volume pvfs0-1 type protocol/client option transport-type tcp option remote-host pvfs0 option transport.socket.nodelay on option transport.remote-port 6996 option remote-subvolume brick1 end-volume volume pvfs1-1 type protocol/client option transport-type tcp option remote-host pvfs1 option transport.socket.nodelay on option transport.remote-port 6996 option remote-subvolume brick1 end-volume volume distribute type cluster/distribute subvolumes pvfs0-1 pvfs1-1 end-volume volume writebehind type performance/write-behind option cache-size 4MB subvolumes distribute end-volume volume readahead type performance/read-ahead option page-count 4 subvolumes writebehind end-volume volume iocache type performance/io-cache option cache-size 1GB option cache-timeout 1 subvolumes readahead end-volume volume quickread type performance/quick-read option cache-timeout 1 option max-file-size 64kB subvolumes iocache end-volume volume statprefetch type performance/stat-prefetch subvolumes quickread end-volume Could you remove the above quickread, stat-prefetch, io-cache and test again. Also could you state the version you are using?. -- Brian Smith Senior Systems Administrator IT Research Computing, University of South Florida 4202 E. Fowler Ave. ENB204 Office Phone: +1 813 974-1467 Organization URL: http://rc.usf.edu ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] FORTRAN Codes and File I/O
Hi, Harshavardhana, Thanks for the reply. My volume files are below. Unfortunately, there is no helpful information in the logs as it seems the log verbosity is set too low. I'll update that and hopefully get some more information. I'm using distribute and not NUFA. ## file auto generated by /usr/bin/glusterfs-volgen (export.vol) # Cmd line: # $ /usr/bin/glusterfs-volgen --name work pvfs0:/pvfs/glusterfs pvfs1:/pvfs/glusterfs volume posix1 type storage/posix option directory /pvfs/glusterfs end-volume volume locks1 type features/locks subvolumes posix1 option mandatory-locks on end-volume volume brick1 type performance/io-threads option thread-count 8 subvolumes locks1 end-volume volume server-tcp type protocol/server option transport-type tcp option auth.addr.brick1.allow * option transport.socket.listen-port 6996 option transport.socket.nodelay on subvolumes brick1 end-volume ## file auto generated by /usr/bin/glusterfs-volgen (mount.vol) # Cmd line: # $ /usr/bin/glusterfs-volgen --name work pvfs0:/pvfs/glusterfs pvfs1:/pvfs/glusterfs # TRANSPORT-TYPE tcp volume pvfs0-1 type protocol/client option transport-type tcp option remote-host pvfs0 option transport.socket.nodelay on option transport.remote-port 6996 option remote-subvolume brick1 end-volume volume pvfs1-1 type protocol/client option transport-type tcp option remote-host pvfs1 option transport.socket.nodelay on option transport.remote-port 6996 option remote-subvolume brick1 end-volume volume distribute type cluster/distribute subvolumes pvfs0-1 pvfs1-1 end-volume volume writebehind type performance/write-behind option cache-size 4MB subvolumes distribute end-volume volume readahead type performance/read-ahead option page-count 4 subvolumes writebehind end-volume volume iocache type performance/io-cache option cache-size 1GB option cache-timeout 1 subvolumes readahead end-volume volume quickread type performance/quick-read option cache-timeout 1 option max-file-size 64kB subvolumes iocache end-volume volume statprefetch type performance/stat-prefetch subvolumes quickread end-volume -- Brian Smith Senior Systems Administrator IT Research Computing, University of South Florida 4202 E. Fowler Ave. ENB204 Office Phone: +1 813 974-1467 Organization URL: http://rc.usf.edu On Fri, 2010-02-12 at 14:47 +0530, Harshavardhana wrote: Hi Brian, Can you share your volume files and log files? are you using NUFA translator?. Running vasp application codes on nufa based configuration we have seen certain issues. -- Harshavardhana On Thu, Feb 11, 2010 at 11:29 PM, Brian Smith b...@usf.edu wrote: H i all, I'm running Gluster 3.0.0 on top of XFS and while running a FORTRAN code that works perfectly well on any other file system, I get runtime errors when trying to open files -- along the lines of: At line 386 of file main.f (unit = 18, file = '') Fortran runtime error: File 'CHGCAR' already exists Are there known issues with FORTRAN I/O and Gluster? Is this some sort of caching artifact? Its not a consistent problem as it only seems to happen when running jobs within my scheduling environment (I use SGE). Let me know if you need more info. Thanks in advance, -Brian -- Brian Smith Senior Systems Administrator IT Research Computing, University of South Florida 4202 E. Fowler Ave. ENB204 Office Phone: +1 813 974-1467 Organization URL: http://rc.usf.edu On Thu, 2010-02-11 at 15:13 +0100, Eros Candelaresi wrote: Hi, for my small webhosting (3 servers, more to come hopefully) I am investigating cluster filesystems. I have seen a few now and I love the flexibility that GlusterFS brings. Still I cannot see a way to adapt it to suit my needs. I have the following hardware: - Server #1 with 160GB S-ATA - Server #2 with 2x 400GB S-ATA - Server #3 with 2x 1,5TB S-ATA I am hoping to find a filesystem that fulfills the following requirements: 1. POSIX compliant (Apache, Postfix, etc. will use it) - GlusterFS has it 2. combine the harddisks of all servers into one single filesystem - DHT/unify seem to do the job 3. redundancy: have a copy of each single file on at least 2 machines such that a single host may fail without people noticing - looks like this may be achieved by having AFR below DHT/Unify 4. after
[Gluster-users] open() issues on 3.0.0
Hi,
I'm new to Gluster, so please forgive any ignorance on my part. I've
tried reading over everything I could find to resolve this myself.
I'm having an issue building executables on our gluster volume (there
are some other similar issues as well, but we'll deal with this one
first). I have the presta infiniband utilities extracted into a
directory in my glusterfs volume and I'm attempting to build them. One
of the source files, util.c (coincidentally, the first one referenced in
the make file) fails to build. I get an error message:
Catastrophic error: could not open source file util.c
Some further investigating reveals the command that causes the issue:
icc -I/opt/priv/openmpi-1.4.1/intel-10.1.018/include -pthread
-L/opt/priv/mx/lib -L/opt/priv/openmpi-1.4.1/intel-10.1.018/lib -lmpi
-lopen-rte -lopen-pal -lmyriexpress -libverbs -lpsm_infinipath -lnuma
-ldl -Wl,--export-dynamic -lnsl -lutil -c util.c
Catastrophic error: could not open source file util.c
compilation aborted for util.c (code 4)
And further, we see an strace of the command:
...
2717 6601 brk(0x957b000)= 0x957b000
2718 6601 open(util.c, O_RDONLY) = 3
2719 6601 stat64(util.c, {st_mode=S_IFREG|0600, st_size=14370, ...})
= 0
2720 6601 close(3) = 0
2721 6601 fstat64(2, {st_mode=S_IFCHR|0620, st_rdev=makedev(136,
1), ...}) = 0
2722 6601 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|
MAP_ANONYMOUS, -1, 0x1000) = 0xf0bce000
2723 6601 write(2, Catastrophic error: could not open source file
\util.c\\n, 56) = 56
2724 6601 write(2, \n, 1) = 1
2725 6601 exit_group(4)
...
We can also see that the file is indeed present:
$ ls -l util.c
-rw--- 1 brs users 14370 Apr 8 2002 util.c
$ stat util.c
File: `util.c'
Size: 14370 Blocks: 32 IO Block: 4096 regular file
Device: 18h/24d Inode: 4295774314 Links: 1
Access: (0600/-rw---) Uid: (1229806/ brs) Gid: (10001/
users)
Access: 2010-01-17 00:27:43.0 -0500
Modify: 2002-04-08 21:13:57.0 -0400
Change: 2010-01-17 00:27:43.0 -0500
If I extract the tar-ball in /tmp, a local ext3 fs, the compile line
above works correctly.
diff /work/b/brs/presta1.2/util.c /tmp/presta1.2/util.c
also appears clean. Any ideas what is happening here? Is there an
issue with mmap2() and glusterfs?
Many thanks in advance,
-Brian
--
Brian Smith
Senior Systems Administrator
IT Research Computing, University of South Florida
4202 E. Fowler Ave. ENB308
Office Phone: +1 813 974-1467
Organization URL: http://rc.usf.edu
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] open() issues on 3.0.0
Good find! That seems to be exactly the issue. My inode numbers for
the files in that directory are literally just 100k away from the 32-bit
cut-off. I've rm'ed and re-extraced to get lesser inode values and
voila! I guess intel must have thought that 2^32 inodes outta be enough
for everyone.
Many thanks,
-Brian
--
Brian Smith
Senior Systems Administrator
IT Research Computing, University of South Florida
4202 E. Fowler Ave. ENB308
Office Phone: +1 813 974-1467
Organization URL: http://rc.usf.edu
On Sun, 2010-01-17 at 13:12 -0500, Aaron Knister wrote:
I ran into a similar problem when compiling code on a gluster mount
with the intel compiler. The problem was intermittent and after
digging into it further it appeared to only happy when compiling from
source files that had very high inode numbers. There's a command run
by icc that's a 32 bit executable and my guess is that somewhere
internally the extremely large inode number wasn't able to be handled
and caused the compilation to fail. The fix was to upgrade the
compiler. I can verify that version 11.0.064 works without a hitch (at
least in this regard).
More Detail:
I can reproduce this error with version 10.0.026 of the intel
compiler.
I have file test.c on a glusterfs with inode num 21479794854 (greater
than 32 bits)
21479794854 -rw-r--r-- 1 aaron users 26 Jan 17 12:58 test.c
Running icc fails:
...
$ icc test.c
Catastrophic error: could not open source file test.c
compilation aborted for test.c (code 4)
...
However if I copy the same file to a directory on the same fs that has
a 32 bit inode number (le 4294967295) (hoping the new file will also
have a 32 bit inode number):
11889296 -rw-r--r-- 1 aaron users 26 Jan 17 13:02 test.c
Compilation is a success:
...
$ !icc
icc test.c
$ echo $?
0
...
If you run icc with the -v option you'll see that a program called
mcpcom gets run and produces the error. At least with version 10.0.026
it was a 32 bit program:
$ file /opt/intel/cce/10.0.026/bin/mcpcom
/opt/intel/cce/10.0.026/bin/mcpcom: ELF 32-bit LSB executable, Intel
80386, version 1 (SYSV), for GNU/Linux 2.2.5, dynamically linked (uses
shared libs), for GNU/Linux 2.2.5, not stripped
Anyway I hope that helps.
-Aaron
On Jan 17, 2010, at 11:13 AM, Brian Smith wrote:
Hi,
I'm new to Gluster, so please forgive any ignorance on my part.
I've
tried reading over everything I could find to resolve this myself.
I'm having an issue building executables on our gluster volume
(there
are some other similar issues as well, but we'll deal with this one
first). I have the presta infiniband utilities extracted into a
directory in my glusterfs volume and I'm attempting to build them.
One
of the source files, util.c (coincidentally, the first one
referenced in
the make file) fails to build. I get an error message:
Catastrophic error: could not open source file util.c
Some further investigating reveals the command that causes the
issue:
icc -I/opt/priv/openmpi-1.4.1/intel-10.1.018/include -pthread
-L/opt/priv/mx/lib -L/opt/priv/openmpi-1.4.1/intel-10.1.018/lib
-lmpi
-lopen-rte -lopen-pal -lmyriexpress -libverbs -lpsm_infinipath
-lnuma
-ldl -Wl,--export-dynamic -lnsl -lutil -c util.c
Catastrophic error: could not open source file util.c
compilation aborted for util.c (code 4)
And further, we see an strace of the command:
...
2717 6601 brk(0x957b000)= 0x957b000
2718 6601 open(util.c, O_RDONLY) = 3
2719 6601 stat64(util.c, {st_mode=S_IFREG|0600,
st_size=14370, ...})
= 0
2720 6601 close(3) = 0
2721 6601 fstat64(2, {st_mode=S_IFCHR|0620, st_rdev=makedev(136,
1), ...}) = 0
2722 6601 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|
MAP_ANONYMOUS, -1, 0x1000) = 0xf0bce000
2723 6601 write(2, Catastrophic error: could not open source file
\util.c\\n, 56) = 56
2724 6601 write(2, \n, 1) = 1
2725 6601 exit_group(4)
...
We can also see that the file is indeed present:
$ ls -l util.c
-rw--- 1 brs users 14370 Apr 8 2002 util.c
$ stat util.c
File: `util.c'
Size: 14370 Blocks: 32 IO Block: 4096 regular file
Device: 18h/24d Inode: 4295774314 Links: 1
Access: (0600/-rw---) Uid: (1229806/ brs) Gid: (10001/
users)
Access: 2010-01-17 00:27:43.0 -0500
Modify: 2002-04-08 21:13:57.0 -0400
Change: 2010-01-17 00:27:43.0 -0500
If I extract the tar-ball in /tmp, a local ext3 fs, the compile line
above works correctly.
diff /work/b/brs/presta1.2/util.c /tmp/presta1.2/util.c
also appears clean. Any ideas what is happening here? Is there an
issue with mmap2() and glusterfs?
Many thanks in advance,
-Brian
--
Brian Smith
Senior Systems Administrator
IT Research Computing
