Re: [Gluster-users] Dose Gluster 3.1 support authorisation control and how to do
Hi, HU Thank for your help. I tried to use your example(1 server ,1 Client) to test authentication function, it's work. But I tried to test it in replication mode (multi-node),FUSE mounting work, but NFS didn't. Any node can mount volume via NFS. And Following is my config. 26: volume gluster-new-volume-server 27: type protocol/server 28: option transport-type tcp 29: option auth.addr./mnt/gluster1.allow 10.18.14.240,10.18.14.248,10.18.14.241,10.18.14.242,10.18.14.243 30: subvolumes /mnt/gluster1 31: end-volume After starting volume, log showed below: +--+ [2011-01-11 01:07:54.188695] E [authenticate.c:235:gf_authenticate] auth: no authentication module is interested in accepting remote-client (null) [2011-01-11 01:07:54.188716] E [server-handshake.c:545:server_setvolume] gluster-new-volume-server: Cannot authenticate client from 127.0.0.1:1017 [2011-01-11 01:07:55.264728] I [server-handshake.c:535:server_setvolume] gluster-new-volume-server: accepted client from 10.18.14.241:995 [2011-01-11 01:07:55.267990] I [server-handshake.c:535:server_setvolume] gluster-new-volume-server: accepted client from 10.18.14.242:1012 [2011-01-11 01:07:55.272025] I [server-handshake.c:535:server_setvolume] gluster-new-volume-server: accepted client from 10.18.14.243:996 Do you know is it necessary to set 127.0.0.1 to allow list? And it can't use host real ip (10.18.14.240) ? But even if I used 127.0.0.1 to replace 10.18.14.240, NFS authentication control still not work. -Original message- From:HU Zhong hz02...@gmail.com To:wei.ch...@m2k.com.tw Cc:gluster-users gluster-users@gluster.org Date:Mon, 10 Jan 2011 11:36:00 +0800 Subject:Re: [Gluster-users] Dose Gluster 3.1 support authorisation control and how to do Hi, Cheng I think you did the configuration in the wrong place. Instead of /etc/glusterd/nfs/nfs-server.vol, you need to modify files under /etc/glusterd/vols/. As a simple example, consider a one-server-one-client system, both server and client are one machine(localhost, ip:192.168.4.112), and export directory /home/huz/share for sharing, the client wants to mount it on /home/huz/mnt. if i modify default configuration /etc/glusterd/vols/testvol/testvol.192.168.4.112.home-huz-share.vol from .. 26 volume testvol-server 27 type protocol/server 28 option transport-type tcp 29 option auth.addr./home/huz/share.allow * 30 subvolumes /home/huz/share 31 end-volume to .. 26 volume testvol-server 27 type protocol/server 28 option transport-type tcp 29 option auth.addr./home/huz/share.reject * 30 subvolumes /home/huz/share 31 end-volume the mount command will fail: $sudo mount -o mountproto=tcp -t nfs localhost:/testvol /home/huz/mnt mount.nfs: mounting localhost:/testvol failed, reason given by server: No such file or directory and the log shows that the authentication error. 11-01-10 11:09:58.203600] E [client-handshake.c:786:client_setvolume_cbk] testvol-client-0: SETVOLUME on remote-host failed: Authentication failed change reject to allow, the mount operation will be ok. you can configure you own ip rule. As for how to use ip auth and usrname/password auth, you can check the attachment. It's a documentation file under the directory doc of glusterfs src project. On Sun, 2011-01-09 at 22:31 +0800, 第二信箱 wrote: Hi, HU: Thanks for your help. I have the following environment: Gluster 3.1.1 Volume Name: gluster-volume Type: Distributed-Replicate Status: Started Number of Bricks: 2 x 2 = 4 Transport-type: tcp Bricks: Brick1: gluster1:/mnt/gluster1 Brick2: gluster2:/mnt/gluster2 Brick3: gluster3:/mnt/gluster3 Brick4: gluster4:/mnt/gluster4 I want to use authenticate module by your suggestion. The way I used below: 1. Stop Volume 2. Edit /etc/glusterd/nfs/nfs-server.vol on Brick1(Gluster1) 3. Modify and Add From volume nfs-server type nfs/server option nfs.dynamic-volumes on option rpc-auth.addr.gluster-volume.allow * option nfs3.gluster-volume.volume-id 907941d9-6950-425b- b3d5-4e43dd420d9e subvolumes gluster-volume end-volume to volume nfs-server type nfs/server option nfs.dynamic-volumes on option rpc-auth.addr.gluster-volume.allow 10.18.14.1 option auth.addr.gluster-volume.allow 10.18.14.1 option nfs3.gluster-volume.volume-id 907941d9-6950-425b-b3d5-4e43dd420d9e subvolumes gluster-volume end-volume 4.Start Volume -- But I still be able to mount volume from 10.18.14.2 by NFS. Anything I missed or be wrong? And I find A. After I started volume , nfs-server.vol was initialed to option rpc-auth.addr.gluster-volume.allow * . B. 4 nodes all have /etc/glusterd/nfs/nfs-server.vol , Should I Edit every .vol file on 4 nodes? -Original message- From:HU
Re: [Gluster-users] Read-only volumes - timeframe for features/filter?
Hello. Has anyone seen error messages like this in /var/log/glusterfs/nfs.log: tail /var/log/glusterfs/nfs.log [2011-01-10 14:22:55.859066] I [afr-self-heal-common.c:1526:afr_self_heal_completion_cbk] pfs-ro1-replicate-3: background meta-data data self-heal completed on / [2011-01-10 14:22:55.859084] I [afr-self-heal-common.c:1526:afr_self_heal_completion_cbk] pfs-ro1-replicate-5: background meta-data data self-heal completed on / [2011-01-10 14:22:57.786088] E [nfs3.c:4346:nfs3_fsstat] nfs-nfsv3: Failed to map FH to vol [2011-01-10 14:22:59.355112] E [nfs3.c:4346:nfs3_fsstat] nfs-nfsv3: Failed to map FH to vol [2011-01-10 14:22:59.415732] E [nfs3.c:675:nfs3_getattr] nfs-nfsv3: Failed to map FH to vol [2011-01-10 14:22:59.455029] E [nfs3.c:675:nfs3_getattr] nfs-nfsv3: Failed to map FH to vol [2011-01-10 14:23:01.800751] E [nfs3.c:675:nfs3_getattr] nfs-nfsv3: Failed to map FH to vol [2011-01-10 14:23:02.127233] E [nfs3.c:675:nfs3_getattr] nfs-nfsv3: Failed to map FH to vol [2011-01-10 14:23:07.834044] E [nfs3.c:4346:nfs3_fsstat] nfs-nfsv3: Failed to map FH to vol [2011-01-10 14:23:09.478852] E [nfs3.c:4346:nfs3_fsstat] nfs-nfsv3: Failed to map FH to vol [2011-01-10 14:40:18.558072] E [afr-self-heal-metadata.c:524:afr_sh_metadata_fix] pfs-ro1-replicate-5: Unable to self-heal permissions/ownership of '/' (possible split-brain). Please fix the file on all backend volumes Mount is done with this command: mount -v -t nfs -o soft,rsize=16384,wsize=16384 jc1lpfsnfsro:/pfs-ro1 /pfs1 Command line being executed is: rsync -av --progress /pfs1/online_archive/2010 . This CentOS 5.5 x86_64, GlusterFS 3.1.1. Currently configured: gluster volume info Volume Name: pfs-ro1 Type: Distributed-Replicate Status: Started Number of Bricks: 10 x 2 = 20 Transport-type: tcp Bricks: Brick1: jc1letgfs17-pfs1:/export/read-only/g01 Brick2: jc1letgfs18-pfs1:/export/read-only/g01 Brick3: jc1letgfs17-pfs1:/export/read-only/g02 Brick4: jc1letgfs18-pfs1:/export/read-only/g02 Brick5: jc1letgfs17-pfs1:/export/read-only/g03 Brick6: jc1letgfs18-pfs1:/export/read-only/g03 Brick7: jc1letgfs17-pfs1:/export/read-only/g04 Brick8: jc1letgfs18-pfs1:/export/read-only/g04 Brick9: jc1letgfs17-pfs1:/export/read-only/g05 Brick10: jc1letgfs18-pfs1:/export/read-only/g05 Brick11: jc1letgfs17-pfs1:/export/read-only/g06 Brick12: jc1letgfs18-pfs1:/export/read-only/g06 Brick13: jc1letgfs17-pfs1:/export/read-only/g07 Brick14: jc1letgfs18-pfs1:/export/read-only/g07 Brick15: jc1letgfs17-pfs1:/export/read-only/g08 Brick16: jc1letgfs18-pfs1:/export/read-only/g08 Brick17: jc1letgfs17-pfs1:/export/read-only/g09 Brick18: jc1letgfs18-pfs1:/export/read-only/g09 Brick19: jc1letgfs17-pfs1:/export/read-only/g10 Brick20: jc1letgfs18-pfs1:/export/read-only/g10 Thanks. James Burnash Unix Engineering. DISCLAIMER: This e-mail, and any attachments thereto, is intended only for use by the addressee(s) named herein and may contain legally privileged and/or confidential information. If you are not the intended recipient of this e-mail, you are hereby notified that any dissemination, distribution or copying of this e-mail, and any attachments thereto, is strictly prohibited. If you have received this in error, please immediately notify me and permanently delete the original and any copy of any e-mail and any printout thereof. E-mail transmission cannot be guaranteed to be secure or error-free. The sender therefore does not accept liability for any errors or omissions in the contents of this message which arise as a result of e-mail transmission. NOTICE REGARDING PRIVACY AND CONFIDENTIALITY Knight Capital Group may, at its discretion, monitor and review the content of all e-mail communications. http://www.knight.com ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
[Gluster-users] How can i turn on the Gluster Debug mode?
Hello, all How can I turn on the Debug mode on 3.1 When I start the volume by command line, is there any way to turn the debug message on brick side? ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
[Gluster-users] infiniband speed
I am testing Infiniband for the first time. It seems that I should be able to get a lot more speed than I am with some pretty basic tests. Maybe someone running Infiniband can confirm that what I am seeing is way out of line, and/or help diagnose? I have two systems connected using 3.1.2qa3. With 3.1.1 infiniband wouldn't even start, it gave an error about unable to intialize rdma. But with the latest version and an upgrade to OFED 1.5.2, everything starts up with no errors and I can create a volume and mount it. The underlying Infiniband seems ok, and a basic ibv_rc_pingpong test shows I can move data pretty fast: 8192 bytes in 0.23 seconds = 2858.45 Mbit/sec 1 iters in 0.23 seconds = 22.93 usec/iter So now I have two volumes created, one that uses tcp over a gig-e link and one that uses rdma. I mount them and do some file copy tests... And they are almost exactly the same? What? gluster volume info Volume Name: test2_volume Type: Replicate Status: Started Number of Bricks: 2 Transport-type: tcp Bricks: Brick1: bravo:/cluster/shadow/test2 Brick2: backup:/cluster/shadow/test2 Volume Name: test_volume Type: Replicate Status: Started Number of Bricks: 2 Transport-type: rdma Bricks: Brick1: bravo:/cluster/shadow/test Brick2: backup:/cluster/shadow/test mount: glusterfs#localhost:/test_volume on /mnt/test type fuse (rw,allow_other,default_permissions,max_read=131072) glusterfs#localhost:/test2_volume on /mnt/test2 type fuse (rw,allow_other,default_permissions,max_read=131072) time cp files.tar /mnt/test2/ real0m11.159s user0m0.123s sys 0m1.244s files.tar is single file, 390MB, so this about 35MB/s. Fine for gig-e. time cp files.tar /mnt/test/ real0m5.656s user0m0.116s sys 0m0.962s 69MB/s... ehhh. Faster at least. On a few runs, this was not any faster at all. Maybe a cache effect? time cp -av /usr/src/kernels /mnt/test2/ real0m49.605s user0m0.681s sys 0m2.593s kernels dir is 34MB of small files. The low latency of IB should really show an improvement here I thought. - time cp -av /usr/src/kernels /mnt/test/ real0m56.046s user0m0.625s sys 0m2.675s It took LONGER? That can't be right. -- And finally, this error is appearing in the rdma mount log every 3 seconds on both nodes: [2011-01-10 19:46:56.728127] E [rdma.c:4428:tcp_connect_finish] test_volume-client-1: tcp connect to failed (Connection refused) [2011-01-10 19:46:59.738291] E [rdma.c:4428:tcp_connect_finish] test_volume-client-1: tcp connect to failed (Connection refused) [2011-01-10 19:47:02.748260] E [rdma.c:4428:tcp_connect_finish] test_volume-client-1: tcp connect to failed (Connection refused) [2011-01-10 19:47:05.758256] E [rdma.c:4428:tcp_connect_finish] test_volume-client-1: tcp connect to failed (Connection refused) [2011-01-10 19:47:08.768299] E [rdma.c:4428:tcp_connect_finish] test_volume-client-1: tcp connect to failed (Connection refused) [2011-01-10 19:47:11.778308] E [rdma.c:4428:tcp_connect_finish] test_volume-client-1: tcp connect to failed (Connection refused) [2011-01-10 19:47:14.788356] E [rdma.c:4428:tcp_connect_finish] test_volume-client-1: tcp connect to failed (Connection refused) [2011-01-10 19:47:17.798381] E [rdma.c:4428:tcp_connect_finish] test_volume-client-1: tcp connect to failed (Connection refused) [2011-01-10 19:47:20.808413] E [rdma.c:4428:tcp_connect_finish] test_volume-client-1: tcp connect to failed (Connection refused) But there are no restrictions in the config. Everything is allow *. So my questions are, can anyone else tell me what kind of basic file copy performance they see using IB? And what can I do to troubleshoot? Thanks List and Devs, Chris ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Dose Gluster 3.1 support authorisation control and how to do
Hi It seems that the node 10.18.14.240 runs both server and client. If not, write the server list and the client list here. As you can see in the log, the node other than above are all accepted by the server, so you can add both 10.18.14.240 and 127.0.0.1 to the ip-allowed list to see whether it works or not. On Tue, 2011-01-11 at 01:25 +0800, W.C Lee wrote: Hi, HU Thank for your help. I tried to use your example(1 server ,1 Client) to test authentication function, it's work. But I tried to test it in replication mode (multi-node),FUSE mounting work, but NFS didn't. Any node can mount volume via NFS. And Following is my config. 26: volume gluster-new-volume-server 27: type protocol/server 28: option transport-type tcp 29: option auth.addr./mnt/gluster1.allow 10.18.14.240,10.18.14.248,10.18.14.241,10.18.14.242,10.18.14.243 30: subvolumes /mnt/gluster1 31: end-volume After starting volume, log showed below: +--+ [2011-01-11 01:07:54.188695] E [authenticate.c:235:gf_authenticate] auth: no authentication module is interested in accepting remote-client (null) [2011-01-11 01:07:54.188716] E [server-handshake.c:545:server_setvolume] gluster-new-volume-server: Cannot authenticate client from 127.0.0.1:1017 [2011-01-11 01:07:55.264728] I [server-handshake.c:535:server_setvolume] gluster-new-volume-server: accepted client from 10.18.14.241:995 [2011-01-11 01:07:55.267990] I [server-handshake.c:535:server_setvolume] gluster-new-volume-server: accepted client from 10.18.14.242:1012 [2011-01-11 01:07:55.272025] I [server-handshake.c:535:server_setvolume] gluster-new-volume-server: accepted client from 10.18.14.243:996 Do you know is it necessary to set 127.0.0.1 to allow list? And it can't use host real ip (10.18.14.240) ? But even if I used 127.0.0.1 to replace 10.18.14.240, NFS authentication control still not work. -Original message- From:HU Zhong hz02...@gmail.com To:wei.ch...@m2k.com.tw Cc:gluster-users gluster-users@gluster.org Date:Mon, 10 Jan 2011 11:36:00 +0800 Subject:Re: [Gluster-users] Dose Gluster 3.1 support authorisation control and how to do Hi, Cheng I think you did the configuration in the wrong place. Instead of /etc/glusterd/nfs/nfs-server.vol, you need to modify files under /etc/glusterd/vols/. As a simple example, consider a one-server-one-client system, both server and client are one machine(localhost, ip:192.168.4.112), and export directory /home/huz/share for sharing, the client wants to mount it on /home/huz/mnt. if i modify default configuration /etc/glusterd/vols/testvol/testvol.192.168.4.112.home-huz-share.vol from .. 26 volume testvol-server 27 type protocol/server 28 option transport-type tcp 29 option auth.addr./home/huz/share.allow * 30 subvolumes /home/huz/share 31 end-volume to .. 26 volume testvol-server 27 type protocol/server 28 option transport-type tcp 29 option auth.addr./home/huz/share.reject * 30 subvolumes /home/huz/share 31 end-volume the mount command will fail: $sudo mount -o mountproto=tcp -t nfs localhost:/testvol /home/huz/mnt mount.nfs: mounting localhost:/testvol failed, reason given by server: No such file or directory and the log shows that the authentication error. 11-01-10 11:09:58.203600] E [client-handshake.c:786:client_setvolume_cbk] testvol-client-0: SETVOLUME on remote-host failed: Authentication failed change reject to allow, the mount operation will be ok. you can configure you own ip rule. As for how to use ip auth and usrname/password auth, you can check the attachment. It's a documentation file under the directory doc of glusterfs src project. On Sun, 2011-01-09 at 22:31 +0800, 第二信箱 wrote: Hi, HU: Thanks for your help. I have the following environment: Gluster 3.1.1 Volume Name: gluster-volume Type: Distributed-Replicate Status: Started Number of Bricks: 2 x 2 = 4 Transport-type: tcp Bricks: Brick1: gluster1:/mnt/gluster1 Brick2: gluster2:/mnt/gluster2 Brick3: gluster3:/mnt/gluster3 Brick4: gluster4:/mnt/gluster4 I want to use authenticate module by your suggestion. The way I used below: 1. Stop Volume 2. Edit /etc/glusterd/nfs/nfs-server.vol on Brick1(Gluster1) 3. Modify and Add From volume nfs-server type nfs/server option nfs.dynamic-volumes on option rpc-auth.addr.gluster-volume.allow * option nfs3.gluster-volume.volume-id 907941d9-6950-425b- b3d5-4e43dd420d9e subvolumes gluster-volume end-volume to volume nfs-server type nfs/server option nfs.dynamic-volumes on option rpc-auth.addr.gluster-volume.allow 10.18.14.1 option auth.addr.gluster-volume.allow 10.18.14.1
[Gluster-users] 4 node replica 2 crash
Hi, I'm using for about 4 weeks a simple 2 node replica 2 cluster; I'm using glusterfs 3.1.1 built on Dec 9 2010 15:41:32 Repository revision: v3.1.1 . I use it to provide images trought Nginx. All works well. Today i've added 2 new brick, and rebalance volume. For about 4 hours work, after the Nginx hang; i've rebooted all server but nothings to do. When I removed two brick all returns ok (I've manually copied file from old brick to the original). What's wrong? ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Infiniband
On 01/09/2011 10:06 PM, Bryan McGuire wrote: Hello, I am looking into GlusterFS as a high availability solution for our email servers. I am new to Infiniband but find it could possibly provide us with the necessary speed. Hi Bryan We've done this for various ISP/email hosting customers. Could someone describe what I would need in the way of Infiniband hardware / software to complete the following. Two to 4 front end email servers with each being a client and server for the GlusterFS file system performing replication of the data. Depends on what sort of machine you use for your front end, and what the software is. Are these servers for IMAP/POP or are these postfix/exim etc? I think I would need the necessary Infiniband cards in each server along with an Infiniband switch. But do not have any background to determine which or even if this is correct. Simplest architecture is a small IB switch, IB HCA's in each node, an IB stack (OFED) in each node, a subnet manager (OpenSM) daemon, IB cables, and then Gluster built against your stack. Make sure your time daemon is up, running, and correct between the nodes. Give serious consideration to really fast disk in each node (fast IOP, so SSD, in RAID10). Once you have that, you are ready to build your volumes (probably replicated distributed, so 4 way at least). Do beware that there are some gotchas in configurating MTA/MUA software for Gluster. Its doable, but such software often abhors shared storage of any sort. You need to make sure that you turn down some of the caching. Thanks in advance. Bryan McGuire Senior Network Engineer NewNet 66 918.231.8063 bmcgu...@newnet66.org Regards, Joe -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics, Inc. email: land...@scalableinformatics.com web : http://scalableinformatics.com http://scalableinformatics.com/sicluster phone: +1 734 786 8423 x121 fax : +1 866 888 3112 cell : +1 734 612 4615 ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] infiniband speed
On 01/10/2011 07:43 PM, Christopher Hawkins wrote: I am testing Infiniband for the first time. It seems that I should be able to get a lot more speed than I am with some pretty basic tests. Maybe someone running Infiniband can confirm that what I am seeing is way out of line, and/or help diagnose? What is the underlying disk speed? Try this: # (on bravo) dd if=/dev/zero of=/cluster/shadow/big.file bs=1M count=20k This will write a 20GB file to the same partition. We need to see how fast that write is (outside of cache) Do the same test on the other machine. Infiniband won't make slow disks fast. So unless you are already getting a gigabyte or two per second (we do on our units), adding Infiniband won't get you any significant additional performance. Put another way, you are bound by the slowest aspect of your storage system. What are your underlying disks? Are these individual drives? RAID? What sort of RAID if it is a RAID? How is it organized? We tune our systems pretty hard, so we start with 2+GB/s for TB sized files before we ever touch the next stack up. Each additional stack you traverse takes performance away (you lose it in stack inefficiency). So you should start out with as fast a system as you possibly can, so you don't lose too much performance. -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics, Inc. email: land...@scalableinformatics.com web : http://scalableinformatics.com http://scalableinformatics.com/sicluster phone: +1 734 786 8423 x121 fax : +1 866 888 3112 cell : +1 734 612 4615 ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Read-only volumes - timeframe for features/filter?
Are any apps on the mount point erroring out with: Invalid argument or Stale NFS file handle? Burnash, James wrote: Hello. Has anyone seen error messages like this in /var/log/glusterfs/nfs.log: tail /var/log/glusterfs/nfs.log [2011-01-10 14:22:55.859066] I [afr-self-heal-common.c:1526:afr_self_heal_completion_cbk] pfs-ro1-replicate-3: background meta-data data self-heal completed on / [2011-01-10 14:22:55.859084] I [afr-self-heal-common.c:1526:afr_self_heal_completion_cbk] pfs-ro1-replicate-5: background meta-data data self-heal completed on / [2011-01-10 14:22:57.786088] E [nfs3.c:4346:nfs3_fsstat] nfs-nfsv3: Failed to map FH to vol [2011-01-10 14:22:59.355112] E [nfs3.c:4346:nfs3_fsstat] nfs-nfsv3: Failed to map FH to vol [2011-01-10 14:22:59.415732] E [nfs3.c:675:nfs3_getattr] nfs-nfsv3: Failed to map FH to vol [2011-01-10 14:22:59.455029] E [nfs3.c:675:nfs3_getattr] nfs-nfsv3: Failed to map FH to vol [2011-01-10 14:23:01.800751] E [nfs3.c:675:nfs3_getattr] nfs-nfsv3: Failed to map FH to vol [2011-01-10 14:23:02.127233] E [nfs3.c:675:nfs3_getattr] nfs-nfsv3: Failed to map FH to vol [2011-01-10 14:23:07.834044] E [nfs3.c:4346:nfs3_fsstat] nfs-nfsv3: Failed to map FH to vol [2011-01-10 14:23:09.478852] E [nfs3.c:4346:nfs3_fsstat] nfs-nfsv3: Failed to map FH to vol [2011-01-10 14:40:18.558072] E [afr-self-heal-metadata.c:524:afr_sh_metadata_fix] pfs-ro1-replicate-5: Unable to self-heal permissions/ownership of '/' (possible split-brain). Please fix the file on all backend volumes Mount is done with this command: mount -v -t nfs -o soft,rsize=16384,wsize=16384 jc1lpfsnfsro:/pfs-ro1 /pfs1 Command line being executed is: rsync -av --progress /pfs1/online_archive/2010 . This CentOS 5.5 x86_64, GlusterFS 3.1.1. Currently configured: gluster volume info Volume Name: pfs-ro1 Type: Distributed-Replicate Status: Started Number of Bricks: 10 x 2 = 20 Transport-type: tcp Bricks: Brick1: jc1letgfs17-pfs1:/export/read-only/g01 Brick2: jc1letgfs18-pfs1:/export/read-only/g01 Brick3: jc1letgfs17-pfs1:/export/read-only/g02 Brick4: jc1letgfs18-pfs1:/export/read-only/g02 Brick5: jc1letgfs17-pfs1:/export/read-only/g03 Brick6: jc1letgfs18-pfs1:/export/read-only/g03 Brick7: jc1letgfs17-pfs1:/export/read-only/g04 Brick8: jc1letgfs18-pfs1:/export/read-only/g04 Brick9: jc1letgfs17-pfs1:/export/read-only/g05 Brick10: jc1letgfs18-pfs1:/export/read-only/g05 Brick11: jc1letgfs17-pfs1:/export/read-only/g06 Brick12: jc1letgfs18-pfs1:/export/read-only/g06 Brick13: jc1letgfs17-pfs1:/export/read-only/g07 Brick14: jc1letgfs18-pfs1:/export/read-only/g07 Brick15: jc1letgfs17-pfs1:/export/read-only/g08 Brick16: jc1letgfs18-pfs1:/export/read-only/g08 Brick17: jc1letgfs17-pfs1:/export/read-only/g09 Brick18: jc1letgfs18-pfs1:/export/read-only/g09 Brick19: jc1letgfs17-pfs1:/export/read-only/g10 Brick20: jc1letgfs18-pfs1:/export/read-only/g10 Thanks. James Burnash Unix Engineering. DISCLAIMER: This e-mail, and any attachments thereto, is intended only for use by the addressee(s) named herein and may contain legally privileged and/or confidential information. If you are not the intended recipient of this e-mail, you are hereby notified that any dissemination, distribution or copying of this e-mail, and any attachments thereto, is strictly prohibited. If you have received this in error, please immediately notify me and permanently delete the original and any copy of any e-mail and any printout thereof. E-mail transmission cannot be guaranteed to be secure or error-free. The sender therefore does not accept liability for any errors or omissions in the contents of this message which arise as a result of e-mail transmission. NOTICE REGARDING PRIVACY AND CONFIDENTIALITY Knight Capital Group may, at its discretion, monitor and review the content of all e-mail communications. http://www.knight.com ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] How can i turn on the Gluster Debug mode?
Hi, For bricks use - gluster volume set volume-name diagnostics.brick-log-level DEBUG For more tunable options refer http://www.gluster.com/community/documentation/index.php/Gluster_3.1:_Setting_Volume_Options -- Cheers, Lakshmipathi.G FOSS Programmer. - Original Message - From: 공용준(yongjoon kong)/Cloud Computing 사업담당 andrew.k...@sk.com To: gluster-users@gluster.org Sent: Tuesday, January 11, 2011 6:17:49 AM Subject: [Gluster-users] How can i turn on the Gluster Debug mode? Hello, all How can I turn on the Debug mode on 3.1 When I start the volume by command line, is there any way to turn the debug message on brick side? ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] infiniband speed
Thanks Joe, you nailed it. These are just test machines and in each case, just a single 10k scsi drive. That is the throughput bottleneck... I was not able to get more than 70MB/s sustained. Chris - Joe Landman land...@scalableinformatics.com wrote: On 01/10/2011 07:43 PM, Christopher Hawkins wrote: I am testing Infiniband for the first time. It seems that I should be able to get a lot more speed than I am with some pretty basic tests. Maybe someone running Infiniband can confirm that what I am seeing is way out of line, and/or help diagnose? What is the underlying disk speed? Try this: # (on bravo) dd if=/dev/zero of=/cluster/shadow/big.file bs=1M count=20k This will write a 20GB file to the same partition. We need to see how fast that write is (outside of cache) Do the same test on the other machine. Infiniband won't make slow disks fast. So unless you are already getting a gigabyte or two per second (we do on our units), adding Infiniband won't get you any significant additional performance. Put another way, you are bound by the slowest aspect of your storage system. What are your underlying disks? Are these individual drives? RAID? What sort of RAID if it is a RAID? How is it organized? We tune our systems pretty hard, so we start with 2+GB/s for TB sized files before we ever touch the next stack up. Each additional stack you traverse takes performance away (you lose it in stack inefficiency). So you should start out with as fast a system as you possibly can, so you don't lose too much performance. -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics, Inc. email: land...@scalableinformatics.com web : http://scalableinformatics.com http://scalableinformatics.com/sicluster phone: +1 734 786 8423 x121 fax : +1 866 888 3112 cell : +1 734 612 4615 ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] infiniband speed
Out of curiosity, what is a typical RAID / spindle count / rpm configuration for you that yields 2 GB/s? - Christopher Hawkins chawk...@bplinux.com wrote: Thanks Joe, you nailed it. These are just test machines and in each case, just a single 10k scsi drive. That is the throughput bottleneck... I was not able to get more than 70MB/s sustained. Chris - Joe Landman land...@scalableinformatics.com wrote: On 01/10/2011 07:43 PM, Christopher Hawkins wrote: I am testing Infiniband for the first time. It seems that I should be able to get a lot more speed than I am with some pretty basic tests. Maybe someone running Infiniband can confirm that what I am seeing is way out of line, and/or help diagnose? What is the underlying disk speed? Try this: # (on bravo) dd if=/dev/zero of=/cluster/shadow/big.file bs=1M count=20k This will write a 20GB file to the same partition. We need to see how fast that write is (outside of cache) Do the same test on the other machine. Infiniband won't make slow disks fast. So unless you are already getting a gigabyte or two per second (we do on our units), adding Infiniband won't get you any significant additional performance. Put another way, you are bound by the slowest aspect of your storage system. What are your underlying disks? Are these individual drives? RAID? What sort of RAID if it is a RAID? How is it organized? We tune our systems pretty hard, so we start with 2+GB/s for TB sized files before we ever touch the next stack up. Each additional stack you traverse takes performance away (you lose it in stack inefficiency). So you should start out with as fast a system as you possibly can, so you don't lose too much performance. -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics, Inc. email: land...@scalableinformatics.com web : http://scalableinformatics.com http://scalableinformatics.com/sicluster phone: +1 734 786 8423 x121 fax : +1 866 888 3112 cell : +1 734 612 4615 ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] infiniband speed
On Tue, Jan 11, 2011 at 5:26 AM, Joe Landman land...@scalableinformatics.com wrote: We tune our systems pretty hard, so we start with 2+GB/s for TB sized files before we ever touch the next stack up. Each additional stack you traverse takes performance away (you lose it in stack inefficiency). So you should start out with as fast a system as you possibly can, so you don't lose too much performance. -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics, Inc. email: land...@scalableinformatics.com web : http://scalableinformatics.com http://scalableinformatics.com/sicluster phone: +1 734 786 8423 x121 fax : +1 866 888 3112 cell : +1 734 612 4615 ___ Wow, 2GB/s is pretty impressive, would you mind telling me how you achieve this? -- Kind Regards Rudi Ahlers SoftDux Website: http://www.SoftDux.com Technical Blog: http://Blog.SoftDux.com Office: 087 805 9573 Cell: 082 554 7532 ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
