Re: [Gluster-users] Infiniband performance issues answered?
On Dec 18, 2012, at 2:15 AM, Sabuj Pattanayek wrote: I have R610's with a similar setup but with HT turned on and I'm getting 3.5GB/s for one way RDMA tests between two QDR connected clients using mellanox connectx x4 PCI-E cards in x8 slots. 1GB/s with IPoIB connections (seem to be limited to 10gbe). Note, I had problems with the 1.x branch of OFED and am using the latest 3.x RC . What are you writing to and from? On Mon, Dec 17, 2012 at 6:44 PM, Joe Julian j...@julianfamily.org wrote: In IRC today, someone who was hitting that same IB performance ceiling that occasionally gets reported had this to say [11:50] nissim first, I ran fedora which is not supported by Mellanox OFED distro [11:50] nissim so I moved to CentOS 6.3 [11:51] nissim next I removed all distibution related infiniband rpms and build the latest OFED package [11:52] nissim disabled ServerSpeed service [11:52] nissim disabled BIOS hyperthreading [11:52] nissim disabled BIOS power mgmt [11:53] nissim ran ib_write_test and goot 5000MB/s [11:53] nissim got 5000MB/s on localhost fwiw, if someone's encountering that issue, between this and the changes since 3.4.0qa5 it might be worth knowing about. http://irclog.perlgeek.de/gluster/2012-12-17#i_6251387 ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Infiniband performance issues answered?
i think qperf just writes to and from memory on both systems so that it can best test the network and not disk, then tosses the packets away On Tue, Dec 18, 2012 at 3:34 AM, Andrew Holway a.hol...@syseleven.de wrote: On Dec 18, 2012, at 2:15 AM, Sabuj Pattanayek wrote: I have R610's with a similar setup but with HT turned on and I'm getting 3.5GB/s for one way RDMA tests between two QDR connected clients using mellanox connectx x4 PCI-E cards in x8 slots. 1GB/s with IPoIB connections (seem to be limited to 10gbe). Note, I had problems with the 1.x branch of OFED and am using the latest 3.x RC . What are you writing to and from? On Mon, Dec 17, 2012 at 6:44 PM, Joe Julian j...@julianfamily.org wrote: In IRC today, someone who was hitting that same IB performance ceiling that occasionally gets reported had this to say [11:50] nissim first, I ran fedora which is not supported by Mellanox OFED distro [11:50] nissim so I moved to CentOS 6.3 [11:51] nissim next I removed all distibution related infiniband rpms and build the latest OFED package [11:52] nissim disabled ServerSpeed service [11:52] nissim disabled BIOS hyperthreading [11:52] nissim disabled BIOS power mgmt [11:53] nissim ran ib_write_test and goot 5000MB/s [11:53] nissim got 5000MB/s on localhost fwiw, if someone's encountering that issue, between this and the changes since 3.4.0qa5 it might be worth knowing about. http://irclog.perlgeek.de/gluster/2012-12-17#i_6251387 ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
[Gluster-users] infiniband replicated distributed setup.--- network setup question...
We have two networks and I have 4 servers I'm trying to turn into a distributed/replicated Gluster storage cluster. Each of the servers has two addresses, one ethernet, one IB. The externet addresses are 155.42.48.1-4 and the Infiniband addresses are 172.24.215.1-4 I login to the first server (1) which is 155.52.48.1. Let's say the servers are named 1,2,1R and 2R (on the ethernet side), but 1-IB, 2-IB, 1-IB-R and 2-IB-R on he Infiniband side. The Volume is to be mounted by other servers in the IB network by native Gluster clients. so, I login to 1 over the ethernet network and do a peer probe. All looks good from 1 with peer probe and peer status. I probe the peers like this, successfully gluster peer probe 2-ib gluster peer probe 1-ib-r gluster peer prove 2-ib-r Peer status looks right. I issue the following command which fails, of course: gluster volume create gf2 replica 2 transport tcp,rdma \ 1-ib:/mnt/d0-0 1-ib-r:/mnt/d0-0 2-ib:/mnt/d0-0 2-ib-r:/mnt/d0-0 From any of those nodes, peer status is wrong: For instance, if I ssh over to 2-ib, and ask for a peer status, it shows the peers to to be 1-ib-r 2-ib-r and *155.52.48.1* (the last is the ethernet side of, not the IB side) I guess my question is this: How do I get the first node to use the ib side when connecting to the other nodes. (both of these networks are public.) Do I have to do something interesting with routes? I need to fix this. Don't most people with IB also have an internet side to things? How do their networks and any local routing info get set up? Am I ,missing something really obvious? Matt -- Matt Temple Director, Research Computing Dana-Farber Cancer Institute. We have two ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] infiniband replicated distributed setup.--- network setup question...
On 12/18/2012 7:33 PM, Matthew Temple wrote: From any of those nodes, peer status is wrong: For instance, if I ssh over to 2-ib, and ask for a peer status, it shows the peers to to be 1-ib-r 2-ib-r and *155.52.48.1* (the last is the ethernet side of, not the IB side) I guess my question is this: How do I get the first node to use the ib side when connecting to the other nodes. (both of these networks are public.) Do I have to do something interesting with routes? Just probe the first server by name from any of the other three. That will fix the peers. Thanks, Shawn ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
[Gluster-users] Infiniband performance issues answered?
In IRC today, someone who was hitting that same IB performance ceiling that occasionally gets reported had this to say [11:50] nissim first, I ran fedora which is not supported by Mellanox OFED distro [11:50] nissim so I moved to CentOS 6.3 [11:51] nissim next I removed all distibution related infiniband rpms and build the latest OFED package [11:52] nissim disabled ServerSpeed service [11:52] nissim disabled BIOS hyperthreading [11:52] nissim disabled BIOS power mgmt [11:53] nissim ran ib_write_test and goot 5000MB/s [11:53] nissim got 5000MB/s on localhost fwiw, if someone's encountering that issue, between this and the changes since 3.4.0qa5 it might be worth knowing about. http://irclog.perlgeek.de/gluster/2012-12-17#i_6251387 ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Infiniband performance issues answered?
I have R610's with a similar setup but with HT turned on and I'm getting 3.5GB/s for one way RDMA tests between two QDR connected clients using mellanox connectx x4 PCI-E cards in x8 slots. 1GB/s with IPoIB connections (seem to be limited to 10gbe). Note, I had problems with the 1.x branch of OFED and am using the latest 3.x RC . On Mon, Dec 17, 2012 at 6:44 PM, Joe Julian j...@julianfamily.org wrote: In IRC today, someone who was hitting that same IB performance ceiling that occasionally gets reported had this to say [11:50] nissim first, I ran fedora which is not supported by Mellanox OFED distro [11:50] nissim so I moved to CentOS 6.3 [11:51] nissim next I removed all distibution related infiniband rpms and build the latest OFED package [11:52] nissim disabled ServerSpeed service [11:52] nissim disabled BIOS hyperthreading [11:52] nissim disabled BIOS power mgmt [11:53] nissim ran ib_write_test and goot 5000MB/s [11:53] nissim got 5000MB/s on localhost fwiw, if someone's encountering that issue, between this and the changes since 3.4.0qa5 it might be worth knowing about. http://irclog.perlgeek.de/gluster/2012-12-17#i_6251387 ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Infiniband performance issues answered?
and yes on some Dells you'll get strange network and RAID controller performance characteristics if you turn on the BIOS power management. On Mon, Dec 17, 2012 at 7:15 PM, Sabuj Pattanayek sab...@gmail.com wrote: I have R610's with a similar setup but with HT turned on and I'm getting 3.5GB/s for one way RDMA tests between two QDR connected clients using mellanox connectx x4 PCI-E cards in x8 slots. 1GB/s with IPoIB connections (seem to be limited to 10gbe). Note, I had problems with the 1.x branch of OFED and am using the latest 3.x RC . On Mon, Dec 17, 2012 at 6:44 PM, Joe Julian j...@julianfamily.org wrote: In IRC today, someone who was hitting that same IB performance ceiling that occasionally gets reported had this to say [11:50] nissim first, I ran fedora which is not supported by Mellanox OFED distro [11:50] nissim so I moved to CentOS 6.3 [11:51] nissim next I removed all distibution related infiniband rpms and build the latest OFED package [11:52] nissim disabled ServerSpeed service [11:52] nissim disabled BIOS hyperthreading [11:52] nissim disabled BIOS power mgmt [11:53] nissim ran ib_write_test and goot 5000MB/s [11:53] nissim got 5000MB/s on localhost fwiw, if someone's encountering that issue, between this and the changes since 3.4.0qa5 it might be worth knowing about. http://irclog.perlgeek.de/gluster/2012-12-17#i_6251387 ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Infiniband performance issues answered?
does anyone have 3.4.0qa5 rpm's available? I'd like to give them a whirl. On Mon, Dec 17, 2012 at 5:17 PM, Sabuj Pattanayek sab...@gmail.com wrote: and yes on some Dells you'll get strange network and RAID controller performance characteristics if you turn on the BIOS power management. On Mon, Dec 17, 2012 at 7:15 PM, Sabuj Pattanayek sab...@gmail.com wrote: I have R610's with a similar setup but with HT turned on and I'm getting 3.5GB/s for one way RDMA tests between two QDR connected clients using mellanox connectx x4 PCI-E cards in x8 slots. 1GB/s with IPoIB connections (seem to be limited to 10gbe). Note, I had problems with the 1.x branch of OFED and am using the latest 3.x RC . On Mon, Dec 17, 2012 at 6:44 PM, Joe Julian j...@julianfamily.org wrote: In IRC today, someone who was hitting that same IB performance ceiling that occasionally gets reported had this to say [11:50] nissim first, I ran fedora which is not supported by Mellanox OFED distro [11:50] nissim so I moved to CentOS 6.3 [11:51] nissim next I removed all distibution related infiniband rpms and build the latest OFED package [11:52] nissim disabled ServerSpeed service [11:52] nissim disabled BIOS hyperthreading [11:52] nissim disabled BIOS power mgmt [11:53] nissim ran ib_write_test and goot 5000MB/s [11:53] nissim got 5000MB/s on localhost fwiw, if someone's encountering that issue, between this and the changes since 3.4.0qa5 it might be worth knowing about. http://irclog.perlgeek.de/gluster/2012-12-17#i_6251387 ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
[Gluster-users] infiniband bonding
Hi folks, Reading this post: http://community.gluster.org/q/port-bonding-link-aggregation-transport-rdma-ib-verbs/ It says that gluster 3.2 does not support bonding of infiniband ports. Does anyone knows whether 3.3 has changed this limitation? Is there any other place where to find information about this subject? Thanks in advance! Samuel. ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] infiniband bonding
Well, it actually says it is a limitation of the Infiniband driver so nothing with Gluster I guess. If the driver allow then in theory should not be a problem for Gluster. Fernando From: gluster-users-boun...@gluster.org [mailto:gluster-users-boun...@gluster.org] On Behalf Of samuel Sent: 21 September 2012 10:56 To: gluster-users@gluster.org Subject: [Gluster-users] infiniband bonding Hi folks, Reading this post: http://community.gluster.org/q/port-bonding-link-aggregation-transport-rdma-ib-verbs/ It says that gluster 3.2 does not support bonding of infiniband ports. Does anyone knows whether 3.3 has changed this limitation? Is there any other place where to find information about this subject? Thanks in advance! Samuel. ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
[Gluster-users] Infiniband support on OpenSolaris
Hi all, I'm currently using an OpenSolaris storage server to serve SRP targets to systems running KVM on CentOs 5.5. I was wanting to test Gluster over RDMA but haven't been able to find any references to Infiniband support on Solaris. Can anyone confirm that this works and if so, what packages are required? Thanks -Matt ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Infiniband support on OpenSolaris
Hello Matthew, Le 8 févr. 2011 à 09:50, Matthew Anderson a écrit : Hi all, I'm currently using an OpenSolaris storage server to serve SRP targets to systems running KVM on CentOs 5.5. I was wanting to test Gluster over RDMA but haven't been able to find any references to Infiniband support on Solaris. Can anyone confirm that this works and if so, what packages are required? It is working basicaly with ipoib... Now using RDMA, it is depending how OpenSolaris does RDMA, since OFUB is more or less a dead project (thanks to Oracle)... OFED like APIs are IMHO not yet ported to OpenSolaris... BTW I really like to have glusterfs/rdma working on OpenSolaris Xavier -- Xavier Beaudouin - x...@soprive.net - http://www.soprive.net/ - Tel: +33 9 5437 9612 So Privé - Le premier acteur dédié au cloud computing en France GPG Fingerprints : A6B2 D563 F93B A3AF C08A CBAC 6BC6 79EB DCC9 9867 ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] infiniband speed
On Mon, 10 Jan 2011, Joe Landman wrote: Try this: # (on bravo) dd if=/dev/zero of=/cluster/shadow/big.file bs=1M count=20k This will write a 20GB file to the same partition. We need to see how fast that write is (outside of cache) Do the same test on the other machine. Infiniband won't make slow disks fast. So unless you are already getting a gigabyte or two per second (we do on our units), adding Infiniband won't get you any significant additional performance. Put another way, you are bound by the slowest aspect of your storage system. What are your underlying disks? Are these individual drives? RAID? What sort of RAID if it is a RAID? How is it organized? We tune our systems pretty hard, so we start with 2+GB/s for TB sized files before we ever touch the next stack up. Each additional stack you traverse takes performance away (you lose it in stack inefficiency). So you should start out with as fast a system as you possibly can, so you don't lose too much performance. What hardware gives you 2+GB/s? I am using the 16 port 3Ware 9750 cards with fast drives and I dont get close to that. -Nathan ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] infiniband speed
On 01/11/2011 10:44 AM, Nathan Stratton wrote: On Mon, 10 Jan 2011, Joe Landman wrote: Try this: # (on bravo) dd if=/dev/zero of=/cluster/shadow/big.file bs=1M count=20k This will write a 20GB file to the same partition. We need to see how fast that write is (outside of cache) Do the same test on the other machine. Infiniband won't make slow disks fast. So unless you are already getting a gigabyte or two per second (we do on our units), adding Infiniband won't get you any significant additional performance. Put another way, you are bound by the slowest aspect of your storage system. What are your underlying disks? Are these individual drives? RAID? What sort of RAID if it is a RAID? How is it organized? We tune our systems pretty hard, so we start with 2+GB/s for TB sized files before we ever touch the next stack up. Each additional stack you traverse takes performance away (you lose it in stack inefficiency). So you should start out with as fast a system as you possibly can, so you don't lose too much performance. What hardware gives you 2+GB/s? I am using the 16 port 3Ware 9750 cards with fast drives and I dont get close to that. Recent results here http://scalability.org/?p=2912 http://scalability.org/?p=2552 and older results http://scalability.org/?p=2356 Not a commercial. Follow the links in the .sig for more info. We've been a Gluster reseller/service provider for a while now (though for some reason we've not been on their site, and have asked them to work on that with us). -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics Inc. email: land...@scalableinformatics.com web : http://scalableinformatics.com http://scalableinformatics.com/sicluster phone: +1 734 786 8423 x121 fax : +1 866 888 3112 cell : +1 734 612 4615 ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
[Gluster-users] infiniband speed
I am testing Infiniband for the first time. It seems that I should be able to get a lot more speed than I am with some pretty basic tests. Maybe someone running Infiniband can confirm that what I am seeing is way out of line, and/or help diagnose? I have two systems connected using 3.1.2qa3. With 3.1.1 infiniband wouldn't even start, it gave an error about unable to intialize rdma. But with the latest version and an upgrade to OFED 1.5.2, everything starts up with no errors and I can create a volume and mount it. The underlying Infiniband seems ok, and a basic ibv_rc_pingpong test shows I can move data pretty fast: 8192 bytes in 0.23 seconds = 2858.45 Mbit/sec 1 iters in 0.23 seconds = 22.93 usec/iter So now I have two volumes created, one that uses tcp over a gig-e link and one that uses rdma. I mount them and do some file copy tests... And they are almost exactly the same? What? gluster volume info Volume Name: test2_volume Type: Replicate Status: Started Number of Bricks: 2 Transport-type: tcp Bricks: Brick1: bravo:/cluster/shadow/test2 Brick2: backup:/cluster/shadow/test2 Volume Name: test_volume Type: Replicate Status: Started Number of Bricks: 2 Transport-type: rdma Bricks: Brick1: bravo:/cluster/shadow/test Brick2: backup:/cluster/shadow/test mount: glusterfs#localhost:/test_volume on /mnt/test type fuse (rw,allow_other,default_permissions,max_read=131072) glusterfs#localhost:/test2_volume on /mnt/test2 type fuse (rw,allow_other,default_permissions,max_read=131072) time cp files.tar /mnt/test2/ real0m11.159s user0m0.123s sys 0m1.244s files.tar is single file, 390MB, so this about 35MB/s. Fine for gig-e. time cp files.tar /mnt/test/ real0m5.656s user0m0.116s sys 0m0.962s 69MB/s... ehhh. Faster at least. On a few runs, this was not any faster at all. Maybe a cache effect? time cp -av /usr/src/kernels /mnt/test2/ real0m49.605s user0m0.681s sys 0m2.593s kernels dir is 34MB of small files. The low latency of IB should really show an improvement here I thought. - time cp -av /usr/src/kernels /mnt/test/ real0m56.046s user0m0.625s sys 0m2.675s It took LONGER? That can't be right. -- And finally, this error is appearing in the rdma mount log every 3 seconds on both nodes: [2011-01-10 19:46:56.728127] E [rdma.c:4428:tcp_connect_finish] test_volume-client-1: tcp connect to failed (Connection refused) [2011-01-10 19:46:59.738291] E [rdma.c:4428:tcp_connect_finish] test_volume-client-1: tcp connect to failed (Connection refused) [2011-01-10 19:47:02.748260] E [rdma.c:4428:tcp_connect_finish] test_volume-client-1: tcp connect to failed (Connection refused) [2011-01-10 19:47:05.758256] E [rdma.c:4428:tcp_connect_finish] test_volume-client-1: tcp connect to failed (Connection refused) [2011-01-10 19:47:08.768299] E [rdma.c:4428:tcp_connect_finish] test_volume-client-1: tcp connect to failed (Connection refused) [2011-01-10 19:47:11.778308] E [rdma.c:4428:tcp_connect_finish] test_volume-client-1: tcp connect to failed (Connection refused) [2011-01-10 19:47:14.788356] E [rdma.c:4428:tcp_connect_finish] test_volume-client-1: tcp connect to failed (Connection refused) [2011-01-10 19:47:17.798381] E [rdma.c:4428:tcp_connect_finish] test_volume-client-1: tcp connect to failed (Connection refused) [2011-01-10 19:47:20.808413] E [rdma.c:4428:tcp_connect_finish] test_volume-client-1: tcp connect to failed (Connection refused) But there are no restrictions in the config. Everything is allow *. So my questions are, can anyone else tell me what kind of basic file copy performance they see using IB? And what can I do to troubleshoot? Thanks List and Devs, Chris ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Infiniband
On 01/09/2011 10:06 PM, Bryan McGuire wrote: Hello, I am looking into GlusterFS as a high availability solution for our email servers. I am new to Infiniband but find it could possibly provide us with the necessary speed. Hi Bryan We've done this for various ISP/email hosting customers. Could someone describe what I would need in the way of Infiniband hardware / software to complete the following. Two to 4 front end email servers with each being a client and server for the GlusterFS file system performing replication of the data. Depends on what sort of machine you use for your front end, and what the software is. Are these servers for IMAP/POP or are these postfix/exim etc? I think I would need the necessary Infiniband cards in each server along with an Infiniband switch. But do not have any background to determine which or even if this is correct. Simplest architecture is a small IB switch, IB HCA's in each node, an IB stack (OFED) in each node, a subnet manager (OpenSM) daemon, IB cables, and then Gluster built against your stack. Make sure your time daemon is up, running, and correct between the nodes. Give serious consideration to really fast disk in each node (fast IOP, so SSD, in RAID10). Once you have that, you are ready to build your volumes (probably replicated distributed, so 4 way at least). Do beware that there are some gotchas in configurating MTA/MUA software for Gluster. Its doable, but such software often abhors shared storage of any sort. You need to make sure that you turn down some of the caching. Thanks in advance. Bryan McGuire Senior Network Engineer NewNet 66 918.231.8063 bmcgu...@newnet66.org Regards, Joe -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics, Inc. email: land...@scalableinformatics.com web : http://scalableinformatics.com http://scalableinformatics.com/sicluster phone: +1 734 786 8423 x121 fax : +1 866 888 3112 cell : +1 734 612 4615 ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] infiniband speed
On 01/10/2011 07:43 PM, Christopher Hawkins wrote: I am testing Infiniband for the first time. It seems that I should be able to get a lot more speed than I am with some pretty basic tests. Maybe someone running Infiniband can confirm that what I am seeing is way out of line, and/or help diagnose? What is the underlying disk speed? Try this: # (on bravo) dd if=/dev/zero of=/cluster/shadow/big.file bs=1M count=20k This will write a 20GB file to the same partition. We need to see how fast that write is (outside of cache) Do the same test on the other machine. Infiniband won't make slow disks fast. So unless you are already getting a gigabyte or two per second (we do on our units), adding Infiniband won't get you any significant additional performance. Put another way, you are bound by the slowest aspect of your storage system. What are your underlying disks? Are these individual drives? RAID? What sort of RAID if it is a RAID? How is it organized? We tune our systems pretty hard, so we start with 2+GB/s for TB sized files before we ever touch the next stack up. Each additional stack you traverse takes performance away (you lose it in stack inefficiency). So you should start out with as fast a system as you possibly can, so you don't lose too much performance. -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics, Inc. email: land...@scalableinformatics.com web : http://scalableinformatics.com http://scalableinformatics.com/sicluster phone: +1 734 786 8423 x121 fax : +1 866 888 3112 cell : +1 734 612 4615 ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] infiniband speed
Thanks Joe, you nailed it. These are just test machines and in each case, just a single 10k scsi drive. That is the throughput bottleneck... I was not able to get more than 70MB/s sustained. Chris - Joe Landman land...@scalableinformatics.com wrote: On 01/10/2011 07:43 PM, Christopher Hawkins wrote: I am testing Infiniband for the first time. It seems that I should be able to get a lot more speed than I am with some pretty basic tests. Maybe someone running Infiniband can confirm that what I am seeing is way out of line, and/or help diagnose? What is the underlying disk speed? Try this: # (on bravo) dd if=/dev/zero of=/cluster/shadow/big.file bs=1M count=20k This will write a 20GB file to the same partition. We need to see how fast that write is (outside of cache) Do the same test on the other machine. Infiniband won't make slow disks fast. So unless you are already getting a gigabyte or two per second (we do on our units), adding Infiniband won't get you any significant additional performance. Put another way, you are bound by the slowest aspect of your storage system. What are your underlying disks? Are these individual drives? RAID? What sort of RAID if it is a RAID? How is it organized? We tune our systems pretty hard, so we start with 2+GB/s for TB sized files before we ever touch the next stack up. Each additional stack you traverse takes performance away (you lose it in stack inefficiency). So you should start out with as fast a system as you possibly can, so you don't lose too much performance. -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics, Inc. email: land...@scalableinformatics.com web : http://scalableinformatics.com http://scalableinformatics.com/sicluster phone: +1 734 786 8423 x121 fax : +1 866 888 3112 cell : +1 734 612 4615 ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] infiniband speed
Out of curiosity, what is a typical RAID / spindle count / rpm configuration for you that yields 2 GB/s? - Christopher Hawkins chawk...@bplinux.com wrote: Thanks Joe, you nailed it. These are just test machines and in each case, just a single 10k scsi drive. That is the throughput bottleneck... I was not able to get more than 70MB/s sustained. Chris - Joe Landman land...@scalableinformatics.com wrote: On 01/10/2011 07:43 PM, Christopher Hawkins wrote: I am testing Infiniband for the first time. It seems that I should be able to get a lot more speed than I am with some pretty basic tests. Maybe someone running Infiniband can confirm that what I am seeing is way out of line, and/or help diagnose? What is the underlying disk speed? Try this: # (on bravo) dd if=/dev/zero of=/cluster/shadow/big.file bs=1M count=20k This will write a 20GB file to the same partition. We need to see how fast that write is (outside of cache) Do the same test on the other machine. Infiniband won't make slow disks fast. So unless you are already getting a gigabyte or two per second (we do on our units), adding Infiniband won't get you any significant additional performance. Put another way, you are bound by the slowest aspect of your storage system. What are your underlying disks? Are these individual drives? RAID? What sort of RAID if it is a RAID? How is it organized? We tune our systems pretty hard, so we start with 2+GB/s for TB sized files before we ever touch the next stack up. Each additional stack you traverse takes performance away (you lose it in stack inefficiency). So you should start out with as fast a system as you possibly can, so you don't lose too much performance. -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics, Inc. email: land...@scalableinformatics.com web : http://scalableinformatics.com http://scalableinformatics.com/sicluster phone: +1 734 786 8423 x121 fax : +1 866 888 3112 cell : +1 734 612 4615 ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] infiniband speed
On Tue, Jan 11, 2011 at 5:26 AM, Joe Landman land...@scalableinformatics.com wrote: We tune our systems pretty hard, so we start with 2+GB/s for TB sized files before we ever touch the next stack up. Each additional stack you traverse takes performance away (you lose it in stack inefficiency). So you should start out with as fast a system as you possibly can, so you don't lose too much performance. -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics, Inc. email: land...@scalableinformatics.com web : http://scalableinformatics.com http://scalableinformatics.com/sicluster phone: +1 734 786 8423 x121 fax : +1 866 888 3112 cell : +1 734 612 4615 ___ Wow, 2GB/s is pretty impressive, would you mind telling me how you achieve this? -- Kind Regards Rudi Ahlers SoftDux Website: http://www.SoftDux.com Technical Blog: http://Blog.SoftDux.com Office: 087 805 9573 Cell: 082 554 7532 ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
[Gluster-users] Infiniband
Hello, I am looking into GlusterFS as a high availability solution for our email servers. I am new to Infiniband but find it could possibly provide us with the necessary speed. Could someone describe what I would need in the way of Infiniband hardware / software to complete the following. Two to 4 front end email servers with each being a client and server for the GlusterFS file system performing replication of the data. I think I would need the necessary Infiniband cards in each server along with an Infiniband switch. But do not have any background to determine which or even if this is correct. Thanks in advance. Bryan McGuire Senior Network Engineer NewNet 66 918.231.8063 bmcgu...@newnet66.org ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
