Re: [Gluster-users] Speed of glusterfs
2011/12/14 Marc Muehlfeld marc.muehlf...@medizinische-genetik.de Am 14.12.2011 14:54, schrieb Marc Muehlfeld: Am 14.12.2011 14:00, schrieb Raphaël Hoareau: You could also try to use two NICs with specific routes. NIC1 knows the route to Server1 and NIC2 knows the route to Server2. Could it be possible that this can't be done or do I miss something? (For my speed test results see the end of this mail) I saw what was wrong: When I # gluster peer probe ... from one node to the other, then glusterfs automatically allows it's own IP from *the same subnet*. If I probe both server from a client and then it works: # gluster peer probe 192.168.20.14 Probe successful # gluster peer probe 192.168.29.15 Probe successful # gluster volume create test replica 2 transport tcp 192.168.20.14:/mnt/ 192.168.29.15:/mnt/ Creation of volume test has been successful. ... Netstat also shows that the client is connected to both nodes with each on a own IP in a separare subnet (and NIC): # netstat -taunp | grep glusterfs tcp0 0 192.168.29.1:1022 192.168.29.15:24011VERBUNDEN 23488/glusterfs tcp0 0 192.168.20.1:1019 192.168.20.14:24011VERBUNDEN 23310/glusterfs Here are the results: Im writing a 10 GB file to the cluster: time dd if=/dev/zero of=/mnt/test.10G bs=1M count=1 1+0 Datensätze ein 1+0 Datensätze aus 1048576 Bytes (10 GB) kopiert, 106,981 s, 98,0 MB/s real1m47.018s user0m0.010s sys 0m5.801s This is a good result for me on a 1GBit connection. Great ! As expected, you doubled your speed. Good to know :) __**_ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/**mailman/listinfo/gluster-usershttp://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Speed of glusterfs
Hi Marc, The GlusterFS client DOES send the data directly to both storage, that's why your speed is about half the NFS speed. Concerning tweaks to write only on one server, and then replicate from one server to another, I think it's not possible given the architecture of GlusterFS. If someone know more details, even tweaks about this... I would be very interested. Regards, Raphaël. 2011/12/14 Marc Muehlfeld marc.muehlf...@medizinische-genetik.de Hi, we're currently testing glusterfs but getting a real slow connection with it. I setup a environment with 2 replicated storage servers. Both machines and the client are connected by 1 Gbit, but if I copy a 2.5 GB file from the client to the volumen mounted with glusterfs protocol, I never get more than 35-40 MB/sec. On the two storage I also run a NFS server. If I copy the same file from the client to one storage by NFS (the NFS server that was shipped with RHEL6), I get never less than 90-95 MB/sec. So the network/HDDs are possible to provide faster transfer. Is there a change to get glusterfs to about the same speed? And why is it so slow? Does the glusterfs send the data directly to both storages? This would explain why it is so slow. But is there a change then to send the data just to one server and this one replicates it (over a second NIC) to the second one)? Regards, Marc __**_ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/**mailman/listinfo/gluster-usershttp://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Speed of glusterfs
In theory yes, but the bandwith will also be limited by the nodes between the client and the server. Unless every part of your network can handle 2Gbit, it won't be a good solution. You could also try to use two NICs with specific routes. NIC1 knows the route to Server1 and NIC2 knows the route to Server2. Something like this : eth0 : route add -net 192.168.0.0 netmask 255.255.255.0 -i eth0# Assuming 192.168.0.0 = network containing Server1 eth1 : route add -net 192.168.1.0 netmask 255.255.255.0 -i eth1# Assuming 192.168.1.0 = network containing Server2 But I really don't know if it could work as I never tried it before. And, as I said, it also depends on the nodes in your network. The most basic configuration would look like this : Server1 Server2 | | | | 1Gbit| |1Gbit | | | | eth0 eth1 Client In that case, you will double your speed. But again, it's just an idea. If you have everything to try it (NIC-teaming and/or NIC specific route), go ahead. And, please, give me your results as I'm very interested in GlusterFS performances. Regards, Raphaël. Le 14 décembre 2011 12:55, Marc Muehlfeld marc.muehlf...@medizinische-genetik.de a écrit : Am 14.12.2011 11:50, schrieb Raphaël Hoareau: The GlusterFS client DOES send the data directly to both storage, that's why your speed is about half the NFS speed. Can I double the speed of GlusterFS by NIC-teaming (2x 1Gbit)? ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Speed of glusterfs
Le 14 décembre 2011 14:54, Marc Muehlfeld marc.muehlf...@medizinische-genetik.de a écrit : Am 14.12.2011 14:00, schrieb Raphaël Hoareau: You could also try to use two NICs with specific routes. NIC1 knows the route to Server1 and NIC2 knows the route to Server2. Could it be possible that this can't be done or do I miss something? Node 1: 192.168.29.14 and 192.168.20.14 Node 2: 192.168.29.15 and 192.168.20.15 Each IP is on a separate own NIC in both machines. Connection over both NICs are fine: # traceroute 192.168.20.14 traceroute to 192.168.20.14 (192.168.20.14), 30 hops max, 60 byte packets 1 192.168.20.14 (192.168.20.14) 0.162 ms 0.148 ms 0.141 ms # traceroute 192.168.29.14 traceroute to 192.168.29.14 (192.168.29.14), 30 hops max, 60 byte packets 1 192.168.29.14 (192.168.29.14) 0.189 ms 0.172 ms 0.244 ms Could you precise on which host the command is executed ? It will help a lot. (Also available for the output above) But (both nodes on different NICs/Subnets): # gluster peer probe 192.168.29.14 Probe successful # gluster volume create test replica 2 transport tcp 192.168.29.14:/mnt/ 192.168.20.15:/mnt/ Operation failed on 192.168.29.14 Both nodes with connection over the same NIC/Subnet: # gluster peer probe 192.168.20.14 Probe successful # gluster volume create test replica 2 transport tcp 192.168.20.14:/mnt/ 192.168.20.15:/mnt/ Creation of volume test has been successful. ... It also works if I create the volume over the 192.168.29.0/24 subnet. But not if I mix them. The log says: [2011-12-14 14:53:45.253837] I [glusterd-handler.c:448:**glusterd_handle_cluster_lock] 0-glusterd: Received LOCK from uuid: 4e765edc-9ca6-4404-8757-** ca170ce938df [2011-12-14 14:53:45.253888] I [glusterd-utils.c:243:**glusterd_lock] 0-glusterd: Cluster lock held by 4e765edc-9ca6-4404-8757-**ca170ce938df [2011-12-14 14:53:45.253938] I [glusterd-handler.c:2651:**glusterd_op_lock_send_resp] 0-glusterd: Responded, ret: 0 [2011-12-14 14:53:45.255287] I [glusterd-handler.c:488:**glusterd_req_ctx_create] 0-glusterd: Received op from uuid: 4e765edc-9ca6-4404-8757-**ca170ce938df [2011-12-14 14:53:45.256600] E [glusterd-op-sm.c:366:** glusterd_op_stage_create_**volume] 0-glusterd: cannot resolve brick: 192.168.20.15:/mnt It seems that 192.168.29.14 can't resolv 192.168.20.15. Can you show the result of route -n executed on the host that produces that log ? [2011-12-14 14:53:45.256635] E [glusterd-op-sm.c:7370:**glusterd_op_ac_stage_op] 0-: Validate failed: 1 [2011-12-14 14:53:45.256681] I [glusterd-handler.c:2743:**glusterd_op_stage_send_resp] 0-glusterd: Responded to stage, ret: 0 [2011-12-14 14:53:45.256994] I [glusterd-handler.c:2693:** glusterd_handle_cluster_**unlock] 0-glusterd: Received UNLOCK from uuid: 4e765edc-9ca6-4404-8757-**ca170ce938df [2011-12-14 14:53:45.257049] I [glusterd-handler.c:2671:**glusterd_op_unlock_send_resp] 0-glusterd: Responded to unlock, ret: 0 I don't have much time right now, but I'll try to help you a bit more later. Regards, Raphaël. ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
