Re: [Gluster-users] Speed of glusterfs

[Raphaël Hoareau]

2011/12/14 Marc Muehlfeld 

> Am 14.12.2011 14:54, schrieb Marc Muehlfeld:
>
>  Am 14.12.2011 14:00, schrieb Raphaël Hoareau:
>>
>>> You could also try to use two NICs with specific routes. NIC1 knows the
>>> route to Server1 and NIC2 knows the route to Server2.
>>>
>>
>> Could it be possible that this can't be done or do I miss something?
>>
>
> (For my speed test results see the end of this mail)
>
>
>
> I saw what was wrong: When I
>
> # gluster peer probe ...
>
> from one node to the other, then glusterfs automatically allows it's own
> IP from *the same subnet*.
>
>
> If I probe both server from a client and then it works:
>
>
> # gluster peer probe 192.168.20.14
> Probe successful
>
> # gluster peer probe 192.168.29.15
> Probe successful
>
> # gluster volume create test replica 2 transport tcp 192.168.20.14:/mnt/
> 192.168.29.15:/mnt/
>
> Creation of volume test has been successful. ...
>
>
> Netstat also shows that the client is connected to both nodes with each on
> a own IP in a separare subnet (and NIC):
>
> # netstat -taunp | grep glusterfs
> tcp0  0 192.168.29.1:1022   192.168.29.15:24011VERBUNDEN  
>  23488/glusterfs
> tcp0  0 192.168.20.1:1019   192.168.20.14:24011VERBUNDEN  
>  23310/glusterfs
> 
>
>
>
>
>
> Here are the results:
>
> Im writing a 10 GB file to the cluster:
>
> time dd if=/dev/zero of=/mnt/test.10G bs=1M count=1
> 1+0 Datensätze ein
> 1+0 Datensätze aus
> 1048576 Bytes (10 GB) kopiert, 106,981 s, 98,0 MB/s
>
> real1m47.018s
> user0m0.010s
> sys 0m5.801s
>
> This is a good result for me on a 1GBit connection.


Great ! As expected, you doubled your speed.
Good to know :)


>
> __**_
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://gluster.org/cgi-bin/**mailman/listinfo/gluster-users<http://gluster.org/cgi-bin/mailman/listinfo/gluster-users>
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] Speed of glusterfs

[Raphaël Hoareau]

Le 14 décembre 2011 14:54, Marc Muehlfeld <
marc.muehlf...@medizinische-genetik.de> a écrit :

> Am 14.12.2011 14:00, schrieb Raphaël Hoareau:
>
>> You could also try to use two NICs with specific routes. NIC1 knows the
>>
>> route to Server1 and NIC2 knows the route to Server2.
>>
>
> Could it be possible that this can't be done or do I miss something?
>
> Node 1: 192.168.29.14 and 192.168.20.14
> Node 2: 192.168.29.15 and 192.168.20.15
> Each IP is on a separate own NIC in both machines.
>
>
>
> Connection over both NICs are fine:
>
> # traceroute 192.168.20.14
> traceroute to 192.168.20.14 (192.168.20.14), 30 hops max, 60 byte packets
>  1  192.168.20.14 (192.168.20.14)  0.162 ms  0.148 ms  0.141 ms
>
> # traceroute 192.168.29.14
> traceroute to 192.168.29.14 (192.168.29.14), 30 hops max, 60 byte packets
>  1  192.168.29.14 (192.168.29.14)  0.189 ms  0.172 ms  0.244 ms
>
>
Could you precise on which host the command is executed ? It will help a
lot. (Also available for the output above)


>
>
> But (both nodes on different NICs/Subnets):
>
> # gluster peer probe 192.168.29.14
> Probe successful
>
> # gluster volume create test replica 2 transport tcp 192.168.29.14:/mnt/
> 192.168.20.15:/mnt/
> Operation failed on 192.168.29.14
>
>
> Both nodes with connection over the same NIC/Subnet:
>
> # gluster peer probe 192.168.20.14
> Probe successful
>
> # gluster volume create test replica 2 transport tcp 192.168.20.14:/mnt/
> 192.168.20.15:/mnt/
> Creation of volume test has been successful. ...
>
> It also works if I create the volume over the 192.168.29.0/24 subnet. But
> not if I mix them.
>
>
>
>
>
> The log says:
> [2011-12-14 14:53:45.253837] I 
> [glusterd-handler.c:448:**glusterd_handle_cluster_lock]
> 0-glusterd: Received LOCK from uuid: 4e765edc-9ca6-4404-8757-**
> ca170ce938df
> [2011-12-14 14:53:45.253888] I [glusterd-utils.c:243:**glusterd_lock]
> 0-glusterd: Cluster lock held by 4e765edc-9ca6-4404-8757-**ca170ce938df
> [2011-12-14 14:53:45.253938] I 
> [glusterd-handler.c:2651:**glusterd_op_lock_send_resp]
> 0-glusterd: Responded, ret: 0
> [2011-12-14 14:53:45.255287] I 
> [glusterd-handler.c:488:**glusterd_req_ctx_create]
> 0-glusterd: Received op from uuid: 4e765edc-9ca6-4404-8757-**ca170ce938df
> [2011-12-14 14:53:45.256600] E [glusterd-op-sm.c:366:**
> glusterd_op_stage_create_**volume] 0-glusterd: cannot resolve brick:
> 192.168.20.15:/mnt
>

It seems that 192.168.29.14 can't resolv 192.168.20.15. Can you show the
result of "route -n" executed on the host that produces that log ?



> [2011-12-14 14:53:45.256635] E 
> [glusterd-op-sm.c:7370:**glusterd_op_ac_stage_op]
> 0-: Validate failed: 1
> [2011-12-14 14:53:45.256681] I 
> [glusterd-handler.c:2743:**glusterd_op_stage_send_resp]
> 0-glusterd: Responded to stage, ret: 0
> [2011-12-14 14:53:45.256994] I [glusterd-handler.c:2693:**
> glusterd_handle_cluster_**unlock] 0-glusterd: Received UNLOCK from uuid:
> 4e765edc-9ca6-4404-8757-**ca170ce938df
> [2011-12-14 14:53:45.257049] I 
> [glusterd-handler.c:2671:**glusterd_op_unlock_send_resp]
> 0-glusterd: Responded to unlock, ret: 0
>

I don't have much time right now, but I'll try to help you a bit more later.

Regards,

Raphaël.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] Speed of glusterfs

[Raphaël Hoareau]

In theory yes, but the bandwith will also be limited by the nodes between
the client and the server.
Unless every part of your network can handle 2Gbit, it won't be a good
solution.

You could also try to use two NICs with specific routes. NIC1 knows the
route to Server1 and NIC2 knows the route to Server2.
Something like this :
eth0 : route add -net 192.168.0.0 netmask 255.255.255.0 -i eth0#
Assuming 192.168.0.0 = network containing Server1
eth1 : route add -net 192.168.1.0 netmask 255.255.255.0 -i eth1#
Assuming 192.168.1.0 = network containing Server2

But I really don't know if it could work as I never tried it before. And,
as I said, it also depends on the nodes in your network.

The most basic configuration would look like this :

 Server1 Server2
  |   |
  |   |
  1Gbit|   |1Gbit
  |   |
  |   |
eth0 eth1
 Client

In that case, you will double your speed. But again, it's just an idea.

If you have everything to try it (NIC-teaming and/or NIC specific route),
go ahead.
And, please, give me your results as I'm very interested in GlusterFS
performances.

Regards,

Raphaël.

Le 14 décembre 2011 12:55, Marc Muehlfeld <
marc.muehlf...@medizinische-genetik.de> a écrit :

> Am 14.12.2011 11:50, schrieb Raphaël Hoareau:
>
>  The GlusterFS client DOES send the data directly to both storage, that's
>> why your speed is about half the NFS speed.
>>
>
> Can I double the speed of GlusterFS by NIC-teaming (2x 1Gbit)?
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] Speed of glusterfs

[Raphaël Hoareau]

Hi Marc,

The GlusterFS client DOES send the data directly to both storage, that's
why your speed is about half the NFS speed.
Concerning tweaks to write only on one server, and then replicate from one
server to another, I think it's not possible given the architecture of
GlusterFS.
If someone know more details, even tweaks about this... I would be very
interested.

Regards,

Raphaël.

2011/12/14 Marc Muehlfeld 

> Hi,
>
> we're currently testing glusterfs but getting a real slow connection with
> it.
>
> I setup a environment with 2 replicated storage servers. Both machines and
> the client are connected by 1 Gbit, but if I copy a 2.5 GB file from the
> client to the volumen mounted with glusterfs protocol, I never get more
> than 35-40 MB/sec.
>
> On the two storage I also run a NFS server. If I copy the same file from
> the client to one storage by NFS (the NFS server that was shipped with
> RHEL6), I get never less than 90-95 MB/sec. So the network/HDDs are
> possible to provide faster transfer.
>
> Is there a change to get glusterfs to about the same speed? And why is it
> so slow? Does the glusterfs send the data directly to both storages? This
> would explain why it is so slow. But is there a change then to send the
> data just to one server and this one replicates it (over a second NIC) to
> the second one)?
>
>
> Regards,
> Marc
>
> __**_
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://gluster.org/cgi-bin/**mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users