[Gluster-users] gluster + interruptions causing high load

2012-04-21 Thread Alex Florescu 
Hello again,

I am now seeking some advices regarding gluster configuration.
I have the following options:
Options Reconfigured:
nfs.disable: 1
network.ping-timeout: 1

>From my tests, I noticed that setting a low network.ping-timeout (1s)
causes gluster to declare the other node dead, and quickly allow access to
the common storage content. But in the last 4 days I have been experiencing
some weird occurrences. I am using gluster in WAN environment replication,
and I am suspecting network connectivity is sometimes timing out.
This causes gluster to stall for ~ 1 minute, like the log says:
[2012-04-21 18:11:31.758139] C
[client-handshake.c:121:rpc_client_ping_timer_expired]
0-comun-vol-client-0: server xxx.xxx.xxx.xxx:24009 has not responded in the
last 42 seconds, disconnecting.
and also causes big load on the server (100+), freezing all other
applications.

I have now reconfigured network.ping-timeout to 10.
What I would like to achieve is to have minimum "downtime" when bricks are
failing. Any other options I can set?
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] Performance issues with striped volume over Infiniband

2012-04-21 Thread Harry Mangalam 
try 'ifstat' to see traffic on all interfaces simultaneously:

501 $ ifstat

  
   eth2ib1
 KB/s in  KB/s out   KB/s in  KB/s out
0.80  0.70  0.00  0.00
0.19  0.15  0.00  0.00
0.07  0.15  0.00  0.00

pkg ifstat in debian/ubuntu.

hjm

On Saturday 21 April 2012 02:45:10 Ionescu, A. wrote:
> Michael,
> 
> Thanks for your suggestion. I had the same intuition as you, but
> then I used "iptraf" and saw no eth0 traffic associated with I/O
> on the Gluster volume. (the tool doesn't show the ib0 interface,
> unfortunately). node01 and node02 are entered into /etc/hosts,
> they resolve to the ipoib addresses and are pingable.
> 
> I will try increasing the number of threads and applying the patch
> Bryan suggested.
> 
> Thanks,
> Adrian

-- 
Harry Mangalam - Research Computing, OIT, Rm 225 MSTB, UC Irvine
[ZOT 2225] / 92697  Google Voice Multiplexer: (949) 478-4487
415 South Circle View Dr, Irvine, CA, 92697 [shipping]
MSTB Lat/Long: (33.642025,-117.844414) (paste into Google Maps)
--
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] multi-interface/IP peers

2012-04-21 Thread David Coulson 
Gluster relies on DNS and/or /etc/hosts to determine the IP for a 
particular cluster member. You can have gluster utilize a different IP 
for *new* connections by updating DNS or /etc/hosts to point the cluster 
peer name to a new IP.


On 4/21/12 7:31 AM, lejeczek wrote:

helo everybody

this I'd imagine must be common scenario, where a peer, or more peers 
are multi if/IPs nodes

for an instance
what happens if a cheaper route to a peer is made available after some 
time a volume has been up?
how one introduces this change to the volume? how one tells the 
gluster to use different IP whereas everything else remains unchanged?


cheers
lejeczek




___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

[Gluster-users] multi-interface/IP peers

2012-04-21 Thread lejeczek 

helo everybody

this I'd imagine must be common scenario, where a peer, or 
more peers are multi if/IPs nodes

for an instance
what happens if a cheaper route to a peer is made available 
after some time a volume has been up?
how one introduces this change to the volume? how one tells 
the gluster to use different IP whereas everything else 
remains unchanged?


cheers
lejeczek


___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] Performance issues with striped volume over Infiniband

2012-04-21 Thread Ionescu, A. 
Michael,

Thanks for your suggestion. I had the same intuition as you, but then I used 
"iptraf" and saw no eth0 traffic associated with I/O on the Gluster volume. 
(the tool doesn't show the ib0 interface, unfortunately). node01 and node02 are 
entered into /etc/hosts, they resolve to the ipoib addresses and are pingable.

I will try increasing the number of threads and applying the patch Bryan 
suggested.

Thanks,
Adrian

From: Michael Mayer [mich...@mayer.cx]
Sent: 21 April 2012 10:21
To: Bryan Whitehead
Cc: Ionescu, A.; gluster-users@gluster.org
Subject: Re: [Gluster-users] Performance issues with striped volume over 
Infiniband

Adrian,

I think you should really try to restart from scratch, i.e.
- setup ipoib with corresponding entries in DNS or /etc/hosts (make sure
the IB hosts are pingable)
- gluster peer probe on the ipoib hostnames
- recreate volume using ipoib hostnames for the bricks
For transport you can still use tcp (I usually set tcp,rdma to be
flexible later on)

I started naively myself a couple of weeks ago and did exactly the same
than you assuming that IB transport would be autodetected. In fact, the
100 MB/s you are seeing indicated very strongly that any communication
is still over Gigabit Ethernet (Which has a theoretical peak performance
of 125 MB/s but given switch quality and protocol overhead you will
reach more like 100-110 MB/s in reality)

Michael.

On 04/20/2012 07:42 PM, Bryan Whitehead wrote:
> Max out the number of IO threads and apply a patch to make gluster
> more agressive about spawning threads as in this thread:
>
> http://gluster.org/pipermail/gluster-users/2012-February/009590.html
>
> (the above thread is actually pretty good for getting performance out
> of gluster with infiniband (I use IP over IB myself - could never get
> the native rDMA to be fast enough).
>
> On Fri, Apr 20, 2012 at 3:27 AM, Ionescu, A.  wrote:
>> I tried this, but the performance is still the same (about ~100MB/s).
>>
>> A strange thing I noticed is that the node on which I run the benchmarking 
>> tool reports ~350 MB/s traffic on the loopback interface.
>>
>> Does anyone have any other suggestions?
>>
>> Thanks,
>> Adrian
>> 
>> From: gluster-users-boun...@gluster.org [gluster-users-boun...@gluster.org] 
>> on behalf of Sabuj Pattanayek [sab...@gmail.com]
>> Sent: 19 April 2012 22:32
>> To: gluster-users@gluster.org
>> Subject: Re: [Gluster-users] Performance issues with striped volume over 
>>Infiniband
>>
>> On Thu, Apr 19, 2012 at 2:00 PM, Ionescu, A.  wrote:
>>> Thanks for your answer, Sabuj. However, I am not sure I understand what you 
>>> mean by trying with ipoib.
>>> Do you mean specifying "transport tcp" and using the ipoib ips/hostnames 
>>> for the bricks?
>>> If you manage to find a way to overcome this performance limitation, please 
>>> share with us your solution.
>> Yes.
>> ___
>> Gluster-users mailing list
>> Gluster-users@gluster.org
>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>> ___
>> Gluster-users mailing list
>> Gluster-users@gluster.org
>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] Performance gluster 3.2.5 + QLogic Infiniband

2012-04-21 Thread Michael Mayer 

Hi all,

thanks for your suggestions,

i think I have "solved" the performance issue now. I had a few too many 
kernel patches included. I am back to the stock RHEL 5.8 kernel with 
stock QLogic OFED and everything works ok (IPoIB). My original intent 
was to explore cachefs on RHEL5 by building a 2.6.32 kernel but while 
cachefs worked like a treat performance for gluster was as bad as 
reported previously - so will go without cachefs for now and reintroduce 
cachefs in an OS upgrade later on.


I even have a nicely working rdma setup now and - using that - 
performance is 900 MB/s + and that consistently so.


Since I have two volumes exported by the same bricks it seems I only can 
get one of them to use RDMA, the other will then refuse to mount and 
only mount if not using rdma on that one - but that is not a real 
problem for now as the second one is only used for backup purposes.


Michael,

On 04/12/2012 01:13 AM, Fabricio Cannini wrote:

Hi there

The only time i setup a gluster "distributed scratch" like Michael is 
doing,
( 3.0.5 Debian packages ) i too choose IPoIB simply because i could 
not get rdma working at all.
Time was short and IPoIB "Just worked" well enough for our demand at 
the time, so i didn't looked into this issue. Plus, pinging and 
ssh'ing into a node through the IB interface comes handy when 
diagnosing and fixing networking issues.


Em quarta-feira, 11 de abril de 2012, Sabuj 
Pattanayekmailto:sab...@gmail.com>> escreveu:

> I wonder if it's possible to have both rdma and ipoib served by a
> single glusterfsd so I can test this? I guess so, since it's just a
> tcp mount?
>
> On Wed, Apr 11, 2012 at 1:43 PM, Harry Mangalam 
mailto:harry.manga...@uci.edu>> wrote:

>> On Tuesday 10 April 2012 15:47:08 Bryan Whitehead wrote:
>>
>>> with my infiniband setup I found my performance was much better by
>>
>>> setting up a TCP network over infiniband and then using pure tcp as
>>
>>> the transport with my gluster volume. For the life of me I couldn't
>>
>>> get rdma to beat tcp.
>>
>> Thanks for that data point, Brian.
>>
>> Very interesting. Is this a common experience? The RDMA experience 
has not
>> been a very smooth one for me and doing everything with IPoIB would 
save a

>> lot of headaches, especially if it's also higher performance.
>>
>> hjm
>>
>> --
>>
>> Harry Mangalam - Research Computing, OIT, Rm 225 MSTB, UC Irvine
>>
>> [ZOT 2225] / 92697 Google Voice Multiplexer: (949) 478-4487
>>
>> 415 South Circle View Dr, Irvine, CA, 92697 [shipping]
>>
>> MSTB Lat/Long: (33.642025,-117.844414) (paste into Google Maps)
>>
>> --
>>
>>
>> ___
>> Gluster-users mailing list
>> Gluster-users@gluster.org 
>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org 
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>


___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users


___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] Performance issues with striped volume over Infiniband

2012-04-21 Thread Michael Mayer 

Adrian,

I think you should really try to restart from scratch, i.e.
- setup ipoib with corresponding entries in DNS or /etc/hosts (make sure 
the IB hosts are pingable)

- gluster peer probe on the ipoib hostnames
- recreate volume using ipoib hostnames for the bricks
For transport you can still use tcp (I usually set tcp,rdma to be 
flexible later on)


I started naively myself a couple of weeks ago and did exactly the same 
than you assuming that IB transport would be autodetected. In fact, the 
100 MB/s you are seeing indicated very strongly that any communication 
is still over Gigabit Ethernet (Which has a theoretical peak performance 
of 125 MB/s but given switch quality and protocol overhead you will 
reach more like 100-110 MB/s in reality)


Michael.

On 04/20/2012 07:42 PM, Bryan Whitehead wrote:

Max out the number of IO threads and apply a patch to make gluster
more agressive about spawning threads as in this thread:

http://gluster.org/pipermail/gluster-users/2012-February/009590.html

(the above thread is actually pretty good for getting performance out
of gluster with infiniband (I use IP over IB myself - could never get
the native rDMA to be fast enough).

On Fri, Apr 20, 2012 at 3:27 AM, Ionescu, A.  wrote:

I tried this, but the performance is still the same (about ~100MB/s).

A strange thing I noticed is that the node on which I run the benchmarking tool 
reports ~350 MB/s traffic on the loopback interface.

Does anyone have any other suggestions?

Thanks,
Adrian

From: gluster-users-boun...@gluster.org [gluster-users-boun...@gluster.org] on 
behalf of Sabuj Pattanayek [sab...@gmail.com]
Sent: 19 April 2012 22:32
To: gluster-users@gluster.org
Subject: Re: [Gluster-users] Performance issues with striped volume over
Infiniband

On Thu, Apr 19, 2012 at 2:00 PM, Ionescu, A.  wrote:

Thanks for your answer, Sabuj. However, I am not sure I understand what you 
mean by trying with ipoib.
Do you mean specifying "transport tcp" and using the ipoib ips/hostnames for 
the bricks?
If you manage to find a way to overcome this performance limitation, please 
share with us your solution.

Yes.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users


___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users