[Gluster-users] simultaneous Gbe/IPoIB clients?

[Harry Mangalam]

We have a use case in our clusters where we're adding IB to some racks  
and would like to use gluster storage over IB in that mix.  If it 
works out on the cluster case, it would be a good argument to scale 
this out much further.  

So, is there an elegant, best-practices-way to use a gluster volume 
that was established using the IPoIB net to provide storage to the 
IPoGBe clients without routing thru a single gateway (and therefore 
losing the multipath advantage of gluster) or creating a DNS hairball?

ie: The IPoIB net is 1.2.3.0 (host names are referenced as 'pbsXib') 
and the IPoGbe net is 1.2.4.0 (names are 'pbsXgb'). All gluster 
storage nodes have 1 IB port on 1.2.3.0 and one Gbe port on 1.2.4.0)

The gluster volume was set up as follows and the host names are 
defined in the local /etc/hosts files for now):

gluster volume create gl \
transport tcp,rdma \
  pbs1ib:/bducgl \
  pbs2ib:/bducgl \
  pbs3ib:/bducgl \
  pbs4ib:/bducgl 

The IPoIB clients (which also have the same /etc/hosts info) have no 
problem accessing the volume as 'pbs1ib:/gl' (and have good large-file 
IO, maxing out at about 250MB/s on a single file copy)

The problem (from the gl.log (5.8K):


seems to be that the Gbe gluster client can contact the Gbe side of 
the gluster volume but cannot resolve the gluster volume since the 
gluster volume was created with pbsXib names.

I can bypass the DNS by pointing the pbsXib names -> pbsXgb addresses 
on a client (in /etc/hosts) and in fact this works fine and I have the 
gluster volume reading and writing clients from both IB and Gbe.

The problem is that this leaves a fragile, tangled nest of DNS 
overrides and manual edits.  Setting this up in the DNS tables would 
solve most of this but the servers would still have to have overrides 
set up in their /etc/hosts to override the DNS.

Is there a more elegant way of doing this?
This is 3.3b3, (which was fairly easy to set up).

Also, with dual-port IB cards, is it recommended to do channel-bonding 
or to use different IP #s?

hjm

-- 
Harry Mangalam - Research Computing, OIT, Rm 225 MSTB, UC Irvine
[ZOT 2225] / 92697  Google Voice Multiplexer: (949) 478-4487
415 South Circle View Dr, Irvine, CA, 92697 [shipping]
MSTB Lat/Long: (33.642025,-117.844414) (paste into Google Maps)
--
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] Recovering out of sync nodes from input/output error

[Jeff Darcy]

FYI, this is now being tracked as

https://bugzilla.redhat.com/show_bug.cgi?id=812963
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] Gluster-users Digest, Vol 48, Issue 18 - Horrible Gluster Performance

[Ben England]

Philip,

What parts of your system perform well?   Can you give a specific example of 
your workload (what you are asking system to do)?  If it's a mixture of 
different workloads that's important too.  What version of Gluster and Linux 
are you using?  My suggestions would be 

a) to reset all your gluster tuning parameters to their default values unless 
you are sure that they actually improve performance, and 

b) try to isolate your performance problem to as simple a workload as possible 
before you try to fix it, and try to determine what workloads DO work well in 
your configuration.  This will make it easier for others to help.  

c) if latency spikes are the issue, this sounds like it could be related to 
writes being excessively buffered by Linux kernel and then being flushed all at 
once, which can block reads.  If so, Use "iostat -kx /dev/sd? 5" or equivalent 
to observe.  You can throttle back "dirty pages" in kernel and avoid buffering 
dirty pages for long periods of time to avoid these spikes.  

http://community.gluster.org/a/linux-kernel-tuning-for-glusterfs/ provides some 
suggestions that may be relevant to your problem, my recommendations are in a 
comment here.  

>Message: 9
>Date: Fri, 13 Apr 2012 11:25:58 +0200
>From: Philip 
>Subject: [Gluster-users] Horrible Gluster Performance
>To: gluster-users@gluster.org
>Message-ID:

>Content-Type: text/plain; charset="iso-8859-1"

>I have a small GlusterFS Cluster providing a replicated volume. Each server
>has 2 SAS disks for the OS and logs and 22 SATA disks for the actual data
>striped together as a RAID10 using MegaRAID SAS 9280-4i4e with this
>configuration: http://pastebin.com/2xj4401J

>Connected to this cluster are a few other servers with the native client
>running nginx to serve files stored on it in the order of 3-10MB.

>Right now a storage server has a outgoing bandwith of 300Mbit/s and the
>busy rate of the raid array is at 30-40%. There are also strange
>side-effects: Sometimes the io-latency skyrockets and there is no access
>possible on the raid for >10 seconds. This happens at 300Mbit/s or
>1000Mbit/s of outgoing bandwidth. The file system used is xfs and it has
>been tuned to match the raid stripe size.

>I've tested all sorts of gluster settings but none seem to have any effect
>because of that I've reset the volume configuration and it is using the
>default one.

>Does anyone have an idea what could be the reason for such a bad
>performance? 22 Disks in a RAID10 should deliver *way* more throughput.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

[Gluster-users] Compiling an running glusterfs on cygwin, host w2008 64X

[Daniel Müller]

Hello to all,

some thoughts. Did someone succeed to compile and run glusterfs on
cygwin/W2008?!

Greetings 
Daniel

---
EDV Daniel Müller

Leitung EDV
Tropenklinik Paul-Lechler-Krankenhaus
Paul-Lechler-Str. 24
72076 Tübingen

Tel.: 07071/206-463, Fax: 07071/206-499
eMail: muel...@tropenklinik.de
Internet: www.tropenklinik.de
---

___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

[Gluster-users] Best practice and various system values

[anthony garnier]

Hi all,

I was just wondering if there is any best practice when you have a private 
Backend for GlusterFS ?

I also wanted to know various system values (like ulimit) you may have

Here is mine : 
# ulimit -a
core file size  (blocks, -c) 1
data seg size   (kbytes, -d) unlimited
scheduling priority (-e) 0
file size   (blocks, -f) unlimited
pending signals (-i) 1029408
max locked memory   (kbytes, -l) 64
max memory size (kbytes, -m) 112008152
open files  (-n) 1024
pipe size(512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority  (-r) 0
stack size  (kbytes, -s) 8192
cpu time   (seconds, -t) unlimited
max user processes  (-u) unlimited
virtual memory  (kbytes, -v) 107097200
file locks  (-x) unlimited

vm.swappiness=60 
vm.vfs_cache_pressure=100
vm.dirty_background_ratio=10
vm.dirty_ratio  =40
sys/block//queue/scheduler = cfq (ssd drive)
/sys/block//queue/nr_requests = 128
 /proc/sys/vm/page-cluster   = 3
/proc/sys/net/ipv4/tcp_fin_timeout =30
/proc/sys/net/ipv4/tcp_rmem=4096262144  4194304
/proc/sys/net/ipv4/tcp_wmem  =4096262144  4194304
/proc/sys/net/ipv4/tcp_retries2 =15
/proc/sys/net/ipv4/tcp_keepalive_intvl   =75


Regards,

Anthony

  ___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users