Re: [Gluster-users] sparse files on EC volume

[Dmitri Chebotarov]

Hi Xavi

At this time I'm using 'plain' bricks with XFS. I'll be moving to LVM
cached bricks.
There is no RAID for data bricks, but I'll be using hardware RAID10 for SSD
cache disks (I can use 'writeback' cache in this case).

'small file performance' is the main reason I'm looking at different
options, i.e. using formated sparse files.
I spent considerable amount of time tuning 10GB/kernel/gluster to reduce
latency - the small file performance improved ~50% but it's still no good
enough, especially when I need to use Gluster for /home folders.

I understand limitations and single point of failure in case with sparse
files. I'm considering different options to provide HA (pacemaker/corosync,
keepalived or using VMs - RHEV - to deliver storage).

Thank you for your reply.


On Tue, Sep 26, 2017 at 3:55 AM, Xavi Hernandez  wrote:

> Hi Dmitri,
>
> On 22/09/17 17:07, Dmitri Chebotarov wrote:
>
>>
>> Hello
>>
>> I'm running some tests to compare performance between Gluster FUSE mount
>> and formated sparse files (located on the same Gluster FUSE mount).
>>
>> The Gluster volume is EC (same for both tests).
>>
>> I'm seeing HUGE difference and trying to figure out why.
>>
>
> Could you explain what hardware configuration are you using ?
>
> Do you have a plain disk for each brick formatted in XFS, or do you have
> some RAID configuration ?
>
>
>> Here is an example:
>>
>> GlusterFUSE mount:
>>
>> # cd /mnt/glusterfs
>> # rm -f testfile1 ; dd if=/dev/zero of=testfile1 bs=1G count=1
>> 1+0 records in
>> 1+0 records out
>> 1073741824 bytes (1.1 GB) copied, 9.74757 s, *110 MB/s*
>>
>> Sparse file (located on GlusterFUSE mount):
>>
>> # truncate -l 100GB /mnt/glusterfs/xfs-100G.img
>> # mkfs.xfs /mnt/glusterfs/xfs-100G.img
>> # mount -o loop /mnt/glusterfs/xfs-100G.img /mnt/xfs-100G
>> # cd /mnt/xfs-100G
>> # rm -f testfile1 ; dd if=/dev/zero of=testfile1 bs=1G count=1
>> 1+0 records in
>> 1+0 records out
>> 1073741824 bytes (1.1 GB) copied, 1.20576 s, *891 MB/s*
>>
>> The same goes for working with small files (i.e. code file, make, etc)
>> with the same data located on FUSE mount vs formated sparse file on the
>> same FUSE mount.
>>
>> What would explain such difference?
>>
>
> First of all, doing tests with relatively small files tends to be
> misleading because of caching capacity of the operating system (to minimize
> that, you can add 'conv=fsync' option to dd). You should do tests with file
> sizes bigger than the amount of physical memory on servers. This way you
> minimize cache effects and see the real sustained performance.
>
> A second important point to note is that gluster is a distributed file
> system that can be accessed simultaneously by more than one client. This
> means that consistency must be assured in all cases, which makes things go
> to bricks sooner than local filesystems normally do.
>
> In your case, all data saved to the fuse volume will most probably be
> present on bricks once the dd command completes. On the other side, the
> test through the formatted sparse file, most probably, is keeping most of
> the data in the cache of the client machine.
>
> Note that using the formatted sparse file makes it possible a better use
> of local cache, improving (relatively) small file access, but on the other
> side, this filesystem can only be used from a single client (single mount).
> If this client fails for some reason, you will loose access to your data.
>
>
>> How does Gluster work with sparse files in general? I may move some of
>> the data on gluster volumes to formated sparse files..
>>
>
> Gluster works fine with sparse files. However you should consider the
> previous points before choosing the formatted sparse files option. I guess
> that the sustained throughput will be very similar for bigger files.
>
> Regards,
>
> Xavi
>
>
>> Thank you.
>>
>>
>> ___
>> Gluster-users mailing list
>> Gluster-users@gluster.org
>> http://lists.gluster.org/mailman/listinfo/gluster-users
>>
>>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Adding bricks to an existing installation.

[Karthik Subrahmanya]

Hey Ludwig,

Yes this configuration is fine. You can add them and do the rebalance after
that.

FYI: Replica 2 volumes are prone to split-brains. Replica 3 or arbiter will
highly reduce the possibility of ending up in split-brains.
If possible consider using one of those configurations. For more
information go through [1].

[1] https://gluster.readthedocs.io/en/latest/Administrator%
20Guide/Split%20brain%20and%20ways%20to%20deal%20with%20it/

Thanks & Regards,
Karthik

On Tue, Sep 26, 2017 at 12:48 AM, Ludwig Gamache 
wrote:

> Sharding is not enabled.
>
> Ludwig
>
> On Mon, Sep 25, 2017 at 2:34 PM,  wrote:
>
>> Do you have sharding enabled ? If yes, don't do it.
>> If no I'll let someone who knows better answer you :)
>>
>> On Mon, Sep 25, 2017 at 02:27:13PM -0400, Ludwig Gamache wrote:
>> > All,
>> >
>> > We currently have a Gluster installation which is made of 2 servers.
>> Each
>> > server has 10 drives on ZFS. And I have a gluster mirror between these
>> 2.
>> >
>> > The current config looks like:
>> > SERVER A-BRICK 1 replicated to SERVER B-BRICK 1
>> >
>> > I now need to add more space and a third server. Before I do the
>> changes, I
>> > want to know if this is a supported config. By adding a third server, I
>> > simply want to distribute the load. I don't want to add extra
>> redundancy.
>> >
>> > In the end, I want to have the following done:
>> > Add a peer to the cluster
>> > Add 2 bricks to the cluster (one on server A and one on SERVER C) to the
>> > existing volume
>> > Add 2 bricks to the cluster (one on server B and one on SERVER C) to the
>> > existing volume
>> > After that, I need to rebalance all the data between the bricks...
>> >
>> > Is this config supported? Is there something I should be careful before
>> I
>> > do this? SHould I do a rebalancing before I add the 3 set of disks?
>> >
>> > Regards,
>> >
>> >
>> > Ludwig
>>
>> > ___
>> > Gluster-users mailing list
>> > Gluster-users@gluster.org
>> > http://lists.gluster.org/mailman/listinfo/gluster-users
>>
>>
>> ___
>> Gluster-users mailing list
>> Gluster-users@gluster.org
>> http://lists.gluster.org/mailman/listinfo/gluster-users
>>
>
>
>
> --
> Ludwig Gamache
> IT Director - Element AI
> 4200 St-Laurent, suite 1200
> 514-704-0564
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] df command shows transport endpoint mount error on gluster client v.3.10.5 + core dump

[Ashish Pandey]

Hi, 

Following are the command to get the debug info for gluster - 
gdb /usr/local/sbin/glusterfs   
Then on gdb prompt you need to type bt or backtrace 
(gdb) bt 
You can also provide the output of "thread apply all bt". 
(gdb) thread apply all bt 
Above commands should be executed on client node on which you have mounted the 
gluster volume and it crashed. 
However, I am not sure if you have enabled the generation of core dump on your 
system and have setup path to coredump. 

https://stackoverflow.com/questions/8305866/how-to-analyze-a-programs-core-dump-file-with-gdb
 
https://unix.stackexchange.com/questions/192716/how-to-set-the-core-dump-file-location-and-name
 
https://stackoverflow.com/questions/2065912/core-dumped-but-core-file-is-not-in-current-directory
 

-- 
Ashish 


- Original Message -

From: "Mauro Tridici"  
To: "Ashish Pandey"  
Cc: "Gluster Users"  
Sent: Tuesday, September 26, 2017 1:21:42 PM 
Subject: Re: [Gluster-users] df command shows transport endpoint mount error on 
gluster client v.3.10.5 + core dump 

Hi Ashish, 

thank you for your answer. 
Do you need complete client log file only or something else in particular? 
Unfortunately, I never used “bt” command. Could you please provide me an usage 
example string? 

I will provide all logs you need. 
Thank you again, 
Mauro 




Il giorno 26 set 2017, alle ore 09:30, Ashish Pandey < aspan...@redhat.com > ha 
scritto: 

Hi Mauro, 

We would require complete log file to debug this issue. 
Also, could you please provide some more information of the core after 
attaching to gdb and using command "bt". 

--- 
Ashish 


- Original Message -

From: "Mauro Tridici" < mauro.trid...@cmcc.it > 
To: "Gluster Users" < gluster-users@gluster.org > 
Sent: Monday, September 25, 2017 5:59:10 PM 
Subject: [Gluster-users] df command shows transport endpoint mount error on 
gluster client v.3.10.5 + core dump 

Dear Gluster Users, 

I implemented a distributed disperse 6x(4+2) gluster (v.3.10.5) volume with the 
following options: 

[root@s01 tier2]# gluster volume info 

Volume Name: tier2 
Type: Distributed-Disperse 
Volume ID: a28d88c5-3295-4e35-98d4-210b3af9358c 
Status: Started 
Snapshot Count: 0 
Number of Bricks: 6 x (4 + 2) = 36 
Transport-type: tcp 
Bricks: 
Brick1: s01-stg:/gluster/mnt1/brick 
Brick2: s02-stg:/gluster/mnt1/brick 
Brick3: s03-stg:/gluster/mnt1/brick 
Brick4: s01-stg:/gluster/mnt2/brick 
Brick5: s02-stg:/gluster/mnt2/brick 
Brick6: s03-stg:/gluster/mnt2/brick 
Brick7: s01-stg:/gluster/mnt3/brick 
Brick8: s02-stg:/gluster/mnt3/brick 
Brick9: s03-stg:/gluster/mnt3/brick 
Brick10: s01-stg:/gluster/mnt4/brick 
Brick11: s02-stg:/gluster/mnt4/brick 
Brick12: s03-stg:/gluster/mnt4/brick 
Brick13: s01-stg:/gluster/mnt5/brick 
Brick14: s02-stg:/gluster/mnt5/brick 
Brick15: s03-stg:/gluster/mnt5/brick 
Brick16: s01-stg:/gluster/mnt6/brick 
Brick17: s02-stg:/gluster/mnt6/brick 
Brick18: s03-stg:/gluster/mnt6/brick 
Brick19: s01-stg:/gluster/mnt7/brick 
Brick20: s02-stg:/gluster/mnt7/brick 
Brick21: s03-stg:/gluster/mnt7/brick 
Brick22: s01-stg:/gluster/mnt8/brick 
Brick23: s02-stg:/gluster/mnt8/brick 
Brick24: s03-stg:/gluster/mnt8/brick 
Brick25: s01-stg:/gluster/mnt9/brick 
Brick26: s02-stg:/gluster/mnt9/brick 
Brick27: s03-stg:/gluster/mnt9/brick 
Brick28: s01-stg:/gluster/mnt10/brick 
Brick29: s02-stg:/gluster/mnt10/brick 
Brick30: s03-stg:/gluster/mnt10/brick 
Brick31: s01-stg:/gluster/mnt11/brick 
Brick32: s02-stg:/gluster/mnt11/brick 
Brick33: s03-stg:/gluster/mnt11/brick 
Brick34: s01-stg:/gluster/mnt12/brick 
Brick35: s02-stg:/gluster/mnt12/brick 
Brick36: s03-stg:/gluster/mnt12/brick 
Options Reconfigured: 
features.scrub: Active 
features.bitrot: on 
features.inode-quota: on 
features.quota: on 
performance.client-io-threads: on 
cluster.min-free-disk: 10 
cluster.quorum-type: auto 
transport.address-family: inet 
nfs.disable: on 
server.event-threads: 4 
client.event-threads: 4 
cluster.lookup-optimize: on 
performance.readdir-ahead: on 
performance.parallel-readdir: off 
cluster.readdir-optimize: on 
features.cache-invalidation: on 
features.cache-invalidation-timeout: 600 
performance.stat-prefetch: on 
performance.cache-invalidation: on 
performance.md-cache-timeout: 600 
network.inode-lru-limit: 5 
performance.io -cache: off 
disperse.cpu-extensions: auto 
performance.io -thread-count: 16 
features.quota-deem-statfs: on 
features.default-soft-limit: 90 
cluster.server-quorum-type: server 
cluster.brick-multiplex: on 
cluster.server-quorum-ratio: 51% 

I also started a long write test (about 69 TB to be written) from different 
gluster clients. 
One of this client returned the error "Transport Endpoint is not connected" 
during the rsync copy process. 
A core dump file has been generated by the issue; this is the core dump 
content: 

GNU gdb (GDB) Red Hat Enterprise Linux (7.2-50.el6) 
Copyright (C) 2010 Free Software Foundation, 

Re: [Gluster-users] sparse files on EC volume

[Xavi Hernandez]

Hi Dmitri,

On 22/09/17 17:07, Dmitri Chebotarov wrote:


Hello

I'm running some tests to compare performance between Gluster FUSE mount 
and formated sparse files (located on the same Gluster FUSE mount).


The Gluster volume is EC (same for both tests).

I'm seeing HUGE difference and trying to figure out why.


Could you explain what hardware configuration are you using ?

Do you have a plain disk for each brick formatted in XFS, or do you have 
some RAID configuration ?




Here is an example:

GlusterFUSE mount:

# cd /mnt/glusterfs
# rm -f testfile1 ; dd if=/dev/zero of=testfile1 bs=1G count=1
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB) copied, 9.74757 s, *110 MB/s*

Sparse file (located on GlusterFUSE mount):

# truncate -l 100GB /mnt/glusterfs/xfs-100G.img
# mkfs.xfs /mnt/glusterfs/xfs-100G.img
# mount -o loop /mnt/glusterfs/xfs-100G.img /mnt/xfs-100G
# cd /mnt/xfs-100G
# rm -f testfile1 ; dd if=/dev/zero of=testfile1 bs=1G count=1
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB) copied, 1.20576 s, *891 MB/s*

The same goes for working with small files (i.e. code file, make, etc) 
with the same data located on FUSE mount vs formated sparse file on the 
same FUSE mount.


What would explain such difference?


First of all, doing tests with relatively small files tends to be 
misleading because of caching capacity of the operating system (to 
minimize that, you can add 'conv=fsync' option to dd). You should do 
tests with file sizes bigger than the amount of physical memory on 
servers. This way you minimize cache effects and see the real sustained 
performance.


A second important point to note is that gluster is a distributed file 
system that can be accessed simultaneously by more than one client. This 
means that consistency must be assured in all cases, which makes things 
go to bricks sooner than local filesystems normally do.


In your case, all data saved to the fuse volume will most probably be 
present on bricks once the dd command completes. On the other side, the 
test through the formatted sparse file, most probably, is keeping most 
of the data in the cache of the client machine.


Note that using the formatted sparse file makes it possible a better use 
of local cache, improving (relatively) small file access, but on the 
other side, this filesystem can only be used from a single client 
(single mount). If this client fails for some reason, you will loose 
access to your data.




How does Gluster work with sparse files in general? I may move some of 
the data on gluster volumes to formated sparse files..


Gluster works fine with sparse files. However you should consider the 
previous points before choosing the formatted sparse files option. I 
guess that the sustained throughput will be very similar for bigger files.


Regards,

Xavi



Thank you.


___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users


___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] df command shows transport endpoint mount error on gluster client v.3.10.5 + core dump

[Mauro Tridici]

Hi Ashish,

thank you for your answer.
Do you need complete client log file only or something else in particular?
Unfortunately, I never used “bt” command. Could you please provide me an usage 
example string?

I will provide all logs you need.
Thank you again,
Mauro

> Il giorno 26 set 2017, alle ore 09:30, Ashish Pandey  ha 
> scritto:
> 
> Hi Mauro,
> 
> We would require complete log file to debug this issue.
> Also, could you please provide some more information of the core after 
> attaching to gdb and using command "bt".
> 
> ---
> Ashish
> 
> 
> From: "Mauro Tridici" 
> To: "Gluster Users" 
> Sent: Monday, September 25, 2017 5:59:10 PM
> Subject: [Gluster-users] df command shows transport endpoint mount error on   
>  gluster client v.3.10.5 + core dump
> 
> Dear Gluster Users,
> 
> I implemented a distributed disperse 6x(4+2) gluster (v.3.10.5) volume with 
> the following options:
> 
> [root@s01 tier2]# gluster volume info
>  
> Volume Name: tier2
> Type: Distributed-Disperse
> Volume ID: a28d88c5-3295-4e35-98d4-210b3af9358c
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 6 x (4 + 2) = 36
> Transport-type: tcp
> Bricks:
> Brick1: s01-stg:/gluster/mnt1/brick
> Brick2: s02-stg:/gluster/mnt1/brick
> Brick3: s03-stg:/gluster/mnt1/brick
> Brick4: s01-stg:/gluster/mnt2/brick
> Brick5: s02-stg:/gluster/mnt2/brick
> Brick6: s03-stg:/gluster/mnt2/brick
> Brick7: s01-stg:/gluster/mnt3/brick
> Brick8: s02-stg:/gluster/mnt3/brick
> Brick9: s03-stg:/gluster/mnt3/brick
> Brick10: s01-stg:/gluster/mnt4/brick
> Brick11: s02-stg:/gluster/mnt4/brick
> Brick12: s03-stg:/gluster/mnt4/brick
> Brick13: s01-stg:/gluster/mnt5/brick
> Brick14: s02-stg:/gluster/mnt5/brick
> Brick15: s03-stg:/gluster/mnt5/brick
> Brick16: s01-stg:/gluster/mnt6/brick
> Brick17: s02-stg:/gluster/mnt6/brick
> Brick18: s03-stg:/gluster/mnt6/brick
> Brick19: s01-stg:/gluster/mnt7/brick
> Brick20: s02-stg:/gluster/mnt7/brick
> Brick21: s03-stg:/gluster/mnt7/brick
> Brick22: s01-stg:/gluster/mnt8/brick
> Brick23: s02-stg:/gluster/mnt8/brick
> Brick24: s03-stg:/gluster/mnt8/brick
> Brick25: s01-stg:/gluster/mnt9/brick
> Brick26: s02-stg:/gluster/mnt9/brick
> Brick27: s03-stg:/gluster/mnt9/brick
> Brick28: s01-stg:/gluster/mnt10/brick
> Brick29: s02-stg:/gluster/mnt10/brick
> Brick30: s03-stg:/gluster/mnt10/brick
> Brick31: s01-stg:/gluster/mnt11/brick
> Brick32: s02-stg:/gluster/mnt11/brick
> Brick33: s03-stg:/gluster/mnt11/brick
> Brick34: s01-stg:/gluster/mnt12/brick
> Brick35: s02-stg:/gluster/mnt12/brick
> Brick36: s03-stg:/gluster/mnt12/brick
> Options Reconfigured:
> features.scrub: Active
> features.bitrot: on
> features.inode-quota: on
> features.quota: on
> performance.client-io-threads: on
> cluster.min-free-disk: 10
> cluster.quorum-type: auto
> transport.address-family: inet
> nfs.disable: on
> server.event-threads: 4
> client.event-threads: 4
> cluster.lookup-optimize: on
> performance.readdir-ahead: on
> performance.parallel-readdir: off
> cluster.readdir-optimize: on
> features.cache-invalidation: on
> features.cache-invalidation-timeout: 600
> performance.stat-prefetch: on
> performance.cache-invalidation: on
> performance.md-cache-timeout: 600
> network.inode-lru-limit: 5
> performance.io-cache: off
> disperse.cpu-extensions: auto
> performance.io-thread-count: 16
> features.quota-deem-statfs: on
> features.default-soft-limit: 90
> cluster.server-quorum-type: server
> cluster.brick-multiplex: on
> cluster.server-quorum-ratio: 51%
> 
> I also started a long write test (about 69 TB to be written) from different 
> gluster clients.
> One of this client returned the error "Transport Endpoint is not connected" 
> during the rsync copy process.
> A core dump file has been generated by the issue; this is the core dump 
> content:
> 
> GNU gdb (GDB) Red Hat Enterprise Linux (7.2-50.el6)
> Copyright (C) 2010 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later 
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-redhat-linux-gnu".
> For bug reporting instructions, please see:
> ...
> Missing separate debuginfo for the main executable file
> Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install 
> /usr/lib/debug/.build-id/fb/4d988b681faa09bb74becacd7a24f4186e8185
> [New Thread 15802]
> [New Thread 15804]
> [New Thread 15805]
> [New Thread 6856]
> [New Thread 30432]
> [New Thread 30486]
> [New Thread 1619]
> [New Thread 15806]
> [New Thread 15810]
> [New Thread 30412]
> [New Thread 15809]
> [New Thread 15799]
> [New Thread 30487]
> [New Thread 15795]
> [New Thread 15797]
> [New Thread 15798]
> [New Thread 15800]
> [New Thread 15801]
> Core was generated by `/usr/sbin/glusterfs 

Re: [Gluster-users] df command shows transport endpoint mount error on gluster client v.3.10.5 + core dump

[Ashish Pandey]

Hi Mauro, 

We would require complete log file to debug this issue. 
Also, could you please provide some more information of the core after 
attaching to gdb and using command "bt". 

--- 
Ashish 


- Original Message -

From: "Mauro Tridici"  
To: "Gluster Users"  
Sent: Monday, September 25, 2017 5:59:10 PM 
Subject: [Gluster-users] df command shows transport endpoint mount error on 
gluster client v.3.10.5 + core dump 

Dear Gluster Users, 

I implemented a distributed disperse 6x(4+2) gluster (v.3.10.5) volume with the 
following options: 

[root@s01 tier2]# gluster volume info 

Volume Name: tier2 
Type: Distributed-Disperse 
Volume ID: a28d88c5-3295-4e35-98d4-210b3af9358c 
Status: Started 
Snapshot Count: 0 
Number of Bricks: 6 x (4 + 2) = 36 
Transport-type: tcp 
Bricks: 
Brick1: s01-stg:/gluster/mnt1/brick 
Brick2: s02-stg:/gluster/mnt1/brick 
Brick3: s03-stg:/gluster/mnt1/brick 
Brick4: s01-stg:/gluster/mnt2/brick 
Brick5: s02-stg:/gluster/mnt2/brick 
Brick6: s03-stg:/gluster/mnt2/brick 
Brick7: s01-stg:/gluster/mnt3/brick 
Brick8: s02-stg:/gluster/mnt3/brick 
Brick9: s03-stg:/gluster/mnt3/brick 
Brick10: s01-stg:/gluster/mnt4/brick 
Brick11: s02-stg:/gluster/mnt4/brick 
Brick12: s03-stg:/gluster/mnt4/brick 
Brick13: s01-stg:/gluster/mnt5/brick 
Brick14: s02-stg:/gluster/mnt5/brick 
Brick15: s03-stg:/gluster/mnt5/brick 
Brick16: s01-stg:/gluster/mnt6/brick 
Brick17: s02-stg:/gluster/mnt6/brick 
Brick18: s03-stg:/gluster/mnt6/brick 
Brick19: s01-stg:/gluster/mnt7/brick 
Brick20: s02-stg:/gluster/mnt7/brick 
Brick21: s03-stg:/gluster/mnt7/brick 
Brick22: s01-stg:/gluster/mnt8/brick 
Brick23: s02-stg:/gluster/mnt8/brick 
Brick24: s03-stg:/gluster/mnt8/brick 
Brick25: s01-stg:/gluster/mnt9/brick 
Brick26: s02-stg:/gluster/mnt9/brick 
Brick27: s03-stg:/gluster/mnt9/brick 
Brick28: s01-stg:/gluster/mnt10/brick 
Brick29: s02-stg:/gluster/mnt10/brick 
Brick30: s03-stg:/gluster/mnt10/brick 
Brick31: s01-stg:/gluster/mnt11/brick 
Brick32: s02-stg:/gluster/mnt11/brick 
Brick33: s03-stg:/gluster/mnt11/brick 
Brick34: s01-stg:/gluster/mnt12/brick 
Brick35: s02-stg:/gluster/mnt12/brick 
Brick36: s03-stg:/gluster/mnt12/brick 
Options Reconfigured: 
features.scrub: Active 
features.bitrot: on 
features.inode-quota: on 
features.quota: on 
performance.client-io-threads: on 
cluster.min-free-disk: 10 
cluster.quorum-type: auto 
transport.address-family: inet 
nfs.disable: on 
server.event-threads: 4 
client.event-threads: 4 
cluster.lookup-optimize: on 
performance.readdir-ahead: on 
performance.parallel-readdir: off 
cluster.readdir-optimize: on 
features.cache-invalidation: on 
features.cache-invalidation-timeout: 600 
performance.stat-prefetch: on 
performance.cache-invalidation: on 
performance.md-cache-timeout: 600 
network.inode-lru-limit: 5 
performance.io-cache: off 
disperse.cpu-extensions: auto 
performance.io-thread-count: 16 
features.quota-deem-statfs: on 
features.default-soft-limit: 90 
cluster.server-quorum-type: server 
cluster.brick-multiplex: on 
cluster.server-quorum-ratio: 51% 

I also started a long write test (about 69 TB to be written) from different 
gluster clients. 
One of this client returned the error "Transport Endpoint is not connected" 
during the rsync copy process. 
A core dump file has been generated by the issue; this is the core dump 
content: 

GNU gdb (GDB) Red Hat Enterprise Linux (7.2-50.el6) 
Copyright (C) 2010 Free Software Foundation, Inc. 
License GPLv3+: GNU GPL version 3 or later  
This is free software: you are free to change and redistribute it. 
There is NO WARRANTY, to the extent permitted by law. Type "show copying" 
and "show warranty" for details. 
This GDB was configured as "x86_64-redhat-linux-gnu". 
For bug reporting instructions, please see: 
... 
Missing separate debuginfo for the main executable file 
Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install 
/usr/lib/debug/.build-id/fb/4d988b681faa09bb74becacd7a24f4186e8185 
[New Thread 15802] 
[New Thread 15804] 
[New Thread 15805] 
[New Thread 6856] 
[New Thread 30432] 
[New Thread 30486] 
[New Thread 1619] 
[New Thread 15806] 
[New Thread 15810] 
[New Thread 30412] 
[New Thread 15809] 
[New Thread 15799] 
[New Thread 30487] 
[New Thread 15795] 
[New Thread 15797] 
[New Thread 15798] 
[New Thread 15800] 
[New Thread 15801] 
Core was generated by `/usr/sbin/glusterfs --volfile-server=s01-stg 
--volfile-id=/tier2 /tier2'. 
Program terminated with signal 6, Aborted. 
#0 0x0032a74328a5 in ?? () 
"/core.15795" is a core file. 
Please specify an executable to debug. 

No problem detected on the gluster servers, no brick down and so on... 

Anyone of us experienced the same problem? 
If yes, how did you resolve it? 

You can find the client "/var/log/messages" and "/var/log/glusterfs" log files 
content. 
Thank you in advance. 
Mauro Tridici