Re: [Gluster-users] stripped volume in 3.4.0qa5 with horrible read performance

[samuel]

Done.

https://bugzilla.redhat.com/show_bug.cgi?id=888174

While testing the system, we found 3.3.0 enables stripped-replicated
volumes and seems to offer a "right" read behaviour in some tests.

Thanks in advance and, please, contact me in case I can offer further help.

Best regards,
Samuel.

On 17 December 2012 16:20, John Mark Walker  wrote:

> Please file a bug. There might be time to fix read performance before the
> 1st beta release.
>
> -JM
>
>
> --
>
> Dear folks,
>
> I've been tried to use replicated stripped volumes with 3.3.
> unsuccessfully due to https://bugzilla.redhat.com/show_bug.cgi?id=861423and I 
> then proceed to try 3.4.0qa5. I then find out that the bug was solved
> and I could use replicated stripped volume with the new version. Amazingly,
> write performance was quite astonishing.
>
> The problem I'm facing now is in the read process: It's horribly slow.
> When I open a file to edit using the gluster native client, it takes a few
> seconds and sometimes I got an error refering to file has been modified
> while I was editing it. There's a ruby application reading the files and I
> got continuously timeout errors.
>
> I'm using 4 bricks with Centos 6.3 with the following structure:
> Type: Striped-Replicate
> Volume ID: 23dbb8dd-5cb3-4c71-9702-7c16ee9a3b3b
> Status: Started
> Number of Bricks: 1 x 2 x 2 = 4
> Transport-type: tcp
> Bricks:
> Brick1: 10.0.51.31:/gfs
> Brick2: 10.0.51.32:/gfs
> Brick3: 10.0.51.33:/gfs
> Brick4: 10.0.51.34:/gfs
> Options Reconfigured:
> performance.quick-read: on
> performance.io-thread-count: 32
> performance.cache-max-file-size: 128MB
> performance.cache-size: 256MB
> performance.io-cache: on
> cluster.stripe-block-size: 2MB
> nfs.disable: on
>
> I started profiling and found out one node with absurd latency figures. I
> stopped the node and the problem moved to another brick:
>  %-latency   Avg-latency   Min-Latency   Max-Latency   No. of
> calls Fop
>  -   ---   ---   ---   
> 
> 99.94  551292.41 us  10.00 us 1996709.00 us361FINODELK
>
> Could anyone provide some information how to debug this problem? Currently
> the volume is not usable due to the horrible delay.
>
> Thank you very much in advance,
> Samuel.
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>
>
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Infiniband performance issues answered?

[Bryan Whitehead]

does anyone have 3.4.0qa5 rpm's available? I'd like to give them a whirl.


On Mon, Dec 17, 2012 at 5:17 PM, Sabuj Pattanayek  wrote:

> and yes on some Dells you'll get strange network and RAID controller
> performance characteristics if you turn on the BIOS power management.
>
> On Mon, Dec 17, 2012 at 7:15 PM, Sabuj Pattanayek 
> wrote:
> > I have R610's with a similar setup but with HT turned on and I'm
> > getting 3.5GB/s for one way RDMA tests between two QDR connected
> > clients using mellanox connectx x4 PCI-E cards in x8 slots. 1GB/s with
> > IPoIB connections (seem to be limited to 10gbe). Note, I had problems
> > with the 1.x branch of OFED and am using the latest 3.x RC .
> >
> > On Mon, Dec 17, 2012 at 6:44 PM, Joe Julian 
> wrote:
> >> In IRC today, someone who was hitting that same IB performance ceiling
> that
> >> occasionally gets reported had this to say
> >>
> >> [11:50]  first, I ran fedora which is not supported by Mellanox
> OFED
> >> distro
> >> [11:50]  so I moved to CentOS 6.3
> >> [11:51]  next I removed all distibution related infiniband rpms
> and
> >> build the latest OFED package
> >> [11:52]  disabled ServerSpeed service
> >> [11:52]  disabled BIOS hyperthreading
> >> [11:52]  disabled BIOS power mgmt
> >> [11:53]  ran ib_write_test and goot 5000MB/s
> >> [11:53]  got 5000MB/s on localhost
> >>
> >> fwiw, if someone's encountering that issue, between this and the changes
> >> since 3.4.0qa5 it might be worth knowing about.
> >>
> >> http://irclog.perlgeek.de/gluster/2012-12-17#i_6251387
> >> ___
> >> Gluster-users mailing list
> >> Gluster-users@gluster.org
> >> http://supercolony.gluster.org/mailman/listinfo/gluster-users
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Infiniband performance issues answered?

[Sabuj Pattanayek]

and yes on some Dells you'll get strange network and RAID controller
performance characteristics if you turn on the BIOS power management.

On Mon, Dec 17, 2012 at 7:15 PM, Sabuj Pattanayek  wrote:
> I have R610's with a similar setup but with HT turned on and I'm
> getting 3.5GB/s for one way RDMA tests between two QDR connected
> clients using mellanox connectx x4 PCI-E cards in x8 slots. 1GB/s with
> IPoIB connections (seem to be limited to 10gbe). Note, I had problems
> with the 1.x branch of OFED and am using the latest 3.x RC .
>
> On Mon, Dec 17, 2012 at 6:44 PM, Joe Julian  wrote:
>> In IRC today, someone who was hitting that same IB performance ceiling that
>> occasionally gets reported had this to say
>>
>> [11:50]  first, I ran fedora which is not supported by Mellanox OFED
>> distro
>> [11:50]  so I moved to CentOS 6.3
>> [11:51]  next I removed all distibution related infiniband rpms and
>> build the latest OFED package
>> [11:52]  disabled ServerSpeed service
>> [11:52]  disabled BIOS hyperthreading
>> [11:52]  disabled BIOS power mgmt
>> [11:53]  ran ib_write_test and goot 5000MB/s
>> [11:53]  got 5000MB/s on localhost
>>
>> fwiw, if someone's encountering that issue, between this and the changes
>> since 3.4.0qa5 it might be worth knowing about.
>>
>> http://irclog.perlgeek.de/gluster/2012-12-17#i_6251387
>> ___
>> Gluster-users mailing list
>> Gluster-users@gluster.org
>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Infiniband performance issues answered?

[Sabuj Pattanayek]

I have R610's with a similar setup but with HT turned on and I'm
getting 3.5GB/s for one way RDMA tests between two QDR connected
clients using mellanox connectx x4 PCI-E cards in x8 slots. 1GB/s with
IPoIB connections (seem to be limited to 10gbe). Note, I had problems
with the 1.x branch of OFED and am using the latest 3.x RC .

On Mon, Dec 17, 2012 at 6:44 PM, Joe Julian  wrote:
> In IRC today, someone who was hitting that same IB performance ceiling that
> occasionally gets reported had this to say
>
> [11:50]  first, I ran fedora which is not supported by Mellanox OFED
> distro
> [11:50]  so I moved to CentOS 6.3
> [11:51]  next I removed all distibution related infiniband rpms and
> build the latest OFED package
> [11:52]  disabled ServerSpeed service
> [11:52]  disabled BIOS hyperthreading
> [11:52]  disabled BIOS power mgmt
> [11:53]  ran ib_write_test and goot 5000MB/s
> [11:53]  got 5000MB/s on localhost
>
> fwiw, if someone's encountering that issue, between this and the changes
> since 3.4.0qa5 it might be worth knowing about.
>
> http://irclog.perlgeek.de/gluster/2012-12-17#i_6251387
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Infiniband performance issues answered?

[Joe Julian]

In IRC today, someone who was hitting that same IB performance ceiling 
that occasionally gets reported had this to say


[11:50]  first, I ran fedora which is not supported by Mellanox 
OFED distro

[11:50]  so I moved to CentOS 6.3
[11:51]  next I removed all distibution related infiniband rpms 
and build the latest OFED package

[11:52]  disabled ServerSpeed service
[11:52]  disabled BIOS hyperthreading
[11:52]  disabled BIOS power mgmt
[11:53]  ran ib_write_test and goot 5000MB/s
[11:53]  got 5000MB/s on localhost

fwiw, if someone's encountering that issue, between this and the changes 
since 3.4.0qa5 it might be worth knowing about.


http://irclog.perlgeek.de/gluster/2012-12-17#i_6251387
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] glusterfs 3.3 : set username and password

[Pavel Piatruk]

Hello,

How can I enable username+password authentication for a volume?
Is it possible by gluster CLI tool?
Or should I edit a file manually? Which one?
Thanks.





___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Transport endpoint

[Robin, Robin]

Yup. I rsync over to a Gluster mounted (and not the actual brick) via client 
over the network.

Robin


From: "Washer, Bryan" mailto:bwas...@netsuite.com>>
Date: Mon, 17 Dec 2012 10:47:38 -0500
To: Joe Julian mailto:j...@julianfamily.org>>, 
"gluster-users@gluster.org" 
mailto:gluster-users@gluster.org>>
Subject: Re: [Gluster-users] Transport endpoint



Just to make sure we don't miss the obviousŠwhen you say sync'd over to
the mount pointŠyou mean where you did a glusterfs mount and not eh actual
location of the brick on one of the mirrors in your replica.

Once you setup the volume and start it.you should NEVER write or delete
directly on the backend brick unless you really know what you are doing.

Bryan

-Original Message-
From: Joe Julian mailto:j...@julianfamily.org>>
Date: Monday, December 17, 2012 9:29 AM
To: "gluster-users@gluster.org" 
mailto:gluster-users@gluster.org>>
Subject: Re: [Gluster-users] Transport endpoint

>On 12/17/2012 06:56 AM, Robin, Robin wrote:
>> Hi,
>>
>> I've got Gluster error: Transport endpoint not connected.
>>
>> It came up twice after trying to rsync 2 TB filesystem over; it
>> reached about 1.8 TB and got the error.
>>
>> Logs on the server side (on reverse time order):
>> [2012-12-15 00:53:24.747934] I
>> [server-helpers.c:629:server_connection_destroy]
>> 0-RedhawkShared-server: destroyed connection of
>>
>>mualhpcp01.hpc.muohio.edu-17684-2012/12/13-17:25:16:994209-RedhawkShared-
>>client-0-0
>> [2012-12-15 00:53:24.743459] I [server-helpers.c:474:do_fd_cleanup]
>> 0-RedhawkShared-server: fd cleanup on
>>
>>/mkennedy/tramelot_nwfs/rpr3/rpr3/rpr3_sparky/matrix/.4d_ccnoesy.ucsf.QTQ
>>swL
>> [2012-12-15 00:53:24.743430] I
>> [server-helpers.c:330:do_lock_table_cleanup] 0-RedhawkShared-server:
>> finodelk released on
>>
>>/mkennedy/tramelot_nwfs/rpr3/rpr3/rpr3_sparky/matrix/.4d_ccnoesy.ucsf.QTQ
>>swL
>> [2012-12-15 00:53:24.743400] I
>> [server-helpers.c:741:server_connection_put] 0-RedhawkShared-server:
>> Shutting down connection
>>
>>mualhpcp01.hpc.muohio.edu-17684-2012/12/13-17:25:16:994209-RedhawkShared-
>>client-0-0
>> [2012-12-15 00:53:24.743368] I [server.c:685:server_rpc_notify]
>> 0-RedhawkShared-server: disconnecting connectionfrom
>>
>>mualhpcp01.hpc.muohio.edu-17684-2012/12/13-17:25:16:994209-RedhawkShared-
>>client-0-0
>> [2012-12-15 00:53:24.740055] W [socket.c:195:__socket_rwv]
>> 0-tcp.RedhawkShared-server: readv failed (Connection reset by peer)
>>
>> I can't find relevant logs on the client side.
>>
>> From the logs, can we judge for sure that this is a network reset
>> problem ?
>>
>When you say, "I can't find relevant logs on the client side," do you
>mean that you can't find the log, or that there's nothing in there from
>around the same timestamp? The client log will be in /var/log/glusterfs
>and will be named based on the mountpoint.
>___
>Gluster-users mailing list
>Gluster-users@gluster.org
>http://supercolony.gluster.org/mailman/listinfo/gluster-users
>


NOTICE: This email and any attachments may contain confidential and proprietary 
information of NetSuite Inc. and is for the sole use of the intended recipient 
for the stated purpose. Any improper use or distribution is prohibited. If you 
are not the intended recipient, please notify the sender; do not review, copy 
or distribute; and promptly delete or destroy all transmitted information. 
Please note that all communications and information transmitted through this 
email system may be monitored by NetSuite or its agents and that all incoming 
email is automatically scanned by a third party spam and filtering service.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Transport endpoint

[Robin, Robin]

Hi,

Joe, there's nothing around the time stamp.

On the client, /var/log/glusterfs/mnt-gluster-RedhawkShared.log:
[2012-12-13 17:25:21.014040] I [fuse-bridge.c:3376:fuse_init]
0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13
kernel 7.10
[2012-12-13 17:25:21.014823] I
[afr-common.c:1964:afr_set_root_inode_on_first_lookup]
0-RedhawkShared-replicate-0: added root inode
[2012-12-15 03:21:37.566345] I [glusterfsd.c:1666:main]
0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.3.0
[2012-12-15 03:21:37.687280] I [io-cache.c:1549:check_cache_size_ok]
0-RedhawkShared-quick-read: Max cache size is 25133539328
[2012-12-15 03:21:37.692566] I [io-cache.c:1549:check_cache_size_ok]
0-RedhawkShared-io-cache: Max cache size is 25133539328
[2012-12-15 03:21:37.694761] I [client.c:2142:notify]
0-RedhawkShared-client-0: parent translators are ready, attempting connect
on transport
[2012-12-15 03:21:37.698557] I [client.c:2142:notify]
0-RedhawkShared-client-1: parent translators are ready, attempting connect
on transport


There's nothing on 2012-12-15, other than the fact that at 03:21 AM, I
manually remount the gluster partition again.

John, I've not filed a bug yet.

Let me do another test on the Gluster protocol.

I'm running keepalived for an NFS floating IP. That could be my
work-around for now.

The downside of using NFS is the fact that I need to use an additional
software like keepalived. What are the other downsides ?

Thanks..
Robin

On 12/17/12 10:29 AM, "Joe Julian"  wrote:

>On 12/17/2012 06:56 AM, Robin, Robin wrote:
>> Hi,
>>
>> I've got Gluster error: Transport endpoint not connected.
>>
>> It came up twice after trying to rsync 2 TB filesystem over; it
>> reached about 1.8 TB and got the error.
>>
>> Logs on the server side (on reverse time order):
>> [2012-12-15 00:53:24.747934] I
>> [server-helpers.c:629:server_connection_destroy]
>> 0-RedhawkShared-server: destroyed connection of
>> 
>>mualhpcp01.hpc.muohio.edu-17684-2012/12/13-17:25:16:994209-RedhawkShared-
>>client-0-0
>> [2012-12-15 00:53:24.743459] I [server-helpers.c:474:do_fd_cleanup]
>> 0-RedhawkShared-server: fd cleanup on
>> 
>>/mkennedy/tramelot_nwfs/rpr3/rpr3/rpr3_sparky/matrix/.4d_ccnoesy.ucsf.QTQ
>>swL
>> [2012-12-15 00:53:24.743430] I
>> [server-helpers.c:330:do_lock_table_cleanup] 0-RedhawkShared-server:
>> finodelk released on
>> 
>>/mkennedy/tramelot_nwfs/rpr3/rpr3/rpr3_sparky/matrix/.4d_ccnoesy.ucsf.QTQ
>>swL
>> [2012-12-15 00:53:24.743400] I
>> [server-helpers.c:741:server_connection_put] 0-RedhawkShared-server:
>> Shutting down connection
>> 
>>mualhpcp01.hpc.muohio.edu-17684-2012/12/13-17:25:16:994209-RedhawkShared-
>>client-0-0
>> [2012-12-15 00:53:24.743368] I [server.c:685:server_rpc_notify]
>> 0-RedhawkShared-server: disconnecting connectionfrom
>> 
>>mualhpcp01.hpc.muohio.edu-17684-2012/12/13-17:25:16:994209-RedhawkShared-
>>client-0-0
>> [2012-12-15 00:53:24.740055] W [socket.c:195:__socket_rwv]
>> 0-tcp.RedhawkShared-server: readv failed (Connection reset by peer)
>>
>> I can't find relevant logs on the client side.
>>
>> From the logs, can we judge for sure that this is a network reset
>> problem ?
>>
>When you say, "I can't find relevant logs on the client side," do you
>mean that you can't find the log, or that there's nothing in there from
>around the same timestamp? The client log will be in /var/log/glusterfs
>and will be named based on the mountpoint.
>___
>Gluster-users mailing list
>Gluster-users@gluster.org
>http://supercolony.gluster.org/mailman/listinfo/gluster-users

___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Transport endpoint

[Washer, Bryan]

Just to make sure we don't miss the obviousŠwhen you say sync'd 
over to
the mount pointŠyou mean where you did a glusterfs mount and not eh actual
location of the brick on one of the mirrors in your replica.

Once you setup the volume and start it.you should NEVER write or delete
directly on the backend brick unless you really know what you are doing.

Bryan

-Original Message-
From: Joe Julian 
Date: Monday, December 17, 2012 9:29 AM
To: "gluster-users@gluster.org" 
Subject: Re: [Gluster-users] Transport endpoint

>On 12/17/2012 06:56 AM, Robin, Robin wrote:
>> Hi,
>>
>> I've got Gluster error: Transport endpoint not connected.
>>
>> It came up twice after trying to rsync 2 TB filesystem over; it
>> reached about 1.8 TB and got the error.
>>
>> Logs on the server side (on reverse time order):
>> [2012-12-15 00:53:24.747934] I
>> [server-helpers.c:629:server_connection_destroy]
>> 0-RedhawkShared-server: destroyed connection of
>> 
>>mualhpcp01.hpc.muohio.edu-17684-2012/12/13-17:25:16:994209-RedhawkShared-
>>client-0-0
>> [2012-12-15 00:53:24.743459] I [server-helpers.c:474:do_fd_cleanup]
>> 0-RedhawkShared-server: fd cleanup on
>> 
>>/mkennedy/tramelot_nwfs/rpr3/rpr3/rpr3_sparky/matrix/.4d_ccnoesy.ucsf.QTQ
>>swL
>> [2012-12-15 00:53:24.743430] I
>> [server-helpers.c:330:do_lock_table_cleanup] 0-RedhawkShared-server:
>> finodelk released on
>> 
>>/mkennedy/tramelot_nwfs/rpr3/rpr3/rpr3_sparky/matrix/.4d_ccnoesy.ucsf.QTQ
>>swL
>> [2012-12-15 00:53:24.743400] I
>> [server-helpers.c:741:server_connection_put] 0-RedhawkShared-server:
>> Shutting down connection
>> 
>>mualhpcp01.hpc.muohio.edu-17684-2012/12/13-17:25:16:994209-RedhawkShared-
>>client-0-0
>> [2012-12-15 00:53:24.743368] I [server.c:685:server_rpc_notify]
>> 0-RedhawkShared-server: disconnecting connectionfrom
>> 
>>mualhpcp01.hpc.muohio.edu-17684-2012/12/13-17:25:16:994209-RedhawkShared-
>>client-0-0
>> [2012-12-15 00:53:24.740055] W [socket.c:195:__socket_rwv]
>> 0-tcp.RedhawkShared-server: readv failed (Connection reset by peer)
>>
>> I can't find relevant logs on the client side.
>>
>> From the logs, can we judge for sure that this is a network reset
>> problem ?
>>
>When you say, "I can't find relevant logs on the client side," do you
>mean that you can't find the log, or that there's nothing in there from
>around the same timestamp? The client log will be in /var/log/glusterfs
>and will be named based on the mountpoint.
>___
>Gluster-users mailing list
>Gluster-users@gluster.org
>http://supercolony.gluster.org/mailman/listinfo/gluster-users
>


NOTICE: This email and any attachments may contain confidential and proprietary 
information of NetSuite Inc. and is for the sole use of the intended recipient 
for the stated purpose. Any improper use or distribution is prohibited. If you 
are not the intended recipient, please notify the sender; do not review, copy 
or distribute; and promptly delete or destroy all transmitted information. 
Please note that all communications and information transmitted through this 
email system may be monitored by NetSuite or its agents and that all incoming 
email is automatically scanned by a third party spam and filtering service

___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Transport endpoint

[Joe Julian]

On 12/17/2012 06:56 AM, Robin, Robin wrote:

Hi,

I've got Gluster error: Transport endpoint not connected.

It came up twice after trying to rsync 2 TB filesystem over; it 
reached about 1.8 TB and got the error.


Logs on the server side (on reverse time order):
[2012-12-15 00:53:24.747934] I 
[server-helpers.c:629:server_connection_destroy] 
0-RedhawkShared-server: destroyed connection of 
mualhpcp01.hpc.muohio.edu-17684-2012/12/13-17:25:16:994209-RedhawkShared-client-0-0
[2012-12-15 00:53:24.743459] I [server-helpers.c:474:do_fd_cleanup] 
0-RedhawkShared-server: fd cleanup on 
/mkennedy/tramelot_nwfs/rpr3/rpr3/rpr3_sparky/matrix/.4d_ccnoesy.ucsf.QTQswL
[2012-12-15 00:53:24.743430] I 
[server-helpers.c:330:do_lock_table_cleanup] 0-RedhawkShared-server: 
finodelk released on 
/mkennedy/tramelot_nwfs/rpr3/rpr3/rpr3_sparky/matrix/.4d_ccnoesy.ucsf.QTQswL
[2012-12-15 00:53:24.743400] I 
[server-helpers.c:741:server_connection_put] 0-RedhawkShared-server: 
Shutting down connection 
mualhpcp01.hpc.muohio.edu-17684-2012/12/13-17:25:16:994209-RedhawkShared-client-0-0
[2012-12-15 00:53:24.743368] I [server.c:685:server_rpc_notify] 
0-RedhawkShared-server: disconnecting connectionfrom 
mualhpcp01.hpc.muohio.edu-17684-2012/12/13-17:25:16:994209-RedhawkShared-client-0-0
[2012-12-15 00:53:24.740055] W [socket.c:195:__socket_rwv] 
0-tcp.RedhawkShared-server: readv failed (Connection reset by peer)


I can't find relevant logs on the client side.

From the logs, can we judge for sure that this is a network reset 
problem ?


When you say, "I can't find relevant logs on the client side," do you 
mean that you can't find the log, or that there's nothing in there from 
around the same timestamp? The client log will be in /var/log/glusterfs 
and will be named based on the mountpoint.

___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Transport endpoint

[John Mark Walker]

- Original Message -

> I have 2 gluster servers, setup with replica 2.

> Gluster versions:
> glusterfs-fuse-3.3.0-1.el6.x86_64
> glusterfs-3.3.0-1.el6.x86_64
> glusterfs-server-3.3.0-1.el6.x86_64

> The error came up after a client mounting via Gluster protocol and I
> manually rsync from another source to the Gluster-mount point as
> destination.
> I've got the same errors twice.

> I did another test: if mount the gluster volume via NFS, the rsync
> was successful.
That's interesting. Did you file a bug? 

-JM 
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] stripped volume in 3.4.0qa5 with horrible read performance

[John Mark Walker]

Please file a bug. There might be time to fix read performance before the 1st 
beta release. 

-JM 

- Original Message -

> Dear folks,

> I've been tried to use replicated stripped volumes with 3.3.
> unsuccessfully due to
> https://bugzilla.redhat.com/show_bug.cgi?id=861423 and I then
> proceed to try 3.4.0qa5. I then find out that the bug was solved and
> I could use replicated stripped volume with the new version.
> Amazingly, write performance was quite astonishing.

> The problem I'm facing now is in the read process: It's horribly
> slow. When I open a file to edit using the gluster native client, it
> takes a few seconds and sometimes I got an error refering to file
> has been modified while I was editing it. There's a ruby application
> reading the files and I got continuously timeout errors.

> I'm using 4 bricks with Centos 6.3 with the following structure:
> Type: Striped-Replicate
> Volume ID: 23dbb8dd-5cb3-4c71-9702-7c16ee9a3b3b
> Status: Started
> Number of Bricks: 1 x 2 x 2 = 4
> Transport-type: tcp
> Bricks:
> Brick1: 10.0.51.31:/gfs
> Brick2: 10.0.51.32:/gfs
> Brick3: 10.0.51.33:/gfs
> Brick4: 10.0.51.34:/gfs
> Options Reconfigured:
> performance.quick-read: on
> performance.io-thread-count: 32
> performance.cache-max-file-size: 128MB
> performance.cache-size: 256MB
> performance.io-cache: on
> cluster.stripe-block-size: 2MB
> nfs.disable: on

> I started profiling and found out one node with absurd latency
> figures. I stopped the node and the problem moved to another brick:
> %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
> - --- --- ---  
> 99.94 551292.41 us 10.00 us 1996709.00 us 361 FINODELK

> Could anyone provide some information how to debug this problem?
> Currently the volume is not usable due to the horrible delay.

> Thank you very much in advance,
> Samuel.

> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Transport endpoint

[Robin, Robin]

I'm not using geo-rep.

I have 2 gluster servers, setup with replica 2.

Gluster versions:
glusterfs-fuse-3.3.0-1.el6.x86_64
glusterfs-3.3.0-1.el6.x86_64
glusterfs-server-3.3.0-1.el6.x86_64

The error came up after a client mounting via Gluster protocol and I manually 
rsync from another source to the Gluster-mount point as destination.
I've got the same errors twice.

I did another test: if mount the gluster volume via NFS, the rsync was 
successful.

Thanks,
Robin


From: John Mark Walker mailto:johnm...@redhat.com>>
Date: Mon, 17 Dec 2012 10:14:06 -0500
To: Microsoft Office User mailto:rob...@muohio.edu>>
Cc: "gluster-users@gluster.org" 
mailto:gluster-users@gluster.org>>
Subject: Re: [Gluster-users] Transport endpoint

Are you using the geo-rep (async replication) feature, or are you manually 
rsync-ing an entire brick?

What exactly do you want to do?

-JM



Hi,

I've got Gluster error: Transport endpoint not connected.

It came up twice after trying to rsync 2 TB filesystem over; it reached about 
1.8 TB and got the error.

Logs on the server side (on reverse time order):
[2012-12-15 00:53:24.747934] I [server-helpers.c:629:server_connection_destroy] 
0-RedhawkShared-server: destroyed connection of 
mualhpcp01.hpc.muohio.edu-17684-2012/12/13-17:25:16:994209-RedhawkShared-client-0-0
[2012-12-15 00:53:24.743459] I [server-helpers.c:474:do_fd_cleanup] 
0-RedhawkShared-server: fd cleanup on 
/mkennedy/tramelot_nwfs/rpr3/rpr3/rpr3_sparky/matrix/.4d_ccnoesy.ucsf.QTQswL
[2012-12-15 00:53:24.743430] I [server-helpers.c:330:do_lock_table_cleanup] 
0-RedhawkShared-server: finodelk released on 
/mkennedy/tramelot_nwfs/rpr3/rpr3/rpr3_sparky/matrix/.4d_ccnoesy.ucsf.QTQswL
[2012-12-15 00:53:24.743400] I [server-helpers.c:741:server_connection_put] 
0-RedhawkShared-server: Shutting down connection 
mualhpcp01.hpc.muohio.edu-17684-2012/12/13-17:25:16:994209-RedhawkShared-client-0-0
[2012-12-15 00:53:24.743368] I [server.c:685:server_rpc_notify] 
0-RedhawkShared-server: disconnecting connectionfrom 
mualhpcp01.hpc.muohio.edu-17684-2012/12/13-17:25:16:994209-RedhawkShared-client-0-0
[2012-12-15 00:53:24.740055] W [socket.c:195:__socket_rwv] 
0-tcp.RedhawkShared-server: readv failed (Connection reset by peer)

I can't find relevant logs on the client side.

>From the logs, can we judge for sure that this is a network reset problem ?

Thanks,
Robin


___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] probable cause of read performance issue

[John Mark Walker]

Thanks for sending to the list. I'm not sure what the solution is, since most 
people need R+W. For 3.4, there are some features around the use case of WORM 
(write once, read many) which might improve performance for the use case you're 
testing. 

Could you perform the same test with the qa5 release from last week? 

-JM 

- Original Message -

> hi, all:
> When I was testing glusterfs performance by iozone with only ONE
> machine, I got the following chart:

> https://dl.dropbox.com/u/33453649/perf_htm_m19da4047.jpg

> Y axis represent disk throughput in kb/s. X axis represent test
> cases.

> It is obvious that R+W case has the worst read performance, R+W means
> you turn on both Read-ahead xlator and Write-behind xlator.
> Unfortunately, that`s the default configuration of glusterfs.

> Though I run glusterfs only on one machine, I think this result
> explains something. Maybe you have more insight on it.

> More info:
> -
> cmdline: ./iozone -a -n 16g -g 16g -i 0 -i 1 -i 2 -i 3 -f /mnt/iozone
> -Rb no-rh.xls -y 4096k
> CPU: intel i3-2120 X 4
> RAM: 8G
> Glusterfs Version: 3.3
> Volume info:
> Volume Name: abc
> Type: Distribute
> Volume ID: 6e0104f1-d32a-4ed4-b011-d29ccf27abe1
> Status: Started
> Number of Bricks: 1
> Transport-type: tcp
> Bricks:
> Brick1: 10.1.82.224:/abc
> Options Reconfigured:
> performance.io-cache: off
> performance.quick-read: off
> performance.read-ahead: on
> performance.write-behind: on
> 

> Best Regards.
> Jules Wang.

> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Transport endpoint

[John Mark Walker]

Are you using the geo-rep (async replication) feature, or are you manually 
rsync-ing an entire brick? 

What exactly do you want to do? 

-JM 

- Original Message -

> Hi,

> I've got Gluster error: Transport endpoint not connected.

> It came up twice after trying to rsync 2 TB filesystem over; it
> reached about 1.8 TB and got the error.

> Logs on the server side (on reverse time order):
> [2012-12-15 00:53:24.747934] I
> [server-helpers.c:629:server_connection_destroy]
> 0-RedhawkShared-server: destroyed connection of
> mualhpcp01.hpc.muohio.edu-17684-2012/12/13-17:25:16:994209-RedhawkShared-client-0-0
> [2012-12-15 00:53:24.743459] I [server-helpers.c:474:do_fd_cleanup]
> 0-RedhawkShared-server: fd cleanup on
> /mkennedy/tramelot_nwfs/rpr3/rpr3/rpr3_sparky/matrix/.4d_ccnoesy.ucsf.QTQswL
> [2012-12-15 00:53:24.743430] I
> [server-helpers.c:330:do_lock_table_cleanup] 0-RedhawkShared-server:
> finodelk released on
> /mkennedy/tramelot_nwfs/rpr3/rpr3/rpr3_sparky/matrix/.4d_ccnoesy.ucsf.QTQswL
> [2012-12-15 00:53:24.743400] I
> [server-helpers.c:741:server_connection_put] 0-RedhawkShared-server:
> Shutting down connection
> mualhpcp01.hpc.muohio.edu-17684-2012/12/13-17:25:16:994209-RedhawkShared-client-0-0
> [2012-12-15 00:53:24.743368] I [server.c:685:server_rpc_notify]
> 0-RedhawkShared-server: disconnecting connectionfrom
> mualhpcp01.hpc.muohio.edu-17684-2012/12/13-17:25:16:994209-RedhawkShared-client-0-0
> [2012-12-15 00:53:24.740055] W [socket.c:195:__socket_rwv]
> 0-tcp.RedhawkShared-server: readv failed (Connection reset by peer)

> I can't find relevant logs on the client side.

> From the logs, can we judge for sure that this is a network reset
> problem ?

> Thanks,
> Robin

> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Transport endpoint

[Robin, Robin]

Hi,

I've got Gluster error: Transport endpoint not connected.

It came up twice after trying to rsync 2 TB filesystem over; it reached about 
1.8 TB and got the error.

Logs on the server side (on reverse time order):
[2012-12-15 00:53:24.747934] I [server-helpers.c:629:server_connection_destroy] 
0-RedhawkShared-server: destroyed connection of 
mualhpcp01.hpc.muohio.edu-17684-2012/12/13-17:25:16:994209-RedhawkShared-client-0-0
[2012-12-15 00:53:24.743459] I [server-helpers.c:474:do_fd_cleanup] 
0-RedhawkShared-server: fd cleanup on 
/mkennedy/tramelot_nwfs/rpr3/rpr3/rpr3_sparky/matrix/.4d_ccnoesy.ucsf.QTQswL
[2012-12-15 00:53:24.743430] I [server-helpers.c:330:do_lock_table_cleanup] 
0-RedhawkShared-server: finodelk released on 
/mkennedy/tramelot_nwfs/rpr3/rpr3/rpr3_sparky/matrix/.4d_ccnoesy.ucsf.QTQswL
[2012-12-15 00:53:24.743400] I [server-helpers.c:741:server_connection_put] 
0-RedhawkShared-server: Shutting down connection 
mualhpcp01.hpc.muohio.edu-17684-2012/12/13-17:25:16:994209-RedhawkShared-client-0-0
[2012-12-15 00:53:24.743368] I [server.c:685:server_rpc_notify] 
0-RedhawkShared-server: disconnecting connectionfrom 
mualhpcp01.hpc.muohio.edu-17684-2012/12/13-17:25:16:994209-RedhawkShared-client-0-0
[2012-12-15 00:53:24.740055] W [socket.c:195:__socket_rwv] 
0-tcp.RedhawkShared-server: readv failed (Connection reset by peer)

I can't find relevant logs on the client side.

>From the logs, can we judge for sure that this is a network reset problem ?

Thanks,
Robin

___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] stripped volume in 3.4.0qa5 with horrible read performance

[samuel]

Dear folks,

I've been tried to use replicated stripped volumes with 3.3. unsuccessfully
due to https://bugzilla.redhat.com/show_bug.cgi?id=861423 and I then
proceed to try 3.4.0qa5. I then find out that the bug was solved and I
could use replicated stripped volume with the new version. Amazingly, write
performance was quite astonishing.

The problem I'm facing now is in the read process: It's horribly slow. When
I open a file to edit using the gluster native client, it takes a few
seconds and sometimes I got an error refering to file has been modified
while I was editing it. There's a ruby application reading the files and I
got continuously timeout errors.

I'm using 4 bricks with Centos 6.3 with the following structure:
Type: Striped-Replicate
Volume ID: 23dbb8dd-5cb3-4c71-9702-7c16ee9a3b3b
Status: Started
Number of Bricks: 1 x 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: 10.0.51.31:/gfs
Brick2: 10.0.51.32:/gfs
Brick3: 10.0.51.33:/gfs
Brick4: 10.0.51.34:/gfs
Options Reconfigured:
performance.quick-read: on
performance.io-thread-count: 32
performance.cache-max-file-size: 128MB
performance.cache-size: 256MB
performance.io-cache: on
cluster.stripe-block-size: 2MB
nfs.disable: on

I started profiling and found out one node with absurd latency figures. I
stopped the node and the problem moved to another brick:
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls
Fop
 -   ---   ---   ---   

99.94  551292.41 us  10.00 us 1996709.00 us361FINODELK

Could anyone provide some information how to debug this problem? Currently
the volume is not usable due to the horrible delay.

Thank you very much in advance,
Samuel.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users