Re: [Gluster-users] high CPU load on all bricks

2013-02-14 Thread Michael Colonno 
Ok, good - I had that right. In that case the bricks each need to mount the 
volume. I mounted the volume on each brick to itself (localhost) so they would 
not depend on each other re: mounting. 
But I am back to square one on the high CPU load / slow write speed issue. I'll 
try a test volume with a single node as brick and client to try any rule out 
anything network-related, though my network is performing well in every other 
way.

Thanks,
Mike C.

On Feb 14, 2013, at 12:31 PM, Bryan Whitehead  wrote:

> Yea, only write to the glusterfs mountpoint. Writing directly to the bricks 
> is bad and shouldn't be done.
> 
> 
> On Thu, Feb 14, 2013 at 11:58 AM, Michael Colonno  
> wrote:
>> Good place to start: do the bricks have to be clients as well? In other 
>> words if I copy a file to a Gluster brick without going through a glusterfs 
>> or NFS mount will that disrupt the parallel file system? I assumed files 
>> need to be routed through a glusterfs mount point for Gluster to be able to 
>> track them(?) What's recommended for bricks which also need i/o to the 
>> entire volume?
>> 
>> Thanks,
>> Mike C.
>> 
>> On Feb 14, 2013, at 10:28 AM, harry mangalam  wrote:
>> 
>> > While I don't understand your 'each brick system also being a client' 
>> > setup -
>> > you mean that each gluster brick is a native gluster client as well?  And 
>> > that
>> > is where much of your gluster access is coming from?  That seems .. 
>> > suboptimal
>> > if that's the setup.  Is there a reason for that setup?
>> >
>> > We have a distributed-only glusterfs feeding a medium cluster over a 
>> > similar
>> > same setup QDR IPoIB with 4 servers with 2 bricks each.  On a fairly busy
>> > system (~80MB/s background), I can get about 100-300MB/s writes to the 
>> > gluster
>> > fs on a large 1.7GB file.  (With tiny writes, the perf decreases
>> > dramatically).
>> >
>> > Here is my config: (if anyone spies something that I should change to 
>> > increase
>> > my perf, please feel free to point out my mistake)
>> >
>> > gluster:
>> > Volume Name: gl
>> > Type: Distribute
>> > Volume ID: 21f480f7-fc5a-4fd8-a084-3964634a9332
>> > Status: Started
>> > Number of Bricks: 8
>> > Transport-type: tcp,rdma
>> > Bricks:
>> > Brick1: bs2:/raid1
>> > Brick2: bs2:/raid2
>> > Brick3: bs3:/raid1
>> > Brick4: bs3:/raid2
>> > Brick5: bs4:/raid1
>> > Brick6: bs4:/raid2
>> > Brick7: bs1:/raid1
>> > Brick8: bs1:/raid2
>> > Options Reconfigured:
>> > performance.write-behind-window-size: 1024MB
>> > performance.flush-behind: on
>> > performance.cache-size: 268435456
>> > nfs.disable: on
>> > performance.io-cache: on
>> > performance.quick-read: on
>> > performance.io-thread-count: 64
>> > auth.allow: 10.2.*.*,10.1.*.*
>> >
>> > my RAID6s (via 3ware 9750s) are mounted with the following options
>> >
>> > /dev/sdc /raid1 xfs rw,noatime,sunit=512,swidth=8192,allocsize=32m 0 0
>> > /dev/sdd /raid2 xfs rw,noatime,sunit=512,swidth=7680,allocsize=32m 0 0
>> > (and should probably be using 'nobarrier,inode64' as well. - testing this 
>> > now)
>> >
>> > There are some good refs on prepping XFS fs for max perf here:
>> > 
>> > The script at:
>> > 
>> > can help to setup the sunit/swidth options.
>> > > > edition/>
>> > Your ib interfaces should be using large mtus (65536)
>> >
>> > hjm
>> >
>> > On Wednesday, February 13, 2013 10:35:12 PM Michael Colonno wrote:
>> >>More data: I got the Infiniband network (QDR) working well and
>> >> switched my gluster volume to the Infiniband fabric (IPoIB, not RDMA since
>> >> it doesn't seem to be supported yet for 3.x). The filesystem was slightly
>> >> faster but still well short of what I would expect by a wide margin. Via 
>> >> an
>> >> informal test (timing the movement of a large file) I'm getting several 
>> >> MB/s
>> >> - well short of even a standard Gb network copy. With the faster network
>> >> the CPU load on the brick systems increased dramatically: now I'm seeing
>> >> 200%-250% usage by glusterfsd and glusterfs.
>> >>
>> >>This leads me to believe that gluster is really not enjoying my
>> >> eight-brick, 2x replication volume with each brick system also being a
>> >> client. I tried a rebalance but no measurable effect. Any suggestions for
>> >> improving the performance? Having each brick be a client of itself seemed
>> >> the most logical choice to remove interdependencies but now I'm doubting 
>> >> the
>> >> setup.
>> >>
>> >>
>> >>
>> >>Thanks,
>> >>
>> >>~Mike C.
>> >>
>> >>
>> >>
>> >> From: gluster-users-boun...@gluster.org
>> >> [mailto:gluster-users-boun...@gluster.org] On Behalf Of Joe Julian
>> >> Sent: Sunday, February 03, 2013 11:47 AM
>> >> To: gluster-users@gluster.org
>> >> Subject: Re: [Gluster-users] high CPU load on all bricks

Re: [Gluster-users] high CPU load on all bricks

2013-02-14 Thread Bryan Whitehead 
Yea, only write to the glusterfs mountpoint. Writing directly to the bricks
is bad and shouldn't be done.


On Thu, Feb 14, 2013 at 11:58 AM, Michael Colonno wrote:

> Good place to start: do the bricks have to be clients as well? In other
> words if I copy a file to a Gluster brick without going through a glusterfs
> or NFS mount will that disrupt the parallel file system? I assumed files
> need to be routed through a glusterfs mount point for Gluster to be able to
> track them(?) What's recommended for bricks which also need i/o to the
> entire volume?
>
> Thanks,
> Mike C.
>
> On Feb 14, 2013, at 10:28 AM, harry mangalam 
> wrote:
>
> > While I don't understand your 'each brick system also being a client'
> setup -
> > you mean that each gluster brick is a native gluster client as well?
>  And that
> > is where much of your gluster access is coming from?  That seems ..
> suboptimal
> > if that's the setup.  Is there a reason for that setup?
> >
> > We have a distributed-only glusterfs feeding a medium cluster over a
> similar
> > same setup QDR IPoIB with 4 servers with 2 bricks each.  On a fairly busy
> > system (~80MB/s background), I can get about 100-300MB/s writes to the
> gluster
> > fs on a large 1.7GB file.  (With tiny writes, the perf decreases
> > dramatically).
> >
> > Here is my config: (if anyone spies something that I should change to
> increase
> > my perf, please feel free to point out my mistake)
> >
> > gluster:
> > Volume Name: gl
> > Type: Distribute
> > Volume ID: 21f480f7-fc5a-4fd8-a084-3964634a9332
> > Status: Started
> > Number of Bricks: 8
> > Transport-type: tcp,rdma
> > Bricks:
> > Brick1: bs2:/raid1
> > Brick2: bs2:/raid2
> > Brick3: bs3:/raid1
> > Brick4: bs3:/raid2
> > Brick5: bs4:/raid1
> > Brick6: bs4:/raid2
> > Brick7: bs1:/raid1
> > Brick8: bs1:/raid2
> > Options Reconfigured:
> > performance.write-behind-window-size: 1024MB
> > performance.flush-behind: on
> > performance.cache-size: 268435456
> > nfs.disable: on
> > performance.io-cache: on
> > performance.quick-read: on
> > performance.io-thread-count: 64
> > auth.allow: 10.2.*.*,10.1.*.*
> >
> > my RAID6s (via 3ware 9750s) are mounted with the following options
> >
> > /dev/sdc /raid1 xfs rw,noatime,sunit=512,swidth=8192,allocsize=32m 0 0
> > /dev/sdd /raid2 xfs rw,noatime,sunit=512,swidth=7680,allocsize=32m 0 0
> > (and should probably be using 'nobarrier,inode64' as well. - testing
> this now)
> >
> > There are some good refs on prepping XFS fs for max perf here:
> > 
> > The script at:
> > 
> > can help to setup the sunit/swidth options.
> > <
> http://www.mysqlperformanceblog.com/2011/12/16/setting-up-xfs-the-simple-
> > edition/>
> > Your ib interfaces should be using large mtus (65536)
> >
> > hjm
> >
> > On Wednesday, February 13, 2013 10:35:12 PM Michael Colonno wrote:
> >>More data: I got the Infiniband network (QDR) working well
> and
> >> switched my gluster volume to the Infiniband fabric (IPoIB, not RDMA
> since
> >> it doesn't seem to be supported yet for 3.x). The filesystem was
> slightly
> >> faster but still well short of what I would expect by a wide margin.
> Via an
> >> informal test (timing the movement of a large file) I'm getting several
> MB/s
> >> - well short of even a standard Gb network copy. With the faster network
> >> the CPU load on the brick systems increased dramatically: now I'm seeing
> >> 200%-250% usage by glusterfsd and glusterfs.
> >>
> >>This leads me to believe that gluster is really not enjoying
> my
> >> eight-brick, 2x replication volume with each brick system also being a
> >> client. I tried a rebalance but no measurable effect. Any suggestions
> for
> >> improving the performance? Having each brick be a client of itself
> seemed
> >> the most logical choice to remove interdependencies but now I'm
> doubting the
> >> setup.
> >>
> >>
> >>
> >>Thanks,
> >>
> >>~Mike C.
> >>
> >>
> >>
> >> From: gluster-users-boun...@gluster.org
> >> [mailto:gluster-users-boun...@gluster.org] On Behalf Of Joe Julian
> >> Sent: Sunday, February 03, 2013 11:47 AM
> >> To: gluster-users@gluster.org
> >> Subject: Re: [Gluster-users] high CPU load on all bricks
> >>
> >>
> >>
> >> On 02/03/2013 11:22 AM, Michael Colonno wrote:
> >>
> >>
> >>
> >>Having taken a lot more data it does seem the glusterfsd and
> >> glusterd processes (along with several ksoftirqd) spike up to near 100%
> on
> >> both client and brick servers during any file transport across the
> mount.
> >> Thankfully this is short-lived for the most part but I'm wondering if
> this
> >> is expected behavior or what others have experienced(?) I'm a little
> >> surprised such a large CPU load would be required to move small files
> and /
> >> or use an application within a Gluster mount point.
> >>
> >>
> >> If you're getting ksoftirq

Re: [Gluster-users] high CPU load on all bricks

2013-02-14 Thread Michael Colonno 
Good place to start: do the bricks have to be clients as well? In other words 
if I copy a file to a Gluster brick without going through a glusterfs or NFS 
mount will that disrupt the parallel file system? I assumed files need to be 
routed through a glusterfs mount point for Gluster to be able to track them(?) 
What's recommended for bricks which also need i/o to the entire volume?

Thanks,
Mike C.

On Feb 14, 2013, at 10:28 AM, harry mangalam  wrote:

> While I don't understand your 'each brick system also being a client' setup - 
> you mean that each gluster brick is a native gluster client as well?  And 
> that 
> is where much of your gluster access is coming from?  That seems .. 
> suboptimal 
> if that's the setup.  Is there a reason for that setup?
> 
> We have a distributed-only glusterfs feeding a medium cluster over a similar 
> same setup QDR IPoIB with 4 servers with 2 bricks each.  On a fairly busy 
> system (~80MB/s background), I can get about 100-300MB/s writes to the 
> gluster 
> fs on a large 1.7GB file.  (With tiny writes, the perf decreases 
> dramatically).
> 
> Here is my config: (if anyone spies something that I should change to 
> increase 
> my perf, please feel free to point out my mistake)
> 
> gluster:
> Volume Name: gl
> Type: Distribute
> Volume ID: 21f480f7-fc5a-4fd8-a084-3964634a9332
> Status: Started
> Number of Bricks: 8
> Transport-type: tcp,rdma
> Bricks:
> Brick1: bs2:/raid1
> Brick2: bs2:/raid2
> Brick3: bs3:/raid1
> Brick4: bs3:/raid2
> Brick5: bs4:/raid1
> Brick6: bs4:/raid2
> Brick7: bs1:/raid1
> Brick8: bs1:/raid2
> Options Reconfigured:
> performance.write-behind-window-size: 1024MB
> performance.flush-behind: on
> performance.cache-size: 268435456
> nfs.disable: on
> performance.io-cache: on
> performance.quick-read: on
> performance.io-thread-count: 64
> auth.allow: 10.2.*.*,10.1.*.*
> 
> my RAID6s (via 3ware 9750s) are mounted with the following options
> 
> /dev/sdc /raid1 xfs rw,noatime,sunit=512,swidth=8192,allocsize=32m 0 0
> /dev/sdd /raid2 xfs rw,noatime,sunit=512,swidth=7680,allocsize=32m 0 0
> (and should probably be using 'nobarrier,inode64' as well. - testing this now)
> 
> There are some good refs on prepping XFS fs for max perf here:
> 
> The script at:
> 
> can help to setup the sunit/swidth options.
>  edition/>
> Your ib interfaces should be using large mtus (65536)
> 
> hjm
> 
> On Wednesday, February 13, 2013 10:35:12 PM Michael Colonno wrote:
>>More data: I got the Infiniband network (QDR) working well and
>> switched my gluster volume to the Infiniband fabric (IPoIB, not RDMA since
>> it doesn't seem to be supported yet for 3.x). The filesystem was slightly
>> faster but still well short of what I would expect by a wide margin. Via an
>> informal test (timing the movement of a large file) I'm getting several MB/s
>> - well short of even a standard Gb network copy. With the faster network
>> the CPU load on the brick systems increased dramatically: now I'm seeing
>> 200%-250% usage by glusterfsd and glusterfs.
>> 
>>This leads me to believe that gluster is really not enjoying my
>> eight-brick, 2x replication volume with each brick system also being a
>> client. I tried a rebalance but no measurable effect. Any suggestions for
>> improving the performance? Having each brick be a client of itself seemed
>> the most logical choice to remove interdependencies but now I'm doubting the
>> setup.
>> 
>> 
>> 
>>Thanks,
>> 
>>~Mike C.
>> 
>> 
>> 
>> From: gluster-users-boun...@gluster.org
>> [mailto:gluster-users-boun...@gluster.org] On Behalf Of Joe Julian
>> Sent: Sunday, February 03, 2013 11:47 AM
>> To: gluster-users@gluster.org
>> Subject: Re: [Gluster-users] high CPU load on all bricks
>> 
>> 
>> 
>> On 02/03/2013 11:22 AM, Michael Colonno wrote:
>> 
>> 
>> 
>>Having taken a lot more data it does seem the glusterfsd and
>> glusterd processes (along with several ksoftirqd) spike up to near 100% on
>> both client and brick servers during any file transport across the mount.
>> Thankfully this is short-lived for the most part but I'm wondering if this
>> is expected behavior or what others have experienced(?) I'm a little
>> surprised such a large CPU load would be required to move small files and /
>> or use an application within a Gluster mount point.
>> 
>> 
>> If you're getting ksoftirqd spikes, that sounds like a hardware issue to me.
>> I never see huge spikes like that on my servers nor clients.
>> 
>> 
>> 
>> 
>> 
>> 
>>I wanted to test this against an NFS mount of the same Gluster
>> volume. I managed to get rstatd installed and running but my attempts to
>> mount the volume via NFS are met with:
>> 
>> 
>> 
>>mount.nfs: requested NFS version

Re: [Gluster-users] high CPU load on all bricks

2013-02-14 Thread Michael Colonno 
Tcp transport and file sizes are nominal (up to a few GB typically). Using 
glusterfs mount (no NFS). There's nothing unusual about the deployment except 
the eight-brick setup server-client setup mentioned below. Is there anything I 
can do to identify the bottleneck(s) and / or tune performance? I'm going to 
try to build the rpms myself though I doubt that will change anything vs. the 
pre-built ones.

Thanks,
Mike C.

On Feb 14, 2013, at 9:52 AM, Bryan Whitehead  wrote:

> is transport tcp or tcp,rdma? I'm using transport=tcp for IPoIB and get 
> pretty fantastic speeds. I noticed when I used tcp,rdma as my transport I had 
> problems.
> 
> Are you mounting via fuse or nfs? I don't have any experience using the nfs 
> but fuse works really well.
> 
> Additionally, how are you using the volume? many small files or big large 
> files? I'm hosting qcow2 files that are between 4 and 250GB.
> 
> 
> On Wed, Feb 13, 2013 at 10:35 PM, Michael Colonno  
> wrote:
>> More data: I got the Infiniband network (QDR) working well and 
>> switched my gluster volume to the Infiniband fabric (IPoIB, not RDMA since 
>> it doesn’t seem to be supported yet for 3.x). The filesystem was slightly 
>> faster but still well short of what I would expect by a wide margin. Via an 
>> informal test (timing the movement of a large file) I’m getting several MB/s 
>> – well short of even a standard Gb network copy. With the faster network the 
>> CPU load on the brick systems increased dramatically: now I’m seeing 
>> 200%-250% usage by glusterfsd and glusterfs.
>> 
>> This leads me to believe that gluster is really not enjoying my 
>> eight-brick, 2x replication volume with each brick system also being a 
>> client. I tried a rebalance but no measurable effect. Any suggestions for 
>> improving the performance? Having each brick be a client of itself seemed 
>> the most logical choice to remove interdependencies but now I’m doubting the 
>> setup…
>> 
>>  
>> 
>> Thanks,
>> 
>> ~Mike C.
>> 
>>  
>> 
>> From: gluster-users-boun...@gluster.org 
>> [mailto:gluster-users-boun...@gluster.org] On Behalf Of Joe Julian
>> 
>> 
>> Sent: Sunday, February 03, 2013 11:47 AM
>> To: gluster-users@gluster.org
>> Subject: Re: [Gluster-users] high CPU load on all bricks
>>  
>> 
>> On 02/03/2013 11:22 AM, Michael Colonno wrote:
>> 
>> 
>> Having taken a lot more data it does seem the glusterfsd and 
>> glusterd processes (along with several ksoftirqd) spike up to near 100% on 
>> both client and brick servers during any file transport across the mount. 
>> Thankfully this is short-lived for the most part but I’m wondering if this 
>> is expected behavior or what others have experienced(?) I’m a little 
>> surprised such a large CPU load would be required to move small files and / 
>> or use an application within a Gluster mount point.
>> 
>> 
>> If you're getting ksoftirqd spikes, that sounds like a hardware issue to me. 
>> I never see huge spikes like that on my servers nor clients.
>> 
>> 
>> 
>>  
>> 
>> I wanted to test this against an NFS mount of the same Gluster 
>> volume. I managed to get rstatd installed and running but my attempts to 
>> mount the volume via NFS are met with:
>> 
>>  
>> 
>> mount.nfs: requested NFS version or transport protocol is not 
>> supported
>> 
>>  
>> 
>> Relevant line in /etc/fstab:
>> 
>>  
>> 
>> node1:/volume/volumenfs 
>> defaults,_netdev,vers=3,mountproto=tcp0 0 
>> 
>>  
>> 
>> It looks like CentOS 6.x has NFS version 4 built into everything. So a few 
>> questions:
>> 
>>  
>> 
>> -   Has anyone else noted significant performance differences between a 
>> glusterfs mount and NFS mount for volumes of 8+ bricks?
>> 
>> -   Is there a straightforward way to make the newer versions of CentOS 
>> play nice with NFS version 3 + Gluster? 
>> 
>> -   Are there any general performance tuning guidelines I can follow to 
>> improve CPU performance? I found a few references to the cache settings but 
>> nothing solid.
>> 
>>  
>> 
>> If the consensus is that NFS will not gain anything then I won’t waste the 
>> time setting it all up.
>> 
>> 
>> NFS gains you the use of FSCache to cache directories and file stats making 
>> directory listings faster, but it adds overhead decreasing the overall 
>> throughput (from all the reports I've seen).
>> 
>> I would suspect that you have the kernel nfs server running on your servers. 
>> Make sure it's disabled.
>> 
>> 
>> 
>>  
>> 
>> Thanks,
>> 
>> ~Mike C.
>> 
>>  
>> 
>>  
>> 
>> From: gluster-users-boun...@gluster.org 
>> [mailto:gluster-users-boun...@gluster.org] On Behalf Of Michael Colonno
>> Sent: Friday, February 01, 2013 4:46 PM
>> To: gluster-users@gluster.org
>> Subject: Re: [Gluster-users] high CPU load on all bricks
>> 
>>  
>> 
>> Update: after a few hours the CPU usage seems to have dropped 
>> down to a sm

Re: [Gluster-users] high CPU load on all bricks

2013-02-14 Thread harry mangalam 
While I don't understand your 'each brick system also being a client' setup - 
you mean that each gluster brick is a native gluster client as well?  And that 
is where much of your gluster access is coming from?  That seems .. suboptimal 
if that's the setup.  Is there a reason for that setup?

We have a distributed-only glusterfs feeding a medium cluster over a similar 
same setup QDR IPoIB with 4 servers with 2 bricks each.  On a fairly busy 
system (~80MB/s background), I can get about 100-300MB/s writes to the gluster 
fs on a large 1.7GB file.  (With tiny writes, the perf decreases 
dramatically).

Here is my config: (if anyone spies something that I should change to increase 
my perf, please feel free to point out my mistake)

gluster:
Volume Name: gl
Type: Distribute
Volume ID: 21f480f7-fc5a-4fd8-a084-3964634a9332
Status: Started
Number of Bricks: 8
Transport-type: tcp,rdma
Bricks:
Brick1: bs2:/raid1
Brick2: bs2:/raid2
Brick3: bs3:/raid1
Brick4: bs3:/raid2
Brick5: bs4:/raid1
Brick6: bs4:/raid2
Brick7: bs1:/raid1
Brick8: bs1:/raid2
Options Reconfigured:
performance.write-behind-window-size: 1024MB
performance.flush-behind: on
performance.cache-size: 268435456
nfs.disable: on
performance.io-cache: on
performance.quick-read: on
performance.io-thread-count: 64
auth.allow: 10.2.*.*,10.1.*.*

my RAID6s (via 3ware 9750s) are mounted with the following options

/dev/sdc /raid1 xfs rw,noatime,sunit=512,swidth=8192,allocsize=32m 0 0
/dev/sdd /raid2 xfs rw,noatime,sunit=512,swidth=7680,allocsize=32m 0 0
(and should probably be using 'nobarrier,inode64' as well. - testing this now)

There are some good refs on prepping XFS fs for max perf here:

The script at:

can help to setup the sunit/swidth options.

Your ib interfaces should be using large mtus (65536)

hjm

On Wednesday, February 13, 2013 10:35:12 PM Michael Colonno wrote:
> More data: I got the Infiniband network (QDR) working well and
> switched my gluster volume to the Infiniband fabric (IPoIB, not RDMA since
> it doesn't seem to be supported yet for 3.x). The filesystem was slightly
> faster but still well short of what I would expect by a wide margin. Via an
> informal test (timing the movement of a large file) I'm getting several MB/s
> - well short of even a standard Gb network copy. With the faster network
> the CPU load on the brick systems increased dramatically: now I'm seeing
> 200%-250% usage by glusterfsd and glusterfs.
> 
> This leads me to believe that gluster is really not enjoying my
> eight-brick, 2x replication volume with each brick system also being a
> client. I tried a rebalance but no measurable effect. Any suggestions for
> improving the performance? Having each brick be a client of itself seemed
> the most logical choice to remove interdependencies but now I'm doubting the
> setup.
> 
> 
> 
> Thanks,
> 
> ~Mike C.
> 
> 
> 
> From: gluster-users-boun...@gluster.org
> [mailto:gluster-users-boun...@gluster.org] On Behalf Of Joe Julian
> Sent: Sunday, February 03, 2013 11:47 AM
> To: gluster-users@gluster.org
> Subject: Re: [Gluster-users] high CPU load on all bricks
> 
> 
> 
> On 02/03/2013 11:22 AM, Michael Colonno wrote:
> 
> 
> 
> Having taken a lot more data it does seem the glusterfsd and
> glusterd processes (along with several ksoftirqd) spike up to near 100% on
> both client and brick servers during any file transport across the mount.
> Thankfully this is short-lived for the most part but I'm wondering if this
> is expected behavior or what others have experienced(?) I'm a little
> surprised such a large CPU load would be required to move small files and /
> or use an application within a Gluster mount point.
> 
> 
> If you're getting ksoftirqd spikes, that sounds like a hardware issue to me.
> I never see huge spikes like that on my servers nor clients.
> 
> 
> 
> 
> 
> 
> I wanted to test this against an NFS mount of the same Gluster
> volume. I managed to get rstatd installed and running but my attempts to
> mount the volume via NFS are met with:
> 
> 
> 
> mount.nfs: requested NFS version or transport protocol is not
> supported
> 
> 
> 
> Relevant line in /etc/fstab:
> 
> 
> 
> node1:/volume/volumenfs
> defaults,_netdev,vers=3,mountproto=tcp0 0
> 
> 
> 
> It looks like CentOS 6.x has NFS version 4 built into everything. So a few
> questions:
> 
> 
> 
> -   Has anyone else noted significant performance differences between a
> glusterfs mount and NFS mount for volumes of 8+ bricks?
> 
> -   Is there a straightforward way to make the newer versions of CentOS
> play nice with NFS version 3 + Gluster?
> 
> -   Are there any general performance tuning guidelines I can f

Re: [Gluster-users] high CPU load on all bricks

2013-02-14 Thread Bryan Whitehead 
is transport tcp or tcp,rdma? I'm using transport=tcp for IPoIB and get
pretty fantastic speeds. I noticed when I used tcp,rdma as my transport I
had problems.

Are you mounting via fuse or nfs? I don't have any experience using the nfs
but fuse works really well.

Additionally, how are you using the volume? many small files or big large
files? I'm hosting qcow2 files that are between 4 and 250GB.


On Wed, Feb 13, 2013 at 10:35 PM, Michael Colonno wrote:

> More data: I got the Infiniband network (QDR) working well and
> switched my gluster volume to the Infiniband fabric (IPoIB, not RDMA since
> it doesn’t seem to be supported yet for 3.x). The filesystem was slightly
> faster but still well short of what I would expect by a wide margin. Via an
> informal test (timing the movement of a large file) I’m getting several
> MB/s – well short of even a standard Gb network copy. With the faster
> network the CPU load on the brick systems increased dramatically: now I’m
> seeing 200%-250% usage by glusterfsd and glusterfs. 
>
> This leads me to believe that gluster is really not enjoying
> my eight-brick, 2x replication volume with each brick system also being a
> client. I tried a rebalance but no measurable effect. Any suggestions for
> improving the performance? Having each brick be a client of itself seemed
> the most logical choice to remove interdependencies but now I’m doubting
> the setup…
>
> ** **
>
> Thanks,
>
> ~Mike C. 
>
> ** **
>
> *From:* gluster-users-boun...@gluster.org [mailto:
> gluster-users-boun...@gluster.org] *On Behalf Of *Joe Julian
>
> *Sent:* Sunday, February 03, 2013 11:47 AM
> *To:* gluster-users@gluster.org
> *Subject:* Re: [Gluster-users] high CPU load on all bricks
>
> ** **
>
> On 02/03/2013 11:22 AM, Michael Colonno wrote:
>
> 
>
> Having taken a lot more data it does seem the glusterfsd and
> glusterd processes (along with several ksoftirqd) spike up to near 100% on
> both client and brick servers during any file transport across the mount.
> Thankfully this is short-lived for the most part but I’m wondering if this
> is expected behavior or what others have experienced(?) I’m a little
> surprised such a large CPU load would be required to move small files and /
> or use an application within a Gluster mount point. 
>
>
> If you're getting ksoftirqd spikes, that sounds like a hardware issue to
> me. I never see huge spikes like that on my servers nor clients.
>
>
> 
>
>  
>
> I wanted to test this against an NFS mount of the same Gluster
> volume. I managed to get rstatd installed and running but my attempts to
> mount the volume via NFS are met with: 
>
>  
>
> mount.nfs: requested NFS version or transport protocol is not
> supported
>
>  
>
> Relevant line in /etc/fstab:
>
>  
>
> node1:/volume/volumenfs
> defaults,_netdev,vers=3,mountproto=tcp0 0  
>
>  
>
> It looks like CentOS 6.x has NFS version 4 built into everything. So a few
> questions:
>
>  
>
> **-   **Has anyone else noted significant performance differences
> between a glusterfs mount and NFS mount for volumes of 8+ bricks? 
>
> **-   **Is there a straightforward way to make the newer versions of
> CentOS play nice with NFS version 3 + Gluster? 
>
> **-   **Are there any general performance tuning guidelines I can
> follow to improve CPU performance? I found a few references to the cache
> settings but nothing solid. 
>
>  
>
> If the consensus is that NFS will not gain anything then I won’t waste the
> time setting it all up. 
>
>
> NFS gains you the use of FSCache to cache directories and file stats
> making directory listings faster, but it adds overhead decreasing the
> overall throughput (from all the reports I've seen).
>
> I would suspect that you have the kernel nfs server running on your
> servers. Make sure it's disabled.
>
>
> 
>
>  
>
> Thanks,
>
> ~Mike C. 
>
>  
>
>  
>
> *From:* gluster-users-boun...@gluster.org [
> mailto:gluster-users-boun...@gluster.org]
> *On Behalf Of *Michael Colonno
> *Sent:* Friday, February 01, 2013 4:46 PM
> *To:* gluster-users@gluster.org
> *Subject:* Re: [Gluster-users] high CPU load on all bricks
>
>  
>
> Update: after a few hours the CPU usage seems to have dropped
> down to a small value. I did not change anything with respect to the
> configuration or unmount / stop anything as I wanted to see if this would
> persist for a long period of time. Both the client and the self-mounted
> bricks are now showing CPU < 1% (as reported by top). Prior to the larger
> CPU loads I installed a bunch of software into the volume (~ 5 GB total).
> Is this kind a transient behavior – by which I mean larger CPU loads after
> a lot of filesystem activity in short time – typical? This is not a problem
> in my

Re: [Gluster-users] server3_1-fops.c:1240 (Cannot allocate memory)

2013-02-14 Thread Joe Julian 

Note that those are Info notices (" I "), not Errors (" E ").

3.2.2 has known critical bugs. You should upgrade (at least within the 
3.2 branch) asap.


On 02/12/2013 10:19 AM, samuel wrote:

Hi folks,

We're using in an old environment gluster 3.2.2 and we've just started 
seeing the following error in /usr/local/var/log/glusterfs/bricks/gfs-log:


[2013-02-12 19:15:05.214430] I 
[server3_1-fops.c:1240:server_writev_cbk] 0-cloud-server: 2400444243: 
WRITEV 7 (37067) ==> -1 (Cannot allocate memory)
[2013-02-12 19:15:19.463087] I 
[server3_1-fops.c:1240:server_writev_cbk] 0-cloud-server: 2249406582: 
WRITEV 15 (52493) ==> -1 (Cannot allocate memory)


in a distributed replicated 2-node environment.

Both nodes have enough disk space (1.3T) but looks like used memory is 
quite high:


node1:
 total   used   free sharedbuffers cached
Mem:   40476804012700  34980  0276 3728652
-/+ buffers/cache: 2837723763908
Swap:  3906244  199163886328

node2:
 total   used   free sharedbuffers cached
Mem:   40476804012488  35192  0   1088 3713512
-/+ buffers/cache: 2978883749792
Swap:  3905532  252443880288

could it be a memory issue?

Best regards,
Samuel.



___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users


___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] HA with CTDB and NFS or CIFS

2013-02-14 Thread John Mark Walker 
If you're interested in setting up high availability with NFS or SMB, you
may be interested in this webinar:

https://vts.inxpo.com/scripts/Server.nxp

Just remember, when you hear "Red Hat Storage" think "GlusterFS" :)

-JM
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users