Re: [Gluster-users] Very slow directory listing and high CPU usage on replicated volume

On 11/06/2012 11:27 AM, Fernando Frediani (Qube) wrote: > Why does it need to rely on FUSE. Why can't it be something that run in > kernel that doesn't have any reliance on FUSE ? I imagine that would require > a lot of engineering but the benefits no need to mention. "A lot" doesn't even capture

Re: [Gluster-users] Very slow directory listing and high CPU usage on replicated volume

On 11/06/2012 04:38 AM, Joe Landman wrote: On 11/06/2012 04:35 AM, Fernando Frediani (Qube) wrote: Joe, I don't think we have to accept this as this is not acceptable thing. I understand your unhappyness with it. But its "free" and you sometimes have to accept what you get for "free". I

Re: [Gluster-users] Very slow directory listing and high CPU usage on replicated volume

Very slow directory listing and high CPU usage on replicated volume On 11/06/2012 04:35 AM, Fernando Frediani (Qube) wrote: > Joe, > > I don't think we have to accept this as this is not acceptable thing. I understand your unhappyness with it. But its "free" and you sometimes h

Re: [Gluster-users] Very slow directory listing and high CPU usage on replicated volume

On 11/06/2012 04:35 AM, Fernando Frediani (Qube) wrote: Joe, I don't think we have to accept this as this is not acceptable thing. I understand your unhappyness with it. But its "free" and you sometimes have to accept what you get for "free". I have seen countless people complaining about

Re: [Gluster-users] Very slow directory listing and high CPU usage on replicated volume

That's a very good point. I've been evaluating glusterfs from version 1.0 and refused to use it for one and only reason: the split-brain problem. With version 3.3 I have finally switched to glusterfs, but after a few months of production usage, I'm thinking of going back to separate servers wit

Re: [Gluster-users] Very slow directory listing and high CPU usage on replicated volume

On Tue, Nov 06, 2012 at 09:35:29AM +, Fernando Frediani (Qube) wrote: > I don't think we have to accept this as this is not acceptable thing. I have > seen countless people complaining about this problem for a while and seems no > improvements have been done. Just a thought: can you try "net

Re: [Gluster-users] Very slow directory listing and high CPU usage on replicated volume

<<< text/html; charset=utf-8: Unrecognized >>> ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Very slow directory listing and high CPU usage on replicated volume

so why Gluster have to ? -Original Message- From: gluster-users-boun...@gluster.org [mailto:gluster-users-boun...@gluster.org] On Behalf Of Joe Landman Sent: 05 November 2012 15:07 To: gluster-users@gluster.org Subject: Re: [Gluster-users] Very slow directory listing and high CPU usage o

Re: [Gluster-users] Very slow directory listing and high CPU usage on replicated volume

: [Gluster-users] Very slow directory listing and high CPU usage on replicated volume On 11/05/2012 09:57 AM, harry mangalam wrote: > Jeff Darcy wrote a nice piece in his hekafs blog about 'the importance > of keeping things sequential' which is essentially about the > contention

Re: [Gluster-users] Very slow directory listing and high CPU usage on replicated volume

I would also watch cpu and memory usage. Maybe try the deadline scheduler. Monitor your switch. Since it works fine at first, look for what changes. Brian Candler wrote: >On Mon, Nov 05, 2012 at 01:10:40PM -0500, Jonathan Lefman wrote: >>Thanks Brian. I tried what you recommended. At firs

Re: [Gluster-users] Very slow directory listing and high CPU usage on replicated volume

On Mon, Nov 05, 2012 at 01:10:40PM -0500, Jonathan Lefman wrote: >Thanks Brian. I tried what you recommended. At first I was very >encouraged when I saw things moving across the wire. But about 15 >minutes into the transfer things ground to a halt. I am currently >running across

Re: [Gluster-users] Very slow directory listing and high CPU usage on replicated volume

Thanks Brian. I tried what you recommended. At first I was very encouraged when I saw things moving across the wire. But about 15 minutes into the transfer things ground to a halt. I am currently running across a GigE channel. Things were moving about 20-40 MB/s but when things stopped moving

Re: [Gluster-users] Very slow directory listing and high CPU usage on replicated volume

If your disks are >1TB with XFS then try mount -o inode64 This has the effect of sequential writes into the same directory being localised next to each other (within the same allocation group). When you skip to the next directory you will probably get a different allocation group. Without this, t

Re: [Gluster-users] Very slow directory listing and high CPU usage on replicated volume

On 11/05/2012 09:57 AM, harry mangalam wrote: Jeff Darcy wrote a nice piece in his hekafs blog about 'the importance of keeping things sequential' which is essentially about the contention for heads between data io and journal io.

Re: [Gluster-users] Very slow directory listing and high CPU usage on replicated volume

Jeff Darcy wrote a nice piece in his hekafs blog about 'the importance of keeping things sequential' which is essentially about the contention for heads between data io and journal io. (also congrats on the Linux Journal

Re: [Gluster-users] Very slow directory listing and high CPU usage on replicated volume

I take it back. Things deteriorated pretty quickly after I began dumping data onto my volume from multiple clients. Initially my transfer rates were okay, not fast, but livable. However after about an hour of copying several terabytes from 3-4 client machines, the rates of transfer often dropped to

Re: [Gluster-users] Very slow directory listing and high CPU usage on replicated volume

I should have also said that my volume is working well now and all is well. -Jon On Fri, Nov 2, 2012 at 1:21 PM, Jonathan Lefman wrote: > Thank you Brian. I'm happy to hear that this behavior is not typical. I am > now using xfs on all of my drives. I also wiped out the entire > /etc/glusterd

Re: [Gluster-users] Very slow directory listing and high CPU usage on replicated volume

Thank you Brian. I'm happy to hear that this behavior is not typical. I am now using xfs on all of my drives. I also wiped out the entire /etc/glusterd directory for good measure. I bet that there was residual information from a previous attempt at a gluster volume that must have caused problems

Re: [Gluster-users] Very slow directory listing and high CPU usage on replicated volume

--- Original Message - From: "Jonathan Lefman" To: "Pranith Kumar Karampuri" Cc: gluster-users@gluster.org Sent: Friday, November 2, 2012 8:46:44 AM Subject: Re: [Gluster-users] Very slow directory listing and high CPU usage on replicated volume Thanks Pranith. We have tried

Re: [Gluster-users] Very slow directory listing and high CPU usage on replicated volume

On Thu, Nov 01, 2012 at 08:03:21PM -0400, Jonathan Lefman wrote: >Soon after loading up about 100 MB of small files (about 300kb each), >the drive usage is at 1.1T. That is very odd. What do you get if you run du and df on the individual bricks themselves? 100MB is only ~330 files of 300KB

Re: [Gluster-users] Very slow directory listing and high CPU usage on replicated volume

From: "Jonathan Lefman" > To: gluster-users@gluster.org > Sent: Friday, November 2, 2012 5:33:21 AM > Subject: [Gluster-users] Very slow directory listing and high CPU usage on > replicated volume > > > Hi all, > > > I am having problems with painfully slow

Re: [Gluster-users] Very slow directory listing and high CPU usage on replicated volume

: "Jonathan Lefman" To: gluster-users@gluster.org Sent: Friday, November 2, 2012 5:33:21 AM Subject: [Gluster-users] Very slow directory listing and high CPU usage on replicated volume Hi all, I am having problems with painfully slow directory listings on a freshly created replica

Re: [Gluster-users] Very slow directory listing and high CPU usage on replicated volume

Thanks for the response. Yes, only gluster bricks on the disk. I know about the large count of small files issue, but we changed how we organized the files. Each directory has about 30 files in it. Am I missing something? On Thu, Nov 1, 2012 at 9:24 PM, Jules Wang wrote: > Hi John: > Glu

Re: [Gluster-users] Very slow directory listing and high CPU usage on replicated volume

Hi John: Glusterfs is not designed for handling large count small files, because it has no meta data server, every lookup operation cost a lot in your situation. The disk usage is abnormal, does your disk only have gluster bricks? Best Regards. Jules Wang At 2012-11-02 08:03:21,"Jona

Re: [Gluster-users] Very slow directory listing and high CPU usage on replicated volume

I am attaching some extra information to help diagnose this issue. Hopefully this is useful. Volume Name: my_gluster_data Type: Distributed-Replicate Volume ID: e8865f37-6e22-476d-956e-29280bb07e75 Status: Started Number of Bricks: 3 x 2 = 6 Transport-type: tcp Bricks: Brick1: server1:/media/data

[Gluster-users] Very slow directory listing and high CPU usage on replicated volume

Hi all, I am having problems with painfully slow directory listings on a freshly created replicated volume. The configuration is as follows: 2 nodes with 3 replicated drives each. The total volume capacity is 5.6T. We would like to expand the storage capacity much more, but first we need to f