Re: [Gluster-users] glusterfsd process thrashing CPU

2014-11-21 Thread Adrian Kan
Sent: Tuesday, November 18, 2014 7:02 PM To: Lindsay Mathieson; gluster-users Subject: Re: [Gluster-users] glusterfsd process thrashing CPU On 11/18/2014 04:14 PM, Lindsay Mathieson wrote: On Tue, 18 Nov 2014 02:36:19 PM Pranith Kumar Karampuri wrote: On 11/18/2014 01:17 PM, Lindsay

Re: [Gluster-users] glusterfsd process thrashing CPU

2014-11-18 Thread Lindsay Mathieson
On 18 November 2014 17:46, Franco Broi franco.b...@iongeo.com wrote: Try strace -Ff -e file -p 'glusterfsd pid' Thanks, Attached Process 27115 attached with 25 threads - interrupt to quit [pid 27122] stat(/mnt/gluster-brick1/datastore, {st_mode=S_IFDIR|0755, st_size=4, ...}) = 0 [pid 11840]

Re: [Gluster-users] glusterfsd process thrashing CPU

2014-11-18 Thread Franco Broi
Can't see how any of that could account for 1000% cpu unless it's just stuck in a loop. On Tue, 2014-11-18 at 18:00 +1000, Lindsay Mathieson wrote: On 18 November 2014 17:46, Franco Broi franco.b...@iongeo.com wrote: Try strace -Ff -e file -p 'glusterfsd pid' Thanks, Attached

Re: [Gluster-users] glusterfsd process thrashing CPU

2014-11-18 Thread Lindsay Mathieson
On 18 November 2014 18:05, Franco Broi franco.b...@iongeo.com wrote: Can't see how any of that could account for 1000% cpu unless it's just stuck in a loop. Currently still varying between 400% to 950% Can glusterfsd be killed without effecting the lgfapi clients? (KVM's)

Re: [Gluster-users] glusterfsd process thrashing CPU

2014-11-18 Thread Pranith Kumar Karampuri
On 11/18/2014 01:17 PM, Lindsay Mathieson wrote: On 18 November 2014 17:40, Pranith Kumar Karampuri pkara...@redhat.com wrote: Sorry didn't see this one. I think this is happening because of 'diff' based self-heal which does full file checksums, that I believe is the root cause. Could you

Re: [Gluster-users] glusterfsd process thrashing CPU

2014-11-18 Thread Lindsay Mathieson
On Tue, 18 Nov 2014 02:36:19 PM Pranith Kumar Karampuri wrote: On 11/18/2014 01:17 PM, Lindsay Mathieson wrote: On 18 November 2014 17:40, Pranith Kumar Karampuri pkara...@redhat.com wrote: However given the files are tens of GB in size, won't it thrash my network? Yes you are right.

Re: [Gluster-users] glusterfsd process thrashing CPU

2014-11-18 Thread Pranith Kumar Karampuri
On 11/18/2014 04:14 PM, Lindsay Mathieson wrote: On Tue, 18 Nov 2014 02:36:19 PM Pranith Kumar Karampuri wrote: On 11/18/2014 01:17 PM, Lindsay Mathieson wrote: On 18 November 2014 17:40, Pranith Kumar Karampuri pkara...@redhat.com wrote: However given the files are tens of GB in size,

[Gluster-users] glusterfsd process thrashing CPU

2014-11-17 Thread Lindsay Mathieson
2 Node replicate setup, Everything has been stable for days untill I had occasion to reboot one of the nodes. Since then (past hour) glusterfsd has been pegging the CPU(s), utilization ranging from 1% to 1000% ! On average its around 500% This is a vm server, so there are only 27 VM images for

Re: [Gluster-users] glusterfsd process thrashing CPU

2014-11-17 Thread Lindsay Mathieson
ps. There is very little network traffic happening -- Lindsay ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] glusterfsd process thrashing CPU

2014-11-17 Thread Lindsay Mathieson
And its happening on both nodes now, they have become near unusable. On 18 November 2014 17:03, Lindsay Mathieson lindsay.mathie...@gmail.com wrote: ps. There is very little network traffic happening -- Lindsay -- Lindsay ___ Gluster-users

Re: [Gluster-users] glusterfsd process thrashing CPU

2014-11-17 Thread Franco Broi
glusterfsd is the filesystem daemon. You could trace strace'ing it to see what it's doing. On Tue, 2014-11-18 at 17:09 +1000, Lindsay Mathieson wrote: And its happening on both nodes now, they have become near unusable. On 18 November 2014 17:03, Lindsay Mathieson

Re: [Gluster-users] glusterfsd process thrashing CPU

2014-11-17 Thread Pranith Kumar Karampuri
On 11/18/2014 12:32 PM, Lindsay Mathieson wrote: 2 Node replicate setup, Everything has been stable for days untill I had occasion to reboot one of the nodes. Since then (past hour) glusterfsd has been pegging the CPU(s), utilization ranging from 1% to 1000% ! On average its around 500% This

Re: [Gluster-users] glusterfsd process thrashing CPU

2014-11-17 Thread Pranith Kumar Karampuri
On 11/18/2014 01:05 PM, Pranith Kumar Karampuri wrote: On 11/18/2014 12:32 PM, Lindsay Mathieson wrote: 2 Node replicate setup, Everything has been stable for days untill I had occasion to reboot one of the nodes. Since then (past hour) glusterfsd has been pegging the CPU(s), utilization

Re: [Gluster-users] glusterfsd process thrashing CPU

2014-11-17 Thread Lindsay Mathieson
Gluster 3.5.2 Very few files - its purely a VM image host, 27 files, 10 - 60GB in size. seems to be undergoing a heal: root@vnb:~# gluster volume heal datastore1 info Brick vnb:/mnt/gluster-brick1/datastore/ /images/108/vm-108-disk-1.qcow2 - Possibly undergoing heal

Re: [Gluster-users] glusterfsd process thrashing CPU

2014-11-17 Thread Lindsay Mathieson
Sorry, meant to send to the list. strace attached. On 18 November 2014 17:35, Pranith Kumar Karampuri pkara...@redhat.com wrote: On 11/18/2014 12:32 PM, Lindsay Mathieson wrote: 2 Node replicate setup, Everything has been stable for days untill I had occasion to reboot one of the nodes.

Re: [Gluster-users] glusterfsd process thrashing CPU

2014-11-17 Thread Franco Broi
Try strace -Ff -e file -p 'glusterfsd pid' On Tue, 2014-11-18 at 17:42 +1000, Lindsay Mathieson wrote: Sorry, meant to send to the list. strace attached. On 18 November 2014 17:35, Pranith Kumar Karampuri pkara...@redhat.com wrote: On 11/18/2014 12:32 PM, Lindsay Mathieson wrote:

Re: [Gluster-users] glusterfsd process thrashing CPU

2014-11-17 Thread Lindsay Mathieson
On 18 November 2014 17:40, Pranith Kumar Karampuri pkara...@redhat.com wrote: Sorry didn't see this one. I think this is happening because of 'diff' based self-heal which does full file checksums, that I believe is the root cause. Could you execute 'gluster volume set volname