Re: [Gluster-users] Glusterfs virtual memory growing up

[Tamas Papp]

On 11/19/2014 02:27 AM, Juan José Pavlik Salles wrote:
Seems like a big jump to take, updating from 3.3.2 to 3.5, is it a 
plug&play upgrade?


Yes, it was easy.

t
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] heal questions

[Ravishankar N]

On 11/19/2014 03:23 AM, Lindsay Mathieson wrote:
> Just some basic question on the heal process, please just point me to the 
> docs 
> if they are there :)
> 
> - How is the need for a heal detected? I presume nodes can detect when they 
> can't sync writes to the other nodes. This is flagged (xattr?) for healing 
> when the other nodes are back up?
> 
> - How is the master file chosen? quorum? latest write?

Self heal is detection/healing is handled by AFR using extended attributes 
which keep track of the type of heal (data/meta-data/entry) and the direction 
of heal (i.e. what the sources and sinks are).
More details can be found here 
:https://github.com/gluster/glusterfs/blob/master/doc/features/afr-v1.md


> 
> - I have a three peer, two node/brick replica with server side quorum. The 
> third peer was added to give an odd number of peers for quorom, but it 
> doesn't 
> participate in bricks. Does this make sense? it helps with deciding which 
> node 
> is master for heals?

Server quorum is useful for detecting split-brains due to graph change, etc but 
I guess it can still lead to data split-brains. Client quorum can be used to 
avoid data split-brains at the cost of
availability. More information here:

https://access.redhat.com/documentation/en-US/Red_Hat_Storage/2.1/html/Administration_Guide/ch10s10.html
https://access.redhat.com/documentation/en-US/Red_Hat_Storage/2.0/html/Administration_Guide/sect-User_Guide-Managing_Volumes-Quorum.html
http://www.gluster.org/community/documentation/index.php/Features/Server-quorum



> 
> - diff heals. I'm guessing the nodes scan the file pairs block by block, 
> calculating a checksum for each block and comparing, if they don't match, the 
> master node block is replicated.

That is correct. It scans in 128 kb chunks (configurable via the 
`cluster.self-heal-window-size` volume set option).
> 
> - Synchronizing the compare over the network must be fun ...
>   
> 
> - Is there anyway to display a file heal progress? it would make the process 
> a 
> lot less anxiety inducing even if it was very slow.

As of now, `gluster volume heal  info` gives only the list of files 
that need healing. But yes, it would be nice to have some sort of a CLI to 
monitor the % progress of files that are
currently under data self-heal.


HTH,
Ravi

> 
> thanks,
> 
> 
> 
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
> 

___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Glusterfs virtual memory growing up

[Juan José Pavlik Salles]

Seems like a big jump to take, updating from 3.3.2 to 3.5, is it a
plug&play upgrade?

2014-11-18 17:51 GMT-03:00 Tamas Papp :

>
> On 11/18/2014 09:41 PM, Juan José Pavlik Salles wrote:
>
>> Did you have the dame problem? Is it a memory leak?
>>
>
> Yes.
>
> After the upgrade it works quite well.
>
> tamas
>



-- 
Pavlik Salles Juan José
Blog - http://viviendolared.blogspot.com
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] heal questions

[Lindsay Mathieson]

Just some basic question on the heal process, please just point me to the docs 
if they are there :)

- How is the need for a heal detected? I presume nodes can detect when they 
can't sync writes to the other nodes. This is flagged (xattr?) for healing 
when the other nodes are back up?

- How is the master file chosen? quorum? latest write?

- I have a three peer, two node/brick replica with server side quorum. The 
third peer was added to give an odd number of peers for quorom, but it doesn't 
participate in bricks. Does this make sense? it helps with deciding which node 
is master for heals?

- diff heals. I'm guessing the nodes scan the file pairs block by block, 
calculating a checksum for each block and comparing, if they don't match, the 
master node block is replicated.

- Synchronizing the compare over the network must be fun ...
  

- Is there anyway to display a file heal progress? it would make the process a 
lot less anxiety inducing even if it was very slow.

thanks,

-- 
Lindsay

signature.asc
Description: This is a digitally signed message part.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Glusterfs virtual memory growing up

[Tamas Papp]

On 11/18/2014 09:41 PM, Juan José Pavlik Salles wrote:

Did you have the dame problem? Is it a memory leak?


Yes.

After the upgrade it works quite well.

tamas
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Glusterfs virtual memory growing up

[Juan José Pavlik Salles]

Did you have the dame problem? Is it a memory leak?

2014-11-18 16:28 GMT-03:00 Tamas Papp :

>   Switching to 3.5 helped us a _lot_.
>
> --
> Sent from mobile
>
> On November 18, 2014 7:48:45 PM Juan José Pavlik Salles <
> jjpav...@gmail.com> wrote:
>
>> Hi guys, I've a small cluster with 5 nodes running gluster 3.3.2 on
>> Centos 6.X, one of them is having an unstoppable growing in its virtual
>> memory consumption:
>>
>> [image: Imágenes integradas 1]
>>
>> Using top you can see that glusterfs is using 117 of virt memory:
>>
>> [image: Imágenes integradas 2]
>>
>> According to the graphs it's been growing for the last 4 days, and I
>> really don't know why because the server isn't doing anything new. Is there
>> any known memory leak? Is it going to crash eventually? There's nothing
>> interesting on the log files.
>>
>> Thanksss
>>
>> --
>> Pavlik Salles Juan José
>> Blog - http://viviendolared.blogspot.com
>>  ___
>> Gluster-users mailing list
>> Gluster-users@gluster.org
>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>
>


-- 
Pavlik Salles Juan José
Blog - http://viviendolared.blogspot.com
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] VFS plug-in for Gluster breaks case sensitivity.

[Jon]

Hello, I was wondering if there has been any progress on reproducing this error 
or if there is any more info I can provide.

Thanks, Jon
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Sparse Files and Heal

[Lindsay Mathieson]

On Tue, 18 Nov 2014 08:26:39 PM you wrote:
>  I can CC you to the bugzilla, so that you can see the update on the bug
> once it is fixed. Do you want to be CCed to the bug?


Yes please,

thanks


p.s switched back to diif and all heals finished this morning :)
-- 
Lindsay

signature.asc
Description: This is a digitally signed message part.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] GlusterFS 3.5.3 and 3.4.6 released

[Dave McAllister]

The Gluster community is please to announce the release of updated 
releases for the 3.4 and 3.5 family. With the release of 3.6 a few weeks 
ago, this is brings all the current members of GlusterFS into a more 
stable, production ready status.


The GlusterFS 3.4.6 release is focused on bug fixes. The release notes 
are available at 
https://github.com/gluster/glusterfs/blob/v3.4.6/doc/release-notes/3.4.6.md. 
Download the latest GlusterFS 3.4.6 at 
http://download.gluster.org/pub/gluster/glusterfs/3.4/3.4.6/


GlusterFS 3.5.3 is also a bug-fix oriented release. The associated 
release notes are at 
https://github.com/gluster/glusterfs/blob/v3.5.3/doc/release-notes/3.5.3.md. 
Download the latest GlusterFS3.5.3 at 
http://download.gluster.org/pub/gluster/glusterfs/3.5/3.5.3/


Also, the latest GlusterFS, 3.6.1, is available for download at 
http://download.gluster.org/pub/gluster/glusterfs/3.6/3.6.1/


Obviously, the Gluster community has been hard at work, and w’re not 
stopping there. We invite you to join in for planning the next releases. 
GlusterFS 3.7 planning 
 
and GlusterFS 4.0 
 
planning would love your input.


--
Dave McAllister

GlusterFS: Your simple solution to scale-out file systems

___
Announce mailing list
annou...@gluster.org
http://supercolony.gluster.org/mailman/listinfo/announce
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Sparse Files and Heal

[Joe Julian]

On 11/18/2014 06:56 AM, Pranith Kumar Karampuri wrote:


On 11/18/2014 05:35 PM, Lindsay Mathieson wrote:


I have a VM image which is a sparse file - 512GB allocated, but only 
32GB used.


root@vnb:~# ls -lh /mnt/gluster-brick1/datastore/images/100

total 31G

-rw--- 2 root root 513G Nov 18 19:57 vm-100-disk-1.qcow2

I switched to full sync and rebooted.

heal was started on the image and it seemed to be just transfering 
the full file from node vnb to vng. iftop showed bandwidth at 500 Mb/s


Eventually the cumulative transfer got to 140GB which seemed odd as 
the real file size was 31G. I logged onto the second node (vng) and 
the *real* file size size was up to 191Gb.


It looks like the heal is not handling sparse files, rather it is 
transferring empty bytes to make up the allocated size. Thats a 
serious problem for the common habit of over committing your disk 
space with vm images. Not to mention the inefficiency.


Ah! this problem doesn't exist in diff self-heal :-(. Because the 
checksums of the files will match in the sparse regions. In full 
self-heal it just reads from the source file and writes to the sink 
file. What we can change there is if the file is a sparse file and the 
data that is read is all zeros (read will return all zeros as data in 
the sparse region) then read the stale file and compare if it is also 
all zeros. If both are 'zeros' then skip the write. I also checked 
that if the sparse file is created while the other brick is down, then 
also it preserves the holes(i.e. sparse regions). This problem only 
appears when both the files in their full size exist on both the 
bricks and full self-heal is done like here :-(.


Thanks for your valuable inputs. So basically you found 2 issues. I 
will raise 2 bugs one for each of the issues you found. I can CC you 
to the bugzilla, so that you can see the update on the bug once it is 
fixed. Do you want to be CCed to the bug?


Hey, nice. This will really speed up full heals for most common vm images.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Sparse Files and Heal

[Pranith Kumar Karampuri]

On 11/18/2014 05:35 PM, Lindsay Mathieson wrote:


I have a VM image which is a sparse file - 512GB allocated, but only 
32GB used.


root@vnb:~# ls -lh /mnt/gluster-brick1/datastore/images/100

total 31G

-rw--- 2 root root 513G Nov 18 19:57 vm-100-disk-1.qcow2

I switched to full sync and rebooted.

heal was started on the image and it seemed to be just transfering the 
full file from node vnb to vng. iftop showed bandwidth at 500 Mb/s


Eventually the cumulative transfer got to 140GB which seemed odd as 
the real file size was 31G. I logged onto the second node (vng) and 
the *real* file size size was up to 191Gb.


It looks like the heal is not handling sparse files, rather it is 
transferring empty bytes to make up the allocated size. Thats a 
serious problem for the common habit of over committing your disk 
space with vm images. Not to mention the inefficiency.


Ah! this problem doesn't exist in diff self-heal :-(. Because the 
checksums of the files will match in the sparse regions. In full 
self-heal it just reads from the source file and writes to the sink 
file. What we can change there is if the file is a sparse file and the 
data that is read is all zeros (read will return all zeros as data in 
the sparse region) then read the stale file and compare if it is also 
all zeros. If both are 'zeros' then skip the write. I also checked that 
if the sparse file is created while the other brick is down, then also 
it preserves the holes(i.e. sparse regions). This problem only appears 
when both the files in their full size exist on both the bricks and full 
self-heal is done like here :-(.


Thanks for your valuable inputs. So basically you found 2 issues. I will 
raise 2 bugs one for each of the issues you found. I can CC you to the 
bugzilla, so that you can see the update on the bug once it is fixed. Do 
you want to be CCed to the bug?


Pranith


thanks,

--

Lindsay



___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users


___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Todays minutes of the Gluster Community Bug Triage meeting

[Niels de Vos]

On Mon, Nov 17, 2014 at 08:57:01PM +0100, Niels de Vos wrote:
> Hi all,
> 
> Tomorrow (Tuesday) we will have an other Gluster Community Bug Triage
> meeting.
> 
> Meeting details:
> - location: #gluster-meeting on Freenode IRC
> - date: every Tuesday
> - time: 12:00 UTC, 13:00 CET (in your terminal, run: date -d "12:00 UTC")
> - agenda: https://public.pad.fsfe.org/p/gluster-bug-triage
> 
> Currently the following items are listed:
> * Roll Call
> * Status of last weeks action items
> * Replacing old/unsupported versions in Bugzilla with "unsupported"?
> * Group Triage
> * Open Floor
> 
> The last two topics have space for additions. If you have a suitable bug
> or topic to discuss, please add it to the agenda.


Minutes: 
http://meetbot.fedoraproject.org/gluster-meeting/2014-11-18/gluster-meeting.2014-11-18-12.01.html
Minutes (text): 
http://meetbot.fedoraproject.org/gluster-meeting/2014-11-18/gluster-meeting.2014-11-18-12.01.txt
Log: 
http://meetbot.fedoraproject.org/gluster-meeting/2014-11-18/gluster-meeting.2014-11-18-12.01.log.html

Meeting summary
---
* Roll Call  (ndevos, 12:02:05)

* Status of last weeks action items  (ndevos, 12:06:28)

* hagarth will look for somebody that can act like a bug assigner
  manager kind of person  (ndevos, 12:06:40)

* jdarcy will try the RSS-feeds and will report  (ndevos, 12:08:19)
  * AGREED: RSS feeds are helpful with keeping up on new bugs  (ndevos,
12:09:57)
  * IDEA: Suggest would-be triagers to use RSS-feeds for triage
(ndevos, 12:10:26)

* Humble will request the replacement of old versions by a new
  "unsupported" version  (ndevos, 12:10:34)

* hagarth should update the MAINTAINERS file, add current maintainers
  and new components like Snapshot  (ndevos, 12:12:07)

* hagarth will look for somebody that can act like a bug assigner
  manager kind of person  (ndevos, 12:13:03)

* Group Triage  (ndevos, 12:13:41)

* How to make it easier for would-be triagers to help out?  (ndevos,
  12:38:51)
  * ACTION: jdarcy will update the Bug Triage wiki page and
meeting-agenda to split FutureFeature bugs from un-Triaged ones
(ndevos, 13:04:24)

Meeting ended at 13:10:49 UTC.


pgpk7WGMbbdLJ.pgp
Description: PGP signature
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Sparse Files and Heal

[Lindsay Mathieson]

I have a VM image which is a sparse file - 512GB allocated, but only 32GB used.


root@vnb:~# ls -lh /mnt/gluster-brick1/datastore/images/100
total 31G
-rw--- 2 root root 513G Nov 18 19:57 vm-100-disk-1.qcow2


I switched to full sync and rebooted.

heal was started on the image and it seemed to be just transfering the full 
file from node
vnb to vng. iftop showed bandwidth at 500 Mb/s

Eventually the cumulative transfer got to 140GB which seemed odd as the real 
file size was
31G. I logged onto the second node (vng) and the *real* file size  size was up 
to 191Gb.

It looks like the heal is not handling sparse files, rather it is transferring 
empty bytes to
make up the allocated size. Thats a serious problem for the common habit of over
committing your disk space with vm images. Not to mention the inefficiency.

thanks,

--
Lindsay


signature.asc
Description: This is a digitally signed message part.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] gluster volume heal info question

[Lindsay Mathieson]

On Tue, 18 Nov 2014 06:16:53 AM you wrote:
> heal info command that you executed basically gives a list of files to be
> healed. So in the above output, 1 entry is possibly getting healed and
> other 7 need to be healed.
> > 
> > 
> >
> > And what is a gfid?
> 
> In glusterfs, gfid (glusterfs ID) is similar to an inode number. It is a
> unique identification number given to each file.


Thanks
-- 
Lindsay

signature.asc
Description: This is a digitally signed message part.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] gluster volume heal info question

[Anuradha Talur]

- Original Message -
> From: "Lindsay Mathieson" 
> To: "gluster-users" 
> Sent: Tuesday, November 18, 2014 4:24:51 PM
> Subject: [Gluster-users] gluster volume heal  info question
> 
> 
> 
> When I run the subject I get:
> 
> 
> 
> root@vnb:~# gluster volume heal datastore1 info
> 
> Brick vnb:/mnt/gluster-brick1/datastore/
> 
> /images/100/vm-100-disk-1.qcow2 - Possibly undergoing heal
> 
> 
> 
> 
> 
> 
> 
> /images/102/vm-102-disk-1.qcow2
> 
> 
> 
> 
> 
> /images/400/vm-400-disk-1.qcow2
> 
> Number of entries: 8
> 
> 
> 
> 
> 
> 
> 
> it has 8 entries but only one is possibly undergoing heal. Whats with the
> other 7?
> 
heal info command that you executed basically gives a list of files to be 
healed. So in the above output, 1 entry is possibly getting healed and other 7 
need to be healed.
> 
> 
> And what is a gfid?

In glusterfs, gfid (glusterfs ID) is similar to an inode number. It is a unique 
identification number given to each file.
> 
> 
> 
> thanks
> 
> 
> 
> 
> 
> --
> 
> Lindsay
> 
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users

-- 
Thanks,
Anuradha.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] glusterfsd process thrashing CPU

[Pranith Kumar Karampuri]

On 11/18/2014 04:14 PM, Lindsay Mathieson wrote:

On Tue, 18 Nov 2014 02:36:19 PM Pranith Kumar Karampuri wrote:

On 11/18/2014 01:17 PM, Lindsay Mathieson wrote:

On 18 November 2014 17:40, Pranith Kumar Karampuri 

wrote:

However given the files are tens of GB in size, won't it thrash my
network?

Yes you are right. I wonder why thrashing of the network is never
reported till now.

Not sure if you are being sarcastic or not :) But from what I've observed,
sync operations seem to self throttle, I've not seen them use more than 50% of
bandwidth, and given most setups have a dedicated network for the servers
maybe they just don't notice if it takes a while?
No, I was not being sarcastic :-). I am genuinely wondering why it is 
not reported till now. May be Joe will have more inputs there, that is 
the reason I CCed him.



I still need to think about how best to solve this problem.

Setup a array of queues for self healing, sorted by size maybe?


Let me tell you a bit more about this issue:
there are two processes which heal the VM images:
1) self-heal-daemon. 2) Mount process.
Self-heal daemon heals one VM image at a time. But mount process
triggers self-heals for all the opened files(VM image is nothing but an
opened file from filesystem's perspective) when a brick goes down and
comes backup.


Thanks, interesting to know.


So we need to come up with a scheme to throttle self-heals
on the mount point to prevent this issue. I will update you as soon as I
come up with a fix. This should not be hard to do. Need some time to
choose the best approach. Thanks a lot for bringing up this issue.

Thanks you for looking at it!

Cheers,




___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users


___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] gluster volume heal info question

[Lindsay Mathieson]

When I run the subject I get:

root@vnb:~# gluster volume heal datastore1 info
Brick vnb:/mnt/gluster-brick1/datastore/
/images/100/vm-100-disk-1.qcow2 - Possibly undergoing heal



/images/102/vm-102-disk-1.qcow2


/images/400/vm-400-disk-1.qcow2
Number of entries: 8



it has 8 entries but only one is possibly undergoing heal. Whats with the other 
7?

And what is a gfid?

thanks


--
Lindsay


signature.asc
Description: This is a digitally signed message part.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] glusterfsd process thrashing CPU

[Lindsay Mathieson]

On Tue, 18 Nov 2014 02:36:19 PM Pranith Kumar Karampuri wrote:
> On 11/18/2014 01:17 PM, Lindsay Mathieson wrote:
> > On 18 November 2014 17:40, Pranith Kumar Karampuri  
wrote:
> > 
> > However given the files are tens of GB in size, won't it thrash my
> > network?
> 
> Yes you are right. I wonder why thrashing of the network is never
> reported till now.

Not sure if you are being sarcastic or not :) But from what I've observed, 
sync operations seem to self throttle, I've not seen them use more than 50% of 
bandwidth, and given most setups have a dedicated network for the servers 
maybe they just don't notice if it takes a while?

> I still need to think about how best to solve this problem.

Setup a array of queues for self healing, sorted by size maybe?

> 
> Let me tell you a bit more about this issue:
> there are two processes which heal the VM images:
> 1) self-heal-daemon. 2) Mount process.
> Self-heal daemon heals one VM image at a time. But mount process
> triggers self-heals for all the opened files(VM image is nothing but an
> opened file from filesystem's perspective) when a brick goes down and
> comes backup.


Thanks, interesting to know.

> So we need to come up with a scheme to throttle self-heals
> on the mount point to prevent this issue. I will update you as soon as I
> come up with a fix. This should not be hard to do. Need some time to
> choose the best approach. Thanks a lot for bringing up this issue.

Thanks you for looking at it!

Cheers,


-- 
Lindsay

signature.asc
Description: This is a digitally signed message part.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] glusterfsd process thrashing CPU

[Pranith Kumar Karampuri]

On 11/18/2014 01:17 PM, Lindsay Mathieson wrote:

On 18 November 2014 17:40, Pranith Kumar Karampuri  wrote:

Sorry didn't see this one. I think this is happening because of 'diff' based
self-heal which does full file checksums, that I believe is the root cause.
Could you execute 'gluster volume set 
cluster.data-self-heal-algorithm full' to prevent this issue in future. But
this option will be effective for the new self-heals that will be triggered
after the execution of the command. The ongoing ones will still use the old
mode of self-heal.

Thanks, makes sense.

However given the files are tens of GB in size, won't it thrash my network?
Yes you are right. I wonder why thrashing of the network is never 
reported till now.
+Joejulian who also uses VMs on gluster(for 5 years now?). He uses this 
option of full self-heal (Thats what I saw in his bug reports).


I still need to think about how best to solve this problem.

Let me tell you a bit more about this issue:
there are two processes which heal the VM images:
1) self-heal-daemon. 2) Mount process.
Self-heal daemon heals one VM image at a time. But mount process 
triggers self-heals for all the opened files(VM image is nothing but an 
opened file from filesystem's perspective) when a brick goes down and 
comes backup. So we need to come up with a scheme to throttle self-heals 
on the mount point to prevent this issue. I will update you as soon as I 
come up with a fix. This should not be hard to do. Need some time to 
choose the best approach. Thanks a lot for bringing up this issue.


Pranith

___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users


___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] glusterfsd process thrashing CPU

[Lindsay Mathieson]

On 18 November 2014 18:05, Franco Broi  wrote:
>
> Can't see how any of that could account for 1000% cpu unless it's just
> stuck in a loop.


Currently still varying between 400% to 950%

Can glusterfsd be killed without effecting the lgfapi clients? (KVM's)
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] glusterfsd process thrashing CPU

[Franco Broi]

Can't see how any of that could account for 1000% cpu unless it's just
stuck in a loop.

On Tue, 2014-11-18 at 18:00 +1000, Lindsay Mathieson wrote: 
> On 18 November 2014 17:46, Franco Broi  wrote:
> >
> > Try strace -Ff -e file -p 'glusterfsd pid'
> 
> Thanks, Attached
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users


___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] glusterfsd process thrashing CPU

[Lindsay Mathieson]

On 18 November 2014 17:46, Franco Broi  wrote:
>
> Try strace -Ff -e file -p 'glusterfsd pid'

Thanks, Attached
Process 27115 attached with 25 threads - interrupt to quit
[pid 27122] stat("/mnt/gluster-brick1/datastore", {st_mode=S_IFDIR|0755, 
st_size=4, ...}) = 0
[pid 11840] lstat("/mnt/gluster-brick1/datastore/", {st_mode=S_IFDIR|0755, 
st_size=4, ...}) = 0
[pid 11840] lgetxattr("/mnt/gluster-brick1/datastore/", 
"system.posix_acl_default", 0x0, 0) = -1 EOPNOTSUPP (Operation not supported)
[pid 11840] lgetxattr("/mnt/gluster-brick1/datastore/", 
"system.posix_acl_access", 0x0, 0) = -1 EOPNOTSUPP (Operation not supported)
[pid 11840] lgetxattr("/mnt/gluster-brick1/datastore/", "trusted.glusterfs.dht" 

[pid 29198] lstat("/mnt/gluster-brick1/datastore/", {st_mode=S_IFDIR|0755, 
st_size=4, ...}) = 0
[pid 29198] lgetxattr("/mnt/gluster-brick1/datastore/", 
"system.posix_acl_default", 0x0, 0) = -1 EOPNOTSUPP (Operation not supported)
[pid 29198] lgetxattr("/mnt/gluster-brick1/datastore/", 
"system.posix_acl_access", 0x0, 0) = -1 EOPNOTSUPP (Operation not supported)
[pid 29198] lgetxattr("/mnt/gluster-brick1/datastore/", "trusted.glusterfs.dht" 

[pid 29197] lstat("/mnt/gluster-brick1/datastore/", {st_mode=S_IFDIR|0755, 
st_size=4, ...}) = 0
[pid 29197] lgetxattr("/mnt/gluster-brick1/datastore/", 
"system.posix_acl_default", 0x0, 0) = -1 EOPNOTSUPP (Operation not supported)
[pid 29197] lgetxattr("/mnt/gluster-brick1/datastore/", 
"system.posix_acl_access", 0x0, 0) = -1 EOPNOTSUPP (Operation not supported)
[pid 29197] lgetxattr("/mnt/gluster-brick1/datastore/", "trusted.glusterfs.dht" 

[pid 11840] <... lgetxattr resumed> , 0x0, 0) = 16
[pid 11840] lgetxattr("/mnt/gluster-brick1/datastore/", 
"trusted.glusterfs.dht", 
"\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\xff\xff\xff\xff", 16) = 16
[pid 11840] lgetxattr("/mnt/gluster-brick1/datastore/", "missing-gfid-ESTALE", 
0x0, 0) = -1 EOPNOTSUPP (Operation not supported)
[pid 11840] lgetxattr("/mnt/gluster-brick1/datastore/", 
"trusted.afr.datastore1-client-0", 0x0, 0) = -1 ENODATA (No data available)
[pid 11840] lgetxattr("/mnt/gluster-brick1/datastore/", 
"trusted.afr.datastore1-client-1", 0x0, 0) = -1 ENODATA (No data available)
[pid 11840] llistxattr("/mnt/gluster-brick1/datastore/", (nil), 0) = 63
[pid 11840] llistxattr("/mnt/gluster-brick1/datastore/", 0x7feae3cfda10, 63) = 
63
[pid 29198] <... lgetxattr resumed> , 0x0, 0) = 16
[pid 29198] lgetxattr("/mnt/gluster-brick1/datastore/", 
"trusted.glusterfs.dht", 
"\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\xff\xff\xff\xff", 16) = 16
[pid 29198] lgetxattr("/mnt/gluster-brick1/datastore/", "missing-gfid-ESTALE", 
0x0, 0) = -1 EOPNOTSUPP (Operation not supported)
[pid 29198] lgetxattr("/mnt/gluster-brick1/datastore/", 
"trusted.afr.datastore1-client-0", 0x0, 0) = -1 ENODATA (No data available)
[pid 29198] lgetxattr("/mnt/gluster-brick1/datastore/", 
"trusted.afr.datastore1-client-1" 
[pid 29197] <... lgetxattr resumed> , 0x0, 0) = 16
[pid 29198] <... lgetxattr resumed> , 0x0, 0) = -1 ENODATA (No data available)
[pid 29197] lgetxattr("/mnt/gluster-brick1/datastore/", "trusted.glusterfs.dht" 

[pid 29198] llistxattr("/mnt/gluster-brick1/datastore/", (nil), 0) = 63
[pid 29198] llistxattr("/mnt/gluster-brick1/datastore/" 
[pid 29197] <... lgetxattr resumed> , 
"\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\xff\xff\xff\xff", 16) = 16
[pid 29198] <... llistxattr resumed> , 0x7feae3ffea10, 63) = 63
[pid 29197] lgetxattr("/mnt/gluster-brick1/datastore/", "missing-gfid-ESTALE", 
0x0, 0) = -1 EOPNOTSUPP (Operation not supported)
[pid 29197] lgetxattr("/mnt/gluster-brick1/datastore/", 
"trusted.afr.datastore1-client-0", 0x0, 0) = -1 ENODATA (No data available)
[pid 29197] lgetxattr("/mnt/gluster-brick1/datastore/", 
"trusted.afr.datastore1-client-1", 0x0, 0) = -1 ENODATA (No data available)
[pid 29197] llistxattr("/mnt/gluster-brick1/datastore/", (nil), 0) = 63
[pid 29197] llistxattr("/mnt/gluster-brick1/datastore/", 0x7feaf0487a10, 63) = 
63
[pid 11846] lstat("/mnt/gluster-brick1/datastore/images", 
{st_mode=S_IFDIR|0755, st_size=27, ...}) = 0
[pid 11846] lgetxattr("/mnt/gluster-brick1/datastore/images", "trusted.gfid" 

[pid 11844] lstat("/mnt/gluster-brick1/datastore/images", 
{st_mode=S_IFDIR|0755, st_size=27, ...}) = 0
[pid 11844] lgetxattr("/mnt/gluster-brick1/datastore/images", "trusted.gfid" 

[pid 11845] lstat("/mnt/gluster-brick1/datastore/images", 
{st_mode=S_IFDIR|0755, st_size=27, ...}) = 0
[pid 11845] lgetxattr("/mnt/gluster-brick1/datastore/images", "trusted.gfid" 

[pid 11844] <... lgetxattr resumed> , "\xbe\x7fIlH\xb0C\xbd\xaaA=BJ6\xca\xb1", 
16) = 16
[pid 11846] <... lgetxattr resumed> , "\xbe\x7fIlH\xb0C\xbd\xaaA=BJ6\xca\xb1", 
16) = 16
[pid 11845] <... lgetxattr resumed> , "\xbe\x7fIlH\xb0C\xbd\xaaA=BJ6\xca\xb1", 
16) = 16
[pid 11846] lgetxattr("/mnt/gluster-brick1/datastore/images", 
"system.posix_acl_default" 
[pid 11845] lgetxattr("/mnt/gluster-brick1/datastore/images", 
"system.posix_acl_d