Re: [Gluster-devel] How to fix wrong telldir/seekdir usage
On Sat, Sep 13, 2014 at 01:34:58PM -0700, Anand Avati wrote: How does the NetBSD nfs server provide stable directory offsets, for the NFS client to resume reading from at a later point in time? Very similar problems are present in that scenario and it might be helpful to see what approaches are taken there (which are probably more tried and tested) I still do not have an answer for this question, but I have a patch to fix the standard-violating seekdir() usage. As discussed in this threead, I remove fd anonymity in afr-selfèheald.c: http://review.gluster.org/8760 The resource cleanup code may be wrong, it needs review. At least the patch lets NetBSD pass self-held.t without getting locked into an infinite loop. For the NFS question, I suspect it is out of scope: seekdir() does not operate at kernel interface level, but at libc level; the data we see from readdir() are cached in userland, and adding a node will not invalidate them. I have to run tess to confirm, though. -- Emmanuel Dreyfus m...@netbsd.org ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] how do you debug ref leaks?
On 09/18/2014 03:13 PM, Niels de Vos wrote:
On Thu, Sep 18, 2014 at 07:43:00AM +0530, Pranith Kumar Karampuri wrote:
hi,
Till now the only method I used to find ref leaks effectively is to find
what operation is causing ref leaks and read the code to find if there is a
ref-leak somewhere. Valgrind doesn't solve this problem because it is
reachable memory from inode-table etc. I am just wondering if there is an
effective way anyone else knows of. Do you guys think we need a better
mechanism of finding refleaks? At least which decreases the search space
significantly i.e. xlator y, fop f etc? It would be better if we can come up
with ways to integrate statedump and this infra just like we did for
mem-accounting.
One way I thought was to introduce new apis called
xl_fop_dict/inode/fd_ref/unref (). Each xl keeps an array of num_fops per
inode/dict/fd and increments/decrements accordingly. Dump this info on
statedump.
I myself am not completely sure about this idea. It requires all xlators to
change.
Any ideas?
From my understanding, you are looking for something like this addition:
struct _xlator {
...
struct {
int64_t inodes[GF_FOP_MAXVALUE - 1];
int64_t fds[GF_FOP_MAXVALUE - 1];
...
} refs;
...
};
Correct.
inode_table_new(.., *xl) gets the xlator passed, so it can in-/decrease
the xl-refs-inodes[_call_frame_t-op] on future inode_new(),
inode_ref() and inode_unref() calls.
Not like this. We are going to provide apis like
xl_fop_inode_ref(inode_t*, xlator_t*, fop op)
xl_fop_inode_unref(inode_t*, xlator_t*, fop op) and so on for fd, dict.
xlator code has to change :-(. That is the problem.
Some of the difficulties I see ATM, are:
- how to get to the _call_frame_t-op
- correctly accounting FOPs like OPEN/CLOSE
Mainly just tossing the idea here, maybe I'm completely missing the
point, or there are other issues why this would be an impossible
solution.
Cheers,
Niels
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-devel
[Gluster-devel] hi, how can i stop/start a brick?
hi, how can i stop a brick ? because that brick need check file system, need a long time, so i need stop it, and start it when repair finish. thank you everybody_en...@163.com ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] GlusterFest Test Week
On Thu, Sep 18, 2014 at 05:42:01PM +0530, Humble Devassy Chirammal wrote: As decided in our last GlusterFS meeting and the 3.6 planning schedule, we shall conduct GlusterFS 3.6 test days starting from next week. I still have a few portability fixes pending in gerrit for NetBSD. Do we plan to merge for 2.6.0, or is 3.6.1 the target here? -- Emmanuel Dreyfus m...@netbsd.org ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] GlusterFest Test Week
On 09/18/2014 06:07 PM, Emmanuel Dreyfus wrote: On Thu, Sep 18, 2014 at 05:42:01PM +0530, Humble Devassy Chirammal wrote: As decided in our last GlusterFS meeting and the 3.6 planning schedule, we shall conduct GlusterFS 3.6 test days starting from next week. I still have a few portability fixes pending in gerrit for NetBSD. Do we plan to merge for 2.6.0, or is 3.6.1 the target here? I intend doing a beta release before the test week. If we can review and merge those before the build happens, that would be great. Which of the patches in gerrit are absolutely necessary for NetBSD? Thanks, Vijay ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] how do you debug ref leaks?
On 09/17/2014 10:13 PM, Pranith Kumar Karampuri wrote: hi, Till now the only method I used to find ref leaks effectively is to find what operation is causing ref leaks and read the code to find if there is a ref-leak somewhere. Valgrind doesn't solve this problem because it is reachable memory from inode-table etc. I am just wondering if there is an effective way anyone else knows of. Do you guys think we need a better mechanism of finding refleaks? At least which decreases the search space significantly i.e. xlator y, fop f etc? It would be better if we can come up with ways to integrate statedump and this infra just like we did for mem-accounting. One way I thought was to introduce new apis called xl_fop_dict/inode/fd_ref/unref (). Each xl keeps an array of num_fops per inode/dict/fd and increments/decrements accordingly. Dump this info on statedump. I myself am not completely sure about this idea. It requires all xlators to change. Any ideas? On a debug build we can use backtrace information stashed per ref and unref, this will give us history of refs taken and released. Which will also give the code path where ref was taken and released. It is heavy weight, so not for non-debug setups, but if a problem is reproducible this could be a quick way to check who is not releasing the ref's or have a history of the refs and unrefs to dig better into code. Shyam ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] how do you debug ref leaks?
On 09/18/2014 07:48 PM, Shyam wrote: On 09/17/2014 10:13 PM, Pranith Kumar Karampuri wrote: hi, Till now the only method I used to find ref leaks effectively is to find what operation is causing ref leaks and read the code to find if there is a ref-leak somewhere. Valgrind doesn't solve this problem because it is reachable memory from inode-table etc. I am just wondering if there is an effective way anyone else knows of. Do you guys think we need a better mechanism of finding refleaks? At least which decreases the search space significantly i.e. xlator y, fop f etc? It would be better if we can come up with ways to integrate statedump and this infra just like we did for mem-accounting. One way I thought was to introduce new apis called xl_fop_dict/inode/fd_ref/unref (). Each xl keeps an array of num_fops per inode/dict/fd and increments/decrements accordingly. Dump this info on statedump. I myself am not completely sure about this idea. It requires all xlators to change. Any ideas? On a debug build we can use backtrace information stashed per ref and unref, this will give us history of refs taken and released. Which will also give the code path where ref was taken and released. It is heavy weight, so not for non-debug setups, but if a problem is reproducible this could be a quick way to check who is not releasing the ref's or have a history of the refs and unrefs to dig better into code. Do you have any ideas for final builds also? Basically when users report leaks it should not take us too long to figure out the problem area. We should just ask them for statedump and should be able to figure out the problem. Pranith Shyam ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] how do you debug ref leaks?
If we could disable/enable ref tracking dynamically, it may only be heavy weight tempoarily while the customer is being observed. You could get a state dump , or another idea is to take a core of the live process. gcore $(pidof processname) - Original Message - From: Pranith Kumar Karampuri pkara...@redhat.com To: Shyam srang...@redhat.com, gluster-devel@gluster.org Sent: Thursday, September 18, 2014 11:34:28 AM Subject: Re: [Gluster-devel] how do you debug ref leaks? On 09/18/2014 07:48 PM, Shyam wrote: On 09/17/2014 10:13 PM, Pranith Kumar Karampuri wrote: hi, Till now the only method I used to find ref leaks effectively is to find what operation is causing ref leaks and read the code to find if there is a ref-leak somewhere. Valgrind doesn't solve this problem because it is reachable memory from inode-table etc. I am just wondering if there is an effective way anyone else knows of. Do you guys think we need a better mechanism of finding refleaks? At least which decreases the search space significantly i.e. xlator y, fop f etc? It would be better if we can come up with ways to integrate statedump and this infra just like we did for mem-accounting. One way I thought was to introduce new apis called xl_fop_dict/inode/fd_ref/unref (). Each xl keeps an array of num_fops per inode/dict/fd and increments/decrements accordingly. Dump this info on statedump. I myself am not completely sure about this idea. It requires all xlators to change. Any ideas? On a debug build we can use backtrace information stashed per ref and unref, this will give us history of refs taken and released. Which will also give the code path where ref was taken and released. It is heavy weight, so not for non-debug setups, but if a problem is reproducible this could be a quick way to check who is not releasing the ref's or have a history of the refs and unrefs to dig better into code. Do you have any ideas for final builds also? Basically when users report leaks it should not take us too long to figure out the problem area. We should just ask them for statedump and should be able to figure out the problem. Pranith Shyam ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] how do you debug ref leaks?
As a wishlist item, I think it'd be nice if debug builds (or some other build-time option) would disable the pools. Then valgrind might be more useful for finding leaks. Maybe for GlusterFS-4.0? On 09/18/2014 11:40 AM, Dan Lambright wrote: If we could disable/enable ref tracking dynamically, it may only be heavy weight tempoarily while the customer is being observed. You could get a state dump , or another idea is to take a core of the live process. gcore $(pidof processname) - Original Message - From: Pranith Kumar Karampuri pkara...@redhat.com To: Shyam srang...@redhat.com, gluster-devel@gluster.org Sent: Thursday, September 18, 2014 11:34:28 AM Subject: Re: [Gluster-devel] how do you debug ref leaks? On 09/18/2014 07:48 PM, Shyam wrote: On 09/17/2014 10:13 PM, Pranith Kumar Karampuri wrote: hi, Till now the only method I used to find ref leaks effectively is to find what operation is causing ref leaks and read the code to find if there is a ref-leak somewhere. Valgrind doesn't solve this problem because it is reachable memory from inode-table etc. I am just wondering if there is an effective way anyone else knows of. Do you guys think we need a better mechanism of finding refleaks? At least which decreases the search space significantly i.e. xlator y, fop f etc? It would be better if we can come up with ways to integrate statedump and this infra just like we did for mem-accounting. One way I thought was to introduce new apis called xl_fop_dict/inode/fd_ref/unref (). Each xl keeps an array of num_fops per inode/dict/fd and increments/decrements accordingly. Dump this info on statedump. I myself am not completely sure about this idea. It requires all xlators to change. Any ideas? On a debug build we can use backtrace information stashed per ref and unref, this will give us history of refs taken and released. Which will also give the code path where ref was taken and released. It is heavy weight, so not for non-debug setups, but if a problem is reproducible this could be a quick way to check who is not releasing the ref's or have a history of the refs and unrefs to dig better into code. Do you have any ideas for final builds also? Basically when users report leaks it should not take us too long to figure out the problem area. We should just ask them for statedump and should be able to figure out the problem. Pranith Shyam ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] how do you debug ref leaks?
On 09/18/2014 09:31 PM, Kaleb KEITHLEY wrote: As a wishlist item, I think it'd be nice if debug builds (or some other build-time option) would disable the pools. Then valgrind might be more useful for finding leaks. Maybe for GlusterFS-4.0? This is already available http://review.gluster.org/7835 Pranith On 09/18/2014 11:40 AM, Dan Lambright wrote: If we could disable/enable ref tracking dynamically, it may only be heavy weight tempoarily while the customer is being observed. You could get a state dump , or another idea is to take a core of the live process. gcore $(pidof processname) - Original Message - From: Pranith Kumar Karampuri pkara...@redhat.com To: Shyam srang...@redhat.com, gluster-devel@gluster.org Sent: Thursday, September 18, 2014 11:34:28 AM Subject: Re: [Gluster-devel] how do you debug ref leaks? On 09/18/2014 07:48 PM, Shyam wrote: On 09/17/2014 10:13 PM, Pranith Kumar Karampuri wrote: hi, Till now the only method I used to find ref leaks effectively is to find what operation is causing ref leaks and read the code to find if there is a ref-leak somewhere. Valgrind doesn't solve this problem because it is reachable memory from inode-table etc. I am just wondering if there is an effective way anyone else knows of. Do you guys think we need a better mechanism of finding refleaks? At least which decreases the search space significantly i.e. xlator y, fop f etc? It would be better if we can come up with ways to integrate statedump and this infra just like we did for mem-accounting. One way I thought was to introduce new apis called xl_fop_dict/inode/fd_ref/unref (). Each xl keeps an array of num_fops per inode/dict/fd and increments/decrements accordingly. Dump this info on statedump. I myself am not completely sure about this idea. It requires all xlators to change. Any ideas? On a debug build we can use backtrace information stashed per ref and unref, this will give us history of refs taken and released. Which will also give the code path where ref was taken and released. It is heavy weight, so not for non-debug setups, but if a problem is reproducible this could be a quick way to check who is not releasing the ref's or have a history of the refs and unrefs to dig better into code. Do you have any ideas for final builds also? Basically when users report leaks it should not take us too long to figure out the problem area. We should just ask them for statedump and should be able to figure out the problem. Pranith Shyam ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] how do you debug ref leaks?
I am going to be bold and throw a suggestion inspired from what I read today. I was reading briefly about how kernel manages its objects using the kobject data structure and establishe 'liveness' (in the garbage collection sense) relationship across objects. It allows one to group objects into 'sets'. This would help us to categorise objects into 'groups' and derive from statedumps which group the leaking object belongs to and infer some context out of that. This is different from the other approaches that have been discussde. I am not familiar with what more could be done with that sort of an infrastructure. I hope this opens up the discussion and doesn't distract it. ~KP - Original Message - On 09/18/2014 09:35 PM, Pranith Kumar Karampuri wrote: On 09/18/2014 09:31 PM, Kaleb KEITHLEY wrote: As a wishlist item, I think it'd be nice if debug builds (or some other build-time option) would disable the pools. Then valgrind might be more useful for finding leaks. Actually there seems to be some issue with running bricks using valgrind. Operations on mount hang when we start the bricks (Ravi confirmed this situation even today). That still needs to be solved, it used to work. Not sure what happened. Pranith. Maybe for GlusterFS-4.0? This is already available http://review.gluster.org/7835 Pranith On 09/18/2014 11:40 AM, Dan Lambright wrote: If we could disable/enable ref tracking dynamically, it may only be heavy weight tempoarily while the customer is being observed. You could get a state dump , or another idea is to take a core of the live process. gcore $(pidof processname) - Original Message - From: Pranith Kumar Karampuri pkara...@redhat.com To: Shyam srang...@redhat.com, gluster-devel@gluster.org Sent: Thursday, September 18, 2014 11:34:28 AM Subject: Re: [Gluster-devel] how do you debug ref leaks? On 09/18/2014 07:48 PM, Shyam wrote: On 09/17/2014 10:13 PM, Pranith Kumar Karampuri wrote: hi, Till now the only method I used to find ref leaks effectively is to find what operation is causing ref leaks and read the code to find if there is a ref-leak somewhere. Valgrind doesn't solve this problem because it is reachable memory from inode-table etc. I am just wondering if there is an effective way anyone else knows of. Do you guys think we need a better mechanism of finding refleaks? At least which decreases the search space significantly i.e. xlator y, fop f etc? It would be better if we can come up with ways to integrate statedump and this infra just like we did for mem-accounting. One way I thought was to introduce new apis called xl_fop_dict/inode/fd_ref/unref (). Each xl keeps an array of num_fops per inode/dict/fd and increments/decrements accordingly. Dump this info on statedump. I myself am not completely sure about this idea. It requires all xlators to change. Any ideas? On a debug build we can use backtrace information stashed per ref and unref, this will give us history of refs taken and released. Which will also give the code path where ref was taken and released. It is heavy weight, so not for non-debug setups, but if a problem is reproducible this could be a quick way to check who is not releasing the ref's or have a history of the refs and unrefs to dig better into code. Do you have any ideas for final builds also? Basically when users report leaks it should not take us too long to figure out the problem area. We should just ask them for statedump and should be able to figure out the problem. Pranith Shyam ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] how do you debug ref leaks?
On 09/18/2014 12:09 PM, Pranith Kumar Karampuri wrote: On 09/18/2014 09:35 PM, Pranith Kumar Karampuri wrote: On 09/18/2014 09:31 PM, Kaleb KEITHLEY wrote: As a wishlist item, I think it'd be nice if debug builds (or some other build-time option) would disable the pools. Then valgrind might be more useful for finding leaks. Actually there seems to be some issue with running bricks using valgrind. Operations on mount hang when we start the bricks (Ravi confirmed this situation even today). That still needs to be solved, it used to work. Not sure what happened. I've been running gluster under valgrind. E.g. see http://download.gluster.org/pub/gluster/glusterfs/dynamic-analysis/master/glusterfs-valgrind/ It's pretty hit or miss, sometimes the client-side glusterfs process (at least IIRC it's the client-side glusterfs) will lock up and consume 100% CPU. And so far I haven't been able to trigger a core dump of the spinning process to see where it's locking up. It would be nice to solve that so that I can do unsupervised nightly valgrind runs. I'll look at the review you referenced for disabling pools. Thanks, -- Kaleb ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel
