Re: [Gluster-devel] how do you debug ref leaks?
On Thu, 18 Sep 2014 12:33:49 -0400 (EDT) Krishnan Parthasarathi kpart...@redhat.com wrote: I am going to be bold and throw a suggestion inspired from what I read today. I was reading briefly about how kernel manages its objects using the kobject data structure and establishe 'liveness' (in the garbage collection sense) relationship across objects. It allows one to group objects into 'sets'. This would help us to categorise objects into 'groups' and derive from statedumps which group the leaking object belongs to and infer some context out of that. This is different from the other approaches that have been discussde. I am not familiar with what more could be done with that sort of an infrastructure. I hope this opens up the discussion and doesn't distract it. Interesting thought. What's the concept with this we'd implement for GlusterFS? eg what are the important points you reckon we should do, and where would be the right places to start? :) + Justin ~KP -- GlusterFS - http://www.gluster.org An open source, distributed file system scaling to several petabytes, and handling thousands of clients. My personal twitter: twitter.com/realjustinclift ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] how do you debug ref leaks?
- Original Message - On Thu, 18 Sep 2014 12:33:49 -0400 (EDT) Krishnan Parthasarathi kpart...@redhat.com wrote: I am going to be bold and throw a suggestion inspired from what I read today. I was reading briefly about how kernel manages its objects using the kobject data structure and establishe 'liveness' (in the garbage collection sense) relationship across objects. It allows one to group objects into 'sets'. This would help us to categorise objects into 'groups' and derive from statedumps which group the leaking object belongs to and infer some context out of that. This is different from the other approaches that have been discussde. I am not familiar with what more could be done with that sort of an infrastructure. I hope this opens up the discussion and doesn't distract it. Interesting thought. What's the concept with this we'd implement for GlusterFS? eg what are the important points you reckon we should do, and where would be the right places to start? :) I was hoping someone in this list with Linux Kernel programming experience would evaluate the merit in this idea :-) I would need some time to explore more on this thought and share my proposal here. But before I sign-off on that note, let me share what I was thinking when I wrote this. In kernel, (almost) all types have an embedded kobject type in them. This kobject structure embeds in it the interface and 'metadata' [sic] for managing the life cycle of the holding type. This allows the kernel to keep the code that manages the life cycle separate from the actual type. This can be loosely termed as polymorphic. In GlusterFS, we use the slab allocator technique to manage data types that are expected to be allocated in large numbers over time. The 'metadata' pertaining to which slab an object was allocated, resides 'outside' (just preceding) the memory area of the object. This by itself is not bad, but turns out to be inconvenient to verify if the 'metadata' is intact, while using gdb. I think using kobject like approach in place of the current way of storing 'metadata' for objects allocated in slab would make inspecting this 'metadata' more convenient. An interesting side effect of imitating the kernel's strategy is we can inherit their time-worn techniques for debugging memory management issues for free. I hope this provokes further discussion ;) ~KP My personal twitter: twitter.com/realjustinclift ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] how do you debug ref leaks?
On 09/18/2014 03:13 PM, Niels de Vos wrote:
On Thu, Sep 18, 2014 at 07:43:00AM +0530, Pranith Kumar Karampuri wrote:
hi,
Till now the only method I used to find ref leaks effectively is to find
what operation is causing ref leaks and read the code to find if there is a
ref-leak somewhere. Valgrind doesn't solve this problem because it is
reachable memory from inode-table etc. I am just wondering if there is an
effective way anyone else knows of. Do you guys think we need a better
mechanism of finding refleaks? At least which decreases the search space
significantly i.e. xlator y, fop f etc? It would be better if we can come up
with ways to integrate statedump and this infra just like we did for
mem-accounting.
One way I thought was to introduce new apis called
xl_fop_dict/inode/fd_ref/unref (). Each xl keeps an array of num_fops per
inode/dict/fd and increments/decrements accordingly. Dump this info on
statedump.
I myself am not completely sure about this idea. It requires all xlators to
change.
Any ideas?
From my understanding, you are looking for something like this addition:
struct _xlator {
...
struct {
int64_t inodes[GF_FOP_MAXVALUE - 1];
int64_t fds[GF_FOP_MAXVALUE - 1];
...
} refs;
...
};
Correct.
inode_table_new(.., *xl) gets the xlator passed, so it can in-/decrease
the xl-refs-inodes[_call_frame_t-op] on future inode_new(),
inode_ref() and inode_unref() calls.
Not like this. We are going to provide apis like
xl_fop_inode_ref(inode_t*, xlator_t*, fop op)
xl_fop_inode_unref(inode_t*, xlator_t*, fop op) and so on for fd, dict.
xlator code has to change :-(. That is the problem.
Some of the difficulties I see ATM, are:
- how to get to the _call_frame_t-op
- correctly accounting FOPs like OPEN/CLOSE
Mainly just tossing the idea here, maybe I'm completely missing the
point, or there are other issues why this would be an impossible
solution.
Cheers,
Niels
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] how do you debug ref leaks?
On 09/17/2014 10:13 PM, Pranith Kumar Karampuri wrote: hi, Till now the only method I used to find ref leaks effectively is to find what operation is causing ref leaks and read the code to find if there is a ref-leak somewhere. Valgrind doesn't solve this problem because it is reachable memory from inode-table etc. I am just wondering if there is an effective way anyone else knows of. Do you guys think we need a better mechanism of finding refleaks? At least which decreases the search space significantly i.e. xlator y, fop f etc? It would be better if we can come up with ways to integrate statedump and this infra just like we did for mem-accounting. One way I thought was to introduce new apis called xl_fop_dict/inode/fd_ref/unref (). Each xl keeps an array of num_fops per inode/dict/fd and increments/decrements accordingly. Dump this info on statedump. I myself am not completely sure about this idea. It requires all xlators to change. Any ideas? On a debug build we can use backtrace information stashed per ref and unref, this will give us history of refs taken and released. Which will also give the code path where ref was taken and released. It is heavy weight, so not for non-debug setups, but if a problem is reproducible this could be a quick way to check who is not releasing the ref's or have a history of the refs and unrefs to dig better into code. Shyam ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] how do you debug ref leaks?
On 09/18/2014 07:48 PM, Shyam wrote: On 09/17/2014 10:13 PM, Pranith Kumar Karampuri wrote: hi, Till now the only method I used to find ref leaks effectively is to find what operation is causing ref leaks and read the code to find if there is a ref-leak somewhere. Valgrind doesn't solve this problem because it is reachable memory from inode-table etc. I am just wondering if there is an effective way anyone else knows of. Do you guys think we need a better mechanism of finding refleaks? At least which decreases the search space significantly i.e. xlator y, fop f etc? It would be better if we can come up with ways to integrate statedump and this infra just like we did for mem-accounting. One way I thought was to introduce new apis called xl_fop_dict/inode/fd_ref/unref (). Each xl keeps an array of num_fops per inode/dict/fd and increments/decrements accordingly. Dump this info on statedump. I myself am not completely sure about this idea. It requires all xlators to change. Any ideas? On a debug build we can use backtrace information stashed per ref and unref, this will give us history of refs taken and released. Which will also give the code path where ref was taken and released. It is heavy weight, so not for non-debug setups, but if a problem is reproducible this could be a quick way to check who is not releasing the ref's or have a history of the refs and unrefs to dig better into code. Do you have any ideas for final builds also? Basically when users report leaks it should not take us too long to figure out the problem area. We should just ask them for statedump and should be able to figure out the problem. Pranith Shyam ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] how do you debug ref leaks?
If we could disable/enable ref tracking dynamically, it may only be heavy weight tempoarily while the customer is being observed. You could get a state dump , or another idea is to take a core of the live process. gcore $(pidof processname) - Original Message - From: Pranith Kumar Karampuri pkara...@redhat.com To: Shyam srang...@redhat.com, gluster-devel@gluster.org Sent: Thursday, September 18, 2014 11:34:28 AM Subject: Re: [Gluster-devel] how do you debug ref leaks? On 09/18/2014 07:48 PM, Shyam wrote: On 09/17/2014 10:13 PM, Pranith Kumar Karampuri wrote: hi, Till now the only method I used to find ref leaks effectively is to find what operation is causing ref leaks and read the code to find if there is a ref-leak somewhere. Valgrind doesn't solve this problem because it is reachable memory from inode-table etc. I am just wondering if there is an effective way anyone else knows of. Do you guys think we need a better mechanism of finding refleaks? At least which decreases the search space significantly i.e. xlator y, fop f etc? It would be better if we can come up with ways to integrate statedump and this infra just like we did for mem-accounting. One way I thought was to introduce new apis called xl_fop_dict/inode/fd_ref/unref (). Each xl keeps an array of num_fops per inode/dict/fd and increments/decrements accordingly. Dump this info on statedump. I myself am not completely sure about this idea. It requires all xlators to change. Any ideas? On a debug build we can use backtrace information stashed per ref and unref, this will give us history of refs taken and released. Which will also give the code path where ref was taken and released. It is heavy weight, so not for non-debug setups, but if a problem is reproducible this could be a quick way to check who is not releasing the ref's or have a history of the refs and unrefs to dig better into code. Do you have any ideas for final builds also? Basically when users report leaks it should not take us too long to figure out the problem area. We should just ask them for statedump and should be able to figure out the problem. Pranith Shyam ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] how do you debug ref leaks?
As a wishlist item, I think it'd be nice if debug builds (or some other build-time option) would disable the pools. Then valgrind might be more useful for finding leaks. Maybe for GlusterFS-4.0? On 09/18/2014 11:40 AM, Dan Lambright wrote: If we could disable/enable ref tracking dynamically, it may only be heavy weight tempoarily while the customer is being observed. You could get a state dump , or another idea is to take a core of the live process. gcore $(pidof processname) - Original Message - From: Pranith Kumar Karampuri pkara...@redhat.com To: Shyam srang...@redhat.com, gluster-devel@gluster.org Sent: Thursday, September 18, 2014 11:34:28 AM Subject: Re: [Gluster-devel] how do you debug ref leaks? On 09/18/2014 07:48 PM, Shyam wrote: On 09/17/2014 10:13 PM, Pranith Kumar Karampuri wrote: hi, Till now the only method I used to find ref leaks effectively is to find what operation is causing ref leaks and read the code to find if there is a ref-leak somewhere. Valgrind doesn't solve this problem because it is reachable memory from inode-table etc. I am just wondering if there is an effective way anyone else knows of. Do you guys think we need a better mechanism of finding refleaks? At least which decreases the search space significantly i.e. xlator y, fop f etc? It would be better if we can come up with ways to integrate statedump and this infra just like we did for mem-accounting. One way I thought was to introduce new apis called xl_fop_dict/inode/fd_ref/unref (). Each xl keeps an array of num_fops per inode/dict/fd and increments/decrements accordingly. Dump this info on statedump. I myself am not completely sure about this idea. It requires all xlators to change. Any ideas? On a debug build we can use backtrace information stashed per ref and unref, this will give us history of refs taken and released. Which will also give the code path where ref was taken and released. It is heavy weight, so not for non-debug setups, but if a problem is reproducible this could be a quick way to check who is not releasing the ref's or have a history of the refs and unrefs to dig better into code. Do you have any ideas for final builds also? Basically when users report leaks it should not take us too long to figure out the problem area. We should just ask them for statedump and should be able to figure out the problem. Pranith Shyam ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] how do you debug ref leaks?
On 09/18/2014 09:31 PM, Kaleb KEITHLEY wrote: As a wishlist item, I think it'd be nice if debug builds (or some other build-time option) would disable the pools. Then valgrind might be more useful for finding leaks. Maybe for GlusterFS-4.0? This is already available http://review.gluster.org/7835 Pranith On 09/18/2014 11:40 AM, Dan Lambright wrote: If we could disable/enable ref tracking dynamically, it may only be heavy weight tempoarily while the customer is being observed. You could get a state dump , or another idea is to take a core of the live process. gcore $(pidof processname) - Original Message - From: Pranith Kumar Karampuri pkara...@redhat.com To: Shyam srang...@redhat.com, gluster-devel@gluster.org Sent: Thursday, September 18, 2014 11:34:28 AM Subject: Re: [Gluster-devel] how do you debug ref leaks? On 09/18/2014 07:48 PM, Shyam wrote: On 09/17/2014 10:13 PM, Pranith Kumar Karampuri wrote: hi, Till now the only method I used to find ref leaks effectively is to find what operation is causing ref leaks and read the code to find if there is a ref-leak somewhere. Valgrind doesn't solve this problem because it is reachable memory from inode-table etc. I am just wondering if there is an effective way anyone else knows of. Do you guys think we need a better mechanism of finding refleaks? At least which decreases the search space significantly i.e. xlator y, fop f etc? It would be better if we can come up with ways to integrate statedump and this infra just like we did for mem-accounting. One way I thought was to introduce new apis called xl_fop_dict/inode/fd_ref/unref (). Each xl keeps an array of num_fops per inode/dict/fd and increments/decrements accordingly. Dump this info on statedump. I myself am not completely sure about this idea. It requires all xlators to change. Any ideas? On a debug build we can use backtrace information stashed per ref and unref, this will give us history of refs taken and released. Which will also give the code path where ref was taken and released. It is heavy weight, so not for non-debug setups, but if a problem is reproducible this could be a quick way to check who is not releasing the ref's or have a history of the refs and unrefs to dig better into code. Do you have any ideas for final builds also? Basically when users report leaks it should not take us too long to figure out the problem area. We should just ask them for statedump and should be able to figure out the problem. Pranith Shyam ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] how do you debug ref leaks?
I am going to be bold and throw a suggestion inspired from what I read today. I was reading briefly about how kernel manages its objects using the kobject data structure and establishe 'liveness' (in the garbage collection sense) relationship across objects. It allows one to group objects into 'sets'. This would help us to categorise objects into 'groups' and derive from statedumps which group the leaking object belongs to and infer some context out of that. This is different from the other approaches that have been discussde. I am not familiar with what more could be done with that sort of an infrastructure. I hope this opens up the discussion and doesn't distract it. ~KP - Original Message - On 09/18/2014 09:35 PM, Pranith Kumar Karampuri wrote: On 09/18/2014 09:31 PM, Kaleb KEITHLEY wrote: As a wishlist item, I think it'd be nice if debug builds (or some other build-time option) would disable the pools. Then valgrind might be more useful for finding leaks. Actually there seems to be some issue with running bricks using valgrind. Operations on mount hang when we start the bricks (Ravi confirmed this situation even today). That still needs to be solved, it used to work. Not sure what happened. Pranith. Maybe for GlusterFS-4.0? This is already available http://review.gluster.org/7835 Pranith On 09/18/2014 11:40 AM, Dan Lambright wrote: If we could disable/enable ref tracking dynamically, it may only be heavy weight tempoarily while the customer is being observed. You could get a state dump , or another idea is to take a core of the live process. gcore $(pidof processname) - Original Message - From: Pranith Kumar Karampuri pkara...@redhat.com To: Shyam srang...@redhat.com, gluster-devel@gluster.org Sent: Thursday, September 18, 2014 11:34:28 AM Subject: Re: [Gluster-devel] how do you debug ref leaks? On 09/18/2014 07:48 PM, Shyam wrote: On 09/17/2014 10:13 PM, Pranith Kumar Karampuri wrote: hi, Till now the only method I used to find ref leaks effectively is to find what operation is causing ref leaks and read the code to find if there is a ref-leak somewhere. Valgrind doesn't solve this problem because it is reachable memory from inode-table etc. I am just wondering if there is an effective way anyone else knows of. Do you guys think we need a better mechanism of finding refleaks? At least which decreases the search space significantly i.e. xlator y, fop f etc? It would be better if we can come up with ways to integrate statedump and this infra just like we did for mem-accounting. One way I thought was to introduce new apis called xl_fop_dict/inode/fd_ref/unref (). Each xl keeps an array of num_fops per inode/dict/fd and increments/decrements accordingly. Dump this info on statedump. I myself am not completely sure about this idea. It requires all xlators to change. Any ideas? On a debug build we can use backtrace information stashed per ref and unref, this will give us history of refs taken and released. Which will also give the code path where ref was taken and released. It is heavy weight, so not for non-debug setups, but if a problem is reproducible this could be a quick way to check who is not releasing the ref's or have a history of the refs and unrefs to dig better into code. Do you have any ideas for final builds also? Basically when users report leaks it should not take us too long to figure out the problem area. We should just ask them for statedump and should be able to figure out the problem. Pranith Shyam ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] how do you debug ref leaks?
On 09/18/2014 12:09 PM, Pranith Kumar Karampuri wrote: On 09/18/2014 09:35 PM, Pranith Kumar Karampuri wrote: On 09/18/2014 09:31 PM, Kaleb KEITHLEY wrote: As a wishlist item, I think it'd be nice if debug builds (or some other build-time option) would disable the pools. Then valgrind might be more useful for finding leaks. Actually there seems to be some issue with running bricks using valgrind. Operations on mount hang when we start the bricks (Ravi confirmed this situation even today). That still needs to be solved, it used to work. Not sure what happened. I've been running gluster under valgrind. E.g. see http://download.gluster.org/pub/gluster/glusterfs/dynamic-analysis/master/glusterfs-valgrind/ It's pretty hit or miss, sometimes the client-side glusterfs process (at least IIRC it's the client-side glusterfs) will lock up and consume 100% CPU. And so far I haven't been able to trigger a core dump of the spinning process to see where it's locking up. It would be nice to solve that so that I can do unsupervised nightly valgrind runs. I'll look at the review you referenced for disabling pools. Thanks, -- Kaleb ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] how do you debug ref leaks?
- Original Message - From: Raghavendra Gowdappa rgowd...@redhat.com To: Pranith Kumar Karampuri pkara...@redhat.com Cc: Gluster Devel gluster-devel@gluster.org Sent: Thursday, September 18, 2014 10:08:15 AM Subject: Re: [Gluster-devel] how do you debug ref leaks? For eg., if a dictionary is not freed because of non-zero refcount, if there is an information on who has held these references would help to narrow down the code path or component. This solution might be rudimentary. However, someone who has worked on things like garbage collection can give better answers I think. This discussion also reminds me of Greenspun's tenth rule [1] [1] http://en.wikipedia.org/wiki/Greenspun%27s_tenth_rule - Original Message - From: Pranith Kumar Karampuri pkara...@redhat.com To: Raghavendra Gowdappa rgowd...@redhat.com Cc: Gluster Devel gluster-devel@gluster.org Sent: Thursday, September 18, 2014 10:05:18 AM Subject: Re: [Gluster-devel] how do you debug ref leaks? On 09/18/2014 09:59 AM, Raghavendra Gowdappa wrote: One thing that would be helpful is allocator info for generic objects like dict, inode, fd etc. That way we wouldn't have to sift through large amount of code. Could you elaborate the idea please. Pranith - Original Message - From: Pranith Kumar Karampuri pkara...@redhat.com To: Gluster Devel gluster-devel@gluster.org Sent: Thursday, September 18, 2014 7:43:00 AM Subject: [Gluster-devel] how do you debug ref leaks? hi, Till now the only method I used to find ref leaks effectively is to find what operation is causing ref leaks and read the code to find if there is a ref-leak somewhere. Valgrind doesn't solve this problem because it is reachable memory from inode-table etc. I am just wondering if there is an effective way anyone else knows of. Do you guys think we need a better mechanism of finding refleaks? At least which decreases the search space significantly i.e. xlator y, fop f etc? It would be better if we can come up with ways to integrate statedump and this infra just like we did for mem-accounting. One way I thought was to introduce new apis called xl_fop_dict/inode/fd_ref/unref (). Each xl keeps an array of num_fops per inode/dict/fd and increments/decrements accordingly. Dump this info on statedump. I myself am not completely sure about this idea. It requires all xlators to change. Any ideas? Pranith ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] how do you debug ref leaks?
On 09/18/2014 10:08 AM, Raghavendra Gowdappa wrote:
For eg., if a dictionary is not freed because of non-zero refcount, if there is
an information on who has held these references would help to narrow down the
code path or component.
Yes that is the aim. The implementation I suggested tries to get that
information per xlator. Are you saying it is better to store in the
dict/inode/fd itself? I actually wrote one patch sometime back to do
something similar (Not tested yet :-) ).
diff --git a/libglusterfs/src/dict.c b/libglusterfs/src/dict.c
index 37a6b2c..bd4c438 100644
--- a/libglusterfs/src/dict.c
+++ b/libglusterfs/src/dict.c
@@ -100,8 +100,10 @@ dict_new (void)
dict = get_new_dict_full(1);
-if (dict)
+if (dict) {
+dict-refs = get_new_dict_full(1);
dict_ref (dict);
+}
return dict;
}
@@ -446,6 +448,7 @@ dict_destroy (dict_t *this)
data_pair_t *pair = this-members_list;
data_pair_t *prev = this-members_list;
+dict_destroy (this-refs);
LOCK_DESTROY (this-lock);
while (prev) {
@@ -495,15 +498,21 @@ dict_unref (dict_t *this)
dict_t *
dict_ref (dict_t *this)
{
+int32_t ref = 0;
+xlat_t *x = NULL;
if (!this) {
gf_log_callingfn (dict, GF_LOG_WARNING, dict is NULL);
return NULL;
}
+x = THIS;
LOCK (this-lock);
this-refcount++;
+dict_get_in32 (this-refs, x-name, ref);
+dict_set_int32 (this-refs, x-name, ref+1);
+
UNLOCK (this-lock);
return this;
@@ -513,15 +522,20 @@ void
data_unref (data_t *this)
{
int32_t ref;
+xlator_t *x = NULL;
if (!this) {
gf_log_callingfn (dict, GF_LOG_WARNING, dict is NULL);
return;
}
+x = THIS;
LOCK (this-lock);
this-refcount--;
+dict_get_in32 (this-refs, x-name, ref);
+dict_set_int32 (this-refs, x-name, ref-1);
+
ref = this-refcount;
UNLOCK (this-lock);
diff --git a/libglusterfs/src/dict.h b/libglusterfs/src/dict.h
index 682c152..33ed7bd 100644
--- a/libglusterfs/src/dict.h
+++ b/libglusterfs/src/dict.h
@@ -93,6 +93,7 @@ struct _dict {
data_pair_t*members_internal;
data_pair_t free_pair;
gf_boolean_tfree_pair_in_use;
+struct _dict *refs;
};
I was not happy with this implementation either.
similar implementation for inode here: http://review.gluster.com/8302
But I am not happy with any of these implementations. Probably because
it is still not granular enough, i.e. fop info is missing. It is better
than no info but still bad. We can't have it on statedump either. So
when User reports high memory usage we still have to spend lot of time
looking over all the places where the dicts are allocated which is bad :-(.
Pranith
- Original Message -
From: Pranith Kumar Karampuri pkara...@redhat.com
To: Raghavendra Gowdappa rgowd...@redhat.com
Cc: Gluster Devel gluster-devel@gluster.org
Sent: Thursday, September 18, 2014 10:05:18 AM
Subject: Re: [Gluster-devel] how do you debug ref leaks?
On 09/18/2014 09:59 AM, Raghavendra Gowdappa wrote:
One thing that would be helpful is allocator info for generic objects
like dict, inode, fd etc. That way we wouldn't have to sift through large
amount of code.
Could you elaborate the idea please.
Pranith
- Original Message -
From: Pranith Kumar Karampuri pkara...@redhat.com
To: Gluster Devel gluster-devel@gluster.org
Sent: Thursday, September 18, 2014 7:43:00 AM
Subject: [Gluster-devel] how do you debug ref leaks?
hi,
Till now the only method I used to find ref leaks effectively is to
find what operation is causing ref leaks and read the code to find if
there is a ref-leak somewhere. Valgrind doesn't solve this problem
because it is reachable memory from inode-table etc. I am just wondering
if there is an effective way anyone else knows of. Do you guys think we
need a better mechanism of finding refleaks? At least which decreases
the search space significantly i.e. xlator y, fop f etc? It would be
better if we can come up with ways to integrate statedump and this infra
just like we did for mem-accounting.
One way I thought was to introduce new apis called
xl_fop_dict/inode/fd_ref/unref (). Each xl keeps an array of num_fops
per inode/dict/fd and increments/decrements accordingly. Dump this info
on statedump.
I myself am not completely sure about this idea. It requires all xlators
to change.
Any ideas?
Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-devel
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-devel
