Re: [Gluster-devel] how do you debug ref leaks?

2014-10-20 Thread Justin Clift
On Thu, 18 Sep 2014 12:33:49 -0400 (EDT)
Krishnan Parthasarathi kpart...@redhat.com wrote:

 I am going to be bold and throw a suggestion inspired from
 what I read today. I was reading briefly about how kernel
 manages its objects using the kobject data structure and
 establishe 'liveness' (in the garbage collection sense)
 relationship across objects. It allows one to group objects
 into 'sets'. This would help us to categorise objects into
 'groups' and derive from statedumps which group the leaking
 object belongs to and infer some context out of that. This is
 different from the other approaches that have been discussde. I am not 
 familiar
 with what more could be done with that sort of an infrastructure. I hope
 this opens up the discussion and doesn't distract it.

Interesting thought.  What's the concept with this we'd implement
for GlusterFS?

  eg what are the important points you reckon we should do, and
 where would be the right places to start? :)

+ Justin

 
 ~KP

-- 
GlusterFS - http://www.gluster.org

An open source, distributed file system scaling to several
petabytes, and handling thousands of clients.

My personal twitter: twitter.com/realjustinclift
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-devel


Re: [Gluster-devel] how do you debug ref leaks?

2014-10-20 Thread Krishnan Parthasarathi


- Original Message -
 On Thu, 18 Sep 2014 12:33:49 -0400 (EDT)
 Krishnan Parthasarathi kpart...@redhat.com wrote:
 
  I am going to be bold and throw a suggestion inspired from
  what I read today. I was reading briefly about how kernel
  manages its objects using the kobject data structure and
  establishe 'liveness' (in the garbage collection sense)
  relationship across objects. It allows one to group objects
  into 'sets'. This would help us to categorise objects into
  'groups' and derive from statedumps which group the leaking
  object belongs to and infer some context out of that. This is
  different from the other approaches that have been discussde. I am not
  familiar
  with what more could be done with that sort of an infrastructure. I hope
  this opens up the discussion and doesn't distract it.
 
 Interesting thought.  What's the concept with this we'd implement
 for GlusterFS?
 
   eg what are the important points you reckon we should do, and
  where would be the right places to start? :)

I was hoping someone in this list with Linux Kernel programming experience
would evaluate the merit in this idea :-) I would need some time to explore
more on this thought and share my proposal here. But before I sign-off on that 
note,
let me share what I was thinking when I wrote this.

In kernel, (almost) all types have an embedded kobject type in them. This 
kobject structure
embeds in it the interface and 'metadata' [sic] for managing the life cycle of 
the holding type.
This allows the kernel to keep the code that manages the life cycle separate 
from the actual type.
This can be loosely termed as polymorphic. 

In GlusterFS, we use the slab allocator technique to manage data types that are
expected to be allocated in large numbers over time. The 'metadata' pertaining 
to
which slab an object was allocated, resides 'outside' (just preceding) the 
memory area
of the object. This by itself is not bad, but turns out to be inconvenient to 
verify
if the 'metadata' is intact, while using gdb.

I think using kobject like approach in place of the current way of storing 
'metadata'
for objects allocated in slab would make inspecting this 'metadata' more 
convenient.
An interesting side effect of imitating the kernel's strategy is we can inherit 
their
time-worn techniques for debugging memory management issues for free.

I hope this provokes further discussion ;)
 
~KP


 My personal twitter: twitter.com/realjustinclift
 
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-devel


Re: [Gluster-devel] how do you debug ref leaks?

2014-09-18 Thread Pranith Kumar Karampuri


On 09/18/2014 03:13 PM, Niels de Vos wrote:

On Thu, Sep 18, 2014 at 07:43:00AM +0530, Pranith Kumar Karampuri wrote:

hi,
 Till now the only method I used to find ref leaks effectively is to find
what operation is causing ref leaks and read the code to find if there is a
ref-leak somewhere. Valgrind doesn't solve this problem because it is
reachable memory from inode-table etc. I am just wondering if there is an
effective way anyone else knows of. Do you guys think we need a better
mechanism of finding refleaks? At least which decreases the search space
significantly i.e. xlator y, fop f etc? It would be better if we can come up
with ways to integrate statedump and this infra just like we did for
mem-accounting.

One way I thought was to introduce new apis called
xl_fop_dict/inode/fd_ref/unref (). Each xl keeps an array of num_fops per
inode/dict/fd and increments/decrements accordingly. Dump this info on
statedump.

I myself am not completely sure about this idea. It requires all xlators to
change.

Any ideas?

 From my understanding, you are looking for something like this addition:

 struct _xlator {
 ...
 struct {
 int64_t inodes[GF_FOP_MAXVALUE - 1];
 int64_t fds[GF_FOP_MAXVALUE - 1];
 ...
 } refs;
 ...
 };

Correct.


inode_table_new(.., *xl) gets the xlator passed, so it can in-/decrease
the xl-refs-inodes[_call_frame_t-op] on future inode_new(),
inode_ref() and inode_unref() calls.

Not like this. We are going to provide apis like
xl_fop_inode_ref(inode_t*, xlator_t*, fop op)
xl_fop_inode_unref(inode_t*, xlator_t*, fop op) and so on for fd, dict.

xlator code has to change :-(. That is the problem.


Some of the difficulties I see ATM, are:
- how to get to the _call_frame_t-op
- correctly accounting FOPs like OPEN/CLOSE

Mainly just tossing the idea here, maybe I'm completely missing the
point, or there are other issues why this would be an impossible
solution.

Cheers,
Niels


___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-devel


Re: [Gluster-devel] how do you debug ref leaks?

2014-09-18 Thread Shyam

On 09/17/2014 10:13 PM, Pranith Kumar Karampuri wrote:

hi,
 Till now the only method I used to find ref leaks effectively is to
find what operation is causing ref leaks and read the code to find if
there is a ref-leak somewhere. Valgrind doesn't solve this problem
because it is reachable memory from inode-table etc. I am just wondering
if there is an effective way anyone else knows of. Do you guys think we
need a better mechanism of finding refleaks? At least which decreases
the search space significantly i.e. xlator y, fop f etc? It would be
better if we can come up with ways to integrate statedump and this infra
just like we did for mem-accounting.

One way I thought was to introduce new apis called
xl_fop_dict/inode/fd_ref/unref (). Each xl keeps an array of num_fops
per inode/dict/fd and increments/decrements accordingly. Dump this info
on statedump.

I myself am not completely sure about this idea. It requires all xlators
to change.

Any ideas?


On a debug build we can use backtrace information stashed per ref and 
unref, this will give us history of refs taken and released. Which will 
also give the code path where ref was taken and released.


It is heavy weight, so not for non-debug setups, but if a problem is 
reproducible this could be a quick way to check who is not releasing the 
ref's or have a history of the refs and unrefs to dig better into code.


Shyam
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-devel


Re: [Gluster-devel] how do you debug ref leaks?

2014-09-18 Thread Pranith Kumar Karampuri


On 09/18/2014 07:48 PM, Shyam wrote:

On 09/17/2014 10:13 PM, Pranith Kumar Karampuri wrote:

hi,
 Till now the only method I used to find ref leaks effectively is to
find what operation is causing ref leaks and read the code to find if
there is a ref-leak somewhere. Valgrind doesn't solve this problem
because it is reachable memory from inode-table etc. I am just wondering
if there is an effective way anyone else knows of. Do you guys think we
need a better mechanism of finding refleaks? At least which decreases
the search space significantly i.e. xlator y, fop f etc? It would be
better if we can come up with ways to integrate statedump and this infra
just like we did for mem-accounting.

One way I thought was to introduce new apis called
xl_fop_dict/inode/fd_ref/unref (). Each xl keeps an array of num_fops
per inode/dict/fd and increments/decrements accordingly. Dump this info
on statedump.

I myself am not completely sure about this idea. It requires all xlators
to change.

Any ideas?


On a debug build we can use backtrace information stashed per ref and 
unref, this will give us history of refs taken and released. Which 
will also give the code path where ref was taken and released.


It is heavy weight, so not for non-debug setups, but if a problem is 
reproducible this could be a quick way to check who is not releasing 
the ref's or have a history of the refs and unrefs to dig better into 
code.


Do you have any ideas for final builds also? Basically when users report 
leaks it should not take us too long to figure out the problem area. We 
should just ask them for statedump and should be able to figure out the 
problem.


Pranith

Shyam
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-devel


___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-devel


Re: [Gluster-devel] how do you debug ref leaks?

2014-09-18 Thread Dan Lambright
If we could disable/enable ref tracking dynamically, it may only be heavy 
weight tempoarily while the customer is being observed.
You could get a state dump , or another idea is to take a core of the live 
process.   gcore $(pidof processname) 

- Original Message -
 From: Pranith Kumar Karampuri pkara...@redhat.com
 To: Shyam srang...@redhat.com, gluster-devel@gluster.org
 Sent: Thursday, September 18, 2014 11:34:28 AM
 Subject: Re: [Gluster-devel] how do you debug ref leaks?
 
 
 On 09/18/2014 07:48 PM, Shyam wrote:
  On 09/17/2014 10:13 PM, Pranith Kumar Karampuri wrote:
  hi,
   Till now the only method I used to find ref leaks effectively is to
  find what operation is causing ref leaks and read the code to find if
  there is a ref-leak somewhere. Valgrind doesn't solve this problem
  because it is reachable memory from inode-table etc. I am just wondering
  if there is an effective way anyone else knows of. Do you guys think we
  need a better mechanism of finding refleaks? At least which decreases
  the search space significantly i.e. xlator y, fop f etc? It would be
  better if we can come up with ways to integrate statedump and this infra
  just like we did for mem-accounting.
 
  One way I thought was to introduce new apis called
  xl_fop_dict/inode/fd_ref/unref (). Each xl keeps an array of num_fops
  per inode/dict/fd and increments/decrements accordingly. Dump this info
  on statedump.
 
  I myself am not completely sure about this idea. It requires all xlators
  to change.
 
  Any ideas?
 
  On a debug build we can use backtrace information stashed per ref and
  unref, this will give us history of refs taken and released. Which
  will also give the code path where ref was taken and released.
 
  It is heavy weight, so not for non-debug setups, but if a problem is
  reproducible this could be a quick way to check who is not releasing
  the ref's or have a history of the refs and unrefs to dig better into
  code.
 
 Do you have any ideas for final builds also? Basically when users report
 leaks it should not take us too long to figure out the problem area. We
 should just ask them for statedump and should be able to figure out the
 problem.
 
 Pranith
  Shyam
  ___
  Gluster-devel mailing list
  Gluster-devel@gluster.org
  http://supercolony.gluster.org/mailman/listinfo/gluster-devel
 
 ___
 Gluster-devel mailing list
 Gluster-devel@gluster.org
 http://supercolony.gluster.org/mailman/listinfo/gluster-devel
 
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-devel


Re: [Gluster-devel] how do you debug ref leaks?

2014-09-18 Thread Kaleb KEITHLEY
As a wishlist item, I think it'd be nice if debug builds (or some other 
build-time option) would disable the pools. Then valgrind might be more 
useful for finding leaks.


Maybe for GlusterFS-4.0?


On 09/18/2014 11:40 AM, Dan Lambright wrote:

If we could disable/enable ref tracking dynamically, it may only be heavy 
weight tempoarily while the customer is being observed.
You could get a state dump , or another idea is to take a core of the live 
process.   gcore $(pidof processname)

- Original Message -

From: Pranith Kumar Karampuri pkara...@redhat.com
To: Shyam srang...@redhat.com, gluster-devel@gluster.org
Sent: Thursday, September 18, 2014 11:34:28 AM
Subject: Re: [Gluster-devel] how do you debug ref leaks?


On 09/18/2014 07:48 PM, Shyam wrote:

On 09/17/2014 10:13 PM, Pranith Kumar Karampuri wrote:

hi,
  Till now the only method I used to find ref leaks effectively is to
find what operation is causing ref leaks and read the code to find if
there is a ref-leak somewhere. Valgrind doesn't solve this problem
because it is reachable memory from inode-table etc. I am just wondering
if there is an effective way anyone else knows of. Do you guys think we
need a better mechanism of finding refleaks? At least which decreases
the search space significantly i.e. xlator y, fop f etc? It would be
better if we can come up with ways to integrate statedump and this infra
just like we did for mem-accounting.

One way I thought was to introduce new apis called
xl_fop_dict/inode/fd_ref/unref (). Each xl keeps an array of num_fops
per inode/dict/fd and increments/decrements accordingly. Dump this info
on statedump.

I myself am not completely sure about this idea. It requires all xlators
to change.

Any ideas?


On a debug build we can use backtrace information stashed per ref and
unref, this will give us history of refs taken and released. Which
will also give the code path where ref was taken and released.

It is heavy weight, so not for non-debug setups, but if a problem is
reproducible this could be a quick way to check who is not releasing
the ref's or have a history of the refs and unrefs to dig better into
code.


Do you have any ideas for final builds also? Basically when users report
leaks it should not take us too long to figure out the problem area. We
should just ask them for statedump and should be able to figure out the
problem.

Pranith

Shyam
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-devel


___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-devel


___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-devel



___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-devel


Re: [Gluster-devel] how do you debug ref leaks?

2014-09-18 Thread Pranith Kumar Karampuri


On 09/18/2014 09:31 PM, Kaleb KEITHLEY wrote:
As a wishlist item, I think it'd be nice if debug builds (or some 
other build-time option) would disable the pools. Then valgrind might 
be more useful for finding leaks.


Maybe for GlusterFS-4.0?

This is already available http://review.gluster.org/7835

Pranith



On 09/18/2014 11:40 AM, Dan Lambright wrote:
If we could disable/enable ref tracking dynamically, it may only be 
heavy weight tempoarily while the customer is being observed.
You could get a state dump , or another idea is to take a core of the 
live process.   gcore $(pidof processname)


- Original Message -

From: Pranith Kumar Karampuri pkara...@redhat.com
To: Shyam srang...@redhat.com, gluster-devel@gluster.org
Sent: Thursday, September 18, 2014 11:34:28 AM
Subject: Re: [Gluster-devel] how do you debug ref leaks?


On 09/18/2014 07:48 PM, Shyam wrote:

On 09/17/2014 10:13 PM, Pranith Kumar Karampuri wrote:

hi,
  Till now the only method I used to find ref leaks 
effectively is to

find what operation is causing ref leaks and read the code to find if
there is a ref-leak somewhere. Valgrind doesn't solve this problem
because it is reachable memory from inode-table etc. I am just 
wondering
if there is an effective way anyone else knows of. Do you guys 
think we

need a better mechanism of finding refleaks? At least which decreases
the search space significantly i.e. xlator y, fop f etc? It would be
better if we can come up with ways to integrate statedump and this 
infra

just like we did for mem-accounting.

One way I thought was to introduce new apis called
xl_fop_dict/inode/fd_ref/unref (). Each xl keeps an array of num_fops
per inode/dict/fd and increments/decrements accordingly. Dump this 
info

on statedump.

I myself am not completely sure about this idea. It requires all 
xlators

to change.

Any ideas?


On a debug build we can use backtrace information stashed per ref and
unref, this will give us history of refs taken and released. Which
will also give the code path where ref was taken and released.

It is heavy weight, so not for non-debug setups, but if a problem is
reproducible this could be a quick way to check who is not releasing
the ref's or have a history of the refs and unrefs to dig better into
code.

Do you have any ideas for final builds also? Basically when users 
report

leaks it should not take us too long to figure out the problem area. We
should just ask them for statedump and should be able to figure out the
problem.

Pranith

Shyam
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-devel


___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-devel


___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-devel



___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-devel


___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-devel


Re: [Gluster-devel] how do you debug ref leaks?

2014-09-18 Thread Krishnan Parthasarathi
I am going to be bold and throw a suggestion inspired from
what I read today. I was reading briefly about how kernel
manages its objects using the kobject data structure and
establishe 'liveness' (in the garbage collection sense)
relationship across objects. It allows one to group objects
into 'sets'. This would help us to categorise objects into
'groups' and derive from statedumps which group the leaking
object belongs to and infer some context out of that. This is
different from the other approaches that have been discussde. I am not familiar
with what more could be done with that sort of an infrastructure. I hope
this opens up the discussion and doesn't distract it.

~KP

- Original Message -
 
 On 09/18/2014 09:35 PM, Pranith Kumar Karampuri wrote:
 
  On 09/18/2014 09:31 PM, Kaleb KEITHLEY wrote:
  As a wishlist item, I think it'd be nice if debug builds (or some
  other build-time option) would disable the pools. Then valgrind might
  be more useful for finding leaks.
 Actually there seems to be some issue with running bricks using
 valgrind. Operations on mount hang when we start the bricks (Ravi
 confirmed this situation even today). That still needs to be solved, it
 used to work. Not sure what happened.
 
 Pranith.
 
  Maybe for GlusterFS-4.0?
  This is already available http://review.gluster.org/7835
 
  Pranith
 
 
  On 09/18/2014 11:40 AM, Dan Lambright wrote:
  If we could disable/enable ref tracking dynamically, it may only be
  heavy weight tempoarily while the customer is being observed.
  You could get a state dump , or another idea is to take a core of
  the live process.   gcore $(pidof processname)
 
  - Original Message -
  From: Pranith Kumar Karampuri pkara...@redhat.com
  To: Shyam srang...@redhat.com, gluster-devel@gluster.org
  Sent: Thursday, September 18, 2014 11:34:28 AM
  Subject: Re: [Gluster-devel] how do you debug ref leaks?
 
 
  On 09/18/2014 07:48 PM, Shyam wrote:
  On 09/17/2014 10:13 PM, Pranith Kumar Karampuri wrote:
  hi,
Till now the only method I used to find ref leaks
  effectively is to
  find what operation is causing ref leaks and read the code to
  find if
  there is a ref-leak somewhere. Valgrind doesn't solve this problem
  because it is reachable memory from inode-table etc. I am just
  wondering
  if there is an effective way anyone else knows of. Do you guys
  think we
  need a better mechanism of finding refleaks? At least which
  decreases
  the search space significantly i.e. xlator y, fop f etc? It would be
  better if we can come up with ways to integrate statedump and
  this infra
  just like we did for mem-accounting.
 
  One way I thought was to introduce new apis called
  xl_fop_dict/inode/fd_ref/unref (). Each xl keeps an array of
  num_fops
  per inode/dict/fd and increments/decrements accordingly. Dump
  this info
  on statedump.
 
  I myself am not completely sure about this idea. It requires all
  xlators
  to change.
 
  Any ideas?
 
  On a debug build we can use backtrace information stashed per ref and
  unref, this will give us history of refs taken and released. Which
  will also give the code path where ref was taken and released.
 
  It is heavy weight, so not for non-debug setups, but if a problem is
  reproducible this could be a quick way to check who is not releasing
  the ref's or have a history of the refs and unrefs to dig better into
  code.
 
  Do you have any ideas for final builds also? Basically when users
  report
  leaks it should not take us too long to figure out the problem
  area. We
  should just ask them for statedump and should be able to figure out
  the
  problem.
 
  Pranith
  Shyam
  ___
  Gluster-devel mailing list
  Gluster-devel@gluster.org
  http://supercolony.gluster.org/mailman/listinfo/gluster-devel
 
  ___
  Gluster-devel mailing list
  Gluster-devel@gluster.org
  http://supercolony.gluster.org/mailman/listinfo/gluster-devel
 
  ___
  Gluster-devel mailing list
  Gluster-devel@gluster.org
  http://supercolony.gluster.org/mailman/listinfo/gluster-devel
 
 
  ___
  Gluster-devel mailing list
  Gluster-devel@gluster.org
  http://supercolony.gluster.org/mailman/listinfo/gluster-devel
 
  ___
  Gluster-devel mailing list
  Gluster-devel@gluster.org
  http://supercolony.gluster.org/mailman/listinfo/gluster-devel
 
 ___
 Gluster-devel mailing list
 Gluster-devel@gluster.org
 http://supercolony.gluster.org/mailman/listinfo/gluster-devel
 
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-devel


Re: [Gluster-devel] how do you debug ref leaks?

2014-09-18 Thread Kaleb KEITHLEY

On 09/18/2014 12:09 PM, Pranith Kumar Karampuri wrote:


On 09/18/2014 09:35 PM, Pranith Kumar Karampuri wrote:


On 09/18/2014 09:31 PM, Kaleb KEITHLEY wrote:

As a wishlist item, I think it'd be nice if debug builds (or some
other build-time option) would disable the pools. Then valgrind might
be more useful for finding leaks.

Actually there seems to be some issue with running bricks using
valgrind. Operations on mount hang when we start the bricks (Ravi
confirmed this situation even today). That still needs to be solved, it
used to work. Not sure what happened.


I've been running gluster under valgrind. E.g. see 
http://download.gluster.org/pub/gluster/glusterfs/dynamic-analysis/master/glusterfs-valgrind/


It's pretty hit or miss, sometimes the client-side glusterfs process (at 
least IIRC it's the client-side glusterfs) will lock up and consume 100% 
CPU. And so far I haven't been able to trigger a core dump of the 
spinning process to see where it's locking up.


It would be nice to solve that so that I can do unsupervised nightly 
valgrind runs.


I'll look at the review you referenced for disabling pools.

Thanks,

--

Kaleb

___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-devel


Re: [Gluster-devel] how do you debug ref leaks?

2014-09-17 Thread Raghavendra Gowdappa


- Original Message -
 From: Raghavendra Gowdappa rgowd...@redhat.com
 To: Pranith Kumar Karampuri pkara...@redhat.com
 Cc: Gluster Devel gluster-devel@gluster.org
 Sent: Thursday, September 18, 2014 10:08:15 AM
 Subject: Re: [Gluster-devel] how do you debug ref leaks?
 
 For eg., if a dictionary is not freed because of non-zero refcount, if there
 is an information on who has held these references would help to narrow down
 the code path or component.

This solution might be rudimentary. However, someone who has worked on things 
like garbage collection can give better answers I think. This discussion also 
reminds me of Greenspun's tenth rule [1]

[1] http://en.wikipedia.org/wiki/Greenspun%27s_tenth_rule

 
 - Original Message -
  From: Pranith Kumar Karampuri pkara...@redhat.com
  To: Raghavendra Gowdappa rgowd...@redhat.com
  Cc: Gluster Devel gluster-devel@gluster.org
  Sent: Thursday, September 18, 2014 10:05:18 AM
  Subject: Re: [Gluster-devel] how do you debug ref leaks?
  
  
  On 09/18/2014 09:59 AM, Raghavendra Gowdappa wrote:
   One thing that would be helpful is allocator info for generic objects
   like dict, inode, fd etc. That way we wouldn't have to sift through large
   amount of code.
  Could you elaborate the idea please.
  
  Pranith
   - Original Message -
   From: Pranith Kumar Karampuri pkara...@redhat.com
   To: Gluster Devel gluster-devel@gluster.org
   Sent: Thursday, September 18, 2014 7:43:00 AM
   Subject: [Gluster-devel] how do you debug ref leaks?
  
   hi,
 Till now the only method I used to find ref leaks effectively is
 to
   find what operation is causing ref leaks and read the code to find if
   there is a ref-leak somewhere. Valgrind doesn't solve this problem
   because it is reachable memory from inode-table etc. I am just wondering
   if there is an effective way anyone else knows of. Do you guys think we
   need a better mechanism of finding refleaks? At least which decreases
   the search space significantly i.e. xlator y, fop f etc? It would be
   better if we can come up with ways to integrate statedump and this infra
   just like we did for mem-accounting.
  
   One way I thought was to introduce new apis called
   xl_fop_dict/inode/fd_ref/unref (). Each xl keeps an array of num_fops
   per inode/dict/fd and increments/decrements accordingly. Dump this info
   on statedump.
  
   I myself am not completely sure about this idea. It requires all xlators
   to change.
  
   Any ideas?
  
   Pranith
   ___
   Gluster-devel mailing list
   Gluster-devel@gluster.org
   http://supercolony.gluster.org/mailman/listinfo/gluster-devel
  
  
  
 ___
 Gluster-devel mailing list
 Gluster-devel@gluster.org
 http://supercolony.gluster.org/mailman/listinfo/gluster-devel
 
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-devel


Re: [Gluster-devel] how do you debug ref leaks?

2014-09-17 Thread Pranith Kumar Karampuri


On 09/18/2014 10:08 AM, Raghavendra Gowdappa wrote:

For eg., if a dictionary is not freed because of non-zero refcount, if there is 
an information on who has held these references would help to narrow down the 
code path or component.
Yes that is the aim. The implementation I suggested tries to get that 
information per xlator. Are you saying it is better to store in the 
dict/inode/fd itself? I actually wrote one patch sometime back to do 
something similar (Not tested yet :-) ).


diff --git a/libglusterfs/src/dict.c b/libglusterfs/src/dict.c
index 37a6b2c..bd4c438 100644
--- a/libglusterfs/src/dict.c
+++ b/libglusterfs/src/dict.c
@@ -100,8 +100,10 @@ dict_new (void)

 dict = get_new_dict_full(1);

-if (dict)
+if (dict) {
+dict-refs = get_new_dict_full(1);
 dict_ref (dict);
+}

 return dict;
 }
@@ -446,6 +448,7 @@ dict_destroy (dict_t *this)
 data_pair_t *pair = this-members_list;
 data_pair_t *prev = this-members_list;

+dict_destroy (this-refs);
 LOCK_DESTROY (this-lock);

 while (prev) {
@@ -495,15 +498,21 @@ dict_unref (dict_t *this)
 dict_t *
 dict_ref (dict_t *this)
 {
+int32_t ref = 0;
+xlat_t  *x = NULL;
 if (!this) {
 gf_log_callingfn (dict, GF_LOG_WARNING, dict is NULL);
 return NULL;
 }

+x = THIS;
 LOCK (this-lock);

 this-refcount++;

+dict_get_in32 (this-refs, x-name, ref);
+dict_set_int32 (this-refs, x-name, ref+1);
+
 UNLOCK (this-lock);

 return this;
@@ -513,15 +522,20 @@ void
 data_unref (data_t *this)
 {
 int32_t ref;
+xlator_t *x = NULL;

 if (!this) {
 gf_log_callingfn (dict, GF_LOG_WARNING, dict is NULL);
 return;
 }

+x = THIS;
 LOCK (this-lock);

 this-refcount--;
+dict_get_in32 (this-refs, x-name, ref);
+dict_set_int32 (this-refs, x-name, ref-1);
+
 ref = this-refcount;

 UNLOCK (this-lock);
diff --git a/libglusterfs/src/dict.h b/libglusterfs/src/dict.h
index 682c152..33ed7bd 100644
--- a/libglusterfs/src/dict.h
+++ b/libglusterfs/src/dict.h
@@ -93,6 +93,7 @@ struct _dict {
 data_pair_t*members_internal;
 data_pair_t free_pair;
 gf_boolean_tfree_pair_in_use;
+struct _dict   *refs;
 };

I was not happy with this implementation either.

similar implementation for inode here: http://review.gluster.com/8302

But I am not happy with any of these implementations. Probably because 
it is still not granular enough, i.e. fop info is missing. It is better 
than no info but still bad. We can't have it on statedump either. So 
when User reports high memory usage we still have to spend lot of time 
looking over all the places where the dicts are allocated which is bad :-(.


Pranith


- Original Message -

From: Pranith Kumar Karampuri pkara...@redhat.com
To: Raghavendra Gowdappa rgowd...@redhat.com
Cc: Gluster Devel gluster-devel@gluster.org
Sent: Thursday, September 18, 2014 10:05:18 AM
Subject: Re: [Gluster-devel] how do you debug ref leaks?


On 09/18/2014 09:59 AM, Raghavendra Gowdappa wrote:

One thing that would be helpful is allocator info for generic objects
like dict, inode, fd etc. That way we wouldn't have to sift through large
amount of code.

Could you elaborate the idea please.

Pranith

- Original Message -

From: Pranith Kumar Karampuri pkara...@redhat.com
To: Gluster Devel gluster-devel@gluster.org
Sent: Thursday, September 18, 2014 7:43:00 AM
Subject: [Gluster-devel] how do you debug ref leaks?

hi,
   Till now the only method I used to find ref leaks effectively is to
find what operation is causing ref leaks and read the code to find if
there is a ref-leak somewhere. Valgrind doesn't solve this problem
because it is reachable memory from inode-table etc. I am just wondering
if there is an effective way anyone else knows of. Do you guys think we
need a better mechanism of finding refleaks? At least which decreases
the search space significantly i.e. xlator y, fop f etc? It would be
better if we can come up with ways to integrate statedump and this infra
just like we did for mem-accounting.

One way I thought was to introduce new apis called
xl_fop_dict/inode/fd_ref/unref (). Each xl keeps an array of num_fops
per inode/dict/fd and increments/decrements accordingly. Dump this info
on statedump.

I myself am not completely sure about this idea. It requires all xlators
to change.

Any ideas?

Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-devel





___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-devel