Re: debugging a memory leak

[Manuel Bouyer]

On Fri, May 20, 2016 at 09:56:42PM +0200, Manuel Bouyer wrote:
> Hello,
> what tools do we have on NetBSD to find a memory leak in a userland
> program (actually OpenCPN - which is a large C++ program with dynamic
> libraries and uses dlopen()) ?
> 
> The memory usage of the process is slowy growing, until the systems
> gets out of ram/swap and kills it (on my evbarm which has no swap it
> takes about 2 days).

thanks for all the replies.
I ended up debugging this on linux with valgrind. It has been very helpfull
at locating the memory leak, and then spotting the double-free that my
fix introduced (the double-free would just cause a segfault in an
unrelated part on NetBSD).

If I had to track this down on NetBSD I think I would have instrumented
malloc/free with utrace(2), including the memory addresses and the
address of the caller.

-- 
Manuel Bouyer 
 NetBSD: 26 ans d'experience feront toujours la difference
--

Re: debugging a memory leak

[tlaronde]

On Sun, May 22, 2016 at 06:22:29PM -0400, Dan LaBell wrote:
> 
> On May 21, 2016, at 4:10 AM, Manuel Bouyer wrote:
> 
> >On Sat, May 21, 2016 at 08:36:10AM +0200, tlaro...@polynum.com wrote:
> >>On Fri, May 20, 2016 at 11:43:11PM +0200, Manuel Bouyer wrote:
> >>>On Fri, May 20, 2016 at 02:20:39PM -0600, Swift Griggs wrote:
> On Fri, 20 May 2016, Manuel Bouyer wrote:
> >what tools do we have on NetBSD to find a memory leak in a
> >userland
> >program (actually OpenCPN - which is a large C++ program with
> >dynamic
> >libraries and uses dlopen()) ?
> 
> >>
> >>I have used boehm-gc in the past on NetBSD. It is a garbage collector
> >>and allows to spot memory leaks too. It's in pkgsrc.
> >
> >thanks for the pointer. But if I understood it properly, the sources
> >needs to be modified ? I fear that in my case this is not an option:
> >opencpn is quite large and uses wxwidgets and glib, the memory may
> >be allocated from here too ...
> >
> I was considering replying about devel/boehm-gc  as well, but you would
> have to modify the build to use .a lib.  It seems to includes a lib that
> is just leak detection, so maybe just re-link.
> What about MALLOC_OPTIONS and /etc/malloc.conf ? And setting it to make
> utrace entries? man 3 jemalloc.

If there is a high-level include file, one has to put an 
"#include leak_detector.h" directive too, IIRC. 

This is probably "the" problem: a cdefs.h or whatever (or adapting
what is used to generate the program---autoconf or whatever to obtain
the same result) has to exist to ease the adaptation.
-- 
Thierry Laronde 
 http://www.kergis.com/
 http://www.arts-po.fr/
Key fingerprint = 0FF7 E906 FBAF FE95 FD89  250D 52B1 AE95 6006 F40C

Re: debugging a memory leak

[Sad Clouds]

On Fri, 20 May 2016 21:56:42 +0200
Manuel Bouyer  wrote:

> Hello,
> what tools do we have on NetBSD to find a memory leak in a userland
> program (actually OpenCPN - which is a large C++ program with dynamic
> libraries and uses dlopen()) ?
> 
> The memory usage of the process is slowy growing, until the systems
> gets out of ram/swap and kills it (on my evbarm which has no swap it
> takes about 2 days).
> 
> -- 
> Manuel Bouyer 
>  NetBSD: 26 ans d'experience feront toujours la difference
> --

What does pmap tell you? Is there a particular segment of memory that
grows over time? Some memory leaks are quite difficult to find, for
example if you create and terminate many threads, but you don't call
pthread_join(), the size of your process will probably keep on growing,
not sure if memory leak tools will be able to detect this.

Re: debugging a memory leak

[William A. Mahaffey III]

On 05/21/16 08:37, Rhialto wrote:

On Sat 21 May 2016 at 00:19:49 +, Valery Ushakov wrote:

Timo Buhrmester  wrote:


but it's limited to Linux, Darwin, and Solaris.

Last time I checked, FreeBSD also had valgrind working.  I wonder
how much effort it would be to port it to NetBSD.

As far as I understand (I had my brushes with valgrind internals) it's
mostly the drudgery of, effectively, writing code that annotates all
the system calls and all ioctls.  And that's not fun, obviously.

I did some research into ports of valgrind. While there seems to be a
FreeBSD version, I get the impression that it lives for years already as
a separate patch and that it for some reason never gets merged into the
mainline.


-uwe

-Olaf.



It's a port, not part of the base OS, if that's the distinction you are 
making. I am using it regularly under FreeBSD 9.3 Release & it seems to 
work AOK  I'm not sure if it is the L version, but it does work 
fine for me 



--

William A. Mahaffey III

 --

"The M1 Garand is without doubt the finest implement of war
 ever devised by man."
   -- Gen. George S. Patton Jr.

Re: debugging a memory leak

[Rhialto]

On Sat 21 May 2016 at 00:19:49 +, Valery Ushakov wrote:
> Timo Buhrmester  wrote:
> 
> > > but it's limited to Linux, Darwin, and Solaris.
> > 
> > Last time I checked, FreeBSD also had valgrind working.  I wonder
> > how much effort it would be to port it to NetBSD.
> 
> As far as I understand (I had my brushes with valgrind internals) it's
> mostly the drudgery of, effectively, writing code that annotates all
> the system calls and all ioctls.  And that's not fun, obviously.

I did some research into ports of valgrind. While there seems to be a
FreeBSD version, I get the impression that it lives for years already as
a separate patch and that it for some reason never gets merged into the
mainline.

> -uwe
-Olaf.
-- 
___ Olaf 'Rhialto' Seibert  -- Wayland: Those who don't understand X
\X/ rhialto/at/xs4all.nl-- are condemned to reinvent it. Poorly.


signature.asc
Description: PGP signature

Re: debugging a memory leak

[tlaronde]

On Fri, May 20, 2016 at 11:43:11PM +0200, Manuel Bouyer wrote:
> On Fri, May 20, 2016 at 02:20:39PM -0600, Swift Griggs wrote:
> > On Fri, 20 May 2016, Manuel Bouyer wrote:
> > > what tools do we have on NetBSD to find a memory leak in a userland 
> > > program (actually OpenCPN - which is a large C++ program with dynamic 
> > > libraries and uses dlopen()) ?
> > 

I have used boehm-gc in the past on NetBSD. It is a garbage collector
and allows to spot memory leaks too. It's in pkgsrc.

FWIW
-- 
Thierry Laronde 
 http://www.kergis.com/
 http://www.arts-po.fr/
Key fingerprint = 0FF7 E906 FBAF FE95 FD89  250D 52B1 AE95 6006 F40C

Re: debugging a memory leak

[Manuel Bouyer]

On Sat, May 21, 2016 at 08:36:10AM +0200, tlaro...@polynum.com wrote:
> On Fri, May 20, 2016 at 11:43:11PM +0200, Manuel Bouyer wrote:
> > On Fri, May 20, 2016 at 02:20:39PM -0600, Swift Griggs wrote:
> > > On Fri, 20 May 2016, Manuel Bouyer wrote:
> > > > what tools do we have on NetBSD to find a memory leak in a userland 
> > > > program (actually OpenCPN - which is a large C++ program with dynamic 
> > > > libraries and uses dlopen()) ?
> > > 
> 
> I have used boehm-gc in the past on NetBSD. It is a garbage collector
> and allows to spot memory leaks too. It's in pkgsrc.

thanks for the pointer. But if I understood it properly, the sources
needs to be modified ? I fear that in my case this is not an option:
opencpn is quite large and uses wxwidgets and glib, the memory may
be allocated from here too ...

-- 
Manuel Bouyer 
 NetBSD: 26 ans d'experience feront toujours la difference
--

Re: debugging a memory leak

[William A. Mahaffey III]

On 05/20/16 15:26, Swift Griggs wrote:

On Fri, 20 May 2016, Manuel Bouyer wrote:

what tools do we have on NetBSD to find a memory leak in a userland
program (actually OpenCPN - which is a large C++ program with dynamic
libraries and uses dlopen()) ?

Manuel, I'm guessing you are a much better C programmer than I, but I can
relate what I use. I do three different things:

1. If it'll compile on Linux, I'll test it with valgrind and just apply
the results on the NetBSD side.

2. ElectricFence, but I doubt it'd work with libs dlopen()'d. Then again,
if you used it with LD_PRELOAD you might be able to pre-empt the symbols
even in the dynamic libs (in theory).

3. I created a cheeseball reference counter similar to the one in glib by
just wrappering malloc(). It just add/drops/prints a linked list of
structs which are entries I'm tracking. When everything is working
perfectly, I remove it. Voila clean memory management without krufty
GC overhead!


The memory usage of the process is slowy growing, until the systems gets
out of ram/swap and kills it (on my evbarm which has no swap it takes
about 2 days).

My guess (which ain't worth much, take it with a grain of salt) you'd find
it pretty darn fast with valgrind. It's never failed me, but it's limited
to Linux, Darwin, and Solaris.

-Swift



I *LOUDLY* 2nd the motion on valgrind. Might be sweet if it got ported 
over to NetBSD 


--

William A. Mahaffey III

 --

"The M1 Garand is without doubt the finest implement of war
 ever devised by man."
   -- Gen. George S. Patton Jr.

Re: debugging a memory leak

[Valery Ushakov]

Timo Buhrmester  wrote:

> > but it's limited to Linux, Darwin, and Solaris.
> 
> Last time I checked, FreeBSD also had valgrind working.  I wonder
> how much effort it would be to port it to NetBSD.

As far as I understand (I had my brushes with valgrind internals) it's
mostly the drudgery of, effectively, writing code that annotates all
the system calls and all ioctls.  And that's not fun, obviously.

-uwe

Re: debugging a memory leak

[Swift Griggs]

On Fri, 20 May 2016, Manuel Bouyer wrote:
> I though ElectricFence would only detect things like use after free or 
> out of bound access, but not memory leaks ?

Hmm, I thought it did. Like if you try to malloc() over a pointer and 
clobber it before you free()'d the previous one (say in a loop/iterative 
situation). A ghetto reference counter + LD_PRELOAD works pretty well for 
that, too. The only problem is that it makes your app fairly slow for 
testing if it does a lot of dynamic allocation. Neither approach is near 
as good as Valgrind, for sure.

-Swift

Re: debugging a memory leak

[Manuel Bouyer]

On Fri, May 20, 2016 at 10:29:02PM +0200, Timo Buhrmester wrote:
> > but it's limited to Linux, Darwin, and Solaris.
> Last time I checked, FreeBSD also had valgrind working.  I wonder how
> much effort it would be to port it to NetBSD.

google found some talks about it for NetBSD but it seems the task
was never completed.

-- 
Manuel Bouyer 
 NetBSD: 26 ans d'experience feront toujours la difference
--

Re: debugging a memory leak

[Manuel Bouyer]

On Fri, May 20, 2016 at 02:20:39PM -0600, Swift Griggs wrote:
> On Fri, 20 May 2016, Manuel Bouyer wrote:
> > what tools do we have on NetBSD to find a memory leak in a userland 
> > program (actually OpenCPN - which is a large C++ program with dynamic 
> > libraries and uses dlopen()) ?
> 
> Manuel, I'm guessing you are a much better C programmer than I, but I can 
> relate what I use. I do three different things:
> 
> 1. If it'll compile on Linux, I'll test it with valgrind and just apply 
> the results on the NetBSD side. 

I tried this, as opencpn is developed for linux. It builds and runs,
but the plugins won't load, the main application lacks some symbols
(but not all !). 
I'm not seeing the lack on Linux; I've not tried running without the
plugins on NetBSD yet.

> 
> 2. ElectricFence, but I doubt it'd work with libs dlopen()'d. Then again, 
> if you used it with LD_PRELOAD you might be able to pre-empt the symbols 
> even in the dynamic libs (in theory). 

I though ElectricFence would only detect things like use after free or
out of bound access, but not memory leaks ?

> 
> 3. I created a cheeseball reference counter similar to the one in glib by 
> just wrappering malloc(). It just add/drops/prints a linked list of 
> structs which are entries I'm tracking. When everything is working 
> perfectly, I remove it. Voila clean memory management without krufty 
> GC overhead! 
> 
> > The memory usage of the process is slowy growing, until the systems gets 
> > out of ram/swap and kills it (on my evbarm which has no swap it takes 
> > about 2 days).
> 
> My guess (which ain't worth much, take it with a grain of salt) you'd find 
> it pretty darn fast with valgrind. It's never failed me, but it's limited 
> to Linux, Darwin, and Solaris.

Yes, valgrind was my first idea. I guess I'll have to debug this
hidden symbols problem...

-- 
Manuel Bouyer 
 NetBSD: 26 ans d'experience feront toujours la difference
--

Re: debugging a memory leak

[Timo Buhrmester]

> but it's limited to Linux, Darwin, and Solaris.
Last time I checked, FreeBSD also had valgrind working.  I wonder how
much effort it would be to port it to NetBSD.

Re: debugging a memory leak

[Swift Griggs]

On Fri, 20 May 2016, Manuel Bouyer wrote:
> what tools do we have on NetBSD to find a memory leak in a userland 
> program (actually OpenCPN - which is a large C++ program with dynamic 
> libraries and uses dlopen()) ?

Manuel, I'm guessing you are a much better C programmer than I, but I can 
relate what I use. I do three different things:

1. If it'll compile on Linux, I'll test it with valgrind and just apply 
the results on the NetBSD side. 

2. ElectricFence, but I doubt it'd work with libs dlopen()'d. Then again, 
if you used it with LD_PRELOAD you might be able to pre-empt the symbols 
even in the dynamic libs (in theory). 

3. I created a cheeseball reference counter similar to the one in glib by 
just wrappering malloc(). It just add/drops/prints a linked list of 
structs which are entries I'm tracking. When everything is working 
perfectly, I remove it. Voila clean memory management without krufty 
GC overhead! 

> The memory usage of the process is slowy growing, until the systems gets 
> out of ram/swap and kills it (on my evbarm which has no swap it takes 
> about 2 days).

My guess (which ain't worth much, take it with a grain of salt) you'd find 
it pretty darn fast with valgrind. It's never failed me, but it's limited 
to Linux, Darwin, and Solaris.

-Swift