On Sunday, 25 September 2016 at 16:23:11 UTC, Matthias Klumpp
wrote:
Hello!
I am working together with others on the D-based
appstream-generator[1] project, which is generating software
metadata for "software centers" and other package-manager
functionality on Linux distributions, and is used by default on
Debian, Ubuntu and Arch Linux.
For Ubuntu, some modifications on the code were needed, and
apparently for them the code is currently crashing in the GC
collection thread: http://paste.debian.net/840490/
The project is running a lot of stuff in parallel and is using
the GC (if the extraction is a few seconds slower due to the GC
being active, it doesn't matter much).
We also link against a lot of 3rd-party libraries and use a big
amount of existing C code in the project.
So, I would like to know the following things:
1) Is there any caveat when linking to C libraries and using
the GC in a project? So far, it seems to be working well, but
there have been a few cases where I was suspicious about the GC
actually doing something to malloc'ed stuff or C structs
present in the bindings.
2) How can one debug issues like the one mentioned above
properly? Since it seems to happen in the GC and doesn't give
me information on where to start searching for the issue, I am
a bit lost.
3) The tool seems to leak memory somewhere and OOMs pretty
quickly on some machines. All the stuff using C code frees
resources properly though, and using Valgrind on the project is
a pain due to large amounts of data being mmapped. I worked
around this a while back, but then the GC interfered with
Valgrind, making information less useful. Is there any
information on how to find memory leaks, or e.g. large structs
the GC cannot free because something is still having a needless
reference on it?
Unfortunately I can't reproduce the crash from 2) myself, it
only seems to happen at Ubuntu (but Ubuntu is using some
different codepaths too).
Any insights would be highly appreciated!
Cheers,
Matthias
[1[: https://github.com/ximion/appstream-generator
First, make sure any C threads calling D code use
Thread.attachThis (thread_attachThis maybe?). Otherwise the GC
will not suspend those threads during a collection which will
cause crashes. I'd guess this is your issue.
Second, tell the GC of non-GC memory that has pointers to GC
memory by using GC.addRange / GC.addRoot as needed. Make sure to
remove them once the non-GC memory is deallocated as well,
otherwise you'll get memory leaks. The GC collector is also
conservative, not precise, so false positives are possible. If
you're using 64 bit programs, this shouldn't be much of an issue
though.
Finally, make sure you're not doing any GC allocations in dtors.
