Garbage collection
I am writing a web application using vibe.d (not sure whether
that is relevant or not), and have run into a memory leak. I
wrote the following code to try and replicate the problem.
import std.algorithm;
import std.range;
import std.format;
import std.stdio;
import core.thread;
import core.memory;
auto f(R)(R r) {
return format("%s", r);
}
void main()
{
while(true)
{
writeln("Starting");
{
auto str = f(iota(1).map!(x=>x+1));
}
writeln("Done");
GC.collect();
Thread.sleep( dur!("msecs")( 3 ) );
}
}
It doesn't replicate the problem but it still doesn't behave as
I would expect. I would expect the memory usage of this code to
grow and shrink. However, I find that the memory usage grows to
about 1.5GB and never decreases. Is there something I am not
understanding?
Garbage Collection Issue
I am encountering a strange problem with the GC on a specific platform: at the first attempt to clear the current memory pool to make room for a new allocation, the GC considers that the page in which the main thread resides (the one created in the init function of the GC) can be freed.. therefore, frees the entire pool and reallocates at the same location; later, when accessing thread's address, it stumbles upon garbage data. The question is: where does the GC expects the address of the thread to be found so that it takes it into consideration? A relevant mention would be that the platform doesn't support TLS so it won't find anything when trying to access that data. Could it be related to this?
Re: Garbage collection
On Saturday, 27 June 2020 at 10:08:15 UTC, James Gray wrote: I am writing a web application using vibe.d (not sure whether that is relevant or not), and have run into a memory leak. I wrote the following code to try and replicate the problem. [...] I now compiled the same code above with ldc2 and it is leaking. Any suggestions?
Re: Garbage collection
On Saturday, 27 June 2020 at 10:08:15 UTC, James Gray wrote: I find that the memory usage grows to about 1.5GB and never decreases. Is there something I am not understanding? How are you measuring that? GC.collect() does not necessarily release the pages to the OS. For that, there's the GC.minimize().
Re: Garbage collection
On Saturday, 27 June 2020 at 11:00:58 UTC, Stanislav Blinov wrote: On Saturday, 27 June 2020 at 10:08:15 UTC, James Gray wrote: I find that the memory usage grows to about 1.5GB and never decreases. Is there something I am not understanding? How are you measuring that? GC.collect() does not necessarily release the pages to the OS. For that, there's the GC.minimize(). I am measuring the memory usage using top from the command line. GC.minimize() does seem to stop the leak. But it doesn't explain why the program isn't releasing essentially all the memory between calls to f (it using around 2GB ram all the time). Is there a way of achieving that?
Re: Garbage collection
On Saturday, 27 June 2020 at 11:11:38 UTC, James Gray wrote: I am measuring the memory usage using top from the command line. GC.minimize() does seem to stop the leak. But it doesn't explain why the program isn't releasing essentially all the memory between calls to f (it using around 2GB ram all the time). Is there a way of achieving that? It's not a leak. The GC allocates memory as it needs it and holds on to it. When something is collected, the GC can reuse then released memory when it needs it.
Re: Garbage collection
On Saturday, 27 June 2020 at 11:11:38 UTC, James Gray wrote: I am measuring the memory usage using top from the command line. GC.minimize() does seem to stop the leak. That is not a memory leak. That's the allocator keeping pages for itself to not have to go to the kernel every time you allocate. But it doesn't explain why the program isn't releasing essentially all the memory between calls to f (it using around 2GB ram all the time). Allocators usually don't do that. They keep (at least some) memory mapped to make allocations more efficient. Is there a way of achieving that? I would think collect + minimize should do the trick. Just keep in mind that that's grossly inefficient.
Re: Garbage collection
On 27/6/20 13:21, Stanislav Blinov wrote: I would think collect + minimize should do the trick. Just keep in mind that that's grossly inefficient. If you are using linux, have in mind that the memory is often not returned to the OS even after a (libc) free. If you check with tools like `top`, it'll still show as assigned to the process. What I had to do (both in D and in C/C++) was to call malloc_trim [1] manually to have the memory actually sent back to the OS. [1]: https://man7.org/linux/man-pages/man3/malloc_trim.3.html
Re: Garbage collection
On Saturday, 27 June 2020 at 11:35:12 UTC, Arafel wrote: If you are using linux, have in mind that the memory is often not returned to the OS even after a (libc) free. That's a good observation. Although a GC implementation is not required to actually use malloc, so depending on that falls into "know what you're doing" territory :)
Re: Garbage collection
On Saturday, 27 June 2020 at 12:07:19 UTC, Stanislav Blinov wrote: On Saturday, 27 June 2020 at 11:35:12 UTC, Arafel wrote: If you are using linux, have in mind that the memory is often not returned to the OS even after a (libc) free. That's a good observation. Although a GC implementation is not required to actually use malloc, so depending on that falls into "know what you're doing" territory :) Thanks for the help, but unfortunately it isn't stopping memory usage growing in the original app. I will try and build a minimal example. In the meantime perhaps someone can suggest how I might figure out what is going on. Repeating the same action is giving memory usage growth as follows. 1.7GB first time (which now drops to about 1GB), then 2.7GB dropping to about 2GB and so on.
Re: Garbage collection
On Saturday, 27 June 2020 at 12:53:01 UTC, James Gray wrote: On Saturday, 27 June 2020 at 12:07:19 UTC, Stanislav Blinov wrote: On Saturday, 27 June 2020 at 11:35:12 UTC, Arafel wrote: [...] That's a good observation. Although a GC implementation is not required to actually use malloc, so depending on that falls into "know what you're doing" territory :) Thanks for the help, but unfortunately it isn't stopping memory usage growing in the original app. I will try and build a minimal example. In the meantime perhaps someone can suggest how I might figure out what is going on. Repeating the same action is giving memory usage growth as follows. 1.7GB first time (which now drops to about 1GB), then 2.7GB dropping to about 2GB and so on. Which eventually results in mac os running out of memory.
Re: Garbage collection
On Saturday, 27 June 2020 at 10:08:15 UTC, James Gray wrote:
have run into a memory leak
Something seems really off indeed. I've run this on Win64 with
DMD (2.092) and LDC (1.22), without any extra cmdline options:
-
import core.memory;
import core.stdc.stdio;
import std.range;
import std.format;
auto f(R)(R r) { return format("%s", r); }
int toMB(ulong size) { return cast(int) (size / 1048576.0 + 0.5);
}
void printGCStats()
{
const stats = GC.stats;
const used = toMB(stats.usedSize);
const free = toMB(stats.freeSize);
const total = toMB(stats.usedSize + stats.freeSize);
printf(" GC stats: %dM used, %dM free, %dM total\n", used,
free, total);
}
void main()
{
printGCStats();
while (true)
{
puts("Starting");
string str = f(iota(100_000_000));
printf(" string size: %dM\n", toMB(str.length));
str = null;
GC.collect();
printGCStats();
}
}
-
Output with DMD (no change with the precise GC via
`--DRT-gcopt=gc:precise`):
-
GC stats: 0M used, 1M free, 1M total
Starting
string size: 943M
GC stats: 1168M used, 1139M free, 2306M total
Starting
string size: 943M
GC stats: 1168M used, 2456M free, 3623M total
Starting
string size: 943M
GC stats: 1168M used, 2456M free, 3623M total
Starting
string size: 943M
GC stats: 1168M used, 2456M free, 3623M total
Starting
string size: 943M
GC stats: 1168M used, 2456M free, 3623M total
-
With LDC:
-
GC stats: 0M used, 1M free, 1M total
Starting
string size: 943M
GC stats: 1168M used, 1139M free, 2306M total
Starting
string size: 943M
GC stats: 2335M used, 1288M free, 3623M total
Starting
string size: 943M
GC stats: 2335M used, 2605M free, 4940M total
Starting
string size: 943M
GC stats: 2335M used, 2605M free, 4940M total
Starting
string size: 943M
GC stats: 2335M used, 2605M free, 4940M total
-
Note that I explicitly clear the `str` slice before GC.collect(),
so that the stack shouldn't contain any refs to the fat string
anymore.
Re: Garbage collection
=> https://issues.dlang.org/show_bug.cgi?id=20983
Re: Garbage collection
On Saturday, 27 June 2020 at 14:12:09 UTC, kinke wrote:
On Saturday, 27 June 2020 at 10:08:15 UTC, James Gray wrote:
have run into a memory leak
Something seems really off indeed. I've run this on Win64 with
DMD (2.092) and LDC (1.22), without any extra cmdline options:
-
import core.memory;
import core.stdc.stdio;
import std.range;
import std.format;
auto f(R)(R r) { return format("%s", r); }
int toMB(ulong size) { return cast(int) (size / 1048576.0 +
0.5); }
void printGCStats()
{
const stats = GC.stats;
const used = toMB(stats.usedSize);
const free = toMB(stats.freeSize);
const total = toMB(stats.usedSize + stats.freeSize);
printf(" GC stats: %dM used, %dM free, %dM total\n", used,
free, total);
}
void main()
{
printGCStats();
while (true)
{
puts("Starting");
string str = f(iota(100_000_000));
printf(" string size: %dM\n", toMB(str.length));
str = null;
GC.collect();
printGCStats();
}
}
-
Output with DMD (no change with the precise GC via
`--DRT-gcopt=gc:precise`):
-
GC stats: 0M used, 1M free, 1M total
Starting
string size: 943M
GC stats: 1168M used, 1139M free, 2306M total
Starting
string size: 943M
GC stats: 1168M used, 2456M free, 3623M total
Starting
string size: 943M
GC stats: 1168M used, 2456M free, 3623M total
Starting
string size: 943M
GC stats: 1168M used, 2456M free, 3623M total
Starting
string size: 943M
GC stats: 1168M used, 2456M free, 3623M total
-
With LDC:
-
GC stats: 0M used, 1M free, 1M total
Starting
string size: 943M
GC stats: 1168M used, 1139M free, 2306M total
Starting
string size: 943M
GC stats: 2335M used, 1288M free, 3623M total
Starting
string size: 943M
GC stats: 2335M used, 2605M free, 4940M total
Starting
string size: 943M
GC stats: 2335M used, 2605M free, 4940M total
Starting
string size: 943M
GC stats: 2335M used, 2605M free, 4940M total
-
Note that I explicitly clear the `str` slice before
GC.collect(), so that the stack shouldn't contain any refs to
the fat string anymore.
Thank you for doing this. I hope my example doesn't obscure what
you show here.
(I borrowed some of your code).
I have produced something which essentially reproduces my problem.
---
import std.range;
import std.algorithm;
import std.format;
import std.stdio;
import core.thread;
import core.memory;
struct Node {
Node* next;
Node* prev;
ulong val;
}
Node* insertAfter(Node* cur, ulong val) {
Node* node = new Node;
if (cur != null) {
node.next = cur.next;
node.prev = cur;
cur.next = node;
node.next.prev = node;
}
else {
node.next = node;
node.prev = node;
}
node.val = val;
return node;
}
int toMB(ulong size) { return cast(int) (size / 1048576.0 + 0.5);
}
void printGCStats()
{
const stats = GC.stats;
const used = toMB(stats.usedSize);
const free = toMB(stats.freeSize);
const total = toMB(stats.usedSize + stats.freeSize);
writef(" GC stats: %dM used, %dM free, %dM total\n", used,
free, total);
}
void main()
{
while(true)
{
printGCStats();
writeln("Starting");
Node* dll;
dll = iota(2).fold!((c,x)=>insertAfter(c,x))(dll);
writef("Last element %s\n", dll.val);
dll = null;
writeln("Done");
GC.collect();
GC.minimize();
Thread.sleep( dur!("msecs")( 1 ) );
}
}
--
With DMD this produces:
GC stats: 0M used, 0M free, 0M total
Starting
Last element 1
Done
GC stats: 6104M used, 51M free, 6155M total
Starting
Last element 1
Done
GC stats: 12207M used, 28M free, 12235M total
With LDC2 this produces:
GC stats: 0M used, 0M free, 0M total
Starting
Last element 1
Done
GC stats: 6104M used, 51M free, 6155M total
Starting
Last element 1
Done
GC stats: 12207M used, 28M free, 12235M total
Re: Garbage collection
On Saturday, 27 June 2020 at 14:12:09 UTC, kinke wrote: Note that I explicitly clear the `str` slice before GC.collect(), so that the stack shouldn't contain any refs to the fat string anymore. Hrm... What happens if you call collect() twice?
Re: Garbage collection
On Saturday, 27 June 2020 at 15:27:34 UTC, Stanislav Blinov wrote: On Saturday, 27 June 2020 at 14:12:09 UTC, kinke wrote: Note that I explicitly clear the `str` slice before GC.collect(), so that the stack shouldn't contain any refs to the fat string anymore. Hrm... What happens if you call collect() twice? Nothing changes, even when collecting 5 times at the end of each iteration. In the filed testcase, I've extracted the stack ref to a dedicated function, so that there really shouldn't be any refs on the stack (this is unoptimized code after all...).
Re: Garbage collection
On Saturday, 27 June 2020 at 16:03:12 UTC, kinke wrote: On Saturday, 27 June 2020 at 15:27:34 UTC, Stanislav Blinov wrote: Hrm... What happens if you call collect() twice? Nothing changes, even when collecting 5 times at the end of each iteration. In the filed testcase, I've extracted the stack ref to a dedicated function, so that there really shouldn't be any refs on the stack (this is unoptimized code after all...). Here on Linux, the double collection results in this output: GC stats: 0M used, 0M free, 0M total Starting string size: 943M GC stats: 0M used, 2306M free, 2306M total Starting string size: 943M GC stats: 0M used, 2306M free, 2306M total Starting string size: 943M GC stats: 0M used, 2306M free, 2306M total Starting string size: 943M GC stats: 0M used, 2306M free, 2306M total
Re: Garbage collection
On Saturday, 27 June 2020 at 14:49:34 UTC, James Gray wrote: On Saturday, 27 June 2020 at 14:12:09 UTC, kinke wrote: [...] Thank you for doing this. I hope my example doesn't obscure what you show here. (I borrowed some of your code). [...] In case it helps, setting all the next and previous pointers in the link list to null allows the garbage collector to collect in the above code.
Re: Garbage collection
On Saturday, 27 June 2020 at 14:49:34 UTC, James Gray wrote: I have produced something which essentially reproduces my problem. What is the problem? Do you have a leak or you want to know how GC works?
Re: Garbage collection
On Tuesday, 30 June 2020 at 06:16:26 UTC, Kagamin wrote: On Saturday, 27 June 2020 at 14:49:34 UTC, James Gray wrote: I have produced something which essentially reproduces my problem. What is the problem? Do you have a leak or you want to know how GC works? I have managed to resolve my problem (which was a memory leak). My code uses a large data structure similar to a link list and the garbage collector was not collecting it. However, if I set all the "links" between the nodes in the data structure to null it is then collected.
Garbage collection and closures.
Hello! I can't understand one thing related to closures and
calling of GC. I have the following demo snippet, where a closure
is passed to `receive` function in a loop.
bool isDone = false;
while(!isDone)
receive((bool val){ isDone = val });
Is GC called every iteration of this loop?
I know that I can move the code to a method of a struct and make
the closure and variable "isDone" as fields of the struct. It
avoids GC calling definitely. But I want to get how many times GC
is called in case of a closure in a loop.
Thanks.
Re: Garbage Collection Issue
On 5/30/20 9:51 PM, Marius Cristian Baciu wrote: I am encountering a strange problem with the GC on a specific platform: at the first attempt to clear the current memory pool to make room for a new allocation, the GC considers that the page in which the main thread resides (the one created in the init function of the GC) can be freed.. therefore, frees the entire pool and reallocates at the same location; later, when accessing thread's address, it stumbles upon garbage data. The question is: where does the GC expects the address of the thread to be found so that it takes it into consideration? A relevant mention would be that the platform doesn't support TLS so it won't find anything when trying to access that data. Could it be related to this? I can't imagine much of druntime working at all without TLS. Indeed, it is a requirement these days. I believe that's where these roots are being stored. -Steve
Re: Garbage Collection Issue
On Sunday, 31 May 2020 at 16:57:06 UTC, Steven Schveighoffer wrote: I can't imagine much of druntime working at all without TLS. Indeed, it is a requirement these days. TLS is evil for async/await when any thread can execute any fiber (case where fiber tied to thread is wrong/dead version of async/await cuz 1st thread has 1000 fibers and 2nd only 10)
Re: Garbage Collection Issue
On Sunday, 31 May 2020 at 16:57:06 UTC, Steven Schveighoffer wrote: I can't imagine much of druntime working at all without TLS. Indeed, it is a requirement these days. I believe that's where these roots are being stored. -Steve I would really like if druntime could remove its TLS variables as much as possible. TLS is really a complicated solution underneath and druntime makes it even more complicated. It requires a hook in thread creation since the raw TLS specification only applies simple variables that can be initialized using memcpy/memset. Any thread that is created outside the druntime will fail if D supports "complex" TLS variables. TLS variables are also slower that normal variables since it often requires a system call in order to obtain the variable. druntime should use stack variables much it can and/or shared variables. If you ever encounter a TLS variable which is global variable in D, try to see if you can solve it with a stack or shared variable.
Re: Garbage Collection Issue
On 6/1/20 5:53 AM, a11e99z wrote: On Sunday, 31 May 2020 at 16:57:06 UTC, Steven Schveighoffer wrote: I can't imagine much of druntime working at all without TLS. Indeed, it is a requirement these days. TLS is evil for async/await when any thread can execute any fiber (case where fiber tied to thread is wrong/dead version of async/await cuz 1st thread has 1000 fibers and 2nd only 10) That would require fiber local storage, which I don't know if that is supported. I think fibers jumping between threads is also not supported, but I don't know for certain. -Steve
Re: Garbage Collection Issue
On 6/1/20 6:51 AM, IGotD- wrote: On Sunday, 31 May 2020 at 16:57:06 UTC, Steven Schveighoffer wrote: I can't imagine much of druntime working at all without TLS. Indeed, it is a requirement these days. I believe that's where these roots are being stored. I would really like if druntime could remove its TLS variables as much as possible. TLS is really a complicated solution underneath and druntime makes it even more complicated. It requires a hook in thread creation since the raw TLS specification only applies simple variables that can be initialized using memcpy/memset. Any thread that is created outside the druntime will fail if D supports "complex" TLS variables. D can use non-D created threads, but they will not be scanned by the GC, or run thread static constructors or destructors. TLS variables are also slower that normal variables since it often requires a system call in order to obtain the variable. I was under the impression that TLS works by altering a global pointer during the context switch. I didn't think accessing a variable involved a system call. For sure they are slower than "normal" variables, but how much slower? I'm not sure. druntime should use stack variables much it can and/or shared variables. druntime does not needlessly use TLS as far as I know. If you find a case that can be switched please file a bug report. If you ever encounter a TLS variable which is global variable in D, try to see if you can solve it with a stack or shared variable. This can only take you so far, when the language uses TLS by default. The GC has to support scanning TLS and so it uses TLS to track thread-specific data. What it sounds like to me is that the OP implemented a "get it to compile" solution for TLS, and this is not working for him. There is no removing TLS, because the language uses it directly for global variables, and many guarantees are enabled by it. For instance the array append runtime uses a lock-free TLS cache to ensure speedy appending. Without TLS, the global lock would be needed for every append. -Steve
Re: Garbage Collection Issue
On Monday, 1 June 2020 at 12:37:05 UTC, Steven Schveighoffer wrote: I was under the impression that TLS works by altering a global pointer during the context switch. I didn't think accessing a variable involved a system call. For sure they are slower than "normal" variables, but how much slower? I'm not sure. It depends, there several different optimizations possible. This is essentially the difference between the -fPIC and -fpie flag GNU compilers. -fpie can optimize TLS so that it is an offset from a certain register (fs or gs with x86). Otherwise the compiler insert __tls_get_addr. Typically shared objects gets this call, but the executable can optimize. So if druntime is a shared object, it will use __tls_get_addr. TLS variables will not be major hit if used moderately, used in a loop, then you will certainly see a performance hit. This can only take you so far, when the language uses TLS by default. The GC has to support scanning TLS and so it uses TLS to track thread-specific data. Yes, this was some of the annoyance I had when porting druntime. The thread startup code needed to use link library (like elf/link.h for linux) in order to obtain the entire TLS area (areas because there a several of them). This includes scanning sections during startup and it becomes even more complicated with runtime loaded modules. Basically there is a lot of boiler plate in druntime just for reading the executable format. druntime has tons of elf stuff in it just to load a program something I'm not too keen on, because it's a lot of code and you need to support all the quirks with different CPU archs and operating systems. You'd want druntime to be more OS agnostic a let the OS services deal with the TLS stuff. The only upside can be that you can have a full symbolic stack trace during aborts when a poking in the executable formats. Well, that's how it is because of GC and there is not really any way around it. A non tracing GC would not have this requirement though. When you dig into these details you realize how heavy the D language really is and some solutions get negative beauty points.
On D's garbage collection
I'm been thinking about using D in conjunction with C11 to develop a set of applications with hard real-time requirements. While initially the goal was to use C++ instead, it has become clear that D's introspection facilities will offer significant advantages. Unfortunately, the project will heavily rely on custom memory allocators written in C, so the presence of garbage collection in the language is a problem. While I'm aware that the nogc attribute exists, I haven't actually seen a way to apply it to a whole project. Is this possible?
Dynamic Array Garbage collection
I have a function defined as:
some_function(int[] array) { ... }; //this function does not ever modify
values of array
When I call this function (once every program cycle) from an object using an
array of type short:
//member variable in an object
short[] x = new short[4];
//object calls this
some_function([ x[0], x[1], x[2], x[3] ]);
After a few seconds of the program running, x[0] and x[1] values are set to 0
and a couple values of another array in the object are modified as well.
I have found that this bug is corrected by calling delete at the end of
some_function of the variable array. This tells me that the garbage collector
is acting funny if I do not call delete on the array. (I have watched the
memory usage of the program and it doesn't fluctuate or pause or anything to
signal the freeing/allocating of memory)
Any Idea what is going on?
Garbage collection in D
I've just started to play around with D, and I'm hoping someone can clarify this. I wrote a very simple program that just allocates lots of objects, in order to benchmark the garbage collector in D. For comparison, I wrote the programs in C++, Java and D: C++: http://gist.github.com/122708 Java: http://gist.github.com/122709 D: http://gist.github.com/121790 With an iteration count of , I get the following numbers: JAVA: 0:01.60 elapsed, 1.25 user, 0.28 system C++: 0:04.99 elapsed, 4.97 user, 0.00 system D: 0:25.28 elapsed, 25.22 user, 0.00 system As you can see, D is abysmally slow compared to C++ and Java. This is using the GNU gdc compiler. I'm hoping the community can give me some insight on what is going on. Thanks, Diwaker
Re: Garbage collection and closures.
On Saturday, 17 June 2017 at 13:03:28 UTC, ANtlord wrote: Is GC called every iteration of this loop? No, it will once on scope entry; where the deepest-referenced variable that is actually captured is defined. The compiler allocates heap space instead of stack space for the locals, then runs the function normally using that space. The current implementation doesn't quite do this - it actually will alloc only once, on function entry, even when there's a variable inside a loop that is supposed to be independent. This is the cause of the commonly-referenced bug with foreach(i; whatever) capture(i); all showing the same number. When that bug is fixed, it is possible to allocate on each loop... but your code doesn't anyway, so you're ok. It will only alloc once. You can prove this with a debugger btw, set a breakpoint on `_d_allocmemory`.
Re: Garbage collection and closures.
On Saturday, 17 June 2017 at 13:13:17 UTC, Adam D. Ruppe wrote: On Saturday, 17 June 2017 at 13:03:28 UTC, ANtlord wrote: Is GC called every iteration of this loop? No, it will once on scope entry; where the deepest-referenced variable that is actually captured is defined. The compiler allocates heap space instead of stack space for the locals, then runs the function normally using that space. Excuse me, I can't get what does it mean "deepest-referenced". What the deep you mean? The deep of a closure or deep of the function where the variable is defined. Can you give an example code? It will only alloc once. You can prove this with a debugger btw, set a breakpoint on `_d_allocmemory`. Is this function called every time when allocation happens in a heap? Thank you. Sorry if my English is not clear.
Re: Garbage collection and closures.
On Saturday, 17 June 2017 at 14:19:34 UTC, ANtlord wrote:
Excuse me, I can't get what does it mean "deepest-referenced".
What the deep you mean? The deep of a closure or deep of the
function where the variable is defined. Can you give an example
code?
Where the variable is defined that is referenced in the closure.
So:
---
void uses(void delegate() dg);
void foo() {
int a;
foreach(b; 0 .. 10) {
uses( () { a++; } ); // #1
uses( () { b++; } ); // #2
}
}
---
In that case, #1 would only be allocated once, at the start of
the foo() function. It only uses `a`, so it doesn't have to
allocate again after the point a is defined.
But #2 might allocate each time through the loop. (It currently
doesn't, but this is filed as an open bug because it is supposed
to.) Since it uses `b` which is defined inside the loop, it will
have to allocate a new copy for each iteration.
Is this function called every time when allocation happens in a
heap?
Not any allocation, it is just the function the compiler uses
when it needs to make a closure.
Re: Garbage collection and closures.
On Saturday, 17 June 2017 at 17:15:50 UTC, Adam D. Ruppe wrote:
On Saturday, 17 June 2017 at 14:19:34 UTC, ANtlord wrote:
Excuse me, I can't get what does it mean "deepest-referenced".
What the deep you mean? The deep of a closure or deep of the
function where the variable is defined. Can you give an
example code?
Where the variable is defined that is referenced in the closure.
So:
---
void uses(void delegate() dg);
void foo() {
int a;
foreach(b; 0 .. 10) {
uses( () { a++; } ); // #1
uses( () { b++; } ); // #2
}
}
---
In that case, #1 would only be allocated once, at the start of
the foo() function. It only uses `a`, so it doesn't have to
allocate again after the point a is defined.
But #2 might allocate each time through the loop. (It currently
doesn't, but this is filed as an open bug because it is
supposed to.) Since it uses `b` which is defined inside the
loop, it will have to allocate a new copy for each iteration.
Is this function called every time when allocation happens in
a heap?
Not any allocation, it is just the function the compiler uses
when it needs to make a closure.
Thanks a lot, Adam! Everything is clear. Except for the bug. I've
got an expected result of a value of the variable from #2. The
value equals to number from sequence plus one in each iteration.
There are ten iterations.
What's wrong? Are there should be five iterations? It doesn't
make sense for me due to the variable assigned value from the
sequence 0..10. Or do I look at the wrong place? Can you give me
a link to the bug?
Thank you again!
Re: Garbage collection and closures.
On Saturday, 17 June 2017 at 17:15:50 UTC, Adam D. Ruppe wrote: On Saturday, 17 June 2017 at 14:19:34 UTC, ANtlord wrote: [...] Where the variable is defined that is referenced in the closure. So: [...] if the uses parma is 'scope': void uses(scope void delegate() dg); will it be not alloc memory?
Re: Garbage collection and closures.
On Monday, 19 June 2017 at 09:10:16 UTC, Dsby wrote:
On Saturday, 17 June 2017 at 17:15:50 UTC, Adam D. Ruppe wrote:
On Saturday, 17 June 2017 at 14:19:34 UTC, ANtlord wrote:
[...]
Where the variable is defined that is referenced in the
closure.
So:
[...]
if the uses parma is 'scope':
void uses(scope void delegate() dg);
will it be not alloc memory?
I test it , if use scope it will not alloc memony.
import std.stdio;
void call(void delegate() fun)
{
fun();
}
void call2(scope void delegate() fun)
{
fun();
}
void main()
{
int a = 10;
// call((){writeln(a);});
call2((){writeln(a);});
}
dmd -vgc ./t.d
it will not print anything.
if use call:
void main()
{
int a = 10;
call((){writeln(a);});
// call2((){writeln(a);});
}
dmd -vgc ./t.d
182ms 2017年06月19日 星期一 17时16分47秒
./t.d(13): vgc: using closure causes GC allocation
Re: Garbage collection and closures.
On Monday, 19 June 2017 at 09:18:56 UTC, Dsby wrote: void uses(scope void delegate() dg); will it be not alloc memory? I test it , if use scope it will not alloc memony. Right, using `scope` at the point the delegate variable is defined means it will never allocate.
Re: On D's garbage collection
On Tuesday, 8 October 2019 at 16:28:51 UTC, Marcel wrote: I'm been thinking about using D in conjunction with C11 to develop a set of applications with hard real-time requirements. While initially the goal was to use C++ instead, it has become clear that D's introspection facilities will offer significant advantages. Unfortunately, the project will heavily rely on custom memory allocators written in C, so the presence of garbage collection in the language is a problem. While I'm aware that the nogc attribute exists, I haven't actually seen a way to apply it to a whole project. Is this possible? I think you may find this interesting: https://www.auburnsounds.com/blog/2016-11-10_Running-D-without-its-runtime.html
Re: On D's garbage collection
On Tuesday, 8 October 2019 at 16:43:23 UTC, Ferhat Kurtulmuş wrote: On Tuesday, 8 October 2019 at 16:28:51 UTC, Marcel wrote: I'm been thinking about using D in conjunction with C11 to develop a set of applications with hard real-time requirements. While initially the goal was to use C++ instead, it has become clear that D's introspection facilities will offer significant advantages. Unfortunately, the project will heavily rely on custom memory allocators written in C, so the presence of garbage collection in the language is a problem. While I'm aware that the nogc attribute exists, I haven't actually seen a way to apply it to a whole project. Is this possible? I think you may find this interesting: https://www.auburnsounds.com/blog/2016-11-10_Running-D-without-its-runtime.html And the code which is actually working: https://github.com/AuburnSounds/Dplug/blob/master/core/dplug/core/nogc.d
Re: On D's garbage collection
On Tuesday, 8 October 2019 at 16:28:51 UTC, Marcel wrote: I'm been thinking about using D in conjunction with C11 to develop a set of applications with hard real-time requirements. While initially the goal was to use C++ instead, it has become clear that D's introspection facilities will offer significant advantages. Unfortunately, the project will heavily rely on custom memory allocators written in C, so the presence of garbage collection in the language is a problem. While I'm aware that the nogc attribute exists, I haven't actually seen a way to apply it to a whole project. Is this possible? Do you want to write D code that just doesn't use the GC or the whole runtime? If the former then use @nogc at the entry point of your D code (This means that - say - main cannot call anything non-@nogc and therefore guarantees the program is @nogc), if the latter then use -betterC IMO, if the interface to your memory allocators is stable then just link with them and write the whole thing in D (Interfacing with C is a solved problem but C is just awful compared to the features you get for free in D)
Re: On D's garbage collection
On Tuesday, 8 October 2019 at 16:48:55 UTC, Ferhat Kurtulmuş wrote: On Tuesday, 8 October 2019 at 16:43:23 UTC, Ferhat Kurtulmuş wrote: On Tuesday, 8 October 2019 at 16:28:51 UTC, Marcel wrote: [...] I think you may find this interesting: https://www.auburnsounds.com/blog/2016-11-10_Running-D-without-its-runtime.html And the code which is actually working: https://github.com/AuburnSounds/Dplug/blob/master/core/dplug/core/nogc.d Wonderful, i'll be looking into it. Thank you!
Freeing of memory (garbage collection)
A couple of questions: 1: Even though D has an automatic garbage collector, is one still allowed to free the memory of a malloced array manually (using free () ), to avoid pauses in the program? 2: One justification on the website for using automatic garbage collection is how "allocated memory will only be freed if system RAM is tight". But surely that's silly, since one may want *lots* of memory free for a completely different application (merely memory being 'tight' before freeing may not be good enough - one may want D to free memory sooner).
Re: Dynamic Array Garbage collection
wolftousen wrote:
> I have a function defined as:
>
> some_function(int[] array) { ... }; //this function does not ever modify
> values of array
>
> When I call this function (once every program cycle) from an object using an
> array of type short:
>
> //member variable in an object
> short[] x = new short[4];
>
> //object calls this
> some_function([ x[0], x[1], x[2], x[3] ]);
>
> After a few seconds of the program running, x[0] and x[1] values are set to 0
> and a couple values of another array in the object are modified as well.
>
> I have found that this bug is corrected by calling delete at the end of
> some_function of the variable array. This tells me that the garbage
> collector is acting funny if I do not call delete on the array. (I have
> watched the memory usage of the program and it doesn't fluctuate or pause or
> anything to signal the freeing/allocating of memory)
>
> Any Idea what is going on?
This sounds like something else is going on, although what I'm not sure.
Maybe it's because I just woke up, but I can't see how that code could
compile anyway, since you can't pass a short[] to a function expecting
an int[]. Do you have a minimal, reproducible test case we can look at?
-- Daniel
Re: Dynamic Array Garbage collection
On Tue, Feb 24, 2009 at 6:45 PM, Daniel Keep wrote: > Maybe it's because I just woke up, but I can't see how that code could > compile anyway, since you can't pass a short[] to a function expecting > an int[]. You missed the array literal.
Re: Dynamic Array Garbage collection
Jarrett Billingsley wrote: > On Tue, Feb 24, 2009 at 6:45 PM, Daniel Keep > wrote: > >> Maybe it's because I just woke up, but I can't see how that code could >> compile anyway, since you can't pass a short[] to a function expecting >> an int[]. > > You missed the array literal. I saw that, but thought that it would be a short[] literal since it's usually the type of the first argument. -- Daniel
Re: Dynamic Array Garbage collection
On Tue, Feb 24, 2009 at 10:42 PM, Daniel Keep wrote: >> >> You missed the array literal. > > I saw that, but thought that it would be a short[] literal since it's > usually the type of the first argument. Odd, it works. And properly too. I give up on trying to figure out what the compiler does to figure out the type of array literals :P
Re: Dynamic Array Garbage collection
Jarrett Billingsley wrote: On Tue, Feb 24, 2009 at 10:42 PM, Daniel Keep wrote: You missed the array literal. I saw that, but thought that it would be a short[] literal since it's usually the type of the first argument. Odd, it works. And properly too. Could it be because of integer promotion rules? I give up on trying to figure out what the compiler does to figure out the type of array literals :P
Re: Dynamic Array Garbage collection
wolftousen wrote:
I have a function defined as:
some_function(int[] array) { ... }; //this function does not ever modify
values of array
When I call this function (once every program cycle) from an object using an
array of type short:
//member variable in an object
short[] x = new short[4];
//object calls this
some_function([ x[0], x[1], x[2], x[3] ]);
After a few seconds of the program running, x[0] and x[1] values are set to 0
and a couple values of another array in the object are modified as well.
I have found that this bug is corrected by calling delete at the end of
some_function of the variable array. This tells me that the garbage collector
is acting funny if I do not call delete on the array. (I have watched the
memory usage of the program and it doesn't fluctuate or pause or anything to
signal the freeing/allocating of memory)
Any Idea what is going on?
Can you use obj2asm and post the assembly of both functions? I'm
guessing DMD is doing some kind of weird optimization with the array
literal allocation.
Re: Dynamic Array Garbage collection
wolftousen wrote: > some_function([ x[0], x[1], x[2], x[3] ]); That may also be written: some_function(x[0 .. 4]); Bye, bearophile
Re: Dynamic Array Garbage collection
On Thu, Mar 12, 2009 at 8:09 AM, bearophile wrote: > wolftousen wrote: >> some_function([ x[0], x[1], x[2], x[3] ]); > > That may also be written: > some_function(x[0 .. 4]); Wrong types.
Re: Dynamic Array Garbage collection
Jarrett Billingsley:
> Wrong types.
Right, I am sorry, thank you for spotting it.
There's the moderate need for a different kind of cast, for collections
(arrays).
At the moment you can do:
import d.func: map, putr, array, xcast;
import std.conv: toInt;
import d.templates: ArrayType1;
void some_function(int[] x) {
putr(typeid(typeof(x)), " ", x);
}
void main() {
short[] x = [cast(short)10_000, 5_000, 100, 6_000, 50, 950];
some_function(map((ArrayType1!(typeof(x)) s){return cast(int)s;}, x[0 ..
4]));
// or less DRY:
some_function(map((short s){return cast(int)s;}, x[0 .. 4]));
// better:
some_function(array(xcast!(int)(x[0 .. 4])));
}
The last version with array and xcast may be acceptable. The advantage of using
it is that the ranges 0..4 can be variables.
Built-in types aren't constructors-functions, so you can't do something like
this:
some_function(map(&int, x[0 .. 4]));
Bye,
bearophile
Re: Garbage collection in D
Diwaker Gupta wrote: I've just started to play around with D, and I'm hoping someone can clarify this. I wrote a very simple program that just allocates lots of objects, in order to benchmark the garbage collector in D. For comparison, I wrote the programs in C++, Java and D: C++: http://gist.github.com/122708 Java: http://gist.github.com/122709 D: http://gist.github.com/121790 With an iteration count of , I get the following numbers: JAVA: 0:01.60 elapsed, 1.25 user, 0.28 system C++: 0:04.99 elapsed, 4.97 user, 0.00 system D: 0:25.28 elapsed, 25.22 user, 0.00 system As you can see, D is abysmally slow compared to C++ and Java. This is using the GNU gdc compiler. I'm hoping the community can give me some insight on what is going on. Thanks, Diwaker Can someone try dmd and ldc as gdc is dying (if not already dead). Also code in the iterations and take out the command line reading and printing so to make it accurate.
Re: Garbage collection in D
On Tue, Jun 2, 2009 at 8:40 PM, Diwaker Gupta wrote: > I've just started to play around with D, and I'm hoping someone can clarify > this. I wrote a very simple program that just allocates lots of objects, in > order to benchmark the garbage collector in D. For comparison, I wrote the > programs in C++, Java and D: > C++: http://gist.github.com/122708 > Java: http://gist.github.com/122709 > D: http://gist.github.com/121790 > > With an iteration count of , I get the following numbers: > JAVA: > 0:01.60 elapsed, 1.25 user, 0.28 system > C++: > 0:04.99 elapsed, 4.97 user, 0.00 system > D: > 0:25.28 elapsed, 25.22 user, 0.00 system > > As you can see, D is abysmally slow compared to C++ and Java. This is using > the GNU gdc compiler. I'm hoping the community can give me some insight on > what is going on. D's GC is not nearly as well-developed as that of Java's, and its performance is not that stellar. Sorry, but you are not the first to discover this by any stretch of the imagination. (On a side note, I have a feeling you and bearophile will get on famously.) Also, benchmarking a GC against manual memory management doesn't do much for you. It's apples and oranges. Though it is funny to see how badly Java beats C++ there.
Re: Garbage collection in D
Hello Jarrett, On Tue, Jun 2, 2009 at 8:40 PM, Diwaker Gupta wrote: I've just started to play around with D, and I'm hoping someone can clarify this. I wrote a very simple program that just allocates lots of objects, in order to benchmark the garbage collector in D. For comparison, I wrote the programs in C++, Java and D: C++: http://gist.github.com/122708 Java: http://gist.github.com/122709 D: http://gist.github.com/121790 With an iteration count of , I get the following numbers: JAVA: 0:01.60 elapsed, 1.25 user, 0.28 system C++: 0:04.99 elapsed, 4.97 user, 0.00 system D: 0:25.28 elapsed, 25.22 user, 0.00 system As you can see, D is abysmally slow compared to C++ and Java. This is using the GNU gdc compiler. I'm hoping the community can give me some insight on what is going on. D's GC is not nearly as well-developed as that of Java's, and its performance is not that stellar. Sorry, but you are not the first to discover this by any stretch of the imagination. (On a side note, I have a feeling you and bearophile will get on famously.) Also, benchmarking a GC against manual memory management doesn't do much for you. It's apples and oranges. Though it is funny to see how badly Java beats C++ there. Java may be able to tell that the allocation never needs to be kept and is just reusing the same space on the stack. Heck it might even not be doing that as you can tell that the class only ever holds the same value as i so it might just be skipping the new all together.
Re: Garbage collection in D
Diwaker Gupta wrote: I've just started to play around with D, and I'm hoping someone can clarify this. I wrote a very simple program that just allocates lots of objects, in order to benchmark the garbage collector in D. For comparison, I wrote the programs in C++, Java and D: C++: http://gist.github.com/122708 Java: http://gist.github.com/122709 D: http://gist.github.com/121790 With an iteration count of , I get the following numbers: JAVA: 0:01.60 elapsed, 1.25 user, 0.28 system C++: 0:04.99 elapsed, 4.97 user, 0.00 system D: 0:25.28 elapsed, 25.22 user, 0.00 system As you can see, D is abysmally slow compared to C++ and Java. This is using the GNU gdc compiler. I'm hoping the community can give me some insight on what is going on. Thanks, Diwaker After porting the D version to tango: D: 6.282s (ldmd -O5 -inline -release -L-s -singleobj gctest.d) C++: 4.435s (g++ -O5 gctest.d) This is on a C2D 2.2Ghz, 2GB RAM, Linux x86-64. I don't have java installed, so can't test that. Maybe if you're planning to use the GC a lot you should consider using tango?
Re: Garbage collection in D
Le Wed, 03 Jun 2009 02:40:11 +0200, Diwaker Gupta a écrit: I've just started to play around with D, and I'm hoping someone can clarify this. I wrote a very simple program that just allocates lots of objects, in order to benchmark the garbage collector in D. For comparison, I wrote the programs in C++, Java and D: C++: http://gist.github.com/122708 Java: http://gist.github.com/122709 D: http://gist.github.com/121790 With an iteration count of , I get the following numbers: JAVA: 0:01.60 elapsed, 1.25 user, 0.28 system C++: 0:04.99 elapsed, 4.97 user, 0.00 system D: 0:25.28 elapsed, 25.22 user, 0.00 system I think the line 14 in the D source is useless. On my linux system : D with line 14 removed : -- ts...@fgabriel:~/dev/DBenchmark$ time ./Benchmark allocations 787459713 real0m28.779s user0m28.778s sys 0m0.004s C++: --- ts...@fgabriel:~/dev/DBenchmark$ time ./a.out allocations Ran allocations of RunAllocations. Final value: 787459713 real0m16.406s user0m16.405s sys 0m0.004s Java : - ts...@fgabriel:~/dev/DBenchmark$ time java Benchmark allocations Ran allocations of RunAllocations. Final value: 787459713 real0m6.679s user0m6.408s sys 0m0.248s But with the use of "scope" keyword at line 13 of the D source : ts...@fgabriel:~/dev/DBenchmark$ time ./Benchmark allocations 787459713 real0m10.752s user0m10.753s sys 0m0.000s
Re: Garbage collection in D
Robert Clipsham wrote: After porting the D version to tango: D: 6.282s (ldmd -O5 -inline -release -L-s -singleobj gctest.d) C++: 4.435s (g++ -O5 gctest.d) This is on a C2D 2.2Ghz, 2GB RAM, Linux x86-64. I don't have java installed, so can't test that. Maybe if you're planning to use the GC a lot you should consider using tango? After reading TSalm's post, I reran the D version with the scope keyword at line 16: D (with scope): 1.098s D: 6.282s C++: 4.435s It seems by using scope and tango you can easily compete with C++.
Re: Garbage collection in D
Jarrett Billingsley: > (On a side note, I have a feeling you and bearophile will get on famously.)< I have found a new friend ;-) Some timings, usual settings, Core2 2 GHz: Timings, N=100_000_000, Windows, seconds: D 1: 40.20 DMD D 2: 21.83 DMD D 2: 18.80 DMD, struct + scope C++: 18.06 D 1: 8.47 DMD D 2: 7.41 DMD + scope Java 1.78 -server Java: 1.44 Timings, N=100_000_000, Pubuntu, seconds: D 1: 25.7 LDC C++: 6.87 D 1: 2.67 LDC + scope Java: 1.49 Poor LDC :-) Bye, bearophile
Re: Garbage collection in D
I have tried the new JavaVM on Win, that optionally performs escape analysis, and the results are nice: Timings, N=100_000_000, Windows, seconds: D 1: 40.20 DMD D 2: 21.83 DMD D 2: 18.80 DMD, struct + scope C++: 18.06 D 1: 8.47 DMD D 2: 7.41 DMD + scope Java: 1.84 V.1.6.0_14, -XX:+UnlockExperimentalVMOptions -XX:+UseG1GC Java 1.78 -server Java: 1.44 Java: 1.38 V.1.6.0_14 Java: 0.28 V.1.6.0_14, -server -XX:+DoEscapeAnalysis Timings, N=100_000_000, Pubuntu, seconds: D 1: 25.7 LDC C++: 6.87 D 1: 2.67 LDC + scope Java: 1.49 Bye, bearophile
Re: Garbage collection in D
Robert Clipsham wrote: > After reading TSalm's post, I reran the D version with the scope keyword > at line 16: > > D (with scope): 1.098s > D: 6.282s > C++: 4.435s > > It seems by using scope and tango you can easily compete with C++. 'scope' eliminates dynamic memory allocation. At this point you're not measuring the speed of the garbage collector at all. For a fair comparison, you should also eliminate the useless dynamic memory allocation from the C++ version. -- Rainer Deyke - rain...@eldwood.com
Re: Garbage collection in D
What's the difference between: D 1: 40.20 DMD D 2: 21.83 DMD D 2: 18.80 DMD, struct + scope and: D 1: 8.47 DMD D 2: 7.41 DMD + scope ...?
Re: Garbage collection in D
Robert Fraser: > What's the difference between: > > D 1: 40.20 DMD > > D 2: 21.83 DMD That's the standard code. > and: > > D 1: 8.47 DMD > > D 2: 7.41 DMD + scope They are both with scope, on D1 and D2. Sorry for my small omission. Bye, bearophile
Re: Garbage collection in D
bearophile Wrote: > I have tried the new JavaVM on Win, that optionally performs escape analysis, > and the results are nice: > > Timings, N=100_000_000, Windows, seconds: > D 1: 40.20 DMD > D 2: 21.83 DMD > D 2: 18.80 DMD, struct + scope > C++: 18.06 > D 1: 8.47 DMD > D 2: 7.41 DMD + scope > Java: 1.84 V.1.6.0_14, -XX:+UnlockExperimentalVMOptions -XX:+UseG1GC > Java 1.78 -server > Java: 1.44 > Java: 1.38 V.1.6.0_14 > Java: 0.28 V.1.6.0_14, -server -XX:+DoEscapeAnalysis > > Timings, N=100_000_000, Pubuntu, seconds: > D 1: 25.7 LDC > C++: 6.87 > D 1: 2.67 LDC + scope > Java: 1.49 > > Bye, > bearophile Sorry for my stepping in... What does this result mean?Does it mean D is slower than Java and C++ is also slower than Java?Or that's true just under certain circumstance? I am really confused and really appreicate if any further explanation. Regards, Sam
Re: Garbage collection in D
Sam Hu wrote: bearophile Wrote: I have tried the new JavaVM on Win, that optionally performs escape analysis, and the results are nice: Timings, N=100_000_000, Windows, seconds: D 1: 40.20 DMD D 2: 21.83 DMD D 2: 18.80 DMD, struct + scope C++: 18.06 D 1: 8.47 DMD D 2: 7.41 DMD + scope Java: 1.84 V.1.6.0_14, -XX:+UnlockExperimentalVMOptions -XX:+UseG1GC Java 1.78 -server Java: 1.44 Java: 1.38 V.1.6.0_14 Java: 0.28 V.1.6.0_14, -server -XX:+DoEscapeAnalysis Timings, N=100_000_000, Pubuntu, seconds: D 1: 25.7 LDC C++: 6.87 D 1: 2.67 LDC + scope Java: 1.49 Bye, bearophile Sorry for my stepping in... What does this result mean?Does it mean D is slower than Java and C++ is also slower than Java?Or that's true just under certain circumstance? I am really confused and really appreicate if any further explanation. Regards, Sam It suggests that for dynamic allocation of many small objects via "new", Java is an order of magnitude faster than C++, which in turn is slightly faster than D.
Re: Garbage collection in D
Sam Hu wrote: > What does this result mean?Does it mean D is slower than Java and C++ > is also slower than Java?Or that's true just under certain > circumstance? I am really confused and really appreicate if any > further explanation. These are the timings for using dynamic memory allocation: >> D 1: 40.20 DMD >> D 2: 21.83 DMD >> C++: 18.06 >> Java: 1.38 V.1.6.0_14 Java is the fastest by a large margin because it has the benefit of a moving garbage collector. This means allocation is a simple pointer bump and deallocation is completely free. The slow aspects of this garbage collector (detection and preservation) aren't really tested by this benchmark. These are the timings without dynamic memory allocation: >> D 1: 8.47 DMD [+ scope] >> D 2: 7.41 DMD + scope >> Java: 0.28 V.1.6.0_14, -server -XX:+DoEscapeAnalysis D's performance is unexpectedly bad, so much that I expect that it might be using dynamic memory allocation anyway despite the 'scope' keyword. Java is clever in that it eliminates unnecessary dynamic memory allocations automatically. C++ is notable absent, but I fully expect it to outperform Java by a significant margin. -- Rainer Deyke - rain...@eldwood.com
Re: Garbage collection in D
> D's performance is unexpectedly bad, so much that I expect that it might > be using dynamic memory allocation anyway despite the 'scope' keyword. I am sorry to hear that,really,really sorry.
Re: Garbage collection in D
Rainer Deyke: >The slow aspects of this garbage collector (detection and preservation) aren't >really tested by this benchmark.< In practice most times real-world Java programs show a good enough performance even taking in account detection and preservation too. >These are the timings without dynamic memory allocation: >>> D 1: 8.47 DMD [+ scope] >>> D 2: 7.41 DMD + scope >>> Java: 0.28 V.1.6.0_14, -server -XX:+DoEscapeAnalysis It's not exactly the same, because in that Java code I have used a program-wide optimization flag (that I guess will become default), while in D I have had to add a "scope" everywhere, and I think adding "scope" is less safe than letting the compiler perform an escape analysis. So I am tempted to put the 0.28 seconds result among the dynamic allocation timings, even if technically it is not, because for the programmer the program "feels" and looks and acts like a dynamic allocation, it's just faster :-) In the end what counts is how well the programs runs after the compiler has done its work. >D's performance is unexpectedly bad, so much that I expect that it might be >using dynamic memory allocation anyway despite the 'scope' keyword. Java is >clever in that it eliminates unnecessary dynamic memory allocations >automatically.< I think Java here is doing a bit more than just removing the dynamic allocation. I don't think D (compiled with LDC) is doing doing any allocation here. I'll ask to the LDC IRC channel. I'll also take a look at the asm generated by the JavaVM (it's not handy to find the asm generated by the JVM, you need to install a debug version of it... how stupid). - Sam Hu: >I am sorry to hear that,really,really sorry.< Wait, things may not be that bad. And even if they are bad, the developers of the LDC compiler may find ways to improve the situation. - Robert Fraser: >It suggests that for dynamic allocation of many small objects via "new", Java >is an order of magnitude faster than C++, which in turn is slightly faster >than D.< Yes, for such tiny benchmarks I have seen several times 10-12 higher allocation performance in Java compared to D1-DMD. But real programs don't use all their time allocating and freeing memory... Bye, bearophile
Re: Garbage collection in D
bearophile wrote: Rainer Deyke: D's performance is unexpectedly bad, so much that I expect that it might be using dynamic memory allocation anyway despite the 'scope' keyword. Java is clever in that it eliminates unnecessary dynamic memory allocations automatically.< I think Java here is doing a bit more than just removing the dynamic allocation. I don't think D (compiled with LDC) is doing doing any allocation here. I'll ask to the LDC IRC channel. LDC actually still does a dynamic allocation there because it doesn't eliminate dynamic allocations in loops. This is unfortunate, but I haven't yet had the time to figure out how to get the optimization passes to prove the allocation can't be live when reached again. (If multiple instances of memory allocated at the same allocation site may be reachable at the same time, it's not safe to use a stack allocation instead of a heap allocation) It's on my to-do list, though.
Re: Garbage collection in D
Frits van Bommel:
> LDC actually still does a dynamic allocation there because it doesn't
> eliminate
> dynamic allocations in loops.
I have compiled the loop in foo() with LDC:
class AllocationItem {
int value;
this(int v) { this.value = v; }
}
int foo(int iters) {
int sum = 0;
for (int i = 0; i < iters; ++i) {
scope auto item = new AllocationItem(i);
sum += item.value;
}
return sum;
}
The asm of the core of the loop:
.LBB2_2:
movl$_D11gc_test2b_d14AllocationItem6__vtblZ, 8(%esp)
movl$0, 12(%esp)
movl%edi, 16(%esp)
movl%ebx, (%esp)
call_d_callfinalizer
incl%edi
cmpl%esi, %edi
jne .LBB2_2
I can see a call to finalizer, but not the allocation?
> This is unfortunate, but I haven't yet had the time to figure out how to get
> the
> optimization passes to prove the allocation can't be live when reached again.
> (If multiple instances of memory allocated at the same allocation site may be
> reachable at the same time, it's not safe to use a stack allocation instead
> of a
> heap allocation)
The new JavaVM with the option I have shown is clearly able to do such things.
Can't you take a look at the source code of the JavaVM? :-)
There's a huge amount of NIH in the open source :-)
Bye,
bearophile
Re: Garbage collection in D
bearophile wrote: Frits van Bommel: LDC actually still does a dynamic allocation there because it doesn't eliminate dynamic allocations in loops. I have compiled the loop in foo() with LDC: [snip] scope auto item = new AllocationItem(i); [snip] The asm of the core of the loop: .LBB2_2: movl$_D11gc_test2b_d14AllocationItem6__vtblZ, 8(%esp) movl$0, 12(%esp) movl%edi, 16(%esp) movl%ebx, (%esp) call_d_callfinalizer incl%edi cmpl%esi, %edi jne .LBB2_2 I can see a call to finalizer, but not the allocation? Sorry, I thought we were talking about the code without 'scope'. Of course the class is indeed stack-allocated if you use scope. (The following: This is unfortunate, but I haven't yet had the time to figure out how to get the optimization passes to prove the allocation can't be live when reached again. (If multiple instances of memory allocated at the same allocation site may be reachable at the same time, it's not safe to use a stack allocation instead of a heap allocation) only applies when 'scope' was not used, and the compiler therefore initially heap-allocated it) The new JavaVM with the option I have shown is clearly able to do such things. Can't you take a look at the source code of the JavaVM? :-) There's a huge amount of NIH in the open source :-) I suspect the Java VM uses a different internal representation of the code than LLVM does...
Re: Garbage collection in D
bearophile wrote: Yes, for such tiny benchmarks I have seen several times 10-12 higher allocation performance in Java compared to D1-DMD. But real programs don't use all their time allocating and freeing memory... Bye, bearophile For the compiler I'm working on now (in D), I wanted to check the affects of allocation on performance. Using a placement new, the time for lex/parse/semantic/type-infer/codegen (on a really huge in-memory file) went from ~6 seconds to ~4 seconds (don't have the exact timings, and can't repro right now since I'm redoing inference). So I'd say that even in real-world applications, these things have an effect. Of course, this only applies to programs which allocate and throw away a lot of small objects. This is a style encouraged by Java and C#'s programming models, much less so by, say, C++'s.
Re: Garbage collection in D
Frits van Bommel wrote: The new JavaVM with the option I have shown is clearly able to do such things. Can't you take a look at the source code of the JavaVM? :-) There's a huge amount of NIH in the open source :-) I suspect the Java VM uses a different internal representation of the code than LLVM does... HotSpot uses 3-argument SSA for IR, AFAIK... I think LLVM is also SSA-based, right? But the Java source is _quite_ complex.
Re: Garbage collection in D
Robert Fraser: > Of course, this only applies to programs which allocate and throw away a > lot of small objects. This is a style encouraged by Java and C#'s > programming models, much less so by, say, C++'s. Right, there are many potential new D programmers coming from Java that may want to use that style that relies a lot on an efficient GC. But you are missing another important style of programming that allocates & frees tons of objects: functional-style programming, where immutable data is the norm. If the D2 language will want to appeal to functional programmers it will have to manage such immutables more efficiently. Bye, bearophile
Re: Garbage collection in D
LDC is a moving target because it's actively developed, and generally things
improve with time.
This is a recent change by the quite active Frits van Bommel:
http://www.dsource.org/projects/ldc/changeset/1486%3A9ed0695cb93c
This is a cleaned up version discussed in this thread:
import tango.stdc.stdio: printf;
import Integer = tango.text.convert.Integer;
class AllocationItem {
int value;
this(int v) { this.value = v; }
}
int foo(int iters) {
int sum = 0;
for (int i = 0; i < iters; ++i) {
auto item = new AllocationItem(i);
sum += item.value;
}
return sum;
}
void main(char[][] args) {
int iters = Integer.parse(args[1]);
printf("%d\n", foo(iters));
}
The asm generated by the last LDC (based on DMD v1.045 and llvm 2.6svn (Tue Jun
9 22:34:25 2009)) (this is just the important part of the asm):
foo:
testl %eax, %eax
jle .LBB2_4
movl%eax, %ecx
xorl%eax, %eax
.align 16
.LBB2_2:
incl%eax
cmpl%ecx, %eax
jne .LBB2_2
leal-2(%ecx), %eax
leal-1(%ecx), %edx
mull%edx
shldl $31, %eax, %edx
leal-1(%edx,%ecx), %eax
ret
.LBB2_4:
xorl%eax, %eax
ret
*/
This is the same code with "scope" added:
import tango.stdc.stdio: printf;
import Integer = tango.text.convert.Integer;
class AllocationItem {
int value;
this(int v) { this.value = v; }
}
int foo(int iters) {
int sum = 0;
for (int i = 0; i < iters; ++i) {
scope auto item = new AllocationItem(i);
sum += item.value;
}
return sum;
}
void main(char[][] args) {
int iters = Integer.parse(args[1]);
printf("%d\n", foo(iters));
}
Its asm:
/*
foo:
pushl %ebx
pushl %edi
pushl %esi
subl$24, %esp
testl %eax, %eax
jle .LBB2_4
movl%eax, %esi
xorl%edi, %edi
leal8(%esp), %ebx
.align 16
.LBB2_2:
movl$_D11gc_test2b_d14AllocationItem6__vtblZ, 8(%esp)
movl$0, 12(%esp)
movl%edi, 16(%esp)
movl%ebx, (%esp)
call_d_callfinalizer
incl%edi
cmpl%esi, %edi
jne .LBB2_2
leal-2(%esi), %eax
leal-1(%esi), %ecx
mull%ecx
shldl $31, %eax, %edx
leal-1(%edx,%esi), %eax
jmp .LBB2_5
.LBB2_4:
xorl%eax, %eax
.LBB2_5:
addl$24, %esp
popl%esi
popl%edi
popl%ebx
ret
*/
The running time:
...$ elaps ./gc_test1 25000
-1782069568
real0m0.170s
user0m0.160s
sys 0m0.010s
The version with "scope":
...$ elaps ./gc_test2 25000
-1782069568
real0m6.430s
user0m6.430s
sys 0m0.000s
(Later I may try again with a less simple and more realistic benchmark, because
this is too much a toy to be interesting.)
Bye,
bearophile
Re: Garbage collection in D
Diwaker Gupta Wrote: > I've just started to play around with D, and I'm hoping someone can clarify > this. I wrote a very simple program that just allocates lots of objects, in > order to benchmark the garbage collector in D. For comparison, I wrote the > programs in C++, Java and D: > C++: http://gist.github.com/122708 > Java: http://gist.github.com/122709 > D: http://gist.github.com/121790 > > With an iteration count of , I get the following numbers: > JAVA: > 0:01.60 elapsed, 1.25 user, 0.28 system > C++: > 0:04.99 elapsed, 4.97 user, 0.00 system > D: > 0:25.28 elapsed, 25.22 user, 0.00 system > > As you can see, D is abysmally slow compared to C++ and Java. This is using > the GNU gdc compiler. I'm hoping the community can give me some insight on > what is going on. > > Thanks, > Diwaker Hi. Inspired by this idea I changed somehow rules to make it more complicated task. So they are: 1. every "AllocationItem" has references to three other items. 2. generate "n_items" in static array "items" of type "AllocationItem" with "value" field set to "0" 3. make random connections between all this items by their reference fields 4. iterate "n_iters" times with the following agorithm a) create new "AllocationArray" with "value" set to "1" b) replace random item from the array "items" with this new item c) add connections to this item d) remove (variant 1) or change (variant 2) three random connections e) now if some object isn't referenced by others it should be removed (GC collected). 5. calculate count of old items (with "value" set to 0) and new ones Here are my programs in D and Java. There is no C++ variant, sorry. In D I used modified for D2 Bill Baxter's weak reference module: http://www.dsource.org/projects/scrapple/browser/trunk/weakref now results: 1) it works strange, as from time to time it gives different results in D2 (approx. 1 in 10) and it's no by RNG. 2) java's version is awfully slow (may be because it's my second java app, first was 8 jears ago). now I hate java even more. 3) java and D give different results. It's all strange. Maybe I made something wrong.
Re: Garbage collection in D
Oh my. I forgot programs, here they are: http://gist.github.com/154958
pointers, assignments, Garbage Collection Oh My?
I hope you like the subject matter and I hope it is not too
simplistic or have been answered before.
Anyway I have a question about how the garbage collector works
in a very specific situation. When passing string type to a
function in a shared library or DLL and assigning it to a
variable of type string inside the function and returning the
internal string. Such as this.
export string mytest(string tstStr)
{
string st = tstStr;
/* abbreviated to protect the innocent but other operations
such as concatenating and deleting may be done to st before
the return
*/
return st;
}
Is the string type a pointer or is it something else? In the
line where tstStr is assigned to st does it copy the address in
tstStr to st or does it copy the value in tstStr? I am just a
bit confused about string types since I come from a C background
and C has no type like this. Also what is returned by this
function? Does this function return a pointer or the contents of
an array? If I do export this what does it do to the Garbage
Collection? Does the Garbage Collection collect tstStr or st?
Also notice the comment in the function.
Temporarily protect array from garbage collection
Is it possible to temporarily prevent the garbage collector from
collecting a memory block even if there are no references to it?
The use case is as follows: I want to call a C library function
which expects to take ownership of a buffer. It looks something
like this:
alias FreeFunc = extern(C) void function(void*, void*)
nothrow;
extern(C) void foo(void* buf, size_t len,
FreeFunc free, void* ctx) nothrow;
Here, 'buf' is a pointer to the buffer, 'len' is the length of
the buffer, 'free' is a function to deallocate the buffer when
the library is done with it, and 'ctx' is a user-supplied context
pointer. Upon deallocation, 'free' receives two parameters; the
pointer to the buffer and the context pointer. The latter can be
anything, even null, as it is just passed to 'free' and not used
for anything else.
Here is the problem: I want to be able to use a
garbage-collected dynamic array with this function, but I don't
want to have to retain a reference to it in my program. (I don't
know when the C library will call the free function.) In other
words, I want something like this:
extern(C) void myFree(void* ptr, void* ctx)
{
enableGCFor(ptr);
}
auto arr = new int[123];
disableGCFor(arr);
foo(arr.ptr, arr.length, &myFree, null);
arr = null;
Is this at all possible?
Thanks,
Lars
reference to delegates and garbage collection
Hello everybody. I am new to D, and I am trying to familiarize
with all the new (to me) features.
I have a question about a strange behavior I cannot understand.
(In case it matters, I am using gdc 2.065 for Windows, but the
same happens with dmd).
Example code :
class A {
void delegate() del;
~this() { writeln("A Destructor called"); }
}
class B {
void fun() {}
}
A makeA() {
auto a = new A();
a.del = &(new B().fun);
return a;
}
void main() {
auto a = makeA();
/* Magic line goes here, see below */
a = null;
foreach(i; 0..10) {
writeln("Collecting");
GC.collect();
}
writeln("End");
}
If I run this code, the instance of A is of course collected at
the first call to GC.collect().
But if I call the delegate (i.e. a.del() ) in the "magic line",
the instance is not destroyed until the end of the main. If I
"wrap" the call to a.del() in another function or method, the
instance is collected.
Can somebody explain why this happens? Is it just a strangeness
of the GC? Does calling the delegate keep a reference to a in the
current scope? Am I going crazy? I spent about half an hour
looking for a non-existent memory leak in my program because of
this behavior, so I'd like to have an explanation.
Thanks in advance!
Re: Freeing of memory (garbage collection)
Reply to Dan, A couple of questions: 1: Even though D has an automatic garbage collector, is one still allowed to free the memory of a malloced array manually (using free () ), to avoid pauses in the program? Yes, in fact if you malloc memory you MUST free it. Only stuff like dynamic arrays, AAs and new'ed stuff gets cleaned up by the GC. 2: One justification on the website for using automatic garbage collection is how "allocated memory will only be freed if system RAM is tight". But surely that's silly, since one may want *lots* of memory free for a completely different application (merely memory being 'tight' before freeing may not be good enough - one may want D to free memory sooner). With virtual memory and swap space, this (almost) no longer matters. If you aren't using memory sooner or later it's not using your RAM but just sitting on the disk.
Re: Freeing of memory (garbage collection)
Dan W: > 1: Even though D has an automatic garbage collector, is one still > allowed to free the memory of a malloced array manually (using free > () ), to avoid pauses in the program? Other people here will just answer your question. But remember that in D manual memory management is done only in special situations, most code of most programs just use the GC. Bye, bearophile
Re: Freeing of memory (garbage collection)
Thanks for that reply. I wonder if extending automatic garbage collection for malloced memory would be a good idea... > Only stuff like dynamic > arrays, AAs and new'ed stuff gets cleaned up by the GC. For the above types of allocating memory, is there a way to 'lock' a variable and say to D, "don't free this memory until I allow you to", and also for it to allow you to free it manually when/if need be?
Re: Freeing of memory (garbage collection)
"Daniel White" wrote
> Thanks for that reply. I wonder if extending automatic garbage
> collection for malloced memory would be a good idea...
>
>> Only stuff like dynamic
>> arrays, AAs and new'ed stuff gets cleaned up by the GC.
>
> For the above types of allocating memory, is there a way to 'lock' a
> variable and say to D, "don't free this memory until I allow you to",
> and also for it to allow you to free it manually when/if need be?
One thing you can do, that nobody has mentioned yet, is delete memory that
you have allocated using the GC.
Deleting a block of memory that the GC has allocated allows you to reuse
that memory without going through a full GC collect cycle, and so most
likely your performance will be better. Using delete in key spots can be
good.
But using manual memory mangement prevents lots of good designs, so use with
caution.
Lastly, if you are allocating lots of temporary classes, try declaring them
as scope. This allocates the class on the stack if possible. Note that any
memory allocated by the constructor will still go through the GC. Also note
that you can't use a scoped class once its scope is finished. e.g.:
class C
{ ... }
...
while(x > 0)
{
scope c = new C(x--); // c is allocated on the stack. scope keyword
implies the type.
c.doSomething();
// c is deleted from stack at end of loop iteration. No heap activity.
}
-Steve
Re: Freeing of memory (garbage collection)
On Tue, Dec 9, 2008 at 9:16 AM, Daniel White <[EMAIL PROTECTED]> wrote: > Thanks for that reply. I wonder if extending automatic garbage > collection for malloced memory would be a good idea... That would be a bad idea. Then how would you do manual memory management in the few cases that absolutely require it? >> Only stuff like dynamic >> arrays, AAs and new'ed stuff gets cleaned up by the GC. > > For the above types of allocating memory, is there a way to 'lock' a > variable and say to D, "don't free this memory until I allow you to", > and also for it to allow you to free it manually when/if need be? Yes, you can use std.gc.addRoot in Phobos (in D1), and GC.addRoot if you're using Tango (import tango.core.Memory) or D2 (import core.memory).
Re: Freeing of memory (garbage collection)
On Tue, Dec 9, 2008 at 9:42 AM, Jarrett Billingsley <[EMAIL PROTECTED]> wrote: > On Tue, Dec 9, 2008 at 9:16 AM, Daniel White <[EMAIL PROTECTED]> wrote: >> Thanks for that reply. I wonder if extending automatic garbage >> collection for malloced memory would be a good idea... > > That would be a bad idea. Then how would you do manual memory > management in the few cases that absolutely require it? > >>> Only stuff like dynamic >>> arrays, AAs and new'ed stuff gets cleaned up by the GC. >> >> For the above types of allocating memory, is there a way to 'lock' a >> variable and say to D, "don't free this memory until I allow you to", >> and also for it to allow you to free it manually when/if need be? > > Yes, you can use std.gc.addRoot in Phobos (in D1), and GC.addRoot if > you're using Tango (import tango.core.Memory) or D2 (import > core.memory). Oh yeah, and you can use 'delete' on arrays, classes, and structs allocated on the heap, but oddly not with AAs.
Re: Freeing of memory (garbage collection)
> That would be a bad idea. Then how would you do manual memory > management in the few cases that absolutely require it? Two ways. Either: a: being able to lock the variable so that the garbage collector can't touch it until you unlock it. b: Using a slightly different version of malloc (say 'mallocl') which again, makes it shielded against the garbage collector's wrath.
Re: Freeing of memory (garbage collection)
On Tue, Dec 9, 2008 at 11:08 AM, Daniel White <[EMAIL PROTECTED]> wrote: >> That would be a bad idea. Then how would you do manual memory >> management in the few cases that absolutely require it? > > Two ways. Either: > > a: being able to lock the variable so that the garbage collector > can't touch it until you unlock it. Why not just use GC-allocated memory and use addRoot then? > b: Using a slightly different version of malloc (say 'mallocl') which again, > makes it shielded against the garbage collector's wrath. At which point you're back to the same situation as we have currently.
Re: Freeing of memory (garbage collection)
Jarrett Billingsley Wrote: > On Tue, Dec 9, 2008 at 11:08 AM, Daniel White <[EMAIL PROTECTED]> wrote: > >> That would be a bad idea. Then how would you do manual memory > >> management in the few cases that absolutely require it? > > > > Two ways. Either: > > > > a: being able to lock the variable so that the garbage collector > > can't touch it until you unlock it. > > Why not just use GC-allocated memory and use addRoot then? Well that question points at the deeper issue of why bother having malloc at all.
Re: Freeing of memory (garbage collection)
On Tue, Dec 9, 2008 at 2:53 PM, Daniel White <[EMAIL PROTECTED]> wrote: > Jarrett Billingsley Wrote: > >> On Tue, Dec 9, 2008 at 11:08 AM, Daniel White <[EMAIL PROTECTED]> wrote: >> >> That would be a bad idea. Then how would you do manual memory >> >> management in the few cases that absolutely require it? >> > >> > Two ways. Either: >> > >> > a: being able to lock the variable so that the garbage collector >> > can't touch it until you unlock it. >> >> Why not just use GC-allocated memory and use addRoot then? > > Well that question points at the deeper issue of why bother having malloc at > all. For interaction with some C libraries. Or for implementing your own memory management without having to deal with the GC.
Re: Freeing of memory (garbage collection)
Daniel White wrote: That would be a bad idea. Then how would you do manual memory management in the few cases that absolutely require it? Two ways. Either: a: being able to lock the variable so that the garbage collector can't touch it until you unlock it. If you have a reference to the memory, the GC won't collect it. Unless you make a habit of hiding pointers inside non-pointer types, that is, which can result in undefined behavior if the hidden pointer points to GC-allocated memory. If you don't have any reference to that memory, then the garbage collector can free it, but in that case, I don't see why it would be an issue, most of the time. If you have a destructor, just be careful -- for instance, if you have an open file in that object and its destructor closes that file. b: Using a slightly different version of malloc (say 'mallocl') which again, makes it shielded against the garbage collector's wrath. malloc is C stdlib malloc. The garbage collector won't touch it.
Re: Freeing of memory (garbage collection)
Steven Schveighoffer: > One thing you can do, that nobody has mentioned yet, is delete memory that > you have allocated using the GC. Leaving the GC manage and free your memory is usually OK. Managing manually your memory allocated from the C heap is doable. But freeing manually and forcefully memory allocated by the GC seems dangerous to me. Bye, bearophile
Re: Freeing of memory (garbage collection)
Tue, 9 Dec 2008 03:25:07 + (UTC), Dan W wrote: > 1: Even though D has an automatic garbage collector, is one still > allowed to free the memory of a malloced array manually (using free > () ), to avoid pauses in the program? Just to clarify. There are 3 types of allocation: 1. std.c.stdlib.malloc(). The good ol' plain C allocation which is entirely manual and requires std.c.stdlib.free() to free memory. 2. std.gc.malloc(), or tango.core.Memory.GC.malloc(), or core.memory.GC.malloc() in recent D2. These allocate an uninitialized but GC-managed and GC-scanned by default memory. That is, this memory gets automatically freed as soon as there are no GC-managed references to it. You can explicitly free this memory using the delete keyword. You can direct GC not to scan this memory for pointers using std.gc.hasNoPointers() or equivalent. 3. Memory allocated via new, arrays, etc. This memory is GC-managed but can be freed explicitly using the delete keyword. > 2: One justification on the website for using automatic garbage > collection is how "allocated memory will only be freed if system RAM > is tight". But surely that's silly, since one may want *lots* of > memory free for a completely different application (merely memory > being 'tight' before freeing may not be good enough - one may want D > to free memory sooner). I don't think this is a valid justification. One of the main GC justifications is an ability to implement flexible data models where data owner is uncertain or unknown. In such cases GC makes it easy to avoid data duplication.
Re: Garbage Collection, Allocators/Deallocators and
Ivo Kasiuk Wrote: > Hi, > > to improve my understanding of the GC and when/how > allocators/deallocators and constructors/destructors get called, I wrote > a little test program. And now I understand even less than before... ... > Running this with DMD 2.049, I observed the following: > > - S1(int), S2(int), C1(), C2(), S1.new and S2.new get invoked in every > cycle of the loop, as you would expect. > > - ~C2() also gets invoked frequently. > > - ~S1(), ~S2(), ~C1(), S1.delete and C1.delete never ever get called. > > - The program does not run out of memory and actually has a relatively > modest memory footprint, so it does not seem to leak memory. (When using > std.c.stdlib.malloc instead of GC.malloc it runs out of memory almost > immediately). > > It is good to see that the GC apparently really frees the unreferenced > memory again. However, I don't understand why the deallocators and the > destructors (apart from ~C2) do not get called. If the memory for the > objects gets freed, as is apparently the case, then why are there no > destructor and deallocator calls for these objects? The deallocator is only called if you delete the object, not when it's finalized by the GC. The GC will only finalize something that is in its memory space, so if this happens there's no need to call the deallocator.
Re: Garbage Collection, Allocators/Deallocators and
Am Samstag, den 18.09.2010, 10:08 -0400 schrieb Sean Kelly:
> Ivo Kasiuk Wrote:
>
> > Hi,
> >
> > to improve my understanding of the GC and when/how
> > allocators/deallocators and constructors/destructors get called, I wrote
> > a little test program. And now I understand even less than before...
> ...
> > Running this with DMD 2.049, I observed the following:
> >
> > - S1(int), S2(int), C1(), C2(), S1.new and S2.new get invoked in every
> > cycle of the loop, as you would expect.
> >
> > - ~C2() also gets invoked frequently.
> >
> > - ~S1(), ~S2(), ~C1(), S1.delete and C1.delete never ever get called.
> >
> > - The program does not run out of memory and actually has a relatively
> > modest memory footprint, so it does not seem to leak memory. (When using
> > std.c.stdlib.malloc instead of GC.malloc it runs out of memory almost
> > immediately).
> >
> > It is good to see that the GC apparently really frees the unreferenced
> > memory again. However, I don't understand why the deallocators and the
> > destructors (apart from ~C2) do not get called. If the memory for the
> > objects gets freed, as is apparently the case, then why are there no
> > destructor and deallocator calls for these objects?
>
> The deallocator is only called if you delete the object, not when it's
> finalized by the GC. The GC will only finalize something that is in its
> memory space, so if this happens there's no need to call the deallocator.
Ok, that makes sense. So the deallocators really should not get called
in this case. But why are the destructors not invoked when the GC
finalizes the objects?
Exploring the example a bit further:
If C's malloc is used instead of GC.malloc then the deallocators also
are not called and the program runs out of memory. How are the objects
supposed to get finalized in this case - do I have to use the delete
keyword explicitly?
An interesting case is when using C's malloc for C1 and using the scope
attribute for c1:
class C1 {
ubyte[1_000_000] buf;
new(size_t size) {
void* ptr = std.c.stdlib.malloc(size);
if (ptr is null)
throw new OutOfMemoryError(__FILE__, __LINE__);
writefln("C1.new(%d) = %x", size, ptr);
return ptr;
}
delete(void* ptr) {
writefln("C1.delete %x", ptr);
if (ptr) std.c.stdlib.free(ptr);
}
this() { writeln("C1()"); }
~this() { writeln("~C1()"); }
}
...
scope C1 c1 = new C1;
In this case, ~C1 gets invoked but not C1.delete. Nevertheless, the
memory appears to get freed (normal memory consumption, no OutOfMemory).
How does this happen?
Re: Garbage Collection, Allocators/Deallocators and
> An interesting case is when using C's malloc for C1 and using the scope
> attribute for c1:
>
> class C1 {
> ubyte[1_000_000] buf;
> new(size_t size) {
> void* ptr = std.c.stdlib.malloc(size);
> if (ptr is null)
> throw new OutOfMemoryError(__FILE__, __LINE__);
> writefln("C1.new(%d) = %x", size, ptr);
>return ptr;
> }
> delete(void* ptr) {
> writefln("C1.delete %x", ptr);
> if (ptr) std.c.stdlib.free(ptr);
> }
> this() { writeln("C1()"); }
> ~this() { writeln("~C1()"); }
> }
> ...
> scope C1 c1 = new C1;
>
> In this case, ~C1 gets invoked but not C1.delete. Nevertheless, the
> memory appears to get freed (normal memory consumption, no OutOfMemory).
> How does this happen?
Ok, I figured that out myself: c1 is allocated on the stack in this
case, so no neither the allocator nor the deallocator need to get
called.
Re: Garbage Collection, Allocators/Deallocators and
Ivo Kasiuk wrote: Ok, that makes sense. So the deallocators really should not get called in this case. But why are the destructors not invoked when the GC finalizes the objects? For S1 and S2, this is a known bug - destructors of structs on the heap don't get called. As for C1, I have no idea what's happening. Definitely a bug, though. Exploring the example a bit further: If C's malloc is used instead of GC.malloc then the deallocators also are not called and the program runs out of memory. How are the objects supposed to get finalized in this case - do I have to use the delete keyword explicitly? If you use C's malloc, you will also have to use C's free. -- Simen
Re: Garbage Collection, Allocators/Deallocators and
> > Exploring the example a bit further: > > If C's malloc is used instead of GC.malloc then the deallocators also > > are not called and the program runs out of memory. How are the objects > > supposed to get finalized in this case - do I have to use the delete > > keyword explicitly? > > If you use C's malloc, you will also have to use C's free. Yes, obviously. But the deallocator which contains the call to C's free has to be invoked by someone, otherwise it's useless. So that is my question: how should finalization and deallocation get triggered in this scenario (where non-GC memory is used)?
Re: Garbage Collection, Allocators/Deallocators and
Ivo Kasiuk wrote: > Exploring the example a bit further: > If C's malloc is used instead of GC.malloc then the deallocators also > are not called and the program runs out of memory. How are the objects > supposed to get finalized in this case - do I have to use the delete > keyword explicitly? If you use C's malloc, you will also have to use C's free. Yes, obviously. But the deallocator which contains the call to C's free has to be invoked by someone, otherwise it's useless. So that is my question: how should finalization and deallocation get triggered in this scenario (where non-GC memory is used)? delete will, as long as it is kept in D, call the deallocator method. Unless the allocated memory is not registered with the garbage collector, it will be ignored by the garbage collector, and you will have to manually delete allocated objects. The point of using C's malloc and free is exactly that - the GC will ignore your objects, and you are free to (that is, have to) manage their lifetimes yourself. -- Simen
Re: Garbage Collection, Allocators/Deallocators and
> >> > Exploring the example a bit further: > >> > If C's malloc is used instead of GC.malloc then the deallocators also > >> > are not called and the program runs out of memory. How are the objects > >> > supposed to get finalized in this case - do I have to use the delete > >> > keyword explicitly? > >> > >> If you use C's malloc, you will also have to use C's free. > > > > Yes, obviously. But the deallocator which contains the call to C's free > > has to be invoked by someone, otherwise it's useless. So that is my > > question: how should finalization and deallocation get triggered in this > > scenario (where non-GC memory is used)? > > delete will, as long as it is kept in D, call the deallocator method. > Unless the allocated memory is not registered with the garbage collector, > it will be ignored by the garbage collector, and you will have to manually > delete allocated objects. The point of using C's malloc and free is > exactly that - the GC will ignore your objects, and you are free to (that > is, have to) manage their lifetimes yourself. > Thanks for the explanation. Summing up what I have learned now: - When GC-allocated memory is used the deallocator is not called when the GC automatically finalizes the object, but only when the delete keyword is used explicitly. - When non-GC memory is used the deallocator is also only called when the delete keyword is used, and deallocation/finalization is never triggered automatically. In other words: the deallocator only ever gets called when the delete keyword is used. What does this mean if the delete keyword is not kept in D: - Will deallocators also be obsolete then? - Is there any way the regular finalization sequence (i.e. including automatic invoking of the destructors) can be run for an object that uses non-GC memory?
Re: Garbage Collection, Allocators/Deallocators and
> > Ok, that makes sense. So the deallocators really should not get called > > in this case. But why are the destructors not invoked when the GC > > finalizes the objects? > > For S1 and S2, this is a known bug - destructors of structs on the heap > don't get called. Ah, I see. You mean #2834. That explains it, thanks. > As for C1, I have no idea what's happening. Definitely a bug, though. I am not sure that it is a bug that ~C1 (and also ~S1) is not invoked. After all, we called GC.malloc manually. So the GC knows about the memory region but is possibly not aware that it contains the object. If this is true it would be logical that the memory is freed but the destructor is not called.
