Re: either me or GC sux badly (GC don't reuse free memory)
On Wednesday, 12 November 2014 at 16:47:47 UTC, ketmar via Digitalmars-d wrote: but this clearly not an issue with sample which does `GC.free()`, and it stops at 1.7GB, while C sample does the same and stops at 2.9GB. GC probably allocates some small blocks for pools and other data. If it gets in a middle of address space, that can cause fragmentation too. Try to add small allocations to the C sample too.
Re: either me or GC sux badly (GC don't reuse free memory)
On Wednesday, 12 November 2014 at 16:47:47 UTC, ketmar via Digitalmars-d wrote: On Wed, 12 Nov 2014 16:40:10 + Sean Kelly via Digitalmars-d wrote: Try following the big allocation with a really small allocation to clear out any registers that may be referencing the large block. but this clearly not an issue with sample which does `GC.free()`, and it stops at 1.7GB, while C sample does the same and stops at 2.9GB. You should get debuginfo by compiling the runtime with the PRINTF debugging flag set: https://github.com/D-Programming-Language/druntime/blob/master/src/gc/gc.d#L20
Re: either me or GC sux badly (GC don't reuse free memory)
On Wed, 12 Nov 2014 16:40:10 + Sean Kelly via Digitalmars-d wrote: > Try following the big allocation with a really small allocation > to clear out any registers that may be referencing the large > block. but this clearly not an issue with sample which does `GC.free()`, and it stops at 1.7GB, while C sample does the same and stops at 2.9GB. signature.asc Description: PGP signature
Re: either me or GC sux badly (GC don't reuse free memory)
It's been a while since I Dded this, but I think the GC will effectively call minimize after collecting, so any collected large allocations should be returned to the OS. Allocations larger than 4K get their own dedicated pool, so fragmentation shouldn't come into play here.
Re: either me or GC sux badly (GC don't reuse free memory)
On Wednesday, 12 November 2014 at 16:20:48 UTC, Steven Schveighoffer wrote: I don't know the internals of C malloc. But I think it should be possible to make D merge segments when it needs to. Yes, but then D must provide a malloc replacement.
Re: either me or GC sux badly (GC don't reuse free memory)
Try following the big allocation with a really small allocation to clear out any registers that may be referencing the large block.
Re: either me or GC sux badly (GC don't reuse free memory)
On Wed, 12 Nov 2014 16:13:39 + via Digitalmars-d wrote: > On Wednesday, 12 November 2014 at 16:06:32 UTC, ketmar via > Digitalmars-d wrote: > > if i'll use libc malloc() for allocating, everything works as i > > expected: address space consumtion is on par with allocation > > size. > > The gc uses C's calloc rather implementing memory handling itself > using the OS so you get fragmentation: > > https://github.com/D-Programming-Language/druntime/blob/master/src/gc/gc.d#L2223 ah, and sorry once again, i was tired. ;-) the C program stops at 2,936,012,800, which is much more realistic. i checked three times and it's really using libc `malloc()` now. so libc malloc is perfectly able to merge segments, while D GC is not. signature.asc Description: PGP signature
Re: either me or GC sux badly (GC don't reuse free memory)
On Wed, 12 Nov 2014 11:20:48 -0500 Steven Schveighoffer via Digitalmars-d wrote: > One thing I am curious about -- it needs to allocate space to deal with > metadata in the heap. That data should be moveable, but I bet it doesn't > get moved. That may be why it can't merge segments. looks like a good GC improvement. ;-) signature.asc Description: PGP signature
Re: either me or GC sux badly (GC don't reuse free memory)
On Wed, 12 Nov 2014 16:23:01 + Kagamin via Digitalmars-d wrote: > On Wednesday, 12 November 2014 at 15:36:48 UTC, ketmar via > Digitalmars-d wrote: > > so heap fragmentation from other allocations can't be the issue. > > Why do you think so? > Try to go in opposite direction: start from 700MB and decrease > allocation size. i mean "from allocations in other places of the progam", not the "previous allocations in this code". sorry. signature.asc Description: PGP signature
Re: either me or GC sux badly (GC don't reuse free memory)
On Wednesday, 12 November 2014 at 15:36:48 UTC, ketmar via Digitalmars-d wrote: so heap fragmentation from other allocations can't be the issue. Why do you think so? Try to go in opposite direction: start from 700MB and decrease allocation size.
Re: either me or GC sux badly (GC don't reuse free memory)
On 11/12/14 11:06 AM, ketmar via Digitalmars-d wrote: i posted the second samle where i'm doing `GC.free()` to reclaim memory. as i said, RES is jumping between "almost nothing" and "several GB", as sample allocates and frees. but VIRT is growing constantly. i believe that GC just can't merge segments, so it keep asking for more and more address space for new segments, leaving old ones unused and unmerged. this way GC has alot of free memory, but when it can't allocate another segment, it throws "out of memory error". Yes, this is what I think is happening. if i'll use libc malloc() for allocating, everything works as i expected: address space consumtion is on par with allocation size. I don't know the internals of C malloc. But I think it should be possible to make D merge segments when it needs to. One thing I am curious about -- it needs to allocate space to deal with metadata in the heap. That data should be moveable, but I bet it doesn't get moved. That may be why it can't merge segments. -Steve
Re: either me or GC sux badly (GC don't reuse free memory)
On Wed, 12 Nov 2014 16:13:39 + via Digitalmars-d wrote: > On Wednesday, 12 November 2014 at 16:06:32 UTC, ketmar via > Digitalmars-d wrote: > > if i'll use libc malloc() for allocating, everything works as i > > expected: address space consumtion is on par with allocation > > size. > > The gc uses C's calloc rather implementing memory handling itself > using the OS so you get fragmentation: > > https://github.com/D-Programming-Language/druntime/blob/master/src/gc/gc.d#L2223 hm. my bad, i was checking malloc() with C program and it happens to use custom allocator. i just carefully re-checked it and it really works the same as D GC. sorry. so this seems to be libc memory manager fault after all. sorry for the noise. signature.asc Description: PGP signature
Re: either me or GC sux badly (GC don't reuse free memory)
On Wednesday, 12 November 2014 at 16:06:32 UTC, ketmar via Digitalmars-d wrote: if i'll use libc malloc() for allocating, everything works as i expected: address space consumtion is on par with allocation size. The gc uses C's calloc rather implementing memory handling itself using the OS so you get fragmentation: https://github.com/D-Programming-Language/druntime/blob/master/src/gc/gc.d#L2223
Re: either me or GC sux badly (GC don't reuse free memory)
On Wed, 12 Nov 2014 15:19:51 + Kagamin via Digitalmars-d wrote: > On Wednesday, 12 November 2014 at 11:05:11 UTC, ketmar via > Digitalmars-d wrote: > > the question is: am i doing something wrong here? how can i > > force GC to > > stop eating my address space and reuse what it already has? > > Try to allocate the arrays with NO_SCAN flag. btw, compiler is smart enough to allocate array with NO_SCAN flag, i checked this with `GC.getAttr()`. signature.asc Description: PGP signature
Re: either me or GC sux badly (GC don't reuse free memory)
On Wed, 12 Nov 2014 10:51:31 -0500
Steven Schveighoffer via Digitalmars-d
wrote:
> On 11/12/14 6:04 AM, ketmar via Digitalmars-d wrote:
> > Hello.
> >
> > let's run this program:
> >
> >import core.sys.posix.unistd;
> >import std.stdio;
> >import core.memory;
> >
> >void main () {
> > uint size = 1024*1024*300;
> > for (;;) {
> >auto buf = new ubyte[](size);
> >writefln("%s", size);
> >sleep(1);
> >size += 1024*1024*100;
> >buf = null;
> >GC.collect();
> >GC.minimize();
> > }
> >}
> >
> > pretty innocent, right? i even trying to help GC here. but...
> >
> >314572800
> >419430400
> >524288000
> >629145600
> >734003200
> >core.exception.OutOfMemoryError@(0)
> >
> > ps.
> >
> > by the way, this is not actually "no more memory", this is "i'm out of
> > address space" (yes, i'm on 32-bit system, GNU/Linux).
> >
> > the question is: am i doing something wrong here? how can i force GC to
> > stop eating my address space and reuse what it already has?
> >
> > sure, i can use libc malloc(), refcounting, and so on, but the question
> > remains: why GC not reusing already allocated and freed memory?
> >
>
> I think I might know what's going on.
>
> You are continually adding 100MB to the allocation size. Memory is
> contiguous from the OS, but can get fragmented inside the GC.
>
> So let's say, you allocate 300MB. Fine. It needs more space from the OS,
> allocates it, and assigns a pool to that 300MB. Now, you add another
> 100MB. At this point, it can't fit into the original pool, so it
> allocates another 400MB. BUT, it doesn't merge the 300MB into that (I
> don't think), so when it adds another 100MB, it has a 300MB space, and a
> 400MB space, neither of which can hold 500MB. And it goes on and on.
> Keep in mind also that it is a frequent error that people make to set a
> pointer to null and expect the data will be collected. For example, buf
> could still be in a register.
>
> I would be interested in how much memory the GC has vs. how much is
> actually used.
i posted the second samle where i'm doing `GC.free()` to reclaim
memory. as i said, RES is jumping between "almost nothing" and "several
GB", as sample allocates and frees. but VIRT is growing constantly.
i believe that GC just can't merge segments, so it keep asking for more
and more address space for new segments, leaving old ones unused and
unmerged. this way GC has alot of free memory, but when it can't
allocate another segment, it throws "out of memory error".
if i'll use libc malloc() for allocating, everything works as i
expected: address space consumtion is on par with allocation size.
signature.asc
Description: PGP signature
Re: either me or GC sux badly (GC don't reuse free memory)
On 11/12/14 6:04 AM, ketmar via Digitalmars-d wrote:
Hello.
let's run this program:
import core.sys.posix.unistd;
import std.stdio;
import core.memory;
void main () {
uint size = 1024*1024*300;
for (;;) {
auto buf = new ubyte[](size);
writefln("%s", size);
sleep(1);
size += 1024*1024*100;
buf = null;
GC.collect();
GC.minimize();
}
}
pretty innocent, right? i even trying to help GC here. but...
314572800
419430400
524288000
629145600
734003200
core.exception.OutOfMemoryError@(0)
ps.
by the way, this is not actually "no more memory", this is "i'm out of
address space" (yes, i'm on 32-bit system, GNU/Linux).
the question is: am i doing something wrong here? how can i force GC to
stop eating my address space and reuse what it already has?
sure, i can use libc malloc(), refcounting, and so on, but the question
remains: why GC not reusing already allocated and freed memory?
I think I might know what's going on.
You are continually adding 100MB to the allocation size. Memory is
contiguous from the OS, but can get fragmented inside the GC.
So let's say, you allocate 300MB. Fine. It needs more space from the OS,
allocates it, and assigns a pool to that 300MB. Now, you add another
100MB. At this point, it can't fit into the original pool, so it
allocates another 400MB. BUT, it doesn't merge the 300MB into that (I
don't think), so when it adds another 100MB, it has a 300MB space, and a
400MB space, neither of which can hold 500MB. And it goes on and on.
Keep in mind also that it is a frequent error that people make to set a
pointer to null and expect the data will be collected. For example, buf
could still be in a register.
I would be interested in how much memory the GC has vs. how much is
actually used.
-Steve
Re: either me or GC sux badly (GC don't reuse free memory)
On 11/12/14 10:19 AM, Kagamin wrote: On Wednesday, 12 November 2014 at 11:05:11 UTC, ketmar via Digitalmars-d wrote: the question is: am i doing something wrong here? how can i force GC to stop eating my address space and reuse what it already has? Try to allocate the arrays with NO_SCAN flag. Really that shouldn't matter. The arrays should all be 0-initialized. -Steve
Re: either me or GC sux badly (GC don't reuse free memory)
On Wed, 12 Nov 2014 15:19:51 + Kagamin via Digitalmars-d wrote: > On Wednesday, 12 November 2014 at 11:05:11 UTC, ketmar via > Digitalmars-d wrote: > > the question is: am i doing something wrong here? how can i > > force GC to > > stop eating my address space and reuse what it already has? > > Try to allocate the arrays with NO_SCAN flag. why this must make any difference in the demonstrated cases? ubyte arrays are initialized to zeroes, so they can't contain false pointers. signature.asc Description: PGP signature
Re: either me or GC sux badly (GC don't reuse free memory)
On Wed, 12 Nov 2014 15:24:08 + Kagamin via Digitalmars-d wrote: > On Wednesday, 12 November 2014 at 12:30:15 UTC, ketmar via > Digitalmars-d wrote: > > this shouldn't fail so soon, right? i'm freeing the memory, > > so... it > > still dying on 1,887,436,800. 1.7GB and that's all? this can't > > be true, > > i have 3GB of free RAM (with 1.2GB used) and 8GB of unused > > swap. and > > yes, it consumed all of the process address space again. > > Maybe you fragmented the heap and don't have 1.7GB of contiguous > memory? i gave two example programs, which demonstrates the effect, and they aren't excerpts. the only allocating `writef` can be removed too, but the effect stays. so heap fragmentation from other allocations can't be the issue. signature.asc Description: PGP signature
Re: either me or GC sux badly (GC don't reuse free memory)
On Wednesday, 12 November 2014 at 12:30:15 UTC, ketmar via Digitalmars-d wrote: this shouldn't fail so soon, right? i'm freeing the memory, so... it still dying on 1,887,436,800. 1.7GB and that's all? this can't be true, i have 3GB of free RAM (with 1.2GB used) and 8GB of unused swap. and yes, it consumed all of the process address space again. Maybe you fragmented the heap and don't have 1.7GB of contiguous memory?
Re: either me or GC sux badly (GC don't reuse free memory)
On Wednesday, 12 November 2014 at 11:05:11 UTC, ketmar via Digitalmars-d wrote: the question is: am i doing something wrong here? how can i force GC to stop eating my address space and reuse what it already has? Try to allocate the arrays with NO_SCAN flag.
Re: either me or GC sux badly (GC don't reuse free memory)
On Wed, 12 Nov 2014 12:42:10 +
Matthias Bentrup via Digitalmars-d wrote:
> On Wednesday, 12 November 2014 at 12:30:15 UTC, ketmar via
> Digitalmars-d wrote:
> > On Wed, 12 Nov 2014 12:05:25 +
> > thedeemon via Digitalmars-d wrote:
> >
> >> On Wednesday, 12 November 2014 at 11:05:11 UTC, ketmar via
> >> Digitalmars-d wrote:
> >> > 734003200
> >> > address space" (yes, i'm on 32-bit system, GNU/Linux).
> >> >
> >> > the question is: am i doing something wrong here? how can i
> >> > force GC to stop eating my address space and reuse what it
> >> > already has?
> >>
> >> Sure: just make the GC precise, not conservative. ;)
> >> With current GC implementation and array this big chances of
> >> having a word on the stack that looks like a pointer to it and
> >> prevents it from being collected are almost 100%. Just don't
> >> store big arrays in GC heap or switch to 64 bits where the
> >> problem is not that bad since address space is much larger and
> >> chances of false pointers are much smaller.
> > but 'mkay, let's change the sample a little:
> >
> > import core.memory;
> > import std.stdio;
> >
> > void main () {
> > uint size = 1024*1024*300;
> > for (;;) {
> > auto buf = new ubyte[](size);
> > writefln("%s", size);
> > size += 1024*1024*100;
> > GC.free(GC.addrOf(buf.ptr));
> > buf = null;
> > GC.collect();
> > GC.minimize();
> > }
> > }
> >
> > this shouldn't fail so soon, right? i'm freeing the memory,
> > so... it
> > still dying on 1,887,436,800. 1.7GB and that's all? this can't
> > be true,
> > i have 3GB of free RAM (with 1.2GB used) and 8GB of unused
> > swap. and
> > yes, it consumed all of the process address space again.
>
> On Linux/x86 you have only 3 GB virtual address space, and this
> has to include the program code + all loaded libraries too. Check
> out /proc//maps, to see where the dlls are loaded, and look
> at the largest chunk of free space available. That is the
> theoretical limit that could be allocated.
i know it. what i can't get is why D allocates more and more address
space with each 'new'. what i expecting is address space consumption on
par with 'size', but it grows alot faster.
seems that i should either read GC code to see what's going on (oh,
boring!) or write memory region dumper (it's funnier).
i bet that something is wrong with GC memory manager though, but can't
prove it for now.
signature.asc
Description: PGP signature
Re: either me or GC sux badly (GC don't reuse free memory)
On Wednesday, 12 November 2014 at 12:30:15 UTC, ketmar via
Digitalmars-d wrote:
On Wed, 12 Nov 2014 12:05:25 +
thedeemon via Digitalmars-d wrote:
On Wednesday, 12 November 2014 at 11:05:11 UTC, ketmar via
Digitalmars-d wrote:
> 734003200
> address space" (yes, i'm on 32-bit system, GNU/Linux).
>
> the question is: am i doing something wrong here? how can i
> force GC to stop eating my address space and reuse what it
> already has?
Sure: just make the GC precise, not conservative. ;)
With current GC implementation and array this big chances of
having a word on the stack that looks like a pointer to it and
prevents it from being collected are almost 100%. Just don't
store big arrays in GC heap or switch to 64 bits where the
problem is not that bad since address space is much larger and
chances of false pointers are much smaller.
but 'mkay, let's change the sample a little:
import core.memory;
import std.stdio;
void main () {
uint size = 1024*1024*300;
for (;;) {
auto buf = new ubyte[](size);
writefln("%s", size);
size += 1024*1024*100;
GC.free(GC.addrOf(buf.ptr));
buf = null;
GC.collect();
GC.minimize();
}
}
this shouldn't fail so soon, right? i'm freeing the memory,
so... it
still dying on 1,887,436,800. 1.7GB and that's all? this can't
be true,
i have 3GB of free RAM (with 1.2GB used) and 8GB of unused
swap. and
yes, it consumed all of the process address space again.
On Linux/x86 you have only 3 GB virtual address space, and this
has to include the program code + all loaded libraries too. Check
out /proc//maps, to see where the dlls are loaded, and look
at the largest chunk of free space available. That is the
theoretical limit that could be allocated.
Re: either me or GC sux badly (GC don't reuse free memory)
On Wed, 12 Nov 2014 12:05:25 + thedeemon via Digitalmars-d wrote: > On Wednesday, 12 November 2014 at 11:05:11 UTC, ketmar via > Digitalmars-d wrote: > > 734003200 > > address space" (yes, i'm on 32-bit system, GNU/Linux). > > > > the question is: am i doing something wrong here? how can i > > force GC to stop eating my address space and reuse what it > > already has? > > Sure: just make the GC precise, not conservative. ;) > With current GC implementation and array this big chances of > having a word on the stack that looks like a pointer to it and > prevents it from being collected are almost 100%. Just don't > store big arrays in GC heap or switch to 64 bits where the > problem is not that bad since address space is much larger and > chances of false pointers are much smaller. for information: yes, RES is jumping high and low as it should. but VIRT is steadily growing until there is no more address space available. so the problem is clearly not in false pointers this time. signature.asc Description: PGP signature
Re: either me or GC sux badly (GC don't reuse free memory)
On Wed, 12 Nov 2014 12:05:25 +
thedeemon via Digitalmars-d wrote:
> On Wednesday, 12 November 2014 at 11:05:11 UTC, ketmar via
> Digitalmars-d wrote:
> > 734003200
> > address space" (yes, i'm on 32-bit system, GNU/Linux).
> >
> > the question is: am i doing something wrong here? how can i
> > force GC to stop eating my address space and reuse what it
> > already has?
>
> Sure: just make the GC precise, not conservative. ;)
> With current GC implementation and array this big chances of
> having a word on the stack that looks like a pointer to it and
> prevents it from being collected are almost 100%. Just don't
> store big arrays in GC heap or switch to 64 bits where the
> problem is not that bad since address space is much larger and
> chances of false pointers are much smaller.
but 'mkay, let's change the sample a little:
import core.memory;
import std.stdio;
void main () {
uint size = 1024*1024*300;
for (;;) {
auto buf = new ubyte[](size);
writefln("%s", size);
size += 1024*1024*100;
GC.free(GC.addrOf(buf.ptr));
buf = null;
GC.collect();
GC.minimize();
}
}
this shouldn't fail so soon, right? i'm freeing the memory, so... it
still dying on 1,887,436,800. 1.7GB and that's all? this can't be true,
i have 3GB of free RAM (with 1.2GB used) and 8GB of unused swap. and
yes, it consumed all of the process address space again.
signature.asc
Description: PGP signature
Re: either me or GC sux badly (GC don't reuse free memory)
On Wed, 12 Nov 2014 12:05:25 + thedeemon via Digitalmars-d wrote: > On Wednesday, 12 November 2014 at 11:05:11 UTC, ketmar via > Digitalmars-d wrote: > > 734003200 > > address space" (yes, i'm on 32-bit system, GNU/Linux). > > > > the question is: am i doing something wrong here? how can i > > force GC to stop eating my address space and reuse what it > > already has? > > Sure: just make the GC precise, not conservative. ;) > With current GC implementation and array this big chances of > having a word on the stack that looks like a pointer to it and > prevents it from being collected are almost 100%. Just don't > store big arrays in GC heap or switch to 64 bits where the > problem is not that bad since address space is much larger and > chances of false pointers are much smaller. even with NO_INTERIOR the sample keeps failing (yet after more iterations). no, really, there is ALWAYS a pointer exactly to the start of the allocated buffer somewhere? i feel something smelly here. signature.asc Description: PGP signature
Re: either me or GC sux badly (GC don't reuse free memory)
On Wednesday, 12 November 2014 at 11:05:11 UTC, ketmar via Digitalmars-d wrote: 734003200 address space" (yes, i'm on 32-bit system, GNU/Linux). the question is: am i doing something wrong here? how can i force GC to stop eating my address space and reuse what it already has? Sure: just make the GC precise, not conservative. ;) With current GC implementation and array this big chances of having a word on the stack that looks like a pointer to it and prevents it from being collected are almost 100%. Just don't store big arrays in GC heap or switch to 64 bits where the problem is not that bad since address space is much larger and chances of false pointers are much smaller.
either me or GC sux badly (GC don't reuse free memory)
Hello.
let's run this program:
import core.sys.posix.unistd;
import std.stdio;
import core.memory;
void main () {
uint size = 1024*1024*300;
for (;;) {
auto buf = new ubyte[](size);
writefln("%s", size);
sleep(1);
size += 1024*1024*100;
buf = null;
GC.collect();
GC.minimize();
}
}
pretty innocent, right? i even trying to help GC here. but...
314572800
419430400
524288000
629145600
734003200
core.exception.OutOfMemoryError@(0)
ps.
by the way, this is not actually "no more memory", this is "i'm out of
address space" (yes, i'm on 32-bit system, GNU/Linux).
the question is: am i doing something wrong here? how can i force GC to
stop eating my address space and reuse what it already has?
sure, i can use libc malloc(), refcounting, and so on, but the question
remains: why GC not reusing already allocated and freed memory?
signature.asc
Description: PGP signature
